Being a good engineer doesn’t make your AI one.

I've spent thirty years building controls in UK financial services, insurance and defence, so I like to think I know how to scope a change and check it worked. My AI builder still managed to wipe every live agent session on my own server.

While I was still building the way we work with agents, I gave a coding agent a bounded job. Partway through, it ran a command that shut down every terminal session on the machine, including the ones running other agents' work. Nothing in my instructions came close to asking for that, and it carried on with its task afterwards as if nothing had happened.

Your expertise doesn't transfer to the assistant

Experienced engineers tend to assume that a precise brief gets a precise result. In my experience the assistant builds what it thinks you meant, in the way it thinks is reasonable, and then reports that it went well.

A lot of the time that's fine, and sometimes it's better than what I asked for. But every so often it's something I'd never have signed off: a shortcut around a control, a change outside the agreed scope, or a cleanup that breaks something else. I usually catch these because I know what to look for, but catching them is my job, not proof that it listened.

A job reporting healthy isn't the same as a job working

Around the same time we hit a quieter version of the same problem. Four of our unattended jobs kept stopping and restarting, and each restart looked normal.

When we dug in, one model call had hung for thirty minutes without writing a single byte. The job had burned through its time allowance while waiting, so every restart after that gave up within seconds. For about three hours it looked as if work was happening and none was. We only found it by opening the output files and the logs, which were empty.

Sorry costs it nothing

When an AI gets it wrong, it isn't accountable. It says sorry, and for it nothing is lost. It doesn't recognise the weeks of work that went into what it just broke, or the weeks that will go into building on top of a mistake nobody caught. The cost lands on you, usually later, when the bad code is already holding up everything else.

Your AI only tests what it knows

Ask an AI to test its own code and it tests the design it already has in its head. It checks that the code does what it built it to do. It doesn't check whether that's what you meant, or what happens just outside the boundaries it imagined, because it can't see past its own assumptions.

That's the gap Agentic Evals is built for. We probe your system from outside its design: we take what your documentation says the code does, and check the running code against those claims, with evidence. It isn't penetration testing and it isn't hardening. It's verification. Does your code do what you think it does, before anything else gets built on top of it?

How I build with agents now

Both of these happened before the controls below existed. They are the reason those controls exist, and neither can happen the same way now.

I still build with AI every day, but I've stopped expecting it to share my standards. These are the rules I work to now.

1. Decide what done means before any code is written. I write the outcome and the acceptance tests first, and the builder isn't allowed to change them, including weakening a test to get a build through.
2. Size the job honestly. I make small fixes myself and only hand a builder genuinely substantial work. A lot of my bad surprises came from pushing a small job through a big process.
3. Use the right model for each role. A fast, inexpensive model writes the code and a different model reviews it, so nothing marks its own homework. The most capable models go where the stakes are highest. I'm a member of Anthropic's Cyber Verification Program, which gives me access to models that aren't generally available. That's useful, but the same rules apply to them.
4. Put the standards into automated checks rather than the prompt. File size, function size and complexity limits are enforced by checks that fail the build. Agents regularly drift from written instructions, but they can't get past a failing check.
5. Stop after two fix rounds. If a repair has failed twice, I go back to the approach rather than asking for a third attempt.
6. Limit what an agent can touch. Agents work in their own space with throwaway credentials, and they're not allowed to stop anything they didn't start.
7. Check the work yourself. I look at the actual change and run the tests myself. The builder's report tells me where to look, but I don't treat it as the result.

What this means for your product

If this happens to someone who has spent a career building controls, I'd assume it's happening in most repositories built this way, including yours.

It matters even more if your product is an agent itself, because the same gap exists inside it. When it says an action was blocked, a task is done or the log is complete, you need some way of checking that's what actually happened.

That's what we test: five checks at a fixed commit, using fake secrets in an isolated environment, with a person reviewing every result. We check that it starts cleanly from a fresh checkout, that blocked actions are really blocked, that it can't reach secrets or get out of its environment, that the logs match what happened, and that "completed" means the action actually ran.

How we evaluate · I stopped reading my code · Tell us about your agent