The record shows what happened
We check that each decision, approval and action is tied to the one before it, not just stated.
How we evaluate
What we check, and how.
If your product was built with AI help, an app, API, command-line tool or an agent that takes actions through tools, APIs, shells or other systems, we review it independently with AI-assisted tooling and a named person signing off: we read the code, test it against your claims, check whether your guardrails stop it when it runs, and whether your logs prove what happened. Products built with AI help are fine; most are. The advanced pack has fixed published prices and agreed test scopes, and the fee does not depend on the result. The report is yours. It is not a certification or a penetration test, and never an endorsement.
What we examine
Every link in the chain
Your software claims a chain of events happened. We check each link, and whether the record of one link actually depends on the one before it, or just sits next to it.
- Policy decision
- Authorisation
- Execution
- Observation
- Durable evidence
- Independent verification
Runtime enforcement and bypass
Does a deny or an approval step actually stop the action at runtime? And are there routes around it that the record never shows?
Tamper, replay and forgery
We alter, replay and forge evidence to see whether anything notices, or whether your own verification waves it through as genuine.
From a fixed commit
Every finding is tied to a named commit, so anyone can rebuild the same code and repeat the test.
Controlled test network
Your code runs in an isolated test environment. Direct internet access is blocked; any permitted model endpoint or local test service is documented in the agreed scope and report. These checks do not establish security outside the tested environment.
What an evaluation can find
The kind of thing your AI won't tell you
Two examples from our evaluation work, anonymised. Neither the tool nor its developer is named.
"Completed" for actions that never ran
A gateway reported actions as completed that it had never performed. The record looked complete, but it was not evidence that anything had run.
A forgery that passed verification
An evidence digest accepted a forged record. We altered an entry and re-hashed it, and the tool's own verification reported it as genuine.
Types of evaluation
Four fixed-price packs.
Agent-powered testing, directed and signed off by Leo Ruocco, with 30 years' experience building controls across financial services, insurance and defence.
Four products, fixed prices: the Honest Second Opinion is £299, the Deny Means Deny certificate is £399, the advanced pack is £499 open source or £999 private (£749 or £1,349 with one re-test), and a deep evaluation is scoped per customer from £2,500 by email. The five-check baseline is included in the advanced pack and is not sold separately. Details and purchase on the buy page. Every evaluation is built specifically for your requirements: your approach specification defines the tests, the report skeleton stays fixed.
How the advanced pack is built
Case inventory: reference baseline and advanced catalogue
Reference baseline: 10 action cases
The standard reference adapter contains these 10 target-action cases, plus two additional channel positive controls. Startup and clean/altered verifier calls are separate. This is a reference set, not a fixed case-count promise for every repository.
allow-write: permitted write.allow-append: permitted append.fake-done: test a completion claim without its effect.fail-partway: failure partway through an action.deny-write: denied write.deny-append: denied append.deny-delete: denied deletion.deny-exec: denied command execution.read-secret: secret-read attempt.egress: outbound-transfer attempt.
Advanced catalogue: 66 templates in 12 families
We select applicable methods and agree repository-specific cases in your approach specification. A catalogue entry is not an executed test; not every family applies to every target class.
| Family | Templates | Scope / readiness |
|---|---|---|
| Path bypass | 7 | Target-dependent |
| Command bypass | 14 | Target-dependent |
| Encoding / Unicode | 4 | Target-dependent |
| Fail-open configuration | 3 | Target-dependent |
| Secret access | 5 | Target-dependent |
| Exfiltration | 5 | Target-dependent |
| Network egress | 5 | Target-dependent |
| Record integrity | 5 | Target-dependent |
| Self-record tampering | 3 | Target-dependent |
| Completion under failure | 3 | Target-dependent |
| Approval handling | 5 | Generic oracle pending |
| MCP edges | 7 | Generic oracle pending |
| Total | 66 | Templates, not executed coverage |
Implementation limit: the 12 approval/MCP templates currently have unimplemented generic oracles. They are not advertised as working or executed coverage. Any tailored evaluation of those surfaces must first have its runnable method and evidence capture confirmed.
Your delivered pack: lists the actual agreed cases, what ran, what did not, expected and observed outcomes, controls and supporting evidence. It does not count overlapping controls or nested campaign requests twice.
Source sets: standard reference adapter and advanced probe catalogue, checked 14 October 2026. Reference cases are not findings or universal customer coverage.
- Five-check baseline, included: the five-check baseline framework, environment and commit record, reviewed outcomes and evidence. We use the target-appropriate profile and explain checks that are not applicable; we do not force an agent test onto an HTTP filter.
- Applicable advanced methods: deeper bypass, approval-substitution, replay, crash/retry and evidence-integrity probes where relevant. We observe actual effects and include positive controls for authorised behaviour, not just rejected attacks.
- Repository-specific cases: your approach specification, source claims, configuration and agreed workflow determine the test plan. We confirm the cases and delivery date in writing before testing. Different kinds of target receive different tests.
The boundary stays fixed: one product, one agreed workflow or control boundary and a fixed commit. This does not include every advanced test family, a whole-product audit or changes to your software.
What you receive
- A reviewed PDF and HTML report combining the five-check baseline and advanced results, with clear verdicts, confirmed findings, disclosed limitations and method gaps kept separate.
- Pinned target-specific test cases, configuration and reproduction instructions for the agreed scope.
- Supporting raw evidence and an evidence index linking outcomes to the captured observations. Sensitive material is handled under the agreed disclosure terms, with any redactions explained.
- Founder-actionable remediation guidance and regression cases where a finding is confirmed, plus explicit coverage limits. You implement any fixes.
The handover supports rerunning the agreed tests; it is not a transfer of our entire proprietary rig. No number of findings or passing outcome is promised. Reports remain private unless publication is agreed in writing; see section 10 of our disclosure terms.
What we hold to
Six principles, applied to every pack.
Agent proposes, software verifies.
No AI decides a verdict. A deterministic checker compares every quoted excerpt against the captured evidence before it appears in a report.
Pass or fail, with the evidence path.
No scores, no points, no weighted average. Each result stands or falls on the evidence file that backs it.
What we did not test is printed.
Gaps are stated in public and never counted as passes. A section without evidence for your product says so.
Every fault we inject, we report.
Negative controls run in every pack. A deliberate fault that the tool misses becomes a finding against the tool, in writing.
The same sections in every pack.
The report skeleton is fixed and is never negotiated per customer. What varies is depth, not structure.
A named person signs.
Every report carries a named signatory, a date and a scope statement. It is not a certification and never an endorsement.
The fixed skeleton
What’s in every report.
Every evaluation, the five-check baseline or an advanced pack, renders the same thirteen sections in the same order. Your evaluation fills them; the structure never moves. A section without evidence for your product says so in public, and it is never marked as a pass.
What was tested
- The boundary. One product, one agreed workflow or control boundary, one pinned commit, the agreed claims table.
- Method in brief. Agent-guided, budget-capped collection; deterministic verification of every quoted excerpt; a named person signs. What the fee covers, and what it does not.
Six links, one verdict each
- Policy decision. Does the stated policy actually gate the action?
- Authorisation. Does a deny or approval stop the action at runtime, with no route around it that the record hides?
- Execution. Did the claimed actions actually happen?
- Observation. Do recorded effects match independently observed effects?
- Durable evidence. Is the record tamper-evident, hash-linked, replay-resistant?
- Independent verification. Does the tool’s own verification catch forgery, or wave it through?
Each link is marked established, partial, failed, or not applicable with a reason. Never blank.
The credibility sections
- Findings. Each as claim, test, result, evidence paths, fix.
- Negative controls. Deliberate faults injected; every missed one becomes a finding.
- What we did not test. Gaps and limits, stated in public. Never counted as a pass.
- Reproduction. Pinned configuration, one command, anyone can rerun.
- Sign-off. A named person, a date, a scope statement. Not a certification, never an endorsement.
The five-check baseline uses the same skeleton with the chain limited to the five-check core. All thirteen sections exist in every delivered pack, even when some are empty, especially then.
Advanced pack
See a sample report
See the advanced pack →
The baseline report example shows the five-check pack. The advanced pack example shows the same skeleton at full depth; actual coverage follows the purchased package: verdicts on each link, four findings with evidence paths, negative controls and reproduction steps.
Independence
What we have a stake in
AgenticBench, our public benchmark of AI coding agents, is separate: it takes no money from the agents it tests, and coding agents on the benchmark can't buy evaluations.
Declared interests: we maintain the open-source HookBus project and the AgentHook standard. We sell no agent-governance product, and we have committed to handing the AgentHook standard to an independent foundation. The full list is on our Trust page.
Enquire
Tell us about your agent
What you build, what its record is meant to show, and what you'd like examined. We reply in writing with the package that fits. Prices are fixed and on the buy page.