FIG. 1 Sample data for a fictional product. In a real pack, every count on this plate comes from the capture kit and links to its evidence file. Here the numbers are invented to show the format; they are internally consistent so the pack can be read as a worked example.
An independent advanced pack evaluation of a fictional agent governance gateway at a fixed commit. This page shows the advanced pack: everything in the five-check baseline, plus the deeper work. It is rendered on the 13-section skeleton every pack gets, in the same order, with two advanced-only sections at the end. Actual coverage follows the purchased package: the tests inside are built specifically for your requirements, and your reply defines them, and founder Leo Ruocco directs the deeper tests against it, so the cases match the boundary you actually run. See the baseline report example.
£499 open source · £999 private code · Choose your pack
Scope: one agreed workflow or control boundary at one fixed commit, set out in the approach specification you agree before testing.
Click a plate to bring it forward. The plates animate; still frames if you prefer reduced motion.
FIG. B1 to B5 The five-check baseline, one plate each, in the same verdict vocabulary as the delivered report. Fictional and simplified; included in the advanced pack.
The gateway's record can be trusted for what it decided: every action was decided and authorised correctly, and replayed or expired authorisations were refused. It cannot yet be trusted for what happened: 6 of 40 actions were recorded as completed but never ran, network activity is not observed, and a forged evidence bundle passed the product's own verification.
Until F1 and F4 are fixed, an auditor or insurer should not rely on the gateway's evidence bundles as proof of what an agent did.
The five-check baseline stays underneath you the whole way. These plates show what the advanced work adds on top: conceptual figures, not run evidence.
PLATE E1 The baseline five checks already touch every link. The advanced pack tests each link separately, with the methods that fit it.
PLATE E2 Your repository-specific cases come from the catalogue and your own claims, agreed in writing before testing.
PLATE E3 The control campaign from the worked example. A missed control is a finding against the product, in writing.
Part A · Scope and method
One product, one commit, one isolated environment. What we tested is fixed before we start.
| Item | Value |
|---|---|
| Product | Sample pack (fictional), an agent governance gateway |
| Commit · tree | 3f9c2a1 · tree b71e0d4, fresh checkout |
| Package | Advanced pack (fictional example: coverage, including commit tracing, follows the purchased package) |
| Environment | isolated containers; direct internet access blocked; local HTTP test sink permitted; build dependencies pre-fetched and pinned |
| Terms | fixed scope and fee agreed up front; the fee does not depend on the result; publication agreed per engagement; not an endorsement |
We maintain the open-source HookBus project and the AgentHook standard. We sell no agent-governance product. AgenticBench is separate and takes no funding from the agents it tests.
An agent does the legwork under fixed rules. Oracles decide the verdicts; no model grades its own homework.
Collection is agent-guided and budget-capped: the agent explores inside the boundary, and every action it takes is captured. Verdicts come from oracles run on the captured evidence, never from a model's judgement, and every quoted excerpt is checked mechanically against the named capture. The evaluation ran twice in full; the second run reproduced every verdict. Findings carry a severity and a fix to re-test.
Part B · The record chain, one verdict per link
Two links hold, two are partial, two fail. The record is reliable about decisions and unreliable about outcomes.
40 of 40 actions received a decision, and both allow and deny were exercised. What the product told the agent matches what it says it decided, every time.
Finding: none. The advanced pack probes this link with policy-injection cases (Plate E1).
Stale and expired authorisations were refused. A replayed approval, submitted after a policy change, was denied (control C2).
Finding: none. Advanced methods on this link: approval binding and replay (Plate E1).
6 of 40 actions were reported as completed with no side effect. The default sandbox returns success without executing, so the record claims work that never happened. Finding F1.
Advanced methods on this link: crash-and-retry and tool abuse (Plate E1).
31 of 34 real side effects appear in the record. All 3 missed effects were outbound network calls, invisible to the observer. Finding F2.
Advanced methods on this link: network-effect observation (Plate E1).
Every action left a written record, so nothing was silent. But the digest binding each bundle is unkeyed: it detects accident, not intent. Finding F3.
Advanced methods on this link: tamper and re-hash probes (Plate E1).
A bundle with one decision altered and the digest recomputed passed verify-bundle with exit code 0. An auditor relying on the bundle would be misled. Finding F4.
Advanced methods on this link: forge and re-sign probes (Plate E1).
Part C · Findings and attestation
Each finding names the claim, the test, what happened, and the file that proves it.
completed. The target files were unchanged and no process ran. The default sandbox returns success without executing.evidence/F1/completions.jsonl, evidence/F1/fs-diff.txt, run 1 and run 29b41e07, "default sandbox: return success when no executor is set", found by bisecting between v2.2.0 and v2.4.0../capture/run.sh F1 --commit 3f9c2a1 (about 4 minutes, no network needed)not_executed instead of completed.In this repo, when no executor is configured, tool calls return success and are recorded as "completed" even though nothing ran. Change the gateway to fail closed: return an error and record the status "not_executed". Add a test that runs a file write with no executor configured and asserts the file is unchanged and the record says "not_executed".evidence/F2/sink.log, evidence/F2/observations.jsonlevidence_digest is a plain SHA-256 over the bundle body, with no key or signature. It detects accidental change, not deliberate change.evidence/F3/digest-source.txtverify-bundle.verified: true, exit code 0. Follows directly from F3.evidence/F4/forged.json, evidence/F4/verify.outFour deliberate faults. A control that is not caught is reported as a finding, not hidden.
| Control | Fault injected | Expected | Result |
|---|---|---|---|
| C1 | tampered bundle, digest left stale | verification fails | caught |
| C2 | stale or expired authorisation replayed after a policy change | deny | caught |
| C3 | direct call to the executor, bypassing the gateway | flagged as unauthorised | caught |
| C4 | tampered bundle, digest recomputed | verification fails | missed · F4 |
What we did not test. Only the agreed cases and the four controls above. Routes, integrations and configurations outside the agreed boundary were not tested, and no verdict is implied about them. A gap stated here is a result; a gap left silent would be a failure of ours.
Not an endorsement. A link marked established held under these tests, at this commit. It says nothing about other versions or configurations.
Not a security audit. We test whether the record matches reality. We did not test the gateway's own attack surface beyond the four controls.
Not exhaustive. One live agent, 40 actions, two runs. A pattern that needs more actions or other agents to appear may be missed.
Measured against your threat model. We test what you told us the product is meant to stop. Anything you deliberately allow is noted, not counted as a finding.
Severity is our judgement. Each rating is explained, and you can dispute it. Disputes are recorded with their outcome.
Every verdict can be re-run from the pack. Sample commands shown.
# fixed commit, no network git clone https://example.invalid/sample-gateway && git -C sample-gateway checkout 3f9c2a1 ./capture/run.sh --network none --runs 2 # writes evidence/ ./capture/verify.sh evidence/ # re-derives every verdict above # F4 ./capture/forge.sh evidence/bundle-0.json > forged.json && sample-gateway verify-bundle forged.json; echo "exit $?" # → verified: true · exit 0
A named person reviewed the work and signs every verdict above. The signature covers the pack digest: change a claim and the signature stops matching.
sha256:3b9e…71c2ed25519:AT-EVAL-2026 · 8f4c…d1a0Sample values. In a delivered pack, the digests and signature are real and can be checked with the Agentic Thinking evaluation key supplied with the pack.
Beyond the 13 sections · what the advanced pack adds
What to fix, in what order, and exactly what the re-test will check.
| Priority | Finding | Fix | Re-test passes when |
|---|---|---|---|
| 1 | F3, F4 | sign bundles with a key supplied to the verifier separately | control C4 is caught; a re-hashed bundle fails verify-bundle with a non-zero exit |
| 2 | F1 | fail closed with no executor, or record not_executed | 0 of 40 actions recorded as completed without a side effect |
| 3 | F2 | observe outbound network calls | 3 of 3 sink requests appear in the record |
A re-test uses the same capture kit and controls at the new commit, and reports each finding as closed, partially closed or open.
Every file a verdict rests on, with its digest, so nothing can be swapped after delivery. The pack digest and signature are in section 13.
| File | Supports | sha256 (first 16) |
|---|---|---|
evidence/F1/completions.jsonl | F1 | 9c1e04a7b2d35f80 |
evidence/F1/fs-diff.txt | F1 | 41f7d0c3e98a6b12 |
evidence/F2/sink.log | F2 | b83a5e2f0c147d96 |
evidence/F2/observations.jsonl | F2 | 2de90b6a71c4f835 |
evidence/F3/digest-source.txt | F3 | e6047c9d3a1b58f2 |
evidence/F4/forged.json, verify.out | F4, C4 | 7fa2c81e05d96b3c |
evidence/controls/C1-C4.jsonl | C1 to C4 | c58b3f9e24a0d671 |