A deterministic implementation of the #1509 pre-execution validation checklist — AgentDojo ASR=0 / FP=0 with real LLM in the loop (open artifacts)

[Lsy1533133]

#1509 proposes pre-execution validation for agent actions with a concrete checklist. We built exactly that layer, ran it through the AgentDojo official benchmark with a real LLM in the loop, and are publishing the artifacts so the community can audit every number. This issue is a working implementation path for #1509, not a competing proposal.

Point-by-point against the #1509 checklist

#1509 asks for	agent-action-verifier provides
agent identity and context	declared plan binding: scope/role checks per action (R3)
action being performed	plan–execution differential: action must be inside the declared plan set (R1)
parameters (schema + expected values)	dangerous value classes + arithmetic guard on parameters (R5/R6)
destination or external system	egress allowlist on host/port/protocol (R2)
timestamp / validity window	declared-plan validity and state checks (R7 initialisation guard)
optional nonce / replay protection	deterministic verdict keyed on declared plan + action tuple
validation fails → execution does not occur	PASS/VETO verdict at the dispatch point; VETO blocks before side effects
"wrapper around execution functions or system-level enforcement layer"	both shapes supported; public contract in verifier_interface.pyi

Repo: https://github.com/Lsy1533133/agent-action-verifier

Closed-loop results (each reported number is backed by a checksummed artifact in the repo)

• AgentDojo official benchmark, real LLM in the loop (97 tasks × 2 rounds, official injection suite, official security() judgment): ASR = 0, FP = 0 in the v2.2 full re-run
• Published fix-cycle trajectory FP 16 → 1 → 0 across v1.2 → v2.1 → v2.2 — same-source iteration, failures included
• Cross-model spot check: GLM subset (24 benign + 24 attack): FP = 0, ASR = 0
• Synthetic stress layer (100,000 seeded scenarios, 7 rule families): FN = 0, FP = 0; interception Wilson-95 lower bound 99.99%+
• White-box adaptive attacks: 49 cases, 16 vector families — 43 hard-blocked, 6 documented boundary cases, 0 bypass

Design properties relevant to AgentScope

• Judge is fixed-constant pure math: no LLM call in the enforcement path, no learned parameters → microsecond-scale per-action cost (P99 = 0.13 ms stress suite) and reproducible verdicts
• Deterministic and seeded — same inputs always yield the same verdict, so it can sit inside an auditable pipeline
• The layer complements monitoring and alignment training; it does not replace them

Honest boundaries
Synthetic scenarios are abstracted from publicly disclosed incident categories, not production traffic. Same-source fix cycles are not independent stability trials. GLM subset vs full run differ in model and sample size and are not directly comparable.

Verify it yourself
python verify_artifacts.py recomputes every SHA-256 chain and the Wilson-95 bounds from raw counts — stdlib only, no trust required.

Feedback on the methodology is very welcome. If the maintainers see a fit, we'd align the contract with AgentScope's execution seams (and with the middleware direction in #1712 / #2022 where relevant).