Re-verifying an item carried unresolved across two revisions of an internal
roadmap document: the claimed "resource exhaustion / agentic DoS" gap in
docs/agents/research/benchmark-2026-06.md.
What the document claims
Primary source: MCPSecBench class 13 "Resource Exhaustion";
MCP-SafetyBench class 6 "Denial of Service". Both independently enumerate
this as a distinct class with concrete test cases.
Presented as the corpus's one confirmed genuine gap, recommended as a new
record (AVE-2026-00052 at the time, since taken by a different record).
What the actual primary sources say
Fetched and read both papers directly (not summaries), including a direct
PDF pull for both to rule out extraction error:
- MCPSecBench (arXiv:2508.13220). Table 1's own footnote enumerates all
17 attack types by name: Prompt Injection, Tool/Service Misuse via
"Confused AI", Schema Inconsistencies, Slash Command Overlap, MCP
Rebinding, Man-in-the-Middle, Tool Shadowing Attack, Data Exfiltration,
Package Name Squatting (tool name), Indirect Prompt Injection, Package
Name Squatting (server name), Tool Poisoning, Rug Pull Attack, Vulnerable
Client, Configuration Drift, Sandbox Escape, Vulnerable Server. No
"Resource Exhaustion" class exists anywhere in this list, at position 13
or otherwise.
- MCP-SafetyBench (arXiv:2512.15163, published ICLR 2026). Table 2 gives
the full, named list of all 20 attack types across MCP Server, Host, and
User sides. No "Denial of Service" class exists anywhere in this list.
Both citations are wrong, not approximately right or renamed since drafting.
Checked the current published/most-recent version of each paper; did not
diff against earlier preprint revisions, so a small residual chance an early
draft briefly carried such a class before publication isn't fully ruled out,
but the currently-citable, currently-correct version of both papers has
never had one.
Disposition
Per the roadmap item's own stated resolution criteria, this is the third
outcome: the benchmark's claim doesn't hold up on inspection. Not "already
covered by AVE-2026-00038" (the near-neighbor, unbounded tool use) either,
since there's no confirmed gap to be covered in the first place.
Dropping this item from the trust-strategy roadmap rather than carrying it
a third time. Not proposing a fix to benchmark-2026-06.md itself in this
issue; that document is a point-in-time research artifact, and rewriting its
conclusion after the fact would remove the negative result rather than
publish it. If a resource-exhaustion-shaped behavioral class is worth adding
to AVE at some point, it needs its own, freshly-verified primary-source
justification, not a revival of this one.
Why this matters beyond one roadmap line
Two independent citation checks this month found real problems: this one
(AVE's own internal research document misciting two papers it apparently
never checked against their actual, final text) and an external one
(the OWASP GenAI Crosswalk's MITRE ATLAS mappings, 5 of ~8 checked wrong or
nonexistent). Same underlying failure mode either way: citing a specific
named source and class without opening the source and checking. Worth
treating as a standing reminder for any future benchmark or gap-analysis
pass, not just this one.
Re-verifying an item carried unresolved across two revisions of an internal
roadmap document: the claimed "resource exhaustion / agentic DoS" gap in
docs/agents/research/benchmark-2026-06.md.What the document claims
Presented as the corpus's one confirmed genuine gap, recommended as a new
record (
AVE-2026-00052at the time, since taken by a different record).What the actual primary sources say
Fetched and read both papers directly (not summaries), including a direct
PDF pull for both to rule out extraction error:
17 attack types by name: Prompt Injection, Tool/Service Misuse via
"Confused AI", Schema Inconsistencies, Slash Command Overlap, MCP
Rebinding, Man-in-the-Middle, Tool Shadowing Attack, Data Exfiltration,
Package Name Squatting (tool name), Indirect Prompt Injection, Package
Name Squatting (server name), Tool Poisoning, Rug Pull Attack, Vulnerable
Client, Configuration Drift, Sandbox Escape, Vulnerable Server. No
"Resource Exhaustion" class exists anywhere in this list, at position 13
or otherwise.
the full, named list of all 20 attack types across MCP Server, Host, and
User sides. No "Denial of Service" class exists anywhere in this list.
Both citations are wrong, not approximately right or renamed since drafting.
Checked the current published/most-recent version of each paper; did not
diff against earlier preprint revisions, so a small residual chance an early
draft briefly carried such a class before publication isn't fully ruled out,
but the currently-citable, currently-correct version of both papers has
never had one.
Disposition
Per the roadmap item's own stated resolution criteria, this is the third
outcome: the benchmark's claim doesn't hold up on inspection. Not "already
covered by AVE-2026-00038" (the near-neighbor, unbounded tool use) either,
since there's no confirmed gap to be covered in the first place.
Dropping this item from the trust-strategy roadmap rather than carrying it
a third time. Not proposing a fix to
benchmark-2026-06.mditself in thisissue; that document is a point-in-time research artifact, and rewriting its
conclusion after the fact would remove the negative result rather than
publish it. If a resource-exhaustion-shaped behavioral class is worth adding
to AVE at some point, it needs its own, freshly-verified primary-source
justification, not a revival of this one.
Why this matters beyond one roadmap line
Two independent citation checks this month found real problems: this one
(AVE's own internal research document misciting two papers it apparently
never checked against their actual, final text) and an external one
(the OWASP GenAI Crosswalk's MITRE ATLAS mappings, 5 of ~8 checked wrong or
nonexistent). Same underlying failure mode either way: citing a specific
named source and class without opening the source and checking. Worth
treating as a standing reminder for any future benchmark or gap-analysis
pass, not just this one.