fix(judge): protect unbounded judge transcript for non-litellm agents#837
Conversation
…gents JudgeAgent's trace-size protection (structure-only digest + expand_trace/ grep_trace discovery) was gated entirely on JudgeSpanCollector spans, which only ever contain spans from autotrack_litellm_calls. An AgentAdapter that talks to its own backend directly (REST/SSE/etc., not via litellm) produces tool-call messages that never become spans, so is_large_trace never tripped for such agents while the raw `transcript` block built from input.messages flowed into the judge prompt with no cap of any kind. A large enough transcript could get "lost in the middle" for a small/cheap judge model, producing a confident but wrong verdict instead of a visible failure. Gate the transcript on its own estimated token count, independent of the span digest. When it's large, render a structure-only skeleton (one line per message: index, role, tool-call name, size) and offer the judge expand_transcript/grep_transcript tools -- mirroring expand_trace/grep_trace but keyed on message position instead of span ID -- so the full content is still reachable on demand instead of force-fed on every call. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughLarge transcripts now use bounded skeleton prompts with ChangesTranscript discovery
Sequence Diagram(s)sequenceDiagram
participant AgentInput
participant JudgeAgent
participant transcript_tools
participant JudgeModel
AgentInput->>JudgeAgent: working_messages
JudgeAgent->>transcript_tools: build_transcript_skeleton(working_messages)
transcript_tools-->>JudgeAgent: skeleton transcript
JudgeAgent->>JudgeModel: prompt with skeleton and discovery tools
JudgeModel->>JudgeAgent: expand_transcript or grep_transcript call
JudgeAgent->>transcript_tools: execute discovery tool
transcript_tools-->>JudgeAgent: bounded transcript content
JudgeAgent->>JudgeModel: discovery result
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/scenario/_judge/judge_utils.py`:
- Around line 180-242: Replace broad Any, Dict, List, and Sequence annotations
with the concrete message and tool-call types already used by the judge APIs,
using built-in dict[...] and list[...] generics. In
python/scenario/_judge/judge_utils.py:180-242, narrow _render_message_line,
render_transcript_lines, and related helpers; in
python/scenario/_judge/transcript_tools.py:20-60 and 77-217, type
_IndexedMessage.message and update helper annotations, marking budget constants
as Final[int]; in python/scenario/judge_agent.py:852-861 and 1020-1022, narrow
working_messages and tool_call to their concrete types.
In `@python/scenario/_judge/transcript_tools.py`:
- Around line 97-120: The skeleton-building logic around the indexed-message
loop must enforce a total token or character budget, including tool-call
metadata, so it cannot exceed the judge context. Cap displayed tool names and
metadata per message, stop adding entries once the budget is reached, and append
a concise indication of omitted contiguous index ranges so omitted messages
remain discoverable.
- Around line 146-162: Limit requested indices and render selected transcript
entries incrementally in the transcript indexing function, stopping when the
character budget is reached instead of joining all selected content before
truncation; preserve invalid-index reporting. In
python/scenario/_judge/transcript_tools.py lines 146-162, update the
selection/rendering flow accordingly. In python/scenario/judge_agent.py lines
815-826, add a matching maxItems constraint to the indices tool schema.
In `@python/tests/test_judge_transcript_size_protection.py`:
- Around line 31-34: Add explicit, parameterized type annotations throughout the
helper, fixture, and callback definitions covered by this change, including
message payloads, tool arguments, fixture yields, and the mock_completion
interface. Replace bare list/dict annotations with concrete generic collection
types and annotate missing parameters and return values, including
create_mock_collector. Leave pytest test methods unannotated to match repository
convention.
- Around line 102-111: Update the setup_config fixture to capture
ScenarioConfig.default_config before replacing it, then restore that saved value
during teardown instead of unconditionally assigning None. Preserve the existing
context_scenario cleanup and test configuration setup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4ea0b0eb-211c-4c26-bcd7-fb177cc59beb
📒 Files selected for processing (5)
python/scenario/_judge/__init__.pypython/scenario/_judge/judge_utils.pypython/scenario/_judge/transcript_tools.pypython/scenario/judge_agent.pypython/tests/test_judge_transcript_size_protection.py
Self-review caught a crash: expand_transcript's indices arg is reachable from an LLM tool call, and sorted(set(indices) - set(valid_range)) raises TypeError the moment a non-integer sneaks into that list (e.g. a lax litellm-routed provider that doesn't enforce the "type": "integer" schema as strictly as OpenAI, sending "2" instead of 2). Verified the crash reproduces, then coerced/filtered indices defensively instead of trusting the schema to hold at runtime -- matching the "never raises" contract this file already applies to tool-call argument rendering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
python/tests/test_judge_transcript_size_protection.py (1)
310-314: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the concrete ignored indices.
The test only checks the generic marker, so it would pass even if the ignored values were omitted. Assert
"Ignored non-integer indices: [None, {}]"to pin the diagnostic contract. Based on the providedexpand_transcriptcontract, ignored values are part of the returned diagnostic.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/tests/test_judge_transcript_size_protection.py` around lines 310 - 314, Update the assertions in the expand_transcript test to verify the complete ignored-index diagnostic, including the concrete values “[None, {}]”, rather than only checking the generic marker. Preserve the existing assistant output assertion and validate the returned message exactly as specified by the diagnostic contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@python/tests/test_judge_transcript_size_protection.py`:
- Around line 310-314: Update the assertions in the expand_transcript test to
verify the complete ignored-index diagnostic, including the concrete values
“[None, {}]”, rather than only checking the generic marker. Preserve the
existing assistant output assertion and validate the returned message exactly as
specified by the diagnostic contract.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7a0addb9-7405-491b-9edc-131d0568de57
📒 Files selected for processing (2)
python/scenario/_judge/transcript_tools.pypython/tests/test_judge_transcript_size_protection.py
🚧 Files skipped from review as they are similar to previous changes (1)
- python/scenario/_judge/transcript_tools.py
Addresses CodeRabbit findings on #837: - build_transcript_skeleton had no cap of its own. A transcript of thousands of tiny messages (streaming/high-frequency tool-call agent) produces a skeleton whose *total* size can exceed the same token_threshold that triggered structure-only rendering in the first place -- the exact failure mode this fix exists to close. Verified: 5000 short messages produced a 32852-token skeleton against an 8192 default threshold. Now bounded to a head+tail window (start frames the scenario, end is what the judge is deciding on) with the elided middle still reachable via expand_transcript/grep_transcript. - expand_transcript's index matching checked membership against a List (O(N*M) for N messages and M requested indices); switched to a set for O(1) lookups. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/tests/test_judge_transcript_size_protection.py`:
- Around line 327-331: Replace the cast(Any, messages) calls in the transcript
skeleton test cases with the project’s concrete parameterized chat-message type,
and annotate the messages fixtures accordingly. Pass the typed messages directly
to build_transcript_skeleton at all referenced cases so Pyright validates their
structure without Any.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 619362a4-9e9e-4e5c-a9c5-e0b480a12db7
📒 Files selected for processing (2)
python/scenario/_judge/transcript_tools.pypython/tests/test_judge_transcript_size_protection.py
🚧 Files skipped from review as they are similar to previous changes (1)
- python/scenario/_judge/transcript_tools.py
Adds direct-unit coverage for the guard clauses that weren't exercised by the earlier repro tests: empty-transcript short-circuits on all three functions, expand_transcript's empty/out-of-range index requests, grep_ transcript's no-match and >MAX_GREP_MATCHES branches, and the char-budget truncation path. Also covers _execute_discovery_tool's JSON-decode-error and unknown-tool-name branches through the real discovery loop, since those sit directly next to the expand_transcript/grep_transcript dispatch arms this fix added. scenario/_judge/transcript_tools.py (the new module) is now at 100% line coverage. judge_utils.py's and judge_agent.py's remaining coverage gaps are pre-existing, in code this PR didn't touch (base64/audio media truncation edge cases, voice modality resolution, deep verdict-parsing branches) -- out of scope for #836. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Aryansharma28
left a comment
Aryansharma28
left a comment
There was a problem hiding this comment.
Deep self-review of #837
Since I authored this fix, I ran it adversarially against itself: exercised every guard clause directly, tried to break expand_transcript/grep_transcript with malformed LLM tool-call input, checked the combined-large-span-and-large-transcript path, and triaged every CodeRabbit finding rather than rubber-stamping or dismissing them. Three real issues surfaced and are fixed in follow-up commits (45bd5bd, 76c56de). No open findings remain.
Verdict: no blockers. Ready to merge once CI's unrelated infra issue (see below) is resolved or waived.
[P1] expand_transcript crashed on non-integer indices from a lax LLM provider — FIXED (45bd5bd)
Why: expand_transcript's indices argument is reachable from an LLM tool call. The tool schema declares "items": {"type": "integer"}, but not every litellm-routed provider enforces function-calling argument types as strictly as OpenAI. I reproduced the crash directly:
expand_transcript(messages, indices=["1", 5])
# TypeError: '<' not supported between instances of 'str' and 'int'sorted(set(indices) - set(valid_range)) raised the moment a non-integer entered the list — an unhandled exception that would have propagated up through _execute_discovery_tool → _run_discovery_loop → call(), crashing the entire judge call instead of degrading gracefully.
Fix: coerce/filter indices defensively (int() with try/except, malformed entries reported back to the LLM instead of raising) — matching the "never raises" contract this file already applies to tool-call argument rendering (_render_tool_arguments's docstring). Covered by test_non_integer_indices_do_not_raise / test_unparseable_indices_are_reported_not_raised.
[P2] build_transcript_skeleton had no cap on its own total size (CodeRabbit) — FIXED (76c56de)
Why: CodeRabbit correctly flagged that the skeleton renders one line per message with no total budget. A transcript of thousands of small messages (a streaming/high-frequency tool-call agent) has a tiny per-message cost but an unbounded total — undercutting the entire point of this fix, since the skeleton is the thing that's supposed to stay under token_threshold. I verified: 5000 short messages produced a 32852-token skeleton against an 8192 default threshold — 4x over.
Fix: bounded to TOOL_RESULT_TOKEN_BUDGET (~4096 tokens) with a head+tail window (start frames the scenario, end is what the judge is deciding on) and the elided middle still reachable via expand_transcript/grep_transcript. Verified the fix brings the same 5000-message case down to 4099 tokens while preserving [0] and [4999]. Covered by TestSkeletonItselfIsBounded.
[P3] expand_transcript index matching was O(N×M) (CodeRabbit, partial) — FIXED (76c56de)
Why: membership-checked entry.index in parsed_indices against a List[int] rather than a set. Cheap, real, no downside to fix.
What I did NOT apply from CodeRabbit's suggestion: the "incrementally render + stop early" rewrite and a maxItems schema cap. The total-transcript-string cost this would guard against is already unconditionally paid every judge call (JudgeUtils.build_transcript_from_messages runs regardless of size to decide is_large_transcript), so the marginal protection is small relative to the added complexity, and expand_trace's existing span_ids schema has no equivalent cap either — adding one only to the new tool would be an asymmetry without a matching real-world risk.
Triaged and intentionally not applied
- CodeRabbit: tighten
Any/Dict/Listto concrete generics acrossjudge_utils.py/transcript_tools.py/judge_agent.py. These match the pre-existing convention in this exact package —_execute_discovery_tool(self, tool_call: Any, spans: Sequence[Any])usedAnyfor both parameters before this PR touched it. Tightening only the new code would make it inconsistent with the file it extends. - CodeRabbit: "use jsonify instead of json.dumps" (ast-grep). False positive — this is a Flask-specific lint rule firing on a non-Flask codebase; there's no Flask response object anywhere near this code.
- CodeRabbit: add concrete type annotations to test helpers. The existing sibling test file (
test_judge_agent_progressive.py) already uses the same loose-annotation style I matched (def create_large_trace() -> list:,def mock_litellm_response(tool_name: str, args: dict):with no return type). Diverging only in the new test file would reduce consistency, not improve it. - CodeRabbit: restore
ScenarioConfig.default_configto its prior value instead ofNonein the test fixture teardown. Valid observation, but it's copy-pasted verbatim from the samesetup_configfixture already duplicated across every judge test file in this suite. Fixing it in isolation in one file doesn't actually improve test isolation (the other files still reset toNone), and a suite-wide fix is out of scope for #836.
Verified directly (adversarial, not just read)
- Reproduced the original bug against unmodified
main: 4/5 new tests fail without the fix, pass with it. - Combined-large-trigger case: built a scenario where both span digest and transcript exceed threshold simultaneously — confirmed all four discovery tools (
expand_trace,grep_trace,expand_transcript,grep_transcript) are offered together with no crash. - Empty-messages, malformed
tool_calls, out-of-range/unparseable indices, no-grep-match, >20-grep-match, and truncation-at-budget paths all exercised directly. scenario/_judge/transcript_tools.py(the new module) is at 100% line coverage. Remaining gaps injudge_utils.py/judge_agent.pyare pre-existing, in code this PR didn't touch (base64/audio media truncation edge cases, voice modality resolution, deep verdict-parsing branches) — out of scope for #836.pyrightclean on every touched file.- Full
pytest tests/(1139 tests): passes; only pre-existing e2e failures requiring live API keys, unrelated to this change.
Things I couldn't verify from the diff alone
- CI's
Test (Examples)step is failing on this PR withAuthenticationError: Incorrect API key providedfor every example test. This is unambiguous: it fails before any of this PR's code runs, on every single example regardless of content, and main's most recent successful run (2026-07-20) passed this exact step — this reads as theOPENAI_API_KEYrepo secret being invalid/rotated as of today, not something introduced by this change. I don't have access to repo secrets to confirm or fix it. Flagging so a maintainer with secret access can check, rather than blocking this PR on it.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
python/tests/test_judge_transcript_size_protection.py (1)
420-500: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract shared helper to remove duplication between the two dispatch tests.
test_malformed_tool_call_arguments_do_not_crash_the_loopandtest_hallucinated_tool_name_reports_unknown_toolrepeat nearly identical MagicMock tool-call construction and an almost-identicalmock_completionclosure. A small helper (e.g._mock_tool_call_response(tool_name, arguments, call_id)plus a shared "capture tool response content" driver) would remove this duplication.♻️ Sketch of a shared helper
+def _mock_tool_call_response(tool_name: str, arguments: str, call_id: str) -> MagicMock: + response = MagicMock() + response.choices = [MagicMock()] + tool_call = MagicMock() + tool_call.id = call_id + tool_call.function.name = tool_name + tool_call.function.arguments = arguments + response.choices[0].message.tool_calls = [tool_call] + response.choices[0].message.content = None + response.choices[0].message.role = "assistant" + return response🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/tests/test_judge_transcript_size_protection.py` around lines 420 - 500, Extract the duplicated tool-call response setup and completion-capture logic from test_malformed_tool_call_arguments_do_not_crash_the_loop and test_hallucinated_tool_name_reports_unknown_tool into small shared test helpers, such as a MagicMock response builder and a completion driver. Keep each test’s distinct tool name, arguments, call ID, expected captured content, call count, and result assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@python/tests/test_judge_transcript_size_protection.py`:
- Around line 420-500: Extract the duplicated tool-call response setup and
completion-capture logic from
test_malformed_tool_call_arguments_do_not_crash_the_loop and
test_hallucinated_tool_name_reports_unknown_tool into small shared test helpers,
such as a MagicMock response builder and a completion driver. Keep each test’s
distinct tool name, arguments, call ID, expected captured content, call count,
and result assertions unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 6f5e4403-a124-4428-a94e-c9d24de593d4
📒 Files selected for processing (1)
python/tests/test_judge_transcript_size_protection.py
Adds examples/test_judge_transcript_size_protection_e2e.py, following this repo's existing examples/ pattern for live-LLM tests. A NonLitellmSalesAgent returns 27 large tool-call/tool-result messages in one turn -- never touching litellm itself, so JudgeSpanCollector stays empty -- with a single fact buried in the middle. A real JudgeAgent (gpt-4.1-mini) must find that fact through the transcript's own discovery tools to pass. Verified locally against a live OpenAI key: PASSED. The judge correctly located and used the buried fact despite the large non-litellm transcript, confirming the fix works end-to-end with a real model, not just in mocked unit tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
github-actions
Bot
added
the
low-risk-change
![github-actions[bot]](https://avatars.github.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/examples/test_judge_transcript_size_protection_e2e.py`:
- Around line 63-71: Update the final assistant message in the test setup so it
is generic and does not repeat NEEDLE_FACT or any equivalent identifying fact.
Keep NEEDLE_FACT present only in the buried tool result, while preserving the
existing success assertions and transcript-discovery flow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9fee658d-882b-461b-a3e8-93ff5e806389
📒 Files selected for processing (1)
python/examples/test_judge_transcript_size_protection_e2e.py
…ummary CodeRabbit review (PR #837): the final assistant message repeated NEEDLE_FACT verbatim, so the judge could pass by reading that summary alone (e.g. a cheap expand_transcript on just the last message) without ever searching the buried tool result -- defeating the point of the test. Made the summary generic so NEEDLE_FACT exists only inside the buried tool-call result. Re-verified live against gpt-4.1-mini: still PASSED, now a genuine proof that the judge searches the transcript rather than reading a conveniently-placed restatement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
github-actions
Bot
removed
the
low-risk-change
…ure state Addresses the remaining unresolved CodeRabbit findings on PR #837: - scenario/_judge/transcript_tools.py: _IndexedMessage.message is now ChatCompletionMessageParam instead of Any; the three budget constants are Final[int]. - scenario/_judge/judge_utils.py: _render_message_line's msg param is now ChatCompletionMessageParam instead of Any. - scenario/judge_agent.py: working_messages (the parameter this fix introduced) is now Sequence[ChatCompletionMessageParam] instead of Sequence[Any] in both _run_discovery_loop and _execute_discovery_tool. - tests/test_judge_transcript_size_protection.py: helper functions (create_mock_collector, create_large_non_litellm_transcript, create_base_input, mock_litellm_response, setup_config) now have concrete parameter/return types instead of bare list/dict/no-annotation; cast(Any, messages) narrowed to cast(List[ChatCompletionMessageParam], messages) everywhere the underlying data is well-formed (the deliberately malformed `indices` casts in the malformed-input tests stay Any, since narrowing those would misrepresent what's being tested). - setup_config fixture now restores ScenarioConfig.default_config to its prior value instead of unconditionally resetting to None. Not applied: switching to lowercase builtin generics (dict[...]/list[...]) in judge_utils.py/transcript_tools.py -- every other file in this package consistently imports List/Dict from typing (verified across trace_tools.py, judge_agent.py, judge_span_digest_formatter.py); doing it only in the new code would make this PR's files inconsistent with the package it extends. Also not applied: rewriting judge_utils.py's pre-existing base64/media-truncation functions' Any usage -- that's unchanged logic outside this PR's diff, not something #836 touches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Automated low-risk assessment This PR was evaluated against the repository's Low-Risk Pull Requests procedure and does not qualify as low risk.
This PR requires a manual review before merging. |
|
All 7 CodeRabbit review threads are now resolved:
Full test suite (113 tests across the judge suite) + pyright: clean after every change. |
2 participants
Fixes #836
Problem
JudgeAgent's trace-size protection (structure-only digest +expand_trace/grep_tracediscovery loop) is gated entirely onis_large_trace, which is computed only from spans collected byJudgeSpanCollector— i.e. spans produced byautotrack_litellm_calls. AnyAgentAdapterthat talks to its own backend directly (REST/SSE/gRPC/etc.) and never routes calls throughlitellmproduces tool-call messages that never become spans.For such agents,
spansstays empty/small forever, sois_large_tracenever trips — while the actualtranscriptblock (built frominput.messagesviaJudgeUtils.build_transcript_from_messages) has no size cap of any kind beyond stripping base64 media. Large tool-call content (SQL result sets, long documents, etc.) flows into the judge prompt completely unbounded.The reporter hit this with a proprietary agent whose 27 tool calls (with large SQL result payloads) never became spans; the judge (
gpt-4o-mini) silently dropped/ignored large portions of the unbounded transcript ("lost in the middle") and returned a confident but wrongfaithfulness: 0.0verdict claiming "no tool calls were made" — a false negative with no error signal at all.Fix
is_large_traceis nowis_large_span_trace OR is_large_transcript, whereis_large_transcriptis computed from the actual transcript's own estimated token count — independent of spans.expand_transcript(indices)andgrep_transcript(pattern), mirrorexpand_trace/grep_tracebut operate on message position instead of span ID, so the judge can still reach full content on demand instead of it being force-fed (or silently dropped) on every call.JudgeUtils.build_transcript_from_messagesis refactored to exposerender_transcript_lines, a per-message renderer shared by the full transcript builder and the new discovery tools, soexpand_transcriptshows byte-identical content to what the full transcript would have shown.Testing
tests/test_judge_transcript_size_protection.pyreproduces the exact bug: emptyspans(simulating a non-litellm agent) + a large transcript (27 tool calls with large payloads, mirroring the reporter's repro). Verified this test fails against unmodifiedmain(4/5 cases) and passes after the fix.test_judge_agent*.py,test_judge_discovery_*.py,test_judge_utilities.py,test_judge_force_verdict_hardening.py) passes unchanged — the span-based path is untouched.pytest tests/run: 1139 passed (the only failures are pre-existing e2e tests requiring live API keys, unrelated to this change).pyrightclean on all touched files.Test plan
mainpyrightclean🤖 Generated with Claude Code