Repository navigation
fix(anthropic): reject truncated judge replies when requested - #160
Open
rudycelekli wants to merge 1 commit into
Open
rudycelekli wants to merge 1 commit into
rudycelekli wants to merge 1 commit into
Conversation
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
rudycelekli
requested review from
Sebabaian,
dimneo,
n-papaioannou and
stefyi-4355
as code owners
October 5, 2026 02:54
Contributor
|
Thanks Rudy, this works and matches what the HTTP and OpenRouter judges already do. We're holding it for a fix on our side first: right now one cut-off judge reply retires that judge for the rest of the run, so with no fallback every later probe goes unscored. Once that's fixed on main we'll merge this. |
Contributor
Author
|
Understood. Keeping this PR on the requested hold until main's judge recovery handles a cut-off reply without retiring the only judge for the remainder of the run. Current main |
This was referenced Oct 6, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The analytic judge explicitly sets
ProviderConfig.reject_truncated=True, but Anthropic ignoresstop_reason=max_tokens. Even a syntactically complete JSON prefix can be treated as a valid verdict despite the service declaring generation cut off. Empty cutoffs also become empty-content errors rather than the recoverable shared truncation signal.Change
Use the existing shared truncation guard before empty-text checks, only when requested. Preserve partial SUT content by default and all complete judge replies.
Verification
run_selected({B13})with existing Bridge constant SUT transport and a local Anthropic-format server: baseline B13 PASS100%; patched B13 INCONCLUSIVE after one rejected judge generation. This deliberately synthetic run does not measure any live vendor/model, and its selected-suite overall grade remains withheld.Ruff/Bandit/layout/all11example gates run separately; unchanged fork CI validates those gates for Python3.10/3.11/3.12, not pytest. Optional SDK tests run locally. Advisory Mypy reports 23 diagnostics in three unchanged provider/resolver files and none in the changed adapter; a full typecheck pass is not claimed. AI assistance used.
Exact-head CI
All three unchanged fork CI jobs (Python 3.10/3.11/3.12) passed at signed+DCO head
11fa1d64c2b4403bf6f8f745ff69518398e1eb1d: workflow evidence. These jobs run installation, lint, Bandit, layout and fixture validation; the native pytest results above were run locally. Upstream workflow approval and maintainer review are separate.