Skip to content

fix(anthropic): reject truncated judge replies when requested - #160

Open
rudycelekli wants to merge 1 commit into
ifixai-ai:mainfrom
rudycelekli:fix/anthropic-judge-cutoff-20261004
Open

rudycelekli wants to merge 1 commit into
ifixai-ai:mainfrom
rudycelekli:fix/anthropic-judge-cutoff-20261004

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

Problem

The analytic judge explicitly sets ProviderConfig.reject_truncated=True, but Anthropic ignores stop_reason=max_tokens. Even a syntactically complete JSON prefix can be treated as a valid verdict despite the service declaring generation cut off. Empty cutoffs also become empty-content errors rather than the recoverable shared truncation signal.

Change

Use the existing shared truncation guard before empty-text checks, only when requested. Preserve partial SUT content by default and all complete judge replies.

Verification

  • Real Anthropic SDK0.107.1 over owned HTTP transport: baseline3failed/3passed; patched6passed, zero skipped. Includes the actual analytic judge + evaluation pipeline with a parseable cut-off verdict and complete-verdict control.
  • Worked production run_selected({B13}) with existing Bridge constant SUT transport and a local Anthropic-format server: baseline B13 PASS100%; patched B13 INCONCLUSIVE after one rejected judge generation. This deliberately synthetic run does not measure any live vendor/model, and its selected-suite overall grade remains withheld.
Illustrative owned-transport B13 scorecard
Before: B13 pass, score1.0; no judge transport failures
After: B13 inconclusive, score0.0 (unscored); owned-judge transport failures1
Reason: ProviderTruncatedError, finish_reason=max_tokens

Ruff/Bandit/layout/all11example gates run separately; unchanged fork CI validates those gates for Python3.10/3.11/3.12, not pytest. Optional SDK tests run locally. Advisory Mypy reports 23 diagnostics in three unchanged provider/resolver files and none in the changed adapter; a full typecheck pass is not claimed. AI assistance used.

Exact-head CI

All three unchanged fork CI jobs (Python 3.10/3.11/3.12) passed at signed+DCO head 11fa1d64c2b4403bf6f8f745ff69518398e1eb1d: workflow evidence. These jobs run installation, lint, Bandit, layout and fixture validation; the native pytest results above were run locally. Upstream workflow approval and maintainer review are separate.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
@n-papaioannou

Copy link
Copy Markdown
Contributor

Thanks Rudy, this works and matches what the HTTP and OpenRouter judges already do. We're holding it for a fix on our side first: right now one cut-off judge reply retires that judge for the rest of the run, so with no fallback every later probe goes unscored. Once that's fixed on main we'll merge this.

@rudycelekli

Copy link
Copy Markdown
Contributor Author

Understood. Keeping this PR on the requested hold until main's judge recovery handles a cut-off reply without retiring the only judge for the remainder of the run. Current main e08f1ca8f481dff49f15f89c6df600920d438a2f still retires a model after JudgeTransportExhaustedError; no changes here override that decision. The provider cutoff patch can be revalidated after the upstream recovery fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants