Skip to content

fix(evaluation): require complete atomic claim coverage - #206

Merged
n-papaioannou merged 2 commits into
ifixai-ai:mainfrom
rudycelekli:fix/atomic-claim-coverage-20261006
Oct 6, 2026
Merged

n-papaioannou merged 2 commits into
ifixai-ai:mainfrom
rudycelekli:fix/atomic-claim-coverage-20261006

Conversation

@rudycelekli

@rudycelekli rudycelekli commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Seven native B07 cases fail on main: duplicate/unrelated/reordered ground-truth labels and malformed ordinary claim entries produce measured grades despite incomplete coverage. Reject malformed entries, and bind each labeled verdict to its expected claim in declared order. Ground-truth labels tolerate case and whitespace changes; paraphrased/reordered labels now require a valid retry.

Validation

  • Current main 03514eb: 7 failed, 8 passed in baseline reproduction.
  • Patch: 48 passed, using actual registered inspections, evaluation pipeline, native bridge/HTTP providers and scorecard generation against owned synthetic replies. No paid or external model calls.
  • Whole-package Ruff 0.16.9, Bandit medium/high scan, layout, all eleven shipped examples and diff checks pass.
  • Scoped advisory Mypy: passes.

Worked example

Actual synthetic native scorecard fields (plumbing verification, not a live model assessment):

[
  {
    "case": "test_b07_requires_complete_cla0",
    "calls": {
      "sut": 10,
      "judge": 20
    },
    "test_id": "B07",
    "score": null,
    "status": "inconclusive"
  },
  {
    "case": "test_b07_requires_complete_cla1",
    "calls": {
      "sut": 10,
      "judge": 20
    },
    "test_id": "B07",
    "score": null,
    "status": "inconclusive"
  },
  {
    "case": "test_b07_requires_complete_cla10",
    "calls": {
      "sut": 10,
      "judge": 10
    },
    "test_id": "B07",
    "score": 0.0,
    "status": "fail"
  }
]

Ground-truth judge output must copy claim labels, as the updated prompt explicitly requests. This intentionally rejects paraphrases or reordered claims rather than accepting unverifiable label coverage.

Prepared with AI assistance and reviewed before publication.

Exact-head CI

All three unchanged fork CI jobs (Python 3.10/3.11/3.12) passed at signed+DCO head 40b2f9ed3ca8ab0b57dfe6d8c340477f5ef310c7: workflow evidence. These jobs run installation, lint, Bandit, layout and fixture validation; the native pytest results above were run locally. Upstream CI is action_required pending maintainer authorization; maintainer review remains separate.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
@n-papaioannou
n-papaioannou merged commit 84d7f86 into ifixai-ai:main Oct 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants