feat: add NDCG benchmark metrics

[Lucas-Bur]

Summary

• add binary NDCG@5/10/20/50 to benchmark measurements, worker evaluation, quality summaries, uncertainty, and reports
• compare NDCG-first direct retrieval with a Recall-first ablation over the same candidate archive and preserve objective-specific guardrails
• reject empty ground truth, correct promotion stability metrics, and make Successive Halving objective-specific
• keep Production retrieval unchanged; this PR is benchmark-only

Validation

• �p check`n- �p test (467 passed, 1 skipped)
• benchmark fixture and focused benchmark tests (31 passed)
• worker-pool tests (13 passed)
• proxy-promotion smoke benchmark
• successive-halving smoke benchmark
• �p run lint:fallow (no duplication; inherited dependency/complexity health findings remain)
• �p run lint:fallow:ci (only inherited findings)

Evidence

The refreshed fd/MiniLM smoke still favors Recall-first on excluded-fold NDCG and R@5 while tying Recall@20 and Context@4k. No Production promotion is claimed.

Closes #172
Parent: #166

Summary by CodeRabbit

• New Features

  • Added binary NDCG metrics at cutoffs 5, 10, 20, and 50 to retrieval evaluations and reports.
  • Added recall-first direct retrieval comparisons alongside NDCG-first evaluation.
  • Added objective-specific candidate selection and guardrail handling.

• Bug Fixes

  • Benchmark validation now rejects empty ground-truth targets and prevents duplicate or overlapping target issues.

• Documentation

  • Updated benchmark documentation, reports, and baseline records to describe the new metrics, objectives, and artifact schema.

[coderabbitai]

Warning

Review limit reached

@Lucas-Bur, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 50 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8c15f836-ecd2-4d65-bbbb-bfcdc9e09aaa

📥 Commits

Reviewing files that changed from the base of the PR and between a0fe039 and 8877dab.

📒 Files selected for processing (5)

• benchmarks/README.md
• benchmarks/retrieval/evaluation/optimization-profiles.ts
• benchmarks/retrieval/evaluation/weight-search.ts
• benchmarks/tests/channels.test.ts
• benchmarks/tests/optimization-profiles.test.ts

📝 Walkthrough

Walkthrough

Schema 26 adds binary NDCG@5/10/20/50 metrics, rejects empty ground truth, adds a recall-first direct objective, applies objective-specific guardrails and archive selection, updates reports, and adds validation for metrics, objectives, artifacts, and router results.

Changes

Retrieval evaluation and objective selection

Layer / File(s)	Summary
NDCG metrics and evaluation contracts benchmarks/retrieval/evaluation/metrics*, benchmarks/retrieval/evaluation/collect.ts, benchmarks/retrieval/evaluation/prepared-fusion-core.mjs, benchmarks/retrieval/evaluation/types.ts, benchmarks/tests/channels.test.ts, benchmarks/tests/corpus.test.ts	Binary NDCG uses unique ranked results and exact file-plus-symbol targets. Query measurements and candidate quality results record NDCG@5/10/20/50. Ground truth must contain at least one location.
Objective quality and guardrails benchmarks/retrieval/evaluation/optimization-profiles.ts, benchmarks/retrieval/evaluation/promotion-evidence.ts, benchmarks/retrieval/evaluation/weight-search.ts, benchmarks/tests/optimization-profiles.test.ts, benchmarks/tests/promotion-evidence.test.ts, benchmarks/tests/channels.test.ts	Direct retrieval prioritizes NDCG@5. direct-recall-first prioritizes Recall@5. Direct guardrails use Recall@20, Recall@50, and ContextRecall@4096. Reranker objectives remain recall-oriented.
Objective-specific archive and router selection benchmarks/retrieval/evaluation/types.ts, benchmarks/retrieval/evaluation/weight-search.ts, benchmarks/tests/retrieval.test.ts, benchmarks/tests/worker-pool.test.ts	Archive candidates are ranked independently for each objective. Eligible candidates are selected first, with ranked fallbacks when guardrails fail. Router results include the recall-first objective.
Schema 26 artifacts and benchmark reporting CONTEXT.md, benchmarks/BASELINE.md, benchmarks/README.md, benchmarks/retrieval/evaluation/report.ts, benchmarks/retrieval/evaluation/types.ts, benchmarks/retrieval/runner.ts, benchmarks/tests/report.test.ts, benchmarks/tests/retrieval.test.ts	Documentation and reports show NDCG metrics, direct-objective comparisons, holdout breakdowns, Schema 26 artifacts, and objective-specific archive behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant QueryMeasurement
  participant normalizedDiscountedCumulativeGain
  participant binaryNdcgAt
  participant QualitySummary
  QueryMeasurement->>normalizedDiscountedCumulativeGain: pass ranked chunks and resolved targets
  normalizedDiscountedCumulativeGain->>binaryNdcgAt: pass chunk indices and cutoff
  binaryNdcgAt-->>normalizedDiscountedCumulativeGain: return normalized binary NDCG
  normalizedDiscountedCumulativeGain-->>QualitySummary: record NDCG@5/10/20/50

Loading

Possibly related PRs

• Lucas-Bur/pix#161: Introduces shared objective-aware evaluation and archive-selection logic extended by this PR.
• Lucas-Bur/pix#167: Adds related evidence-router optimization and archive-selection behavior.
• Lucas-Bur/pix#169: Shares prepared fusion evaluation, metrics, and weight-search changes extended with NDCG support.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: adding NDCG benchmark metrics.
Linked Issues check	✅ Passed	The changes implement the linked issue requirements for NDCG metrics, objective comparison, guardrails, validation, and benchmark-only scope.
Out of Scope Changes check	✅ Passed	The changes remain focused on benchmark metrics, evaluation, reporting, tests, and objective-specific selection without Production retrieval changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/172-ndcg-direct-objective

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmarks/README.md`:
- Line 12: Update the benchmark comparison text around the Recall@K and binary
NDCG@K question to use DBSF as the production fusion method, or explicitly
identify RRF as a historical diagnostic rather than production-weighted,
consistent with CONTEXT.md.

In `@benchmarks/retrieval/evaluation/optimization-profiles.ts`:
- Around line 189-190: Update the objective priority lists in optimization
profiles, including the entries near ndcgAt20 and ndcgAt50, so reranker-top20
and reranker-top50 remain recall-oriented after ties among recall, context
recall, and MRR. Remove the NDCG priorities from those reranker lists unless an
explicit ordering contract is already defined.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 509895b4-9fbf-4c41-85c1-c1121ada462b

📥 Commits

Reviewing files that changed from the base of the PR and between 24f728c and a0fe039.

📒 Files selected for processing (21)

• CONTEXT.md
• benchmarks/BASELINE.md
• benchmarks/README.md
• benchmarks/retrieval/evaluation/collect.ts
• benchmarks/retrieval/evaluation/metrics-core.d.mts
• benchmarks/retrieval/evaluation/metrics-core.mjs
• benchmarks/retrieval/evaluation/metrics.ts
• benchmarks/retrieval/evaluation/optimization-profiles.ts
• benchmarks/retrieval/evaluation/prepared-fusion-core.mjs
• benchmarks/retrieval/evaluation/promotion-evidence.ts
• benchmarks/retrieval/evaluation/report.ts
• benchmarks/retrieval/evaluation/types.ts
• benchmarks/retrieval/evaluation/weight-search.ts
• benchmarks/retrieval/runner.ts
• benchmarks/tests/channels.test.ts
• benchmarks/tests/corpus.test.ts
• benchmarks/tests/optimization-profiles.test.ts
• benchmarks/tests/promotion-evidence.test.ts
• benchmarks/tests/report.test.ts
• benchmarks/tests/retrieval.test.ts
• benchmarks/tests/worker-pool.test.ts