perf: establish objective-aware retrieval benchmark baseline - #161
Merged
Conversation
📝 WalkthroughWalkthroughAdded an opt-in retrieval-quality benchmark with pinned corpora, retrieval metrics, fusion and evidence-router optimization, embedding caching, SQLite-backed execution, reports, tests, and baseline visualization. Shared query routing and automatic embedder selection were added. The JavaScript dense scorer was removed. ChangesRetrieval benchmark foundation
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant Runner
participant Corpus
participant Cache
participant SQLite
participant Report
Runner->>Corpus: Prepare pinned repository
Corpus->>Cache: Load or write embeddings
Runner->>SQLite: Persist indexes and execute searches
SQLite-->>Runner: Return rankings and measurements
Runner->>Report: Render benchmark artifacts
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
benchmarks/retrieval/weight-search.ts (2)
866-897: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBuild the validation evidence once per call.
prepareEvidenceSamples(validation)runs inside themapcallback, so it repeats for every objective inROUTER_OBJECTIVES.buildRoutingEvidencesorts the dense score list per sample, so this triples the cost without changing the result. Hoist it above themap.♻️ Proposed refactor
const dynamicSelection = selectBestEvidenceRouter(development, fusion) const productionValidation = summarizeProductionRrf(validation) + const validationEvidence = prepareEvidenceSamples(validation) return dynamicSelection.selections.map((selection) => { @@ - validation: summarizeEvidenceRouter( - prepareEvidenceSamples(validation), - selection.config, - fusion, - ), + validation: summarizeEvidenceRouter(validationEvidence, selection.config, fusion),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/retrieval/weight-search.ts` around lines 866 - 897, Hoist the prepareEvidenceSamples(validation) call out of the dynamicSelection.selections.map callback and compute it once before mapping. Reuse the resulting evidence collection in each summarizeEvidenceRouter call while preserving the existing selection results and validation behavior.
45-48: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winBind the Halton prime count to the parameter count.
coefficientParameterscurrently has exactly 28 entries (7 influences × 4 channels), andHALTON_PRIMEShas exactly 28 primes. If a future influence is added,HALTON_PRIMES[parameterIndex]becomesundefined. The non-null assertion then hides the fault:radicalInverse(index, undefined)returns0-basedNaNarithmetic,valueIndexbecomesNaN, andvalues[NaN]!yieldsundefined. The scout seeds are then silently corrupted instead of failing.Add an explicit guard so the mismatch fails fast.
♻️ Proposed guard
if (baseSeeds.length === 0) return [] const coefficientParameters = parameters.slice(CHANNELS.length) + if (coefficientParameters.length > HALTON_PRIMES.length) + throw new Error( + `Halton sequence needs ${coefficientParameters.length} primes, got ${HALTON_PRIMES.length}`, + )Also applies to: 563-578
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/retrieval/weight-search.ts` around lines 45 - 48, Add an explicit length validation tying HALTON_PRIMES to coefficientParameters before the Halton sampling loop uses HALTON_PRIMES[parameterIndex], and fail immediately with a clear error when the counts differ. Remove or supersede the non-null assertion so an out-of-range prime cannot silently produce invalid scout seeds; preserve the existing sampling behavior when the lengths match.benchmarks/retrieval/report.ts (1)
20-48: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMerge the three weighted average helpers.
weightedRouterAverage,weightedFusionAverage, andweightedProductionAveragehave identical bodies. They differ only in the row type. Replace them with one generic helper over rows that exposevalidationQueries.♻️ Proposed refactor
-const weightedRouterAverage = ( - rows: readonly EvidenceRouterSearchResult[], - select: (row: EvidenceRouterSearchResult) => number, -): number => { - const samples = rows.reduce((sum, row) => sum + row.validationQueries, 0) - return samples === 0 - ? 0 - : rows.reduce((sum, row) => sum + select(row) * row.validationQueries, 0) / samples -} - -const weightedFusionAverage = ( - rows: readonly FusionSearchResult[], - select: (row: FusionSearchResult) => number, -): number => { - const samples = rows.reduce((sum, row) => sum + row.validationQueries, 0) - return samples === 0 - ? 0 - : rows.reduce((sum, row) => sum + select(row) * row.validationQueries, 0) / samples -} - -const weightedProductionAverage = ( - rows: readonly ProductionRrfSearchResult[], - select: (row: ProductionRrfSearchResult) => number, -): number => { - const samples = rows.reduce((sum, row) => sum + row.validationQueries, 0) - return samples === 0 - ? 0 - : rows.reduce((sum, row) => sum + select(row) * row.validationQueries, 0) / samples -} +const weightedAverage = <T extends { readonly validationQueries: number }>( + rows: readonly T[], + select: (row: T) => number, +): number => { + const samples = rows.reduce((sum, row) => sum + row.validationQueries, 0) + return samples === 0 + ? 0 + : rows.reduce((sum, row) => sum + select(row) * row.validationQueries, 0) / samples +}Then replace each call site with
weightedAverage.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/retrieval/report.ts` around lines 20 - 48, Merge weightedRouterAverage, weightedFusionAverage, and weightedProductionAverage into a single generic weightedAverage helper constrained to row objects exposing validationQueries and accepting a row selector. Update every call site to use weightedAverage while preserving the existing zero-sample handling and weighted calculation.benchmarks/retrieval/runner.ts (1)
384-388: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReuse the fold key builder from
folds.ts.This line rebuilds the grouped-fold key with the literal separator
\0.benchmarks/retrieval/folds.tsowns the same format in its privatefoldKeyhelper. If either side changes, the lookup fails at runtime and only thegroupedFold === undefinedguard reports it. ExportfoldKeyfromfolds.tsand call it here.♻️ Proposed refactor
-import { assignGroupedFolds } from "./folds.js" +import { assignGroupedFolds, foldKey } from "./folds.js" @@ - const groupedFold = groupedFoldAssignments.get(`${manifest.id}\0${question.id}`) + const groupedFold = groupedFoldAssignments.get(foldKey(manifest.id, question.id))In
benchmarks/retrieval/folds.ts:-const foldKey = (repositoryId: string, questionId: string): string => +/** Key identifying one question inside one repository manifest. */ +export const foldKey = (repositoryId: string, questionId: string): string => `${repositoryId}\0${questionId}`🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/retrieval/runner.ts` around lines 384 - 388, Export the existing foldKey helper from folds.ts, then update the groupedFoldAssignments lookup in the runner flow to call foldKey(manifest.id, question.id) instead of rebuilding the key with a literal separator. Keep the existing undefined guard and failure behavior unchanged.benchmarks/retrieval/sqlite-index.ts (1)
30-40: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider flattening the nested
Layer.provideMergecalls with.pipe(...).The two nested
Layer.provideMergecalls are harder to read than a.pipe(Layer.provideMerge(...), Layer.provideMerge(...))chain, which is the idiomatic Effect composition style shown in Effect's own layer-composition examples.♻️ Proposed readability refactor
-const sqliteBenchmarkIndexLayer = (model: string, dtype: EmbeddingDtype) => - Layer.provideMerge( - Layer.provideMerge( - SqliteIndexStoreBase, - Layer.merge( - Layer.succeed(ConfigStore, benchmarkConfigStore(benchmarkConfig(model, dtype))), - sqliteIndexDatabaseLayer(":memory:"), - ), - ), - layerNoop({}), - ) +const sqliteBenchmarkIndexLayer = (model: string, dtype: EmbeddingDtype) => + SqliteIndexStoreBase.pipe( + Layer.provideMerge( + Layer.merge( + Layer.succeed(ConfigStore, benchmarkConfigStore(benchmarkConfig(model, dtype))), + sqliteIndexDatabaseLayer(":memory:"), + ), + ), + Layer.provideMerge(layerNoop({})), + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/retrieval/sqlite-index.ts` around lines 30 - 40, Refactor sqliteBenchmarkIndexLayer to flatten the nested Layer.provideMerge calls into a .pipe(...) composition chain. Preserve the existing merge order and all layers, including SqliteIndexStoreBase, the benchmark configuration/database layer, and layerNoop({}).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/plot-baseline.html`:
- Around line 244-297: Move the r50 object out of the mini.r20 object so it
becomes a sibling of r20, r10, and ctx under mini, matching the structure used
by bge. Preserve the existing r50 grouped, loro, and fitall data so
renderModelPanel can access DATA.mini.r50 directly.
In `@benchmarks/retrieval/runner.ts`:
- Around line 131-138: Update selectManifests to validate every identifier from
PIX_BENCH_REPOS against manifests, matching the unknown-value validation
behavior already implemented by selectModels. Reject unknown identifiers before
filtering, while preserving the existing smoke-profile default and
valid-selection behavior.
- Around line 307-320: Update the unresolved-target validation in the
manifest.questions mapping to return an Effect.fail with the existing Error
message instead of throwing directly. Preserve the current question ID and
unresolved target details, matching the typed failure pattern used by
runRetrievalBenchmark’s neighboring failure paths.
---
Nitpick comments:
In `@benchmarks/retrieval/report.ts`:
- Around line 20-48: Merge weightedRouterAverage, weightedFusionAverage, and
weightedProductionAverage into a single generic weightedAverage helper
constrained to row objects exposing validationQueries and accepting a row
selector. Update every call site to use weightedAverage while preserving the
existing zero-sample handling and weighted calculation.
In `@benchmarks/retrieval/runner.ts`:
- Around line 384-388: Export the existing foldKey helper from folds.ts, then
update the groupedFoldAssignments lookup in the runner flow to call
foldKey(manifest.id, question.id) instead of rebuilding the key with a literal
separator. Keep the existing undefined guard and failure behavior unchanged.
In `@benchmarks/retrieval/sqlite-index.ts`:
- Around line 30-40: Refactor sqliteBenchmarkIndexLayer to flatten the nested
Layer.provideMerge calls into a .pipe(...) composition chain. Preserve the
existing merge order and all layers, including SqliteIndexStoreBase, the
benchmark configuration/database layer, and layerNoop({}).
In `@benchmarks/retrieval/weight-search.ts`:
- Around line 866-897: Hoist the prepareEvidenceSamples(validation) call out of
the dynamicSelection.selections.map callback and compute it once before mapping.
Reuse the resulting evidence collection in each summarizeEvidenceRouter call
while preserving the existing selection results and validation behavior.
- Around line 45-48: Add an explicit length validation tying HALTON_PRIMES to
coefficientParameters before the Halton sampling loop uses
HALTON_PRIMES[parameterIndex], and fail immediately with a clear error when the
counts differ. Remove or supersede the non-null assertion so an out-of-range
prime cannot silently produce invalid scout seeds; preserve the existing
sampling behavior when the lengths match.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b12a35b5-4c5c-44d5-9649-157c0e5312fc
⛔ Files ignored due to path filters (1)
.gitignoreis excluded by!.gitignore
📒 Files selected for processing (38)
CONTEXT.mdbenchmarks/BASELINE.mdbenchmarks/README.mdbenchmarks/corpus/effect-v4.jsonbenchmarks/corpus/fastapi.jsonbenchmarks/corpus/fd.jsonbenchmarks/plot-baseline.htmlbenchmarks/retrieval/corpus.tsbenchmarks/retrieval/embedding-cache.tsbenchmarks/retrieval/evidence-router.tsbenchmarks/retrieval/folds.tsbenchmarks/retrieval/fusion.tsbenchmarks/retrieval/metrics.tsbenchmarks/retrieval/prepare.tsbenchmarks/retrieval/ranking.tsbenchmarks/retrieval/report.tsbenchmarks/retrieval/runner.tsbenchmarks/retrieval/sqlite-index.tsbenchmarks/retrieval/types.tsbenchmarks/retrieval/weight-search.tsbenchmarks/tests/channels.test.tsbenchmarks/tests/corpus.test.tsbenchmarks/tests/retrieval.test.tsbenchmarks/tsconfig.jsonbenchmarks/vite.config.tsdocs/adr/0008-embedding-internal-representation.mddocs/adr/0018-sqlite-index-and-vector-search.mdpackage.jsonsrc/application/query-project.tssrc/lib/retrieval/dense.test.tssrc/lib/retrieval/dense.tssrc/lib/retrieval/routing.tssrc/lib/vectors/cosine.test.tssrc/lib/vectors/cosine.tssrc/services/chunker.tssrc/services/device-detect.tssrc/services/embedder.tssrc/services/index-store.test.ts
💤 Files with no reviewable changes (5)
- src/lib/vectors/cosine.ts
- src/lib/retrieval/dense.test.ts
- src/lib/retrieval/dense.ts
- src/lib/vectors/cosine.test.ts
- src/services/index-store.test.ts
Comment on lines
+244
to
+297
| r20: { | ||
| grouped: [ | ||
| { s: 2, v: 78.9, m: "Ø der 4 Query-Formen (per-Form Gewichte)" }, | ||
| { s: 3, v: 77.5, m: "Evidence-Router (gegruppt)" }, | ||
| { s: 4, v: 78.0, m: "dynamischer Router (gegruppt, Vergleich im BGE-Run)" }, | ||
| { s: 5, v: 78.1, m: "positive Basen (gegruppt)" }, | ||
| { s: 6, v: 76.4, m: "Log2-Kernel (gegruppt)" }, | ||
| { s: 7, v: 82.8, m: "Relative-Score-Fusion (kein dynamischer Router)" }, | ||
| { s: 8, v: 80.8, m: "DBSF dynamisch (gegruppt 5-fold)" }, | ||
| { s: 9, v: 82.2, m: "Relative-Score dynamisch (gegruppt)" }, | ||
| { s: 10, v: 82.2, m: "Relative-Score dynamisch (gegruppt)" }, | ||
| { s: 11, v: 83.3, m: "Relative-Score dynamisch (gegruppt)" }, | ||
| { s: 12, v: 82.8, m: "Relative-Score dynamisch (gegruppt, Vergleich)" }, | ||
| { s: 14, v: 78.1, m: "DBSF dynamisch (gegruppt 5-fold)" }, | ||
| { s: 15, v: 78.1, m: "DBSF dynamisch (gegruppt 5-fold)" }, | ||
| { s: 16, v: 80.3, m: "Objective reranker-top50 (gegruppt)" }, | ||
| ], | ||
| loro: [ | ||
| { s: 2, v: 79.7, m: "Ø der 4 Query-Formen (per-Form Gewichte)" }, | ||
| { s: 3, v: 78.6, m: "Evidence-Router (LORO)" }, | ||
| { s: 4, v: 80.3, m: "dynamischer Router (LORO, Vergleich im BGE-Run)" }, | ||
| { s: 5, v: 80.3, m: "positive Basen (LORO)" }, | ||
| { s: 6, v: 81.1, m: "Log2-Kernel (LORO)" }, | ||
| { s: 7, v: 83.3, m: "Relative-Score-Fusion (kein dynamischer Router)" }, | ||
| { s: 8, v: 80.3, m: "DBSF dynamisch (LORO)" }, | ||
| { s: 9, v: 82.8, m: "Relative-Score dynamisch (LORO)" }, | ||
| { s: 10, v: 83.3, m: "Relative-Score dynamisch (LORO)" }, | ||
| { s: 11, v: 82.8, m: "Relative-Score dynamisch (LORO)" }, | ||
| { s: 12, v: 83.6, m: "Relative-Score dynamisch (LORO, Vergleich)" }, | ||
| { s: 14, v: 80.0, m: "DBSF dynamisch (LORO)" }, | ||
| { s: 15, v: 80.0, m: "DBSF dynamisch (LORO)" }, | ||
| { s: 16, v: 81.9, m: "Objective direct (LORO)" }, | ||
| ], | ||
| fitall: [ | ||
| { s: 3, v: 82.5, m: "Evidence-Router" }, | ||
| { s: 4, v: 83.1, m: "dynamischer Router" }, | ||
| { s: 5, v: 83.1, m: "positive Basen" }, | ||
| { s: 6, v: 83.1, m: "Log2-Kernel" }, | ||
| { s: 8, v: 83.9, m: "DBSF" }, | ||
| { s: 9, v: 85.3, m: "Relative-Score" }, | ||
| { s: 10, v: 85.3, m: "Relative-Score" }, | ||
| { s: 11, v: 85.3, m: "Relative-Score" }, | ||
| { s: 12, v: 85.8, m: "Relative-Score (Vergleich)" }, | ||
| { s: 14, v: 83.9, m: "DBSF" }, | ||
| { s: 15, v: 83.9, m: "DBSF" }, | ||
| { s: 16, v: 84.2, m: "Objective reranker-top20" }, | ||
| ], | ||
| r50: { | ||
| grouped: [{ s: 16, v: 89.2, m: "Objective reranker-top50 (gegruppt)" }], | ||
| loro: [{ s: 16, v: 89.4, m: "Objective reranker-top20 (LORO)" }], | ||
| fitall: [{ s: 16, v: 92.5, m: "Objective reranker-top50" }], | ||
| }, | ||
| }, | ||
| r10: { |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix the misplaced r50 key: it breaks the dashboard.
r50 (Lines 291-295) is nested inside mini.r20 instead of being a sibling of r20, r10, and ctx under mini. Compare to bge (Line 407), where r50 is correctly a sibling.
renderModelPanel reads DATA.mini.r50 directly (Line 655, via modelData[meta.key]). Since that path is undefined, renderChart dereferences metricData[meta.key] (Line 539) on undefined and throws. This halts the synchronous rendering loop, so the MiniLM Recall@10/Context charts, the entire BGE panel, and the Schema 1 note never render.
🐛 Proposed fix: move `r50` out of `r20`
fitall: [
{ s: 3, v: 82.5, m: "Evidence-Router" },
{ s: 4, v: 83.1, m: "dynamischer Router" },
{ s: 5, v: 83.1, m: "positive Basen" },
{ s: 6, v: 83.1, m: "Log2-Kernel" },
{ s: 8, v: 83.9, m: "DBSF" },
{ s: 9, v: 85.3, m: "Relative-Score" },
{ s: 10, v: 85.3, m: "Relative-Score" },
{ s: 11, v: 85.3, m: "Relative-Score" },
{ s: 12, v: 85.8, m: "Relative-Score (Vergleich)" },
{ s: 14, v: 83.9, m: "DBSF" },
{ s: 15, v: 83.9, m: "DBSF" },
{ s: 16, v: 84.2, m: "Objective reranker-top20" },
],
- r50: {
- grouped: [{ s: 16, v: 89.2, m: "Objective reranker-top50 (gegruppt)" }],
- loro: [{ s: 16, v: 89.4, m: "Objective reranker-top20 (LORO)" }],
- fitall: [{ s: 16, v: 92.5, m: "Objective reranker-top50" }],
- },
},
+ r50: {
+ grouped: [{ s: 16, v: 89.2, m: "Objective reranker-top50 (gegruppt)" }],
+ loro: [{ s: 16, v: 89.4, m: "Objective reranker-top20 (LORO)" }],
+ fitall: [{ s: 16, v: 92.5, m: "Objective reranker-top50" }],
+ },
r10: {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| r20: { | |
| grouped: [ | |
| { s: 2, v: 78.9, m: "Ø der 4 Query-Formen (per-Form Gewichte)" }, | |
| { s: 3, v: 77.5, m: "Evidence-Router (gegruppt)" }, | |
| { s: 4, v: 78.0, m: "dynamischer Router (gegruppt, Vergleich im BGE-Run)" }, | |
| { s: 5, v: 78.1, m: "positive Basen (gegruppt)" }, | |
| { s: 6, v: 76.4, m: "Log2-Kernel (gegruppt)" }, | |
| { s: 7, v: 82.8, m: "Relative-Score-Fusion (kein dynamischer Router)" }, | |
| { s: 8, v: 80.8, m: "DBSF dynamisch (gegruppt 5-fold)" }, | |
| { s: 9, v: 82.2, m: "Relative-Score dynamisch (gegruppt)" }, | |
| { s: 10, v: 82.2, m: "Relative-Score dynamisch (gegruppt)" }, | |
| { s: 11, v: 83.3, m: "Relative-Score dynamisch (gegruppt)" }, | |
| { s: 12, v: 82.8, m: "Relative-Score dynamisch (gegruppt, Vergleich)" }, | |
| { s: 14, v: 78.1, m: "DBSF dynamisch (gegruppt 5-fold)" }, | |
| { s: 15, v: 78.1, m: "DBSF dynamisch (gegruppt 5-fold)" }, | |
| { s: 16, v: 80.3, m: "Objective reranker-top50 (gegruppt)" }, | |
| ], | |
| loro: [ | |
| { s: 2, v: 79.7, m: "Ø der 4 Query-Formen (per-Form Gewichte)" }, | |
| { s: 3, v: 78.6, m: "Evidence-Router (LORO)" }, | |
| { s: 4, v: 80.3, m: "dynamischer Router (LORO, Vergleich im BGE-Run)" }, | |
| { s: 5, v: 80.3, m: "positive Basen (LORO)" }, | |
| { s: 6, v: 81.1, m: "Log2-Kernel (LORO)" }, | |
| { s: 7, v: 83.3, m: "Relative-Score-Fusion (kein dynamischer Router)" }, | |
| { s: 8, v: 80.3, m: "DBSF dynamisch (LORO)" }, | |
| { s: 9, v: 82.8, m: "Relative-Score dynamisch (LORO)" }, | |
| { s: 10, v: 83.3, m: "Relative-Score dynamisch (LORO)" }, | |
| { s: 11, v: 82.8, m: "Relative-Score dynamisch (LORO)" }, | |
| { s: 12, v: 83.6, m: "Relative-Score dynamisch (LORO, Vergleich)" }, | |
| { s: 14, v: 80.0, m: "DBSF dynamisch (LORO)" }, | |
| { s: 15, v: 80.0, m: "DBSF dynamisch (LORO)" }, | |
| { s: 16, v: 81.9, m: "Objective direct (LORO)" }, | |
| ], | |
| fitall: [ | |
| { s: 3, v: 82.5, m: "Evidence-Router" }, | |
| { s: 4, v: 83.1, m: "dynamischer Router" }, | |
| { s: 5, v: 83.1, m: "positive Basen" }, | |
| { s: 6, v: 83.1, m: "Log2-Kernel" }, | |
| { s: 8, v: 83.9, m: "DBSF" }, | |
| { s: 9, v: 85.3, m: "Relative-Score" }, | |
| { s: 10, v: 85.3, m: "Relative-Score" }, | |
| { s: 11, v: 85.3, m: "Relative-Score" }, | |
| { s: 12, v: 85.8, m: "Relative-Score (Vergleich)" }, | |
| { s: 14, v: 83.9, m: "DBSF" }, | |
| { s: 15, v: 83.9, m: "DBSF" }, | |
| { s: 16, v: 84.2, m: "Objective reranker-top20" }, | |
| ], | |
| r50: { | |
| grouped: [{ s: 16, v: 89.2, m: "Objective reranker-top50 (gegruppt)" }], | |
| loro: [{ s: 16, v: 89.4, m: "Objective reranker-top20 (LORO)" }], | |
| fitall: [{ s: 16, v: 92.5, m: "Objective reranker-top50" }], | |
| }, | |
| }, | |
| r10: { | |
| r20: { | |
| grouped: [ | |
| { s: 2, v: 78.9, m: "Ø der 4 Query-Formen (per-Form Gewichte)" }, | |
| { s: 3, v: 77.5, m: "Evidence-Router (gegruppt)" }, | |
| { s: 4, v: 78.0, m: "dynamischer Router (gegruppt, Vergleich im BGE-Run)" }, | |
| { s: 5, v: 78.1, m: "positive Basen (gegruppt)" }, | |
| { s: 6, v: 76.4, m: "Log2-Kernel (gegruppt)" }, | |
| { s: 7, v: 82.8, m: "Relative-Score-Fusion (kein dynamischer Router)" }, | |
| { s: 8, v: 80.8, m: "DBSF dynamisch (gegruppt 5-fold)" }, | |
| { s: 9, v: 82.2, m: "Relative-Score dynamisch (gegruppt)" }, | |
| { s: 10, v: 82.2, m: "Relative-Score dynamisch (gegruppt)" }, | |
| { s: 11, v: 83.3, m: "Relative-Score dynamisch (gegruppt)" }, | |
| { s: 12, v: 82.8, m: "Relative-Score dynamisch (gegruppt, Vergleich)" }, | |
| { s: 14, v: 78.1, m: "DBSF dynamisch (gegruppt 5-fold)" }, | |
| { s: 15, v: 78.1, m: "DBSF dynamisch (gegruppt 5-fold)" }, | |
| { s: 16, v: 80.3, m: "Objective reranker-top50 (gegruppt)" }, | |
| ], | |
| loro: [ | |
| { s: 2, v: 79.7, m: "Ø der 4 Query-Formen (per-Form Gewichte)" }, | |
| { s: 3, v: 78.6, m: "Evidence-Router (LORO)" }, | |
| { s: 4, v: 80.3, m: "dynamischer Router (LORO, Vergleich im BGE-Run)" }, | |
| { s: 5, v: 80.3, m: "positive Basen (LORO)" }, | |
| { s: 6, v: 81.1, m: "Log2-Kernel (LORO)" }, | |
| { s: 7, v: 83.3, m: "Relative-Score-Fusion (kein dynamischer Router)" }, | |
| { s: 8, v: 80.3, m: "DBSF dynamisch (LORO)" }, | |
| { s: 9, v: 82.8, m: "Relative-Score dynamisch (LORO)" }, | |
| { s: 10, v: 83.3, m: "Relative-Score dynamisch (LORO)" }, | |
| { s: 11, v: 82.8, m: "Relative-Score dynamisch (LORO)" }, | |
| { s: 12, v: 83.6, m: "Relative-Score dynamisch (LORO, Vergleich)" }, | |
| { s: 14, v: 80.0, m: "DBSF dynamisch (LORO)" }, | |
| { s: 15, v: 80.0, m: "DBSF dynamisch (LORO)" }, | |
| { s: 16, v: 81.9, m: "Objective direct (LORO)" }, | |
| ], | |
| fitall: [ | |
| { s: 3, v: 82.5, m: "Evidence-Router" }, | |
| { s: 4, v: 83.1, m: "dynamischer Router" }, | |
| { s: 5, v: 83.1, m: "positive Basen" }, | |
| { s: 6, v: 83.1, m: "Log2-Kernel" }, | |
| { s: 8, v: 83.9, m: "DBSF" }, | |
| { s: 9, v: 85.3, m: "Relative-Score" }, | |
| { s: 10, v: 85.3, m: "Relative-Score" }, | |
| { s: 11, v: 85.3, m: "Relative-Score" }, | |
| { s: 12, v: 85.8, m: "Relative-Score (Vergleich)" }, | |
| { s: 14, v: 83.9, m: "DBSF" }, | |
| { s: 15, v: 83.9, m: "DBSF" }, | |
| { s: 16, v: 84.2, m: "Objective reranker-top20" }, | |
| ], | |
| }, | |
| r50: { | |
| grouped: [{ s: 16, v: 89.2, m: "Objective reranker-top50 (gegruppt)" }], | |
| loro: [{ s: 16, v: 89.4, m: "Objective reranker-top20 (LORO)" }], | |
| fitall: [{ s: 16, v: 92.5, m: "Objective reranker-top50" }], | |
| }, | |
| r10: { |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/plot-baseline.html` around lines 244 - 297, Move the r50 object
out of the mini.r20 object so it becomes a sibling of r20, r10, and ctx under
mini, matching the structure used by bge. Preserve the existing r50 grouped,
loro, and fitall data so renderModelPanel can access DATA.mini.r50 directly.
Comment on lines
+131
to
+138
| const selectManifests = ( | ||
| manifests: readonly CorpusManifest[], | ||
| profile: BenchmarkProfile, | ||
| ): readonly CorpusManifest[] => { | ||
| const selected = selectValues(process.env.PIX_BENCH_REPOS) | ||
| if (selected) return manifests.filter((manifest) => selected.has(manifest.id)) | ||
| return profile === "smoke" ? manifests.filter((manifest) => manifest.id === "fd") : manifests | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Reject unknown PIX_BENCH_REPOS values.
selectManifests silently drops identifiers that match no manifest. A typo then produces an empty manifest list, and the benchmark writes an artifact with zero repositories and zero measurements. selectModels already validates unknown values at Line 147. Apply the same check here.
🛡️ Proposed validation
const selected = selectValues(process.env.PIX_BENCH_REPOS)
- if (selected) return manifests.filter((manifest) => selected.has(manifest.id))
+ if (selected) {
+ const unknown = [...selected].filter((id) => !manifests.some((manifest) => manifest.id === id))
+ if (unknown.length > 0)
+ throw new Error(`Unknown PIX_BENCH_REPOS values: ${unknown.join(", ")}`)
+ return manifests.filter((manifest) => selected.has(manifest.id))
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const selectManifests = ( | |
| manifests: readonly CorpusManifest[], | |
| profile: BenchmarkProfile, | |
| ): readonly CorpusManifest[] => { | |
| const selected = selectValues(process.env.PIX_BENCH_REPOS) | |
| if (selected) return manifests.filter((manifest) => selected.has(manifest.id)) | |
| return profile === "smoke" ? manifests.filter((manifest) => manifest.id === "fd") : manifests | |
| } | |
| const selectManifests = ( | |
| manifests: readonly CorpusManifest[], | |
| profile: BenchmarkProfile, | |
| ): readonly CorpusManifest[] => { | |
| const selected = selectValues(process.env.PIX_BENCH_REPOS) | |
| if (selected) { | |
| const unknown = [...selected].filter((id) => !manifests.some((manifest) => manifest.id === id)) | |
| if (unknown.length > 0) | |
| throw new Error(`Unknown PIX_BENCH_REPOS values: ${unknown.join(", ")}`) | |
| return manifests.filter((manifest) => selected.has(manifest.id)) | |
| } | |
| return profile === "smoke" ? manifests.filter((manifest) => manifest.id === "fd") : manifests | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/retrieval/runner.ts` around lines 131 - 138, Update
selectManifests to validate every identifier from PIX_BENCH_REPOS against
manifests, matching the unknown-value validation behavior already implemented by
selectModels. Reject unknown identifiers before filtering, while preserving the
existing smoke-profile default and valid-selection behavior.
Comment on lines
+307
to
+320
| const targetsByQuestion = manifest.questions.map((question) => { | ||
| const targets = resolveGoldTargets( | ||
| question.groundTruth, | ||
| corpus.chunks, | ||
| corpus.identifiersByChunk, | ||
| ) | ||
| const unresolved = question.groundTruth.filter((_, index) => targets[index].size === 0) | ||
| if (unresolved.length > 0) { | ||
| throw new Error( | ||
| `${question.id} has unresolved gold targets: ${unresolved.map((target) => `${target.file}::${target.symbol}`).join(", ")}`, | ||
| ) | ||
| } | ||
| return targets | ||
| }) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Report unresolved gold targets through the typed error channel.
This throw runs inside the Effect.gen body, so Effect treats it as a defect and the fiber dies. The declared error type of runRetrievalBenchmark is Error, and the neighboring failures at Line 325 and Line 386 use Effect.fail. Convert this validation to the same typed failure so callers can handle it.
♻️ Proposed refactor
- const targetsByQuestion = manifest.questions.map((question) => {
- const targets = resolveGoldTargets(
- question.groundTruth,
- corpus.chunks,
- corpus.identifiersByChunk,
- )
- const unresolved = question.groundTruth.filter((_, index) => targets[index].size === 0)
- if (unresolved.length > 0) {
- throw new Error(
- `${question.id} has unresolved gold targets: ${unresolved.map((target) => `${target.file}::${target.symbol}`).join(", ")}`,
- )
- }
- return targets
- })
+ const targetsByQuestion: (readonly ReadonlySet<number>[])[] = []
+ for (const question of manifest.questions) {
+ const targets = resolveGoldTargets(
+ question.groundTruth,
+ corpus.chunks,
+ corpus.identifiersByChunk,
+ )
+ const unresolved = question.groundTruth.filter((_, index) => targets[index].size === 0)
+ if (unresolved.length > 0)
+ return yield* Effect.fail(
+ new Error(
+ `${question.id} has unresolved gold targets: ${unresolved.map((target) => `${target.file}::${target.symbol}`).join(", ")}`,
+ ),
+ )
+ targetsByQuestion.push(targets)
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const targetsByQuestion = manifest.questions.map((question) => { | |
| const targets = resolveGoldTargets( | |
| question.groundTruth, | |
| corpus.chunks, | |
| corpus.identifiersByChunk, | |
| ) | |
| const unresolved = question.groundTruth.filter((_, index) => targets[index].size === 0) | |
| if (unresolved.length > 0) { | |
| throw new Error( | |
| `${question.id} has unresolved gold targets: ${unresolved.map((target) => `${target.file}::${target.symbol}`).join(", ")}`, | |
| ) | |
| } | |
| return targets | |
| }) | |
| const targetsByQuestion: (readonly ReadonlySet<number>[])[] = [] | |
| for (const question of manifest.questions) { | |
| const targets = resolveGoldTargets( | |
| question.groundTruth, | |
| corpus.chunks, | |
| corpus.identifiersByChunk, | |
| ) | |
| const unresolved = question.groundTruth.filter((_, index) => targets[index].size === 0) | |
| if (unresolved.length > 0) | |
| return yield* Effect.fail( | |
| new Error( | |
| `${question.id} has unresolved gold targets: ${unresolved.map((target) => `${target.file}::${target.symbol}`).join(", ")}`, | |
| ), | |
| ) | |
| targetsByQuestion.push(targets) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/retrieval/runner.ts` around lines 307 - 320, Update the
unresolved-target validation in the manifest.questions mapping to return an
Effect.fail with the existing Error message instead of throwing directly.
Preserve the current question ID and unresolved target details, matching the
typed failure pattern used by runRetrievalBenchmark’s neighboring failure paths.
This was referenced Aug 2, 2026
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
vp checkvp test(415 passed)vp test --config benchmarks/vite.config.ts --run benchmarks/tests/channels.test.ts(16 passed)vp test --config benchmarks/vite.config.ts --run benchmarks/tests/retrieval.test.ts -t runs-the-smoke-retrieval-profile(passed)benchmarks/results/retrieval-2026-07-31T23-54-20.351Z.jsonFallow still reports the repository's known dependency, duplication, and complexity health findings; dead exports are at 0.0%.
Summary by CodeRabbit
New Features
Documentation
Tests