From 4ec8357793b5c9bd6398705f870c5d8e048bacdc Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:22:29 -0400 Subject: [PATCH 01/15] docs(planned): add infra_ir_measures_migration spec + plan + pipeline_status Adds the approved planning artifacts for replacing the abandoned pytrec_eval library with ir_measures in backend/app/eval/scoring.py. Cross-model review trajectory: - Spec: 3 GPT-5.5 cycles (11 -> 6 -> 1 findings, all accepted) - Plan: 3 GPT-5.5 cycles (10 -> 4 -> 1 findings, 14 accepted + 1 rejected with cited counter-evidence at scoring.py:74-78) Scope: 8 stories in 1 epic, single-PR migration, no schema change. Public API of scoring.py frozen; persisted JSONB key shape frozen; aggregate computed via ir_measures.iter_calc + manual mean (not calc_aggregate); per- query universe filtered to pytrec_eval's historical contract. Includes the pre-commit hook's MVP1_DASHBOARD.md regeneration which picks up the new planned-feature folder in its inventory. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/00_overview/DASHBOARD.md | 2 +- docs/00_overview/MVP1_DASHBOARD.md | 29 +- docs/00_overview/dashboard.html | 2 +- docs/00_overview/mvp1_dashboard.html | 61 +- .../feature_spec.md | 725 +++++++++++ .../implementation_plan.md | 1150 +++++++++++++++++ .../pipeline_status.md | 28 + 7 files changed, 1959 insertions(+), 38 deletions(-) create mode 100644 docs/02_product/planned_features/infra_ir_measures_migration/feature_spec.md create mode 100644 docs/02_product/planned_features/infra_ir_measures_migration/implementation_plan.md create mode 100644 docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md diff --git a/docs/00_overview/DASHBOARD.md b/docs/00_overview/DASHBOARD.md index 969aea85..a6f76954 100644 --- a/docs/00_overview/DASHBOARD.md +++ b/docs/00_overview/DASHBOARD.md @@ -6,7 +6,7 @@ _Top-level index across MVP1 → GA v1+ as of **2026-05-22**. Click a release na | Release | Theme | Progress | Status | |---|---|---|---| -| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 65 / 65 scoped done · 6 remaining | **In progress** | +| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 65 / 66 scoped done · 7 remaining | **In progress** | | [MVP2 / v0.2](MVP2_DASHBOARD.md) | Observable | 1 / 1 scoped done · 1 remaining | **In progress** | | MVP3 / v0.3 | Production Stacks | — | **Not yet scoped** | | MVP4 / v0.4 | Multi-tenant, Multi-LLM | — | **Not yet scoped** | diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index f2872095..970ac287 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -6,23 +6,29 @@ _Reflects feature-folder state as of **2026-05-22** (latest mtime of any planned ## Next up -All scoped MVP1 features shipped 🎉 +**[infra_ir_measures_migration](../02_product/planned_features/infra_ir_measures_migration/feature_spec.md)** — Infra, currently in **Plan** -Pull from the Idea backlog or capture a new feature spec. +> `scoring.py` imports `ir_measures` rather than `pytrec_eval`; the user-facing `score(qrels, run, metrics) -> ScoreResult` signature is unchanged; the wire forms (`ndcg@10`, `map@5`, `mrr`, `map`) the rest of the codebase consumes are unchan + +Plan approved; run /impl-execute to ship + +```bash +/impl-execute docs/02_product/planned_features/infra_ir_measures_migration/implementation_plan.md --all +``` ## MVP1 Progress | Metric | Value | |---|---| -| Scoped items done | **65 / 65** (100%) — feat_/infra_/chore_/epic_ past idea stage | +| Scoped items done | **65 / 66** (98%) — feat_/infra_/chore_/epic_ past idea stage | | Pending work | **14** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | | → P1 | **0** high-value, ready when P0 clears | | → P2 (default) | 13 important to file, not blocking | | → Backlog | 1 captured for record, not planned | | Open bugs | 2 | -| Legacy "Path to MVP1" | 6 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 8 idea-only feat/infra (not yet scoped into MVP1) | +| Legacy "Path to MVP1" | 7 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | +| Backlog ideas | 7 idea-only feat/infra (not yet scoped into MVP1) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -113,15 +119,17 @@ Pull from the Idea backlog or capture a new feature spec. _None._ -### Plan (0) +### Plan (1) -_None._ +| Priority | Feature | Type | One-liner | Depends on | Status | +|---|---|---|---|---|---| +| P2 | [infra_ir_measures_migration](../02_product/planned_features/infra_ir_measures_migration/feature_spec.md) | Infra | `scoring.py` imports `ir_measures` rather than `pytrec_eval`; the user-facing `score(qrels, run, metrics) -> ScoreResult` signature is unchanged; the wire forms (`ndcg@10`, `map@5`, `mrr`, `map`) the | — | [PR #197](https://github.com/SoundMindsAI/relyloop/pull/197) | ### Spec (0) _None._ -### Idea (14) +### Idea (13) | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---| @@ -131,7 +139,6 @@ _None._ | P2 | [feat_home_demo_reseed_endpoint](../02_product/planned_features/feat_home_demo_reseed_endpoint/idea.md) | Feature | Phase 1 ships the banner + badges that signal "this is demo data." It does NOT close the recovery loop for operators who blew away their dev DB and want to re-seed the meaningful demos from inside the | — | Idea — deferred Phase 2 work from `feat_home_first_run_demo_nudge` (Phase 1 merged 2026-05-22 as PR #188 squash `21325432`) | | P2 | [feat_study_baseline_trial](../02_product/planned_features/feat_study_baseline_trial/idea.md) | Feature | `studies.baseline_metric` exists as a column on the `studies` table (declared in `feat_study_lifecycle` Phase 1, [`backend/app/db/models/study.py:76`](../../backend/app/db/models/study.py#L76)) with t | — | Idea — deferred Phase 2 work from `feat_pr_metric_confidence` (Phase 1 merged 2026-05-21 as PR #180 squash `d0a8358`). | | P2 | [feat_study_clone_from_previous](../02_product/planned_features/feat_study_clone_from_previous/idea.md) | Feature | A relevance engineer's normal workflow after the first study completes: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| P2 | [infra_ir_measures_migration](../02_product/planned_features/infra_ir_measures_migration/idea.md) | Infra | `pytrec_eval` is unmaintained — the canonical repo's last commit is 2020-09-07 and there have been zero GitHub releases. The package still works (it's a thin C wrapper over the stable `trec_eval` bina | — | Idea — surfaced during blog drafting; noticed `pytrec_eval` is effectively abandoned (last commit 2020-09-07) | | P2 | [infra_study_preflight_real_engine_integration](../02_product/planned_features/infra_study_preflight_real_engine_integration/idea.md) | Infra | `feat_study_preflight_overlap_probe`'s integration tests (AC-1 through AC-4b in [`backend/tests/integration/test_studies_api.py`](../../backend/tests/integration/test_studies_api.py)) use… | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | | P2 | [chore_studies_post_arq_spy_fixture](../02_product/planned_features/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | | P2 | [chore_study_default_stop_conditions](../02_product/planned_features/chore_study_default_stop_conditions/idea.md) | Chore | The server-side `StudyConfigSpec` validator at [`backend/app/api/v1/schemas.py:572-580`](../../backend/app/api/v1/schemas.py) correctly **requires** at least one of `max_trials` or `time_budget_min` — | — | Idea — surfaced during the 2026-05-21 Karpathy-loop audit; recommendation grounded in measured per-trial cost from the local dev DB. | @@ -151,6 +158,8 @@ graph LR classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + infra_ir_measures_migration["ir measures migration"] + class infra_ir_measures_migration plan; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] @@ -314,6 +323,7 @@ graph LR infra_e2e_wire_seed_helper_into_studies_spec --> chore_tutorial_polish infra_foundation --> chore_tutorial_polish infra_frontend_stack_refresh --> chore_tutorial_polish + infra_ir_measures_migration --> chore_tutorial_polish infra_make_targets_split_backend_only --> chore_tutorial_polish infra_nvmrc --> chore_tutorial_polish infra_optuna_eval --> chore_tutorial_polish @@ -347,6 +357,7 @@ graph LR infra_e2e_wire_seed_helper_into_studies_spec --> feat_chat_agent infra_foundation --> feat_chat_agent infra_frontend_stack_refresh --> feat_chat_agent + infra_ir_measures_migration --> feat_chat_agent infra_make_targets_split_backend_only --> feat_chat_agent infra_nvmrc --> feat_chat_agent infra_optuna_eval --> feat_chat_agent diff --git a/docs/00_overview/dashboard.html b/docs/00_overview/dashboard.html index d3768a89..389427bd 100644 --- a/docs/00_overview/dashboard.html +++ b/docs/00_overview/dashboard.html @@ -384,7 +384,7 @@

Releases

MVP1 / v0.1
The Loop
-
65 / 65 scoped done · 6 remaining
+
65 / 66 scoped done · 7 remaining
In progress
diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index 5e82e95e..c84e0643 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -382,12 +382,12 @@

RelyLoop MVP1 Dashboard

-
-
Next up
-
All scoped MVP1 features shipped 🎉
-
- Pull from the Idea backlog or capture a new feature spec. -
+
+
Next up — Infra, currently in Plan
+
Ir Measures Migration
+
`scoring.py` imports `ir_measures` rather than `pytrec_eval`; the user-facing `score(qrels, run, metrics) -> ScoreResult` signature is unchanged; the wire forms (`ndcg@10`, `map@5`, `mrr`, `map`) the rest of the codebase consumes are unchan
+
Plan approved; run /impl-execute to ship
+ /impl-execute docs/02_product/planned_features/infra_ir_measures_migration/implementation_plan.md --all
@@ -395,11 +395,11 @@

RelyLoop MVP1 Dashboard

MVP1 Progress

-
+
Scoped items done
-
65 / 65
-
100% of feat_/infra_/chore_/epic_ items past idea stage
-
+
65 / 66
+
98% of feat_/infra_/chore_/epic_ items past idea stage
+
Pending work
@@ -435,14 +435,14 @@

MVP1 Progress

Legacy "Path to MVP1"
-
6
+
7
scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)
Backlog ideas: - 8 idea-only feat/infra folders (not yet scoped into MVP1) + 7 idea-only feat/infra folders (not yet scoped into MVP1) In flight: @@ -463,7 +463,7 @@

Pipeline

-

Idea 14

+

Idea 13

Auto Followup Studies
@@ -543,19 +543,6 @@

Idea 14

-
-
Ir Measures Migration
-
- Infra - P2 - -
-
`pytrec_eval` is unmaintained — the canonical repo's last commit is 2020-09-07 and there have been zero GitHub releases. The package still works (it's a thin C wrapper over the stable `trec_eval` bina
- - -
- -
Study Preflight Real Engine Integration
@@ -654,7 +641,19 @@

Spec 0

-

Plan 0

+

Plan 1

+ +
+
Ir Measures Migration
+
+ Infra + P2 + PR #197 +
+
`scoring.py` imports `ir_measures` rather than `pytrec_eval`; the user-facing `score(qrels, run, metrics) -> ScoreResult` signature is unchanged; the wire forms (`ndcg@10`, `map@5`, `mrr`, `map`) the
+ + +
@@ -1679,6 +1678,8 @@

Dependency graph (feat_ + infra_)

classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + infra_ir_measures_migration["ir measures migration"] + class infra_ir_measures_migration plan; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] @@ -1842,6 +1843,7 @@

Dependency graph (feat_ + infra_)

infra_e2e_wire_seed_helper_into_studies_spec --> chore_tutorial_polish infra_foundation --> chore_tutorial_polish infra_frontend_stack_refresh --> chore_tutorial_polish + infra_ir_measures_migration --> chore_tutorial_polish infra_make_targets_split_backend_only --> chore_tutorial_polish infra_nvmrc --> chore_tutorial_polish infra_optuna_eval --> chore_tutorial_polish @@ -1875,6 +1877,7 @@

Dependency graph (feat_ + infra_)

infra_e2e_wire_seed_helper_into_studies_spec --> feat_chat_agent infra_foundation --> feat_chat_agent infra_frontend_stack_refresh --> feat_chat_agent + infra_ir_measures_migration --> feat_chat_agent infra_make_targets_split_backend_only --> feat_chat_agent infra_nvmrc --> feat_chat_agent infra_optuna_eval --> feat_chat_agent @@ -1908,6 +1911,8 @@

Dependency graph (feat_ + infra_)

classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + infra_ir_measures_migration["ir measures migration"] + class infra_ir_measures_migration plan; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] @@ -2071,6 +2076,7 @@

Dependency graph (feat_ + infra_)

infra_e2e_wire_seed_helper_into_studies_spec --> chore_tutorial_polish infra_foundation --> chore_tutorial_polish infra_frontend_stack_refresh --> chore_tutorial_polish + infra_ir_measures_migration --> chore_tutorial_polish infra_make_targets_split_backend_only --> chore_tutorial_polish infra_nvmrc --> chore_tutorial_polish infra_optuna_eval --> chore_tutorial_polish @@ -2104,6 +2110,7 @@

Dependency graph (feat_ + infra_)

infra_e2e_wire_seed_helper_into_studies_spec --> feat_chat_agent infra_foundation --> feat_chat_agent infra_frontend_stack_refresh --> feat_chat_agent + infra_ir_measures_migration --> feat_chat_agent infra_make_targets_split_backend_only --> feat_chat_agent infra_nvmrc --> feat_chat_agent infra_optuna_eval --> feat_chat_agent diff --git a/docs/02_product/planned_features/infra_ir_measures_migration/feature_spec.md b/docs/02_product/planned_features/infra_ir_measures_migration/feature_spec.md new file mode 100644 index 00000000..f54b9c99 --- /dev/null +++ b/docs/02_product/planned_features/infra_ir_measures_migration/feature_spec.md @@ -0,0 +1,725 @@ +# Feature Specification — Replace `pytrec_eval` with `ir_measures` for IR metric scoring + +**Date:** 2026-05-22 +**Status:** Draft +**Owners:** soundminds.ai (initial maintainer) +**Related docs:** +- [`idea.md`](./idea.md) +- [`docs/01_architecture/optimization.md`](../../../01_architecture/optimization.md) +- [`docs/01_architecture/tech-stack.md`](../../../01_architecture/tech-stack.md) +- Upstream library: [terrierteam/ir_measures](https://github.com/terrierteam/ir_measures) + +--- + +## 1) Purpose + +Replace the unmaintained `pytrec_eval` library with `ir_measures` as the IR-evaluation engine sitting behind [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py). `pytrec_eval`'s canonical repo at `cvangysel/pytrec_eval` last shipped a commit on 2020-09-07 with zero GitHub releases since; the package still installs and computes correct numbers, but every Python release going forward is a roll-the-dice for C-extension wheel availability, every `trec_eval` upstream bug report has no maintainer to file against, and every Apple-Silicon / arm64 first-build pays the gcc-compile tax. `ir_measures` (PyTerrier team, active) wraps `pytrec_eval` + `gdeval` + `judged_as_relevant` behind a provider interface, exposes a typed metric-object DSL (`nDCG@10`, `AP@5`, `RR`, etc.), and supports the per-query iteration shape (`iter_calc()`) that the next confidence wave (paired-comparison + Fisher randomization) needs. + +- **Problem:** Direct dependency on an abandoned-upstream C-extension library with single-maintainer bus-factor risk and no path forward for Python 3.14+. +- **Outcome:** `scoring.py` imports `ir_measures` rather than `pytrec_eval`; the user-facing `score(qrels, run, metrics) -> ScoreResult` signature is unchanged; the wire forms (`ndcg@10`, `map@5`, `mrr`, `map`) the rest of the codebase consumes are unchanged; every parity-tested metric value matches `pytrec_eval` to 6 decimal places. `pyproject.toml` no longer pins `pytrec_eval` directly. +- **Non-goal:** Replacing the pure-Python `bootstrap_ci_95()` / runner-up-gap / late-trial-stddev / convergence helpers in [`backend/app/domain/study/confidence.py`](../../../../backend/app/domain/study/confidence.py). Those helpers consume user-facing per-query metric dicts (already keyed by `ndcg@10` etc., not pytrec_eval wire forms) and do not import the library directly — they are unaffected by this migration. + +## 2) Current state audit + +### Existing implementations + +| File | What it does | Notes | +|---|---|---| +| [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) (~195 LOC) | The **only direct call site** of `import pytrec_eval`. Owns `_translate_metric_name()` (user-facing → wire form), `objective_metric_key()` (used by `confidence.py` + studies endpoint + worker), `SUPPORTED_METRICS` + `SUPPORTED_K_VALUES` frozensets (the source-of-truth allowlist for `studies.objective.metric` / `studies.objective.k`), and `score(qrels, run, metrics) -> ScoreResult`. | The `evaluator = pytrec_eval.RelevanceEvaluator(qrels, wire_set); evaluator.evaluate(run)` call lives on line 176; re-keying back to user-facing names lives at lines 180–192. The translation table maps `ndcg@k` → `ndcg_cut_k`, `map@k` → `map_cut_k`, `precision@k` → `P_k`, `recall@k` → `recall_k`, `map` → `map`, `mrr` → `recip_rank`. | +| [`backend/app/eval/qrels_loader.py`](../../../../backend/app/eval/qrels_loader.py) | Loads judgments from the `judgments` table into the dict-of-dicts shape `pytrec_eval` (and `ir_measures`) consume. Docstring at line 45 says "an empty dict… which `pytrec_eval` treats as a no-op". | The shape `{query_id: {doc_id: rating}}` is identical between `pytrec_eval` and `ir_measures` per the [ir_measures README](https://github.com/terrierteam/ir_measures) — no loader change needed. | +| [`backend/app/db/models/trial.py:19,83`](../../../../backend/app/db/models/trial.py) | `per_query_metrics` JSONB column docstring references `pytrec_eval` scores from `scoring.py::score()`. Two mentions: module-level docstring (line 19) **and** the column-level docstring on line 83. | Both docstrings describe a contract that survives this migration — the persisted shape is keyed by user-facing tokens (`ndcg@10`, `map@10`, `mrr`, plain `map`), NOT the wire forms. | +| [`backend/app/api/v1/schemas.py:534`](../../../../backend/app/api/v1/schemas.py) | `ObjectiveSpec` docstring says `k` is required "per pytrec_eval semantics: those metrics are computed at a cutoff rank". | The cutoff semantics are an IR convention, not a `pytrec_eval` invention — rewording without losing the constraint. | +| [`backend/app/api/v1/studies.py:270,313`](../../../../backend/app/api/v1/studies.py) | Two error-message strings reference `pytrec_eval`: (a) inline comment at 270 ("pytrec_eval scores 0 on every trial by construction"), (b) user-facing error message at 313 ("pytrec_eval will likely score 0 on every trial"). Both are part of `JUDGMENT_TARGET_MISMATCH` / `INSUFFICIENT_JUDGMENT_OVERLAP` handlers. | **The user-facing error message at 313 is wire-visible to operators** AND it is **currently pinned by a contract-test substring assertion** in `backend/tests/contract/test_studies_api_contract.py` (verified by grep). Architecturally, the contract is the `error_code` + `retryable` fields per `api-conventions.md`; in practice the test-enforced substring makes the message text part of what the PR must atomically update. Both must move in lock-step. | +| [`migrations/versions/0015_trials_per_query_metrics.py:17`](../../../../migrations/versions/0015_trials_per_query_metrics.py) | Docstring says the persisted column is keyed by user-facing names "NOT the pytrec_eval wire forms". | Historical migration — see §15. Rewording vs. leaving alone is a decision (see §19 Q1). | +| [`backend/app/domain/study/confidence.py`](../../../../backend/app/domain/study/confidence.py) | Imports `objective_metric_key` from `scoring.py` but **does NOT import `pytrec_eval` directly.** `bootstrap_ci_95()` at line 247 consumes a list of floats; the per-query dict is keyed by user-facing tokens. | **No code change required**, including no rewording — confidence.py doesn't name the library. | +| `pyproject.toml:47` | Direct pin `pytrec-eval>=0.5`. | Drop and replace with `ir-measures>=0.4.3`. | +| `pyproject.toml:156-158` | `[[tool.mypy.overrides]]` block setting `ignore_missing_imports = true` for module `pytrec_eval`. | `ir_measures` likely ships type hints (see §19 Q2); drop the override if confirmed, otherwise repoint to `ir_measures`. | +| [`Dockerfile`](../../../../Dockerfile) stage-2, lines 44–54 | Installs `gcc`, `g++`, `python3-dev` headers so `pytrec_eval`'s C extension can compile on first install (no prebuilt wheels). The deps stage is discarded; runtime is slim. | Conditional change — depends on whether `ir_measures` resolves a transitive C-extension backend (see §19 Q3). | + +### Doc-rewrite inventory (current-state docs that name `pytrec_eval`) + +Verified via `grep -rn 'pytrec_eval\|pytrec-eval'` on `main` HEAD (2026-05-22). This list is **canonical** for the migration — the idea file's list is a subset; the additional files below were missed and are required. + +| Doc | Lines | Treatment | +|---|---|---| +| [`README.md`](../../../../README.md) | 9 | Update — current-state project README. | +| [`CLAUDE.md`](../../../../CLAUDE.md) | 15, 29 | Update both mentions. | +| [`architecture.md`](../../../../architecture.md) | 131 | Update — "eval/ pytrec_eval scoring + Optuna runtime helpers". | +| [`release-notes-v0.1.0-draft.md`](../../../../release-notes-v0.1.0-draft.md) | 12 | Update — release notes for the not-yet-tagged v0.1.0 / v0.2.0 cycle. | +| [`docs/00_overview/product/relevance-copilot-spec.md`](../../../00_overview/product/relevance-copilot-spec.md) | 12, 155, 688, 690, 692–693, 711, 2192, 2302, 2513, 2658, 2722 (~11 mentions) | Update — durable umbrella spec, NOT a historical artifact. Includes the "Engine: pytrec_eval everywhere" subsection (lines 688–693) that needs reframing as the provider-abstracted engine choice. | +| [`docs/01_architecture/optimization.md`](../../../01_architecture/optimization.md) | 1 (title), 3, 15, 48, 50, 52–53, 69, 76, 87, 90, 176 (~10 mentions) | Update — **the canonical IR-evaluation architecture page**, including the code example block at lines 87–90. Title `# Optimization (Optuna + pytrec_eval)` becomes `# Optimization (Optuna + ir_measures)`. | +| [`docs/01_architecture/tech-stack.md`](../../../01_architecture/tech-stack.md) | 41 | Update — IR evaluation row in the stack table. | +| [`docs/01_architecture/system-overview.md`](../../../01_architecture/system-overview.md) | 76 | Update — component table row. | +| [`docs/01_architecture/README.md`](../../../01_architecture/README.md) | 21 | Update — directory readme cross-references `optimization.md`. | +| [`docs/01_architecture/data-model.md`](../../../01_architecture/data-model.md) | 52, 231 | Update — `judgments` consumer pointer + `per_query_metrics` description. | +| [`docs/01_architecture/cluster-lifecycle.md`](../../../01_architecture/cluster-lifecycle.md) | 159 | Update — "Wire up Optuna's RDBStorage + pytrec_eval" cluster-lifecycle step. | +| [`docs/02_product/mvp1-user-stories.md`](../../../02_product/mvp1-user-stories.md) | 40 | Update — US-7 narrative ("nDCG@10, MAP, and P@10"). | +| [`docs/08_guides/workflows-overview.md`](../../../08_guides/workflows-overview.md) | 123, 277 | Update — tenant-facing operator guide. | +| [`ui/public/docs/workflows-overview.md`](../../../../ui/public/docs/workflows-overview.md) | 123, 277 | Update — runtime-served mirror of the same doc; both must move in lock-step. | +| [`ui/public/guides/05_import_judgments_and_calibrate/script.md`](../../../../ui/public/guides/05_import_judgments_and_calibrate/script.md) | 6 | Update. | +| [`ui/public/guides/06_create_and_monitor_study/script.md`](../../../../ui/public/guides/06_create_and_monitor_study/script.md) | 8 | Update. | +| [`ui/public/guides/06_create_and_monitor_study/metadata.json`](../../../../ui/public/guides/06_create_and_monitor_study/metadata.json) | 26 | Update — `caption` field; same content shape as `script.md`. | +| [`ui/src/components/studies/create-study-modal.tsx`](../../../../ui/src/components/studies/create-study-modal.tsx) | 60 | Update — `// Source-of-truth: backend/app/eval/scoring.py:32 (metric → pytrec_eval token …)` source-of-truth comment. | +| [`ui/src/__tests__/components/studies/k-ignored.test.ts`](../../../../ui/src/__tests__/components/studies/k-ignored.test.ts) | 4 | Update — matching source-of-truth comment. | +| [`ui/src/lib/types.ts`](../../../../ui/src/lib/types.ts) | 1889 | Update — `pytrec_eval semantics` comment. | +| [`docs/00_overview/MVP1_DASHBOARD.md`](../../../00_overview/MVP1_DASHBOARD.md) | 64, 134 | **Auto-generated** — no manual edit. Regen ([`scripts/build_mvp1_dashboard.py`](../../../../scripts/build_mvp1_dashboard.py)) will pick up the planned-feature folder + spec changes. | + +**Historical artifacts — explicitly leave alone:** +- [`state.md`](../../../../state.md) — add a new dated entry describing this migration when it lands; do NOT back-edit existing entries. +- Anything under `docs/00_overview/implemented_features/` — every implemented-features folder is frozen at shipping date. Includes the `infra_optuna_eval` implementation plan, the `feat_pr_metric_confidence` spec/plan, etc. +- Dated blog posts under `docs/blog/` — datestamped at writing time; same convention as implemented_features. (Mentions of `pytrec_eval` in `2026-05-22-elevator-pitch-search-platform.md` and others are point-in-time references.) + +### Code-comment / docstring sweep (current-state, beyond the doc-rewrite list above) + +Verified via the same grep. Update each in lock-step with the scoring.py rewrite; treat as part of the same PR so the codebase doesn't ship with rotted references. + +| File | Lines | Treatment | +|---|---|---| +| [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) | 1, 4, 22, 52, 156, 176 | Module docstring + `_translate_metric_name` notes + `score()` docstring + the `import pytrec_eval` itself + the `RelevanceEvaluator` call. Most are the structural code change, not a comment-only sweep. | +| [`backend/app/eval/qrels_loader.py`](../../../../backend/app/eval/qrels_loader.py) | 45 | Reword "treats as a no-op" docstring. | +| [`backend/app/db/models/trial.py`](../../../../backend/app/db/models/trial.py) | 19, 83 | Two docstrings — module-level (line 19) and column-level on `per_query_metrics` (line 83). | +| [`backend/app/api/v1/schemas.py`](../../../../backend/app/api/v1/schemas.py) | 534 | `ObjectiveSpec` k-cutoff comment. | +| [`backend/app/api/v1/studies.py`](../../../../backend/app/api/v1/studies.py) | 270, 313 | TWO mentions, NOT one (idea understates). Line 270 is an inline comment; line 313 is part of a user-facing error message string. | +| [`migrations/versions/0015_trials_per_query_metrics.py`](../../../../migrations/versions/0015_trials_per_query_metrics.py) | 17 | See §19 Q1 — historical migration; decision is whether to reword the docstring or leave the historical reference. | +| `backend/tests/unit/eval/test_scoring.py` | 4, 143, 156 | Reword. | +| `backend/tests/unit/eval/test_scoring_metric_tokens.py` | 1, 82 | Reword. | +| `backend/tests/unit/eval/test_qrels_loader.py` | 53 | Reword. | +| `backend/tests/contract/test_trial_row_shape.py` | 6, 109, 113 | **Behavioral assertion**: line 113 asserts `metrics` keys do NOT start with `pytrec_eval` wire prefixes. The assertion itself must move to checking for the *new* wire-form leakage surface (the `ir_measures` metric-object `repr` strings) — see FR-2 / AC-3. Reword the docstrings on lines 6 + 109 accordingly. | +| `backend/tests/contract/test_studies_api_contract.py` | 156 | Reword. | +| `backend/tests/integration/test_run_trial_per_query_persistence.py` | 53, 111, 119 | Reword; the assertion on line 111 also lives under the "no pytrec_eval wire-form leakage" contract — update to assert no `ir_measures`-shaped leakage. | +| `backend/tests/integration/fixtures/handbuilt_qrels.py` | 75 | Reword `build_zero_scoring_hits_response` helper docstring. | +| `backend/tests/benchmarks/test_scoring_perf.py` | 56 | Reword the warm-up comment. | +| [`backend/app/services/test_seeding.py`](../../../../backend/app/services/test_seeding.py) | 127, 142 | **Inline bug fix (bundled per C2-F5).** Both occurrences of `"p@10"` in seed `metrics` dicts → `"precision@10"` to match the canonical user-facing token emitted by `score()` + `objective_metric_key()`. Pre-existing inconsistency surfaced by AC-3's strict regex; the 2-character fix is well under the inline-fix budget. | + +### Reader inventory — every code path that reads the user-facing token keys + +Verified via `grep -rn '\.metrics\b\|\.per_query_metrics\b\|\.primary_metric\b\|objective_metric_key' backend/app backend/workers` (2026-05-22). Every reader below consumes either `primary_metric` (scalar float, unaffected) or `metrics` / `per_query_metrics` JSONB keyed by user-facing tokens. FR-1's signature-preservation + FR-3's key invariant guarantee these readers continue to work without source edits. + +| Reader | File:line | What it reads | Why preserved | +|---|---|---|---| +| Trial list endpoint serialization | [`backend/app/api/v1/studies.py:551,566-567`](../../../../backend/app/api/v1/studies.py) | `t.primary_metric`, `t.metrics` (JSONB) | API response shape; user-facing tokens flow straight to wire. Preserved by FR-3. | +| Trial pagination + sort | [`backend/app/db/repo/trial.py:127-154`](../../../../backend/app/db/repo/trial.py) | `Trial.primary_metric` (scalar) | Scalar comparison; unaffected. | +| "Best trial" lookup | [`backend/app/db/repo/trial.py:224,234`](../../../../backend/app/db/repo/trial.py) | `func.max(Trial.primary_metric)` | Scalar aggregate; unaffected. | +| Confidence orchestrator | [`backend/app/domain/study/confidence.py:568,570,609,612`](../../../../backend/app/domain/study/confidence.py) | `winner_trial.primary_metric`, `winner_trial.per_query_metrics`, `runner_up_trial.per_query_metrics` | Calls `objective_metric_key()` to derive lookup key (preserved by FR-1). | +| Confidence service | [`backend/app/services/study_confidence.py:59,66,83-91`](../../../../backend/app/services/study_confidence.py) | Same as above | Same. | +| Digest worker — top trials | [`backend/workers/digest.py:318,632`](../../../../backend/workers/digest.py) | `t.primary_metric` | Scalar. | +| Orchestrator — cancel-streak | [`backend/workers/orchestrator.py:360`](../../../../backend/workers/orchestrator.py) | `Trial.status, Trial.primary_metric` | Scalar. | +| `objective_metric_key()` callers | [`backend/app/eval/scoring.py:106`](../../../../backend/app/eval/scoring.py) (definition); [`backend/app/domain/study/confidence.py:40,556`](../../../../backend/app/domain/study/confidence.py); [`backend/app/services/study_confidence.py:30,85`](../../../../backend/app/services/study_confidence.py); plus 2 test files | Function returns user-facing token string. | Preserved verbatim by FR-1. | + +### Write-surface audit — every code path that WRITES user-facing token keys + +Verified via `grep -rn 'metrics\s*=\|per_query_metrics\s*=\|Trial(' backend/app backend/workers migrations` (2026-05-22). There are **two production write paths**, both flowing through helpers that this migration preserves: + +| Write site | File:line | What it writes | Invariant boundary | +|---|---|---|---| +| **Happy-path scoring write** | [`backend/workers/trials.py:446-447`](../../../../backend/workers/trials.py) | `metrics=scored["aggregate"]`, `per_query_metrics=scored["per_query"]` — full dicts straight from `score()`. | `score()`'s re-keying step (preserved by FR-1) — re-keys `ir_measures` metric-object outputs back to user-facing tokens before returning. | +| **Idempotency-replay write** | [`backend/workers/trials.py:178`](../../../../backend/workers/trials.py) (COMPLETE path); lines 191, 203, 507 emit `metrics={}` for FAIL/PRUNED paths (no token contract to enforce on an empty dict) | `metrics = {objective_key: snapshot.value}` — synthesized single-key dict when Optuna already has the result cached (no re-run of search/score). | `objective_key = objective_metric_key(study.objective)` — function returns user-facing token (preserved by FR-1). | + +Both paths therefore funnel through user-facing tokens via FR-1's preserved API. No additional write surfaces exist in production code. **Test-only Trial constructors** ([`backend/tests/integration/test_pagination.py:264`](../../../../backend/tests/integration/test_pagination.py), [`backend/tests/integration/test_sort_pagination.py:419`](../../../../backend/tests/integration/test_sort_pagination.py)) write hand-rolled dicts that match the user-facing token shape — no AC-3 conflict. + +**Inline fix bundled with this PR (per GPT-5.5 cycle-2 C2-F5):** [`backend/app/services/test_seeding.py:127,142`](../../../../backend/app/services/test_seeding.py) currently writes `metrics={"ndcg@10": ..., "map": ..., "p@10": ...}` — the `"p@10"` literal is a pre-existing inconsistency with the canonical user-facing token (`precision@10`) that `score()` emits. Left alone, it would fail AC-3's strict regex. The fix is two character-substitutions (`p@10` → `precision@10`); bundled inline per CLAUDE.md "Inline-fix vs idea-file rubric" (≤50 LOC, no new tests beyond what AC-3 already runs, same subsystem). The fix is captured in §15 doc/code update list. + +### Navigation and link impact + +None. No URL paths change. No frontend route changes. + +### Existing test impact + +| Test file | Pattern | Count | Required change | +|---|---|---|---| +| `backend/tests/unit/eval/test_scoring.py` | Asserts metric values + re-keying | unchanged | Pass unchanged (public API preserved). Comments reworded. | +| `backend/tests/unit/eval/test_scoring_metric_tokens.py` | Asserts `_translate_metric_name` raises on bad input | unchanged | Pass unchanged — the translation table simplifies but `ValueError` paths are preserved. | +| `backend/tests/unit/eval/test_qrels_loader.py` | Loader returns empty dict on unknown id | unchanged | Pass unchanged. | +| `backend/tests/contract/test_trial_row_shape.py` | Asserts no `pytrec_eval` wire prefixes leak into `trials.metrics` | **updated assertion** | The "no wire-form leakage" contract moves from "no `ndcg_cut_`/`P_`/`recip_rank`/`map_cut_`/`recall_` keys" to "no `ir_measures` metric-object `repr` strings (e.g. `nDCG@10`-style PascalCase tokens) AND the existing pytrec_eval prefix check is preserved (since `ir_measures` may use pytrec_eval as a transitive backend whose wire names could still surface if re-keying is wrong)". Line 113's assertion expression itself widens; the test stays. | +| `backend/tests/integration/test_run_trial_per_query_persistence.py` | Same "no wire-form leakage" assertion at integration layer | **updated assertion** | Same as above. | +| `backend/tests/benchmarks/test_scoring_perf.py` | Per-call timing for `score()` | unchanged (modulo possible delta) | Should pass unchanged; the benchmark numbers may shift slightly under `ir_measures`. The benchmark is not part of the default test layer (marked `@pytest.mark.benchmark`) so no CI gate impact. | + +### Existing behaviors affected by scope change + +- **Behavior:** `pyproject.toml` direct pin on `pytrec-eval>=0.5`. + - Current: pinned; `uv sync` installs the C extension via gcc. + - New: dropped; `ir-measures>=0.4.3` pinned instead. `pytrec_eval` may remain transitively (resolves at impl-plan time — Q3). + - Decision needed: No — locked. +- **Behavior:** `[[tool.mypy.overrides]]` for `pytrec_eval`. + - Current: `ignore_missing_imports = true` for module `pytrec_eval`. + - New: if `ir_measures` ships type hints, drop the override entirely. If it doesn't, repoint the override to `ir_measures` (or both, if `pytrec_eval` remains a transitive backend that's still nominally importable). + - Decision needed: Yes (Q2) — empirical, resolved at impl-plan time. +- **Behavior:** Dockerfile stage-2 toolchain install. + - Current: installs `gcc`/`g++`/`python3-dev`. + - New: stays as-is if `ir_measures` resolves a transitive `pytrec_eval` C extension; can be dropped if no transitive C extension is needed. + - Decision needed: Yes (Q3) — empirical, resolved at impl-plan time. **Do NOT drop speculatively.** +- **Behavior:** `_translate_metric_name()` returning wire strings. + - Current: returns strings like `"ndcg_cut_10"` to pass into `pytrec_eval.RelevanceEvaluator`. + - New: returns `ir_measures` metric *objects* per the locked FR-1 mapping table — `nDCG@10`, `AP@10`, `AP` (plain), `P@10`, `R@10`, `RR`. Uncut `nDCG` / `P` / `R` are NEVER returned because the user-facing uncut forms remain invalid (the existing "requires an @ cut" `ValueError` paths are preserved). These objects are suitable to pass into `ir_measures.iter_calc()`. + - Decision needed: No — locked. + +--- + +## 3) Scope + +### In scope + +1. Replace `pytrec_eval` with `ir_measures` in [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) — preserve the public `score(qrels, run, metrics) -> ScoreResult` signature exactly; preserve `SUPPORTED_METRICS` / `SUPPORTED_K_VALUES` / `objective_metric_key()`; rewrite `_translate_metric_name()` to return `ir_measures` metric objects instead of wire strings. +2. Update [`pyproject.toml`](../../../../pyproject.toml) — drop `pytrec-eval>=0.5`, add `ir-measures>=0.4.3`, conditionally drop the `pytrec_eval` mypy override (per §19 Q2 resolution). +3. Add **parity test** at `backend/tests/unit/eval/test_scoring_parity.py` that runs both libraries against a fixed (qrels, run) fixture and asserts identical values to 6 decimal places for every valid `(metric, k)` pair in `SUPPORTED_METRICS × (SUPPORTED_K_VALUES ∪ {None})` per FR-2's per-metric k-rules — exactly 30 parametrized cases. Verifies per-query SHAPE parity (same outer qids, same inner metric keys per query, same handling of empty-overlap / qrel-only / run-only queries) per FR-3. The test is **a permanent CI gate**, kept active by adding `pytrec-eval>=0.5` to `[dependency-groups.dev]` (FR-4) so both libs remain reachable in the test environment indefinitely. +4. Update the "no wire-form leakage" assertions in `test_trial_row_shape.py:109-113` and `test_run_trial_per_query_persistence.py:111` to cover both the legacy `pytrec_eval` wire prefixes AND the new `ir_measures` metric-object `repr` shapes. +5. Doc-rewrite sweep across all current-state docs and code comments listed in §2 — both backend and frontend (UI source-of-truth comments). +6. Update the **user-visible error message** in `studies.py:313` (`INSUFFICIENT_JUDGMENT_OVERLAP` handler) — this is wire-visible to operators. +7. Conditionally update [`Dockerfile`](../../../../Dockerfile) stage-2 (per §19 Q3 resolution). + +### Out of scope + +- Replacing the pure-Python `bootstrap_ci_95()` / runner-up-gap / late-trial-stddev / convergence helpers in [`backend/app/domain/study/confidence.py`](../../../../backend/app/domain/study/confidence.py). They consume user-facing per-query metric dicts (already keyed by `ndcg@10`, etc.) and do not import the library directly. +- Adding ERR@k or any other metric not currently in `SUPPORTED_METRICS`. The metric allowlist is unchanged. +- Adding paired-bootstrap / Fisher randomization / paired-comparison helpers. Those are queued for the next confidence wave (referenced in idea §"Relationship to other work") and will sit on `ir_measures.iter_calc()` once that primitive is the per-query iteration surface; this migration unblocks but does not deliver them. +- Updating any doc under `docs/00_overview/implemented_features/` or dated blog posts in `docs/blog/`. Those are point-in-time historical artifacts (per CLAUDE.md `state.md` convention: "Don't back-edit them"). +- Replacing the qrels-loader implementation. The dict-of-dicts shape is identical between `pytrec_eval` and `ir_measures`. +- Changing the persisted `trials.metrics` / `trials.per_query_metrics` JSONB keys. The user-facing tokens (`ndcg@10`, `map@10`, `mrr`, plain `map`) are the durable contract — this migration must preserve them byte-identically (FR-1c). +- Adding `ir-measures[ranx]` as an extra. `ranx` remains available as a fast-follow if the hand-rolled paired-bootstrap grows uncomfortable; not added speculatively. +- Multi-PR phasing. The whole migration ships in one PR (see §3 Phase boundaries). + +### API convention check + +- **Endpoint prefix convention:** `/api/v1/` for business endpoints; unprefixed for `/healthz`. Verified in [`backend/app/api/v1/`](../../../../backend/app/api/v1/). **This migration adds zero endpoints.** +- **Router namespace for this feature's endpoints:** N/A — no new routes. +- **HTTP methods for CRUD:** N/A. +- **Non-auth error envelope shape:** existing operator-visible error message at `studies.py:313` lives inside the `_err()` helper which emits the standard envelope `{ "detail": { "error_code": "INSUFFICIENT_JUDGMENT_OVERLAP", "message": "", "retryable": false } }` per [`docs/01_architecture/api-conventions.md`](../../../01_architecture/api-conventions.md). The migration only changes the `message` text inside that envelope; envelope shape unchanged. +- **Auth error shape:** N/A in MVP1–3. + +### Phase boundaries (if multi-phase) + +**Single-phase.** The migration is one PR — scoring rewrite + parity test + pyproject change + doc sweep + UI comment sweep all together. Rationale: +- The doc sweep would create grep-divergence (some docs say `ir_measures`, some say `pytrec_eval`) if landed separately — operators reading docs would see inconsistency. +- The parity test depends on both libraries being installable; running it once at PR-merge time is sufficient and there's no reason to defer. +- The UI source-of-truth comments at `create-study-modal.tsx:60` and `k-ignored.test.ts:4` reference `scoring.py:32` directly — leaving the comments stale while scoring.py moves means the next reader doesn't know which file is the source of truth. + +No `phase2_idea.md` is required — there is only one phase. + +## 4) Product principles and constraints + +- **Public API of `scoring.py` is frozen.** Callers (the `run_trial` worker + tests + `confidence.py`) must require zero source edits. `score()`, `objective_metric_key()`, `SUPPORTED_METRICS`, `SUPPORTED_K_VALUES`, `ScoreResult`, `Qrels`, `Run` all preserve their existing signatures and shapes. +- **Persisted JSONB shape is frozen.** `trials.metrics` and `trials.per_query_metrics` JSONB columns must continue to be keyed by the user-facing tokens currently in production — `ndcg@10`, `map@10`, `precision@10`, `recall@10`, `mrr`, plain `map`. Any drift here breaks `confidence.py`'s `objective_metric_key()` consumer + the digest worker + the PR-body renderer + the trials API + any read-back from historical rows (every trial in production was persisted with these keys). +- **Six-decimal parity is a hard gate.** The migration is invalid if `ir_measures` and `pytrec_eval` disagree on any metric for the fixed fixture by ≥ 1e-6. +- **No `pytrec_eval` wire-form leakage outside `scoring.py`.** Same invariant as today — extended to also forbid `ir_measures` metric-object `repr` shapes leaking past the function. The contract is that everything outside `scoring.py` sees only user-facing tokens. +- **Forward-only.** No DEPRECATED markers, no compatibility shims, no "fall back to pytrec_eval if ir_measures fails". Single migration; complete cutover. + +### Anti-patterns + +- **Do not** preserve `import pytrec_eval` as a fallback path inside `scoring.py`. The migration is a cutover. A "try ir_measures; except: fall back to pytrec_eval" branch turns the parity test into a no-op and locks in the abandoned-upstream risk we're trying to escape. +- **Do not** invent a new persisted metric-key shape (e.g., `nDCG@10` PascalCase). The `trials.metrics` keys are a durable contract; rows that were persisted yesterday must still be readable tomorrow. `ir_measures.iter_calc()` yields `Metric` namedtuples whose `measure` field is the metric *object* — `str(measure)` produces PascalCase like `nDCG@10`. The migration must re-key back to the user-facing lowercase token (`ndcg@10`) before returning from `score()`. +- **Do not** rely on `ir_measures` calling `pytrec_eval` transitively as "good enough" without verifying parity. The PyTerrier team may route some metrics through `gdeval` or another backend that produces subtly different values for tie-handling or normalization. The fixed-fixture parity test is the only way to know. +- **Do not** drop the Dockerfile gcc/g++/python3-dev install speculatively. If `ir_measures` resolves `pytrec_eval` transitively, those headers are still required at install time. The verification recipe (`pip install ir-measures && pip show pytrec_eval`) settles this empirically at impl-plan time. +- **Do not** edit anything under `docs/00_overview/implemented_features/` or dated blog posts. Those are frozen historical artifacts; any back-edit muddies the project's "what shipped when" narrative. +- **Do not** ship the rewrite without also updating `studies.py:313`. That string is operator-facing in the `INSUFFICIENT_JUDGMENT_OVERLAP` error response — leaving it stale means production-grade incident messages still reference the abandoned library. + +## 5) Assumptions and dependencies + +- **Dependency:** [terrierteam/ir_measures](https://github.com/terrierteam/ir_measures) (`ir-measures>=0.4.3` on PyPI). + - Why required: Becomes the IR-evaluation engine behind `scoring.py`. + - Status: Implemented + actively maintained (PyTerrier team). + - Risk if missing: N/A — the library is the substrate this feature swaps onto. If PyPI is unreachable at install time, `uv sync` fails with a normal dependency-resolution error. +- **Dependency:** `pytrec-eval>=0.5` as a permanent `[dependency-groups.dev]` pin. + - Why required: The parity test (FR-2) is a **permanent CI gate**, not a one-shot pre-merge check. It imports both `pytrec_eval` and `ir_measures` side-by-side and asserts value-equivalence on every CI run. Keeping `pytrec-eval` in `[dependency-groups.dev]` ensures the test environment can always import it, regardless of whether `ir_measures` resolves it transitively (per §19 Q3). + - Status: Pinned by FR-4 — added in the same PR as the migration; remains until a future dedicated chore explicitly removes or replaces the parity gate. + - Risk if missing: Parity gate goes dark. Acceptable future-drag: when `pytrec-eval` eventually fails to install (e.g., Python 3.14+ with no wheels available), the parity test gracefully `xfail`s or skips; at that point a `chore_pytrec_eval_dev_dep_removal` idea file is filed and the gate retires. We accept this future drag today in exchange for the permanent live parity gate. +- **Dependency:** No external service / no LLM / no operator-environment change. Purely an in-process library swap. + +## 6) Actors and roles + +- **Primary actor:** the `run_trial` worker (an in-process consumer of `scoring.py::score()`). No human actor. +- **Role model:** N/A — RelyLoop is single-tenant + no auth through MVP3 per [`docs/01_architecture/tech-stack.md` §"Canonical release matrix"](../../../01_architecture/tech-stack.md). +- **Permission boundaries:** N/A. + +### Authorization + +N/A — single-tenant install, no auth surface. + +### Audit events + +N/A — audit_log lands at MVP2. + +## 7) Functional requirements + +### FR-1: `scoring.py` swaps to `ir_measures` + +- Requirement: + - The system **MUST** replace the `import pytrec_eval` statement in [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) with `import ir_measures` (and explicit metric-object imports as needed: `from ir_measures import nDCG, AP, RR, P, R`). + - The system **MUST** preserve the public function signatures `score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult`, `objective_metric_key(objective: dict[str, object]) -> str`, and the type aliases `Qrels = dict[str, dict[str, int]]`, `Run = dict[str, dict[str, float]]`, `ScoreResult = TypedDict("ScoreResult", {"aggregate": dict[str, float], "per_query": dict[str, dict[str, float]]})` byte-identically. + - The system **MUST** preserve the values of `SUPPORTED_METRICS = frozenset({"ndcg", "map", "precision", "recall", "mrr"})` and `SUPPORTED_K_VALUES = frozenset({1, 3, 5, 10, 20, 50, 100})`. + - The system **MUST** rewrite `_translate_metric_name(user_facing: str)` to return an `ir_measures` *metric object* per the mapping table below, rather than a `pytrec_eval` wire string. The function's `ValueError` paths for malformed tokens (`"unknown metric base"`, `"k value is not an integer"`, `"k not in allowlist"`, `"metric does not accept an @ cut"`, `"requires an @ cut"`) **MUST** all be preserved with the same triggering inputs. In particular, uncut `ndcg`, `precision`, and `recall` **MUST** continue to raise the existing "requires an @ cut" error — no new "plain metric" path is opened up. + + **Locked metric-object mapping** (per `ir_measures` README; uses MIT-licensed naming: `AP` for average precision, `RR` for reciprocal rank, `P`/`R` for precision/recall): + + | User-facing token | `ir_measures` metric object | Notes | + |---|---|---| + | `ndcg@` | `nDCG@` | k ∈ `SUPPORTED_K_VALUES`; uncut `ndcg` still rejected. | + | `map@` | `AP@` | k ∈ `SUPPORTED_K_VALUES`. | + | `map` (plain) | `AP` | Full-recall MAP; matches `pytrec_eval`'s plain `map`. | + | `precision@` | `P@` | k ∈ `SUPPORTED_K_VALUES`; uncut `precision` still rejected. | + | `recall@` | `R@` | k ∈ `SUPPORTED_K_VALUES`; uncut `recall` still rejected. | + | `mrr` | `RR` | k ignored; only the plain form is valid. | + - The system **MUST** re-key the per-query and aggregate results back to user-facing tokens (`ndcg@10`, `map@10`, `precision@10`, `recall@10`, `mrr`, plain `map`) before returning from `score()` — the persisted JSONB keys do not change. + - The system **MUST NOT** retain a `pytrec_eval` import-time fallback or runtime branch inside `scoring.py`. + - **Aggregate-computation contract (per GPT-5.5 cycle-2 C2-F4).** The implementation **MUST** compute the `ScoreResult.aggregate` dict by iterating per-query results and taking the arithmetic mean over **exactly the qid set the per-query dict reports for each metric** — mirroring `score()`'s current logic at [`backend/app/eval/scoring.py:187-192`](../../../../backend/app/eval/scoring.py). The implementation **MUST NOT** delegate the aggregate to `ir_measures.calc_aggregate(...)` because that helper aggregates over `ir_measures`' provider-defined query universe, which may include qrel-only or run-only topics that `pytrec_eval`'s `RelevanceEvaluator(...).evaluate(run)` excludes today. The required pattern: build the per-query dict via `ir_measures.iter_calc([metric_obj, ...], qrels, run)` (yielding `Metric(query_id, measure, value)` tuples), filter/re-key to user-facing tokens, THEN compute the aggregate as `sum(values) / len(values)` over the per-query dict's metric-present entries. +- Notes: The translation table simplifies from "metric → wire string" to "metric → ir_measures object" — one entry per metric family, with `@k` applied via the `metric @ k` operator. The wire-form re-keying step that converts the engine's output back to the user-facing token shape is preserved as today. The deliberate avoidance of `calc_aggregate()` is the load-bearing design choice for FR-3's per-query shape parity — see FR-2 fixture's qrel-only / run-only / empty-overlap edge cases. + +### FR-2: Parity test pins value-equivalence to 6 decimal places + +- Requirement: + - The system **MUST** ship `backend/tests/unit/eval/test_scoring_parity.py` (new file) that loads a fixed (qrels, run) fixture and asserts, for every valid `(metric, k)` pair in the parametrized cross of `SUPPORTED_METRICS × (SUPPORTED_K_VALUES ∪ {None})` (where the pairing respects the per-metric k-rules: `ndcg`/`precision`/`recall` require k; `map` accepts both; `mrr` ignores k → only `(mrr, None)` is parametrized), value-equivalence to 6 decimal places between: + - The aggregate value emitted by **the migrated `score(qrels, run, {token})["aggregate"][token]`** (i.e., the function under test, which computes the aggregate via per-query iteration per FR-1's aggregate-computation contract — NOT via `ir_measures.calc_aggregate()`), AND + - The aggregate value emitted by **`pytrec_eval.RelevanceEvaluator(qrels, {wire_set}).evaluate(run)` followed by the same mean-across-queries computation** the current `score()` performs (lines 187–192 of `scoring.py`). + - Tolerance: `abs(a - b) < 1e-6`. + - The parity test **MUST NOT** call `ir_measures.calc_aggregate()` directly — comparing `calc_aggregate()` to `pytrec_eval` would test whether `ir_measures`' native aggregate matches `pytrec_eval`, not whether the migrated `score()` is correct. The contract is that `score()`'s output is unchanged, so the test compares the new `score()` against the old `pytrec_eval`-based evaluation. + - The system **MUST** parametrize over every valid `(metric, k)` pair so each individual case is independently observable in pytest output. The cross-product yields exactly **30 cases**: `ndcg × 7 k-values = 7`, `precision × 7 = 7`, `recall × 7 = 7`, `map × 7 + plain map = 8`, `mrr (plain) = 1` (mrr ignores k so any other k-value would be a duplicate). + - The system **MUST** use the same fixture as the existing `test_scoring.py` (or a sibling fixture in `backend/tests/unit/eval/fixtures/`) so the inputs are diff-reviewable. Required fixture coverage: ≥ 8 queries, ≥ 5 docs each, mixed ratings (0/1/2/3), AND the following edge cases each represented by at least one query: (a) a query with no relevant docs (zero-score path), (b) a query in qrels with no matching docs in the run (qrel-only / missing-from-run), (c) a query in the run with no entry in qrels (run-only / unjudged), (d) a query whose run has no overlap at all with the qrels (the literal study2 scenario). +- Notes: The parity test runs **as a permanent CI gate**, not as a one-shot pre-merge check. To keep both libraries reachable in CI, this migration adds `pytrec-eval>=0.5` to `[dependency-groups.dev]` (FR-4) so the test environment can import both libs side-by-side even after `pytrec_eval` is no longer a runtime/`[project].dependencies` pin. Runtime images (the API + worker Compose services) no longer ship `pytrec_eval` directly — only `ir_measures` (which may resolve `pytrec_eval` transitively per §19 Q3). The parity test never runs against a runtime image. + +### FR-3: No wire-form leakage (extended) + per-query shape parity + +- Requirement: + - The system **MUST** preserve the existing "no `pytrec_eval` wire-form leakage" assertions in `backend/tests/contract/test_trial_row_shape.py:113` and `backend/tests/integration/test_run_trial_per_query_persistence.py:111`. + - The system **MUST** extend those assertions to also forbid `ir_measures`-shaped metric-object `repr` strings (PascalCase tokens like `nDCG@10`, `P@10`, `RR`, `AP@5`, `R@10`) from leaking into `trials.metrics` or `trials.per_query_metrics` JSONB. + - The system **MUST** keep the persisted JSONB key set restricted to the existing user-facing tokens: `ndcg@`, `map@`, `map` (plain), `precision@`, `recall@`, `mrr` — for `k ∈ SUPPORTED_K_VALUES`. + - **Per-query shape parity (added per GPT-5.5 cycle-1 F5).** The system **MUST** preserve the per-query result shape `score()` emits today: + - Outer dict keys (`query_id`) **MUST** be exactly the set of queries `pytrec_eval`'s evaluator currently returns — every query that has at least one rated doc in `qrels` AND at least one entry in `run`, no more and no less. + - For each outer key, inner dict keys **MUST** include exactly the same set of metric tokens as today's output — i.e., every requested metric token where the underlying backend produced a value. Today's logic at [`backend/app/eval/scoring.py:180-184`](../../../../backend/app/eval/scoring.py) conditionally omits a metric for a query when the wire-form key is absent from `raw_per_query[qid]`. The new implementation **MUST** maintain the same conditional-inclusion semantics — `ir_measures.iter_calc()` yielding fewer per-(qid, metric) tuples than `pytrec_eval`'s evaluator on the same input is a parity failure. + - For each outer key, every present inner value **MUST** equal the `pytrec_eval`-emitted value to 1e-6. + - Per-query shape parity is verified by the same parity test (FR-2 fixture extended with the four edge cases listed there) AND by a new integration test that loads a synthetic-but-realistic trial and asserts per-(qid, metric) tuple equality between the old and new implementations. +- Notes: This invariant is what makes `confidence.py::compute_outcome_summary` and `digest.py`'s prompt rendering work without per-metric translation. Any drift here cascades to incorrect per-query analytics on every existing study. `ir_measures.iter_calc()` returns `Metric(query_id, measure, value)` tuples that *may* omit pairs the caller didn't request, but its handling of edge cases (qrel-only-query, run-only-query, empty-overlap) needs explicit fixture-level pinning to confirm equivalence — see FR-2 fixture requirements. + +### FR-4: `pyproject.toml` updates + +- Requirement: + - The system **MUST** remove `pytrec-eval>=0.5` from `[project].dependencies` and add `ir-measures>=0.4.3` in its place. + - The system **MUST** add `pytrec-eval>=0.5` to `[dependency-groups.dev]` so the parity test (FR-2) can import both libraries side-by-side as a permanent CI gate. This pin is the parity-test infrastructure; it never ships into the runtime image because the Dockerfile installs only the `[project]` runtime deps via `uv sync --frozen --no-dev`. + - The system **MUST** either remove the `[[tool.mypy.overrides]] module = "pytrec_eval"` block (if `ir_measures` ships type hints AND `pytrec_eval` is no longer directly imported) OR keep one repointed at `module = "ir_measures"` (if it doesn't ship type hints) OR keep both blocks (the existing `pytrec_eval` override AND a new `ir_measures` override) if both modules are imported anywhere (the parity test imports `pytrec_eval` even when scoring.py doesn't). Resolution at impl-plan time per Q2; spec-level constraint: the override set **MUST** match the actual import surface after the migration. +- Notes: `uv.lock` regenerates on `uv lock` — that's a normal side-effect, not a manual edit. Keeping `pytrec-eval>=0.5` as a permanent dev-group dependency means future Python/uv upgrades may eventually break it (the C extension has no Python-3.14+ wheels guaranteed); when that happens, the parity test gracefully becomes `xfail`-or-skip and a `chore_pytrec_eval_dev_dep_removal` idea file is filed. We accept this future drag in exchange for the live parity gate today. + +### FR-5: Operator-visible string at `studies.py:313` is reworded + +- Requirement: + - The system **MUST** change the string `"pytrec_eval will likely score 0 on every trial"` in the `INSUFFICIENT_JUDGMENT_OVERLAP` error message (`backend/app/api/v1/studies.py:313`) to a wording that names `ir_measures` (or, equivalently, names the behavior without naming any library — e.g. `"every trial will score 0 on every metric"`). + - The system **MUST** update the contract-test substring assertion in `backend/tests/contract/test_studies_api_contract.py` that currently pins the pre-migration text. The impl-plan author confirms the exact assertion location and updates it atomically with the source change. +- Notes: Architecturally, per [`docs/01_architecture/api-conventions.md`](../../../01_architecture/api-conventions.md), the envelope contract is the `error_code` + HTTP status + `retryable` fields — the `message` field is free-text intended for human display. **In practice**, this repo's contract suite asserts on substrings of the message field for several error codes (a pre-existing pattern across `test_studies_api_contract.py`), so the message change is effectively a test-enforced contract for this PR. The atomic update of source + contract test is captured as a checklist item under §18. The `INSUFFICIENT_JUDGMENT_OVERLAP` error code itself, HTTP 422 status, and `retryable=false` field are all preserved unchanged. + +### FR-6: Conditional Dockerfile change + +- Requirement: + - The system **MUST** verify at impl-plan time whether `ir_measures` keeps `pytrec_eval` as a transitive backend for the metrics in `SUPPORTED_METRICS × SUPPORTED_K_VALUES`. Verification recipe: `pip install ir-measures && pip show pytrec_eval` (or `uv tree | grep pytrec_eval` after a fresh `uv sync`). + - If a C-extension backend is still resolved transitively, the [`Dockerfile`](../../../../Dockerfile) stage-2 install of `gcc` / `g++` / `python3-dev` (lines 44–54) **MUST** stay as-is, and the comment block at lines 44–48 explaining "pytrec_eval (added by infra_optuna_eval) ships as a sdist with NO prebuilt wheels…" **MUST** be reworded to credit `ir_measures` (or the actual transitive C-extension dependency) as the reason. + - If no C-extension is resolved transitively, the system **MAY** drop the gcc/g++/python3-dev install block. The decision must cite the empirical verification. +- Notes: Saves the speculative "drop the install" mistake that would surface as a slow next-feature debug session. + +### FR-7: Doc-rewrite sweep is complete + +- Requirement: + - The system **MUST** update every file in the doc-rewrite inventory in §2 ("Doc-rewrite inventory" table) AND every file in the code-comment / docstring sweep table. + - The system **MUST** include an explicit story / task in the implementation plan for regenerating [`docs/00_overview/MVP1_DASHBOARD.md`](../../../00_overview/MVP1_DASHBOARD.md) via `scripts/build_mvp1_dashboard.py` so the dashboard's two existing `pytrec_eval` mentions (lines 64 + 134, verified by grep) are picked up by the rewrite. Without an explicit task, the dashboard regen is silently optional and the merge-time grep gate (below) will fail. + - Verification gate at merge time: `grep -rn 'pytrec_eval\|pytrec-eval' .` on the working tree (excluding `node_modules`, `.venv`, `.git`) **MUST** return only: + - Lines inside `docs/00_overview/implemented_features/` (historical), + - Lines inside `docs/blog/` (dated historical), + - Lines inside `state.md` (historical entries; the new state.md entry describing the migration may name `pytrec_eval` once to reference "the library being replaced"), + - The parity-test file (`backend/tests/unit/eval/test_scoring_parity.py` — imports `pytrec_eval` for side-by-side comparison), + - The `pyproject.toml` `[dependency-groups.dev]` line (per FR-4), + - The Dockerfile comment IF it's reworded to explain `ir_measures`' transitive C-extension dependency (per FR-6). + - **Expanded wire-form sweep (per GPT-5.5 cycle-1 F11).** The merge-time check **MUST** also verify that the legacy `pytrec_eval` wire-form terms are gone from live-state docs and comments. Run `grep -rEn '(RelevanceEvaluator|ndcg_cut_|map_cut_|recip_rank|recall_[0-9]|\\bP_[0-9])' . --exclude-dir=node_modules --exclude-dir=.venv --exclude-dir=.git` and confirm every remaining match is inside the same allowlist as above (historical, dated, parity test, dependency-groups, Dockerfile). Specifically, code examples in [`docs/01_architecture/optimization.md`](../../../01_architecture/optimization.md) that show `pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut_10", "map", "P_10"}).evaluate(run)` MUST be rewritten to the `ir_measures.calc_aggregate([nDCG@10, AP, P@10], qrels, run)` shape. + - The system **MUST NOT** leave a single live-state doc, code comment, or UI source-of-truth annotation that names `pytrec_eval` OR cites a pytrec_eval wire-form metric token after the PR lands. +- Notes: The verification grep is itself part of the PR-time CI check. Spec §16 captures it as a release gate. + +## 8) API and data contract baseline + +### 8.1 Endpoint surface + +**N/A.** This migration adds zero endpoints, changes zero request/response shapes (except the wire-string in the `INSUFFICIENT_JUDGMENT_OVERLAP` envelope's `message` field, which is a free-text field not part of the contract). + +### 8.2 Contract rules + +- The user-visible error message string in the `INSUFFICIENT_JUDGMENT_OVERLAP` envelope is a non-contractual `message` field per [`docs/01_architecture/api-conventions.md`](../../../01_architecture/api-conventions.md) — the contract is the `error_code` + `retryable` fields, both unchanged. +- All other API contracts (`POST /api/v1/studies`, `GET /api/v1/studies/{id}`, `GET /api/v1/trials/...`, the digest endpoint, the proposal endpoints) are unchanged. + +### 8.3 Response examples + +**N/A** for new responses (none added). The `INSUFFICIENT_JUDGMENT_OVERLAP` envelope existing shape (which this migration rewords inside the `message` field) is documented in [`feat_study_preflight_overlap_probe/feature_spec.md`](../../../00_overview/implemented_features/2026_05_22_feat_study_preflight_overlap_probe/feature_spec.md). No change to status code (422), no change to `error_code` value, no change to `retryable` (false). + +### 8.4 Enumerated value contracts + +The wire-value enumerations consumed by the create-study flow (`objective.metric`, `objective.k`) are owned by `scoring.py`'s `SUPPORTED_METRICS` and `SUPPORTED_K_VALUES` frozensets: + +| Field | Accepted values (exact) | Backend source of truth | Frontend call site(s) | +|---|---|---|---| +| `objective.metric` | `ndcg`, `map`, `precision`, `recall`, `mrr` | [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) — `SUPPORTED_METRICS` frozenset | Create-study modal Step 5 metric `` | + +**This migration MUST preserve both frozensets' values byte-identically.** The frontend's hardcoded option arrays already match these via the source-of-truth comments at `create-study-modal.tsx:60` and `k-ignored.test.ts:4` (which this spec updates per §2's UI source-of-truth comment sweep). + +### 8.5 Error code catalog + +**No new error codes.** The existing `INSUFFICIENT_JUDGMENT_OVERLAP` code (owned by `feat_study_preflight_overlap_probe`) is unchanged in code value, status, and retryability — only its message text is reworded. + +## 9) Data model and state transitions + +**No data-model changes.** No new tables, no new columns, no migration, no `alembic_version` head bump. + +The persisted shapes that this migration explicitly preserves: + +- `trials.metrics` JSONB — keys are user-facing tokens (`ndcg@10`, `map@10`, `precision@10`, `recall@10`, `mrr`, plain `map`). +- `trials.per_query_metrics` JSONB — outer key is `query_id` (UUIDv7 string); inner dict keyed by the same user-facing tokens; value is `float` per-query score. + +### Required invariants + +- **JSONB key invariant:** Every key in `trials.metrics` and the inner dicts of `trials.per_query_metrics` MUST be in the set `{f"{metric}@{k}" for metric in {"ndcg","precision","recall"} for k in SUPPORTED_K_VALUES} | {f"map@{k}" for k in SUPPORTED_K_VALUES} | {"map", "mrr"}`. No `pytrec_eval` wire prefixes (`ndcg_cut_`, `P_`, `recip_rank`, `map_cut_`, `recall_`). No `ir_measures` PascalCase reprs (`nDCG@10`, `P@10`, `RR`, `AP@5`). +- **Value-equivalence invariant:** For any fixed (qrels, run, metric), the value emitted by the new `score()` MUST equal the value the old `score()` would have emitted to within 1e-6. Verified at PR-merge time by the parity test (FR-2). +- **Existing-row read invariant:** Every existing `trials` row in production was persisted with the user-facing token keys. After this migration, every existing row must still be readable by `confidence.py::compute_study_confidence` (which calls `objective_metric_key()` to derive the per-query lookup key). The contract preservation in FR-1 + FR-3 guarantees this — no row-rewrite, no backfill, no migration. + +### State transitions + +N/A — no state machines added or modified. + +### Idempotency/replay behavior + +N/A — no event-driven surfaces. + +## 10) Security, privacy, and compliance + +- **Threats:** None new. The migration is an in-process library swap. +- **Controls:** Same as today — `ir_measures` is an MIT-licensed open-source library on PyPI; no secrets, no network calls, no PII handling. +- **Secrets/key handling:** N/A. +- **Auditability:** `state.md` gains a new dated entry (per CLAUDE.md "After completing a task" convention) describing the migration: date, Alembic head (unchanged), test counts, parity result. +- **Data retention/deletion/export impact:** None. +- **Supply-chain note:** `ir_measures` adds one new pip dependency (`ir-measures`) plus its transitive set. Verify at impl-plan time that the transitive set (e.g., `cwl-eval`, `pandas` if it's a hard dep, etc.) is acceptable. No license-incompatible dependencies are expected — `ir_measures` itself is MIT. + +## 11) UX flows and edge cases + +### Information architecture + +**N/A** — no UI route changes, no new pages, no new components. Two distinct categories of operator-visible change exist (per GPT-5.5 cycle-1 F8 — keep them disjoint mentally): + +1. **Runtime API/UI behavior change (one surface):** The reworded error message inside the `INSUFFICIENT_JUDGMENT_OVERLAP` envelope (delivered as JSON to the create-study modal's error toast; verified at `studies.py:313`). This is what shows up live in the running app post-merge. +2. **Operator-visible documentation copy change (many surfaces):** The guide scripts at `/guide/05` + `/guide/06`, the workflows-overview docs at `/guide/workflows-overview` + the duplicate at `ui/public/docs/workflows-overview.md`, and the guide-06 caption in `metadata.json`. These are static markdown / JSON served by the UI from `ui/public/`. They do not change runtime behavior but they ARE what a tenant reads when learning the product. Listed under §15 documentation update requirements; release-notes copy should call them out. + +### Tooltips and contextual help + +**N/A** — no new UI elements. No glossary keys are added or modified. Existing glossary entries (`ui/src/lib/glossary.ts`) that reference IR metric semantics (`ndcg@10`, etc.) describe the semantics via the standard IR convention — they do not name `pytrec_eval`. Verified by grep on `ui/src/lib/glossary.ts` returning zero `pytrec` matches. + +### Primary flows + +1. **Operator creates a study.** Backend renders the search-space → enqueues `run_trial` jobs. The worker imports `scoring.py::score()`; under the hood, `score()` now uses `ir_measures` rather than `pytrec_eval`. Same `trials` rows persist, same `trials.metrics` keys, same `primary_metric` value, same digest, same PR body. Operator sees zero behavioral difference. +2. **Operator creates a study with insufficient judgment overlap.** Same flow up to the preflight probe at `studies.py:286`. The 422 envelope returns with the same `error_code` and `retryable` fields; the `message` text now says "ir_measures" instead of "pytrec_eval". The error toast in the create-study modal shows the updated text — visually the only operator-noticeable change in this whole migration. +3. **Operator reads a guide (`/guide/06/...`).** The guide-06 caption + script (currently mentioning "scores via pytrec_eval") now mentions "scores via ir_measures". Both `ui/public/guides/06_*/script.md` and `metadata.json` ship the rewording. + +### Edge/error flows + +- **`ir_measures` import fails at app boot.** Same failure mode as today's `pytrec_eval` import failure — the worker's process exits at import time, surfaced via the Compose healthcheck for the worker container. No new error path; same operator-recovery flow (rebuild image, retry). +- **Parity test fails on one metric.** Block the PR. The migration is invalid until the discrepancy is investigated. Most-likely cause: `ir_measures` routes the metric through a non-`pytrec_eval` backend (e.g., `gdeval`) with subtly different tie-handling. Mitigation: pin `ir_measures` to use the `pytrec_eval` provider for that metric specifically, or accept the discrepancy and document it. +- **`pytrec_eval` not resolvable in CI dev environment.** Permanent mitigation: `pytrec-eval>=0.5` lives in `[dependency-groups.dev]` (FR-4); the test environment always has it. The transitive question (whether `ir_measures` ALSO pulls it) is only relevant to Dockerfile decisions (FR-6 / Q3), not to parity-test viability. + +## 12) Given/When/Then acceptance criteria + +### AC-1: scoring.py imports ir_measures, not pytrec_eval + +- Given the merged PR for this migration is on `main`. +- When `grep -n 'import pytrec_eval\|import ir_measures' backend/app/eval/scoring.py` runs. +- Then exactly one match returns and it is `import ir_measures` (or `import ir_measures` + explicit metric imports). Zero `import pytrec_eval` lines remain in `scoring.py`. +- Example values: + - Input: `grep -n 'import pytrec_eval' backend/app/eval/scoring.py` + - Expected: exit code 1 (no matches). + +### AC-2: Parity test asserts ≤ 1e-6 value drift on every supported (metric, k) + +- Given a fixed graded-qrels-and-run fixture in `backend/tests/unit/eval/fixtures/` (≥ 8 queries, mixed ratings). +- When `pytest backend/tests/unit/eval/test_scoring_parity.py -v` runs. +- Then every parametrized case passes; for each `(metric, k)` pair in `SUPPORTED_METRICS × (SUPPORTED_K_VALUES ∪ {None})`, `abs(ir_measures_aggregate - pytrec_eval_aggregate) < 1e-6`. +- Example values: + - Input: pytest parametrize over `[("ndcg", 10), ("ndcg", 5), ("map", None), ("map", 10), ("precision", 10), ("recall", 10), ("mrr", None), …]` + - Expected: zero failures, ≥ 30 parametrized cases (5 metrics × ~7 k values minus the invalid combinations). + +### AC-3: trials.metrics keys are exactly the user-facing token set + +- Given the test database after running `make test-integration`. +- When `backend/tests/contract/test_trial_row_shape.py` and `backend/tests/integration/test_run_trial_per_query_persistence.py` execute. +- Then every key in `trials.metrics` and every inner key in `trials.per_query_metrics` matches the **strict** regex `^(?:mrr|map|(?:ndcg|precision|recall|map)@(?:1|3|5|10|20|50|100))$` (i.e., user-facing tokens only, with uncut `ndcg` / `precision` / `recall` excluded per the k-rules in `objective_metric_key()`); no `pytrec_eval` wire prefixes (`ndcg_cut_`, `P_`, `recip_rank`, `map_cut_`, `recall_`); no `ir_measures` PascalCase reprs (`nDCG@`, `P@`, `RR`, `AP@`, `R@`). +- Negative cases (must be REJECTED by the assertion to prove the regex is strict — implementer adds explicit `pytest.raises(AssertionError)` cases for these): + - `ndcg` (uncut — forbidden by `objective_metric_key()`), + - `precision` (uncut — same), + - `recall` (uncut — same), + - `nDCG@10` (PascalCase `ir_measures` repr), + - `P@10` (`ir_measures` repr), + - `RR` (`ir_measures` repr), + - `AP@5` (`ir_measures` repr), + - `R@10` (`ir_measures` repr), + - `ndcg_cut_10` (`pytrec_eval` wire), + - `recip_rank` (`pytrec_eval` wire), + - `map_cut_10` (`pytrec_eval` wire), + - `P_10` (`pytrec_eval` wire), + - `recall_10` (`pytrec_eval` wire). +- Positive cases (must be ACCEPTED): `ndcg@10`, `map@10`, `map`, `mrr`, `precision@10`, `recall@10`, `ndcg@5`, `precision@50`, `map@1`, etc. — every value in the cross-product `{ndcg|precision|recall} × {1,3,5,10,20,50,100}` plus `{map@1,...,map@100, map, mrr}`. +- Example values: + - Input: any successful trial's `trials.metrics` JSONB. + - Expected: every key matches the strict regex; the assertion is a substantive guard, not a tautology that passes on every string. + +### AC-4a: pyproject.toml `[project].dependencies` has NO `pytrec-eval` pin + +- Given the merged PR. +- When a section-aware TOML check runs (e.g., `python -c "import tomllib, sys; data = tomllib.load(open('pyproject.toml','rb')); deps = data['project']['dependencies']; assert not any(d.startswith('pytrec-eval') for d in deps), deps"`). +- Then exit code 0 — `pytrec-eval` is absent from the runtime dependency list. +- Example values: + - Input: the python check above. + - Expected: zero matches in `[project].dependencies`. + +### AC-4b: pyproject.toml `[dependency-groups.dev]` HAS the `pytrec-eval` pin + +- Given the merged PR. +- When the same section-aware check runs against the dev group (`data['dependency-groups']['dev']`). +- Then exit code 0 — a line `pytrec-eval>=0.5` (or tighter pin) IS present in `[dependency-groups.dev]`. This is the permanent parity-gate infrastructure pin per FR-4. +- Example values: + - Input: `python -c "import tomllib; d=tomllib.load(open('pyproject.toml','rb')); assert any(x.startswith('pytrec-eval') for x in d['dependency-groups']['dev'])"` + - Expected: zero exit code. + +### AC-4c: mypy overrides match the import surface + +- Given the merged PR. +- When the impl-plan author inspects `[[tool.mypy.overrides]]` blocks in `pyproject.toml`. +- Then every `module = ""` block matches an actual `import ` somewhere in the source tree (`scoring.py` imports `ir_measures`; the parity-test imports both `pytrec_eval` and `ir_measures`). Stale overrides for modules no longer imported anywhere in the codebase are removed. +- Example values: + - Acceptable: an `ir_measures` override iff `ir_measures` doesn't ship type hints (resolved by Q2); a `pytrec_eval` override iff the parity test or any other source still imports it. + - Unacceptable: a `pytrec_eval` override when nothing in the source tree imports `pytrec_eval`. + +### AC-5: ir-measures direct pin exists + +- Given the merged PR. +- When `grep -nE '^\s*"ir-measures' pyproject.toml` runs. +- Then exactly one match returns, in `[project].dependencies`, with version constraint `>=0.4.3` (or a tighter pin if the impl-plan author chose). +- Example values: + - Input: `grep '"ir-measures' pyproject.toml` + - Expected: `"ir-measures>=0.4.3",` + +### AC-6: Doc-rewrite verification grep is clean + +- Given the merged PR. +- When `grep -rn 'pytrec_eval\|pytrec-eval' . --exclude-dir=node_modules --exclude-dir=.venv --exclude-dir=.git` runs. +- Then every remaining match is inside one of: + - `docs/00_overview/implemented_features/` (historical), + - `docs/blog/` (dated historical), + - `state.md` (historical entries; the new entry describing the migration may name `pytrec_eval` once as "the library being replaced"), + - The parity test file (`backend/tests/unit/eval/test_scoring_parity.py`), + - The Dockerfile comment IF reworded to credit `ir_measures`' transitive C-extension dependency (per FR-6), + - The `pyproject.toml` `[dependency-groups.dev]` line (per FR-4 — REQUIRED, not optional), + - The `pyproject.toml` `[[tool.mypy.overrides]]` block for `pytrec_eval` if AC-4c keeps it. +- Example values: + - Input: the grep above. + - Expected: no matches in `backend/app/`, `backend/tests/` (other than the parity test), `docs/01_architecture/`, `docs/02_product/` (other than this spec's own §15 docstring inventory and §19 decision log naming the library being replaced), `docs/08_guides/`, `ui/src/`, `ui/public/`, `README.md`, `CLAUDE.md`, `architecture.md`, `release-notes-v0.1.0-draft.md`. + +### AC-7: studies.py:313 error message names ir_measures (or no library) + +- Given the merged PR. +- When `grep -n "ir_measures\|pytrec_eval" backend/app/api/v1/studies.py` runs. +- Then the inline comment (currently line 270) and the error-message string (currently line 313) both name `ir_measures` instead of `pytrec_eval`, OR are reworded to name no library at all (e.g., "every trial will score 0 on every metric"). Either is acceptable. +- Example values: + - Input: `grep -n "score 0 on every trial" backend/app/api/v1/studies.py` + - Expected: one match, with `ir_measures` (or no library) named in the surrounding string. + +### AC-8: Existing tests still pass unchanged + +- Given the merged PR. +- When `make test-unit test-integration test-contract` runs (with running Postgres + ES + OpenSearch). +- Then every pre-existing test passes — no test assertion is weakened, skipped, or deleted to accommodate the migration. The only test changes are: (a) new `test_scoring_parity.py` file added (the **permanent** CI gate per FR-2), (b) the "no wire-form leakage" assertions in `test_trial_row_shape.py:113` and `test_run_trial_per_query_persistence.py:111` are *extended* (not weakened) to also forbid `ir_measures` PascalCase reprs (`nDCG@`, `P@`, `RR`, `AP@`, `R@`), (c) the `test_studies_api_contract.py` case that pins the `INSUFFICIENT_JUDGMENT_OVERLAP` envelope message substring is updated to match the new wording, (d) a new integration test (per AC-12) loads a pre-migration-shaped trial fixture and exercises the consumers without re-scoring. +- Example values: + - Input: `pytest backend/tests/unit backend/tests/integration backend/tests/contract -v` + - Expected: zero new failures; total count = previous count + ~30 (parity test parametrized cases) + 1 (existing-row regression). + +### AC-12: Existing-row read regression — pre-migration JSONB shapes still hydrate consumers + +- Given a synthetic-but-realistic `Trial` row inserted via a test fixture with `metrics = {"ndcg@10": 0.82, "map@10": 0.71, "map": 0.65, "mrr": 0.91}` AND `per_query_metrics = {"q1": {"ndcg@10": 0.83, "map@10": 0.7, "mrr": 1.0}, "q2": {"ndcg@10": 0.81, "map@10": 0.72, "mrr": 0.83}, ...}` (exactly the shape rows persisted before this migration carry today). +- When the test invokes `compute_study_confidence` against that trial via the existing `fetch_study_confidence` service, AND the trial-list endpoint serializes the row, AND the digest worker's top-trials section runs on it. +- Then every consumer returns its expected output without raising: `compute_study_confidence` returns a `ConfidenceShape` whose `headline.metric` matches the trial's objective, whose `ci_95` populates from the per-query values, and whose `per_query_outcomes.top_regressors` enumerates the named regressors correctly; the trial-list response includes the JSONB unchanged; the digest sees the row in its top-trials list. +- This AC is the load-bearing test for the "no-migration / no-backfill" claim — it proves the new code path on the OLD persisted shape works without re-scoring. +- Example values: + - Input: insert a fixture trial with the JSONB above; call `GET /api/v1/studies/{study_id}` and assert the `confidence` block populates. + - Expected: `confidence.headline.value == 0.82`, `confidence.headline.n_queries > 0`, no exceptions, no NULL ConfidenceShape return. + +### AC-9: Coverage gate stays green + +- Given the merged PR. +- When `make test-unit && coverage report` runs. +- Then the 80% coverage gate from `pyproject.toml [tool.coverage.report].fail_under` is satisfied. The migration touches only `scoring.py` (already well-covered) and shouldn't move the needle. +- Example values: + - Expected: coverage report shows ≥ 80% for `backend.app.eval.scoring`. + +### AC-10: Dockerfile change matches the empirical verification + +- Given the merged PR. +- When the impl-plan author runs `pip install ir-measures && pip show pytrec_eval` in a clean environment (per FR-6 verification recipe). +- Then either: + - (a) `pytrec_eval` IS resolved as a transitive backend → Dockerfile lines 44–54 stay; comment is reworded to credit `ir_measures` as the reason for needing gcc/g++/python3-dev headers. + - (b) `pytrec_eval` is NOT resolved transitively → Dockerfile lines 44–54 are dropped; alpine/arm64 first-builds get the wheel install path. +- The implementation plan's Story for this conditional change documents which branch was taken AND cites the verification output. +- Example values: + - Branch (a) command: `docker build .` succeeds on a fresh checkout; the `deps` stage still installs gcc. + - Branch (b) command: `docker build .` succeeds with gcc removed. + +### AC-11: state.md gains a new dated entry + +- Given the merged PR. +- When the user reads `state.md` after merge. +- Then the "Most recent meaningful changes" section's newest entry describes the migration: PR number, date, scope summary ("scoring.py swapped from pytrec_eval to ir_measures"), parity test result, Alembic head ("unchanged at `0015_trials_per_query_metrics` — application-layer-only feature"). The entry is *new*; no previous entry is back-edited. + +## 13) Non-functional requirements + +- **Performance:** `score()` p99 latency unchanged ± 10%. `ir_measures` adds a thin layer over the same backend; the parametrized-fixture benchmark at `backend/tests/benchmarks/test_scoring_perf.py` should report numbers within 10% of pre-migration baseline. The benchmark is not part of the default test gate (marked `@pytest.mark.benchmark`); spot-check after merge. +- **Reliability:** No change in error semantics. The library swap doesn't introduce new failure modes — same C-extension load path (if transitive), same input shape, same return shape. +- **Operability:** No new env vars, no new secrets, no new logs, no new metrics. The structlog events emitted by `run_trial` (`trial.scored`, etc.) carry the same payload shape — they reference `metric` keys by user-facing token, which is unchanged. +- **Accessibility/usability:** N/A — no UI change. + +## 14) Test strategy requirements (spec-level) + +| Layer | Path | Required coverage | +|---|---|---| +| Unit | `backend/tests/unit/eval/test_scoring.py` | Existing tests pass unchanged. Comments reworded (no new test cases needed). | +| Unit | `backend/tests/unit/eval/test_scoring_metric_tokens.py` | Existing tests pass unchanged. `_translate_metric_name`'s ValueError paths still trigger on the same inputs (the function now returns metric objects instead of strings, but the exception surface is preserved). | +| Unit | `backend/tests/unit/eval/test_qrels_loader.py` | Existing tests pass unchanged. | +| Unit | **NEW** `backend/tests/unit/eval/test_scoring_parity.py` | Parametrized parity test per FR-2 / AC-2. Imports both `pytrec_eval` and `ir_measures`; asserts 6-decimal equivalence per `(metric, k)` pair across the 30-case cross. Permanent CI gate (FR-4 keeps `pytrec-eval` as a dev/test dep so this test stays runnable). The fixture covers the four edge cases listed in FR-2 (no-relevant-docs query, qrel-only query, run-only query, empty-overlap query). | +| Unit | **NEW (or extended)** `backend/tests/unit/eval/test_scoring_per_query_shape.py` | Per-query shape parity per FR-3 / AC-2 (extended). Asserts the outer-key set and per-query inner-key set match `pytrec_eval`'s output 1:1 on the same edge-case fixture. | +| Contract | `backend/tests/contract/test_trial_row_shape.py` | The "no wire-form leakage" assertion at line 113 is **extended** (not replaced) to also forbid `ir_measures` PascalCase reprs (`nDCG@`, `P@`, `RR`, `AP@`, `R@`). AC-3's negative-case enumeration verifies the assertion is substantive (rejects each forbidden token explicitly). Tests still pass against the user-facing token contract. | +| Contract | `backend/tests/contract/test_studies_api_contract.py` | The `INSUFFICIENT_JUDGMENT_OVERLAP` envelope test that pins the message substring (per FR-5) is updated to match the new wording. | +| Integration | `backend/tests/integration/test_run_trial_per_query_persistence.py` | The "no wire-form leakage" assertion at line 111 extended same as contract test. Existing happy-path tests pass unchanged — the worker invokes `score()` which now returns the same shape via `ir_measures`. | +| Integration | **NEW** `backend/tests/integration/test_existing_row_read_compat.py` | Per AC-12: insert a synthetic-but-realistic trial with pre-migration JSONB shape, then exercise `fetch_study_confidence` + the trial-list endpoint + the digest's top-trials section without re-scoring. The load-bearing test for the "no migration / no backfill" claim. | +| E2E | N/A | No E2E coverage required. No UI flow changes (error-toast text change is a string update, not a flow change). | + +## 15) Documentation update requirements + +- `docs/01_architecture/optimization.md` — title + 10 mentions reworded to `ir_measures`. Code example block at lines 87–90 (`pytrec_eval.RelevanceEvaluator(qrels, ...).evaluate(run)`) rewritten to the `ir_measures.calc_aggregate([nDCG@10, AP, P@10, ...], qrels, run)` shape. The "Engine: pytrec_eval everywhere" subsection (if mirrored from the umbrella spec) reframed as "Engine: provider-abstracted via `ir_measures`". +- `docs/01_architecture/tech-stack.md` — line 41 IR-evaluation row updated. +- `docs/01_architecture/system-overview.md` — line 76 component table row updated. +- `docs/01_architecture/README.md` — line 21 cross-reference updated. +- `docs/01_architecture/data-model.md` — lines 52, 231 reworded. +- `docs/01_architecture/cluster-lifecycle.md` — line 159 reworded. +- `docs/00_overview/product/relevance-copilot-spec.md` — umbrella spec; ~11 mentions reworded. The "Engine: pytrec_eval everywhere" subsection (lines 688–693) reframed as a provider-abstraction discussion. Stack table at line 155 + line 2513 + decision log at 2658 + appendix at 2722 all updated. +- `docs/02_product/mvp1-user-stories.md` — US-7 narrative reworded. +- `docs/02_product/planned_features/feat_study_baseline_trial/idea.md` — sibling-coordination: line 56 ("scores via `pytrec_eval`") reworded to `ir_measures`. Same-PR update. +- `docs/02_product/planned_features/feat_auto_followup_studies/idea.md` — sibling-coordination: line 47 ("Optuna + pytrec_eval are deterministic") reworded to `Optuna + ir_measures`. Same-PR update. +- `docs/08_guides/workflows-overview.md` — lines 123, 277 reworded. +- `ui/public/docs/workflows-overview.md` — same content as above; lock-step. +- `ui/public/guides/05_import_judgments_and_calibrate/script.md` — line 6 reworded. +- `ui/public/guides/06_create_and_monitor_study/script.md` — line 8 reworded. +- `ui/public/guides/06_create_and_monitor_study/metadata.json` — line 26 `caption` reworded. +- `README.md` — line 9 reworded. +- `CLAUDE.md` — lines 15, 29 reworded. +- `architecture.md` — line 131 reworded. +- `release-notes-v0.1.0-draft.md` — line 12 reworded. +- `state.md` — new dated entry per AC-11. **Do not back-edit existing entries.** + +**Auto-regenerated (still a required PR task):** +- `docs/00_overview/MVP1_DASHBOARD.md` — regenerates via `scripts/build_mvp1_dashboard.py`. The implementation plan **MUST** include an explicit story for running the regen so the dashboard's two pre-existing `pytrec_eval` mentions (line 64 in the `infra_optuna_eval` row, line 134 in the `infra_ir_measures_migration` row that this very feature is closing) are picked up at merge time. Without an explicit regen task, the merge-time AC-6 grep gate fails (per GPT-5.5 cycle-1 F11). + +## 16) Rollout and migration readiness + +- **Feature flags / staged rollout:** None. The migration is a library swap with a parity test as the gate. There's no operator-facing surface to flag. +- **Migration/backfill expectations:** None. No DB migration, no data backfill — every existing `trials` row continues to be readable because the persisted JSONB key shape is preserved (FR-1c / FR-3). +- **Operational readiness gates:** + - Parity test passes for every (metric, k) in the 30-case cross (AC-2). + - Per-query shape parity passes for the four edge-case queries in the fixture (AC-2 extended via FR-3). + - Doc-sweep grep is clean AND the broader wire-form sweep (`RelevanceEvaluator|ndcg_cut_|map_cut_|recip_rank|recall_[0-9]|\bP_[0-9]`) is also clean per FR-7 (AC-6). + - Existing test suites all green at unit + integration + contract layers (AC-8). + - Existing-row read regression passes — pre-migration JSONB shapes hydrate every consumer without raising (AC-12). + - 80% coverage gate met (AC-9). + - Dockerfile change matches empirical verification (AC-10). + - MVP1_DASHBOARD.md regenerated (per §15 explicit regen task). +- **Release gate:** PR-time CI runs the full test matrix (including the new parity test); the GPT-5.5 cross-model review and Gemini Code Assist review pass; the canonical grep verification (AC-6) runs as part of the impl-plan's pre-push gate (or as a dedicated CI step). No staged rollout needed — the migration is atomic at merge time. + +## 17) Traceability matrix + +| FR ID | Acceptance Criteria IDs | Planned stories/tasks | Test files/suites | Docs to update | +|---|---|---|---|---| +| FR-1 (scoring.py swap) | AC-1, AC-3 | Story 1: rewrite scoring.py with locked metric-object mapping | `test_scoring.py`, `test_scoring_metric_tokens.py`, `test_qrels_loader.py` (all pass unchanged); contract tests verify JSONB keys | `scoring.py` docstrings + `qrels_loader.py:45` | +| FR-2 (parity test, permanent CI gate) | AC-2 | Story 2: write parity test + fixture (30 parametrized cases, 4 edge-case queries) | NEW `test_scoring_parity.py` | N/A | +| FR-3 (no wire-form leakage + per-query shape parity) | AC-3, AC-8, AC-12 | Story 3: extend leakage assertions, add per-query shape parity test, add existing-row regression | `test_trial_row_shape.py`, `test_run_trial_per_query_persistence.py`, NEW `test_scoring_per_query_shape.py`, NEW `test_existing_row_read_compat.py` | N/A | +| FR-4 (pyproject.toml — runtime + dev) | AC-4, AC-5 | Story 4: drop pytrec-eval from `[project]`, add `ir-measures` to `[project]`, add `pytrec-eval` to `[dependency-groups.dev]`, update mypy overrides | N/A (verified by AC-4/AC-5 grep) | N/A | +| FR-5 (studies.py:313 wording) | AC-7 | Story 5: reword inline comment (270) + error message (313) + contract-test substring | `test_studies_api_contract.py` (envelope substring) | studies.py inline + error message | +| FR-6 (Dockerfile conditional) | AC-10 | Story 6: verify transitive backend; update Dockerfile | N/A (verified by docker build) | Dockerfile lines 44–54 + comment | +| FR-7 (doc sweep + dashboard regen + broader grep) | AC-6, AC-11 | Story 7: full doc + code-comment + UI source-of-truth sweep + MVP1_DASHBOARD regen | N/A (verified by AC-6 grep AND broader wire-form grep) | Everything in §15 doc list + dashboard regen | + +## 18) Definition of feature done + +This feature is complete when: + +- [ ] All acceptance criteria (AC-1 through AC-12) pass in CI. +- [ ] `make test-unit`, `make test-integration`, `make test-contract`, `make typecheck`, `make lint` all green. +- [ ] The grep gate per AC-6 passes (no live-state `pytrec_eval` mentions outside the allowlist). +- [ ] The broader wire-form sweep passes per FR-7 (`RelevanceEvaluator`, `ndcg_cut_`, `map_cut_`, `recip_rank`, `recall_[0-9]`, `\bP_[0-9]` are all clean outside the allowlist). +- [ ] The parity test (AC-2) passes for all 30 parametrized `(metric, k)` cases. +- [ ] Per-query shape parity passes for the 4 edge-case queries (no-relevant, qrel-only, run-only, empty-overlap). +- [ ] The existing-row read regression (AC-12) passes — pre-migration JSONB shapes hydrate confidence + trial-list + digest without raising. +- [ ] `docs/01_architecture/optimization.md` + `tech-stack.md` + `system-overview.md` + all docs in §15 are updated and merged in the same PR. The umbrella spec (`docs/00_overview/product/relevance-copilot-spec.md`)'s "Engine: pytrec_eval everywhere" subsection is reframed as a provider-abstraction discussion. +- [ ] `state.md` has a new dated entry describing the migration (per AC-11). +- [ ] `docs/00_overview/MVP1_DASHBOARD.md` regenerated via `scripts/build_mvp1_dashboard.py`. +- [ ] Q1/Q2/Q3/Q4/Q5 resolutions are recorded in the decision log with cited verification output (per §19). +- [ ] Dockerfile change decision (drop vs. keep gcc/g++/python3-dev) is documented with the empirical verification output that drove it. +- [ ] Sibling planned-feature idea files (`feat_study_baseline_trial/idea.md:56`, `feat_auto_followup_studies/idea.md:47`) updated to name `ir_measures` instead of `pytrec_eval`. +- [ ] `studies.py:313` operator-visible error message AND its contract-test substring assertion updated atomically. +- [ ] `pytrec-eval>=0.5` lives only in `[dependency-groups.dev]` after the PR; no occurrence in `[project].dependencies`. +- [ ] No open questions remain in §19. + +## 19) Open questions and decision log + +### Open questions + +All five resolve at implementation-plan time. They are empirical and bounded — none require a product decision. + +**Q1: Does the historical migration `0015_trials_per_query_metrics.py:17` docstring get reworded?** + +- Context: The migration file is part of `docs/00_overview/implemented_features/2026_05_21_feat_pr_metric_confidence/`'s shipped scope, but the *file itself* lives at `migrations/versions/0015_trials_per_query_metrics.py` — under `migrations/`, not under `implemented_features/`. The CLAUDE.md / spec convention says implemented-features folders are frozen, but `migrations/` lives outside that convention. +- Options: + - (a) Treat as historical (leave alone): the migration was authored when `pytrec_eval` was the engine; rewording rewrites history. + - (b) Treat as current-state (reword): the migration file is in the active source tree and will be read by future engineers (or `alembic upgrade --sql` output); leaving stale references is misleading. +- Recommended: **(b) reword.** The migration file is read by future engineers; the docstring is a forward-looking explanation of the persisted shape, not a record of authoring history. +- Owner: spec author / impl-plan author — Due: before plan-creation. (Recommended decision is the default; deferred to impl-plan time only if the recommendation is rejected.) + +**Q2: Does `ir_measures` ship PEP 561 type hints (a `py.typed` marker)?** + +- Context: `pyproject.toml` line 156–158 currently has `[[tool.mypy.overrides]] module = "pytrec_eval" ignore_missing_imports = true` because `pytrec_eval` ships no type info. If `ir_measures` does ship type info, the override can be dropped entirely; if not, it must be repointed. +- Verification recipe: `pip install ir-measures && find $(python -c 'import ir_measures, os; print(os.path.dirname(ir_measures.__file__))') -name py.typed` — empty output means no `py.typed` marker; one path means it does. +- Owner: impl-plan author — Due: before writing pyproject.toml updates. + +**Q3: Does `ir-measures>=0.4.3` keep `pytrec_eval` as a default transitive backend for the metrics in `SUPPORTED_METRICS × SUPPORTED_K_VALUES`?** + +- Context: Drives the Dockerfile change decision (FR-6), the `[dependency-groups.dev]` decision (FR-4), and the parity-test fixture infrastructure (FR-2). +- Verification recipe: `pip install ir-measures && pip show pytrec_eval` — exit code 0 + non-empty output means `pytrec_eval` is reachable transitively; exit code 1 means it isn't. Also confirmable via `uv tree | grep pytrec_eval` after a fresh `uv sync`. +- Two outcomes for the **Dockerfile / runtime image** (the parity-test viability is already guaranteed independently by FR-4's `[dependency-groups.dev]` pin, regardless of transitive routing): + - (a) `pytrec_eval` is transitively present for at least one `SUPPORTED_METRICS` value. Dockerfile gcc/g++/python3-dev install stays (the runtime image needs the C-extension toolchain at install time); the comment block is reworded to credit `ir_measures`' transitive backend. + - (b) `pytrec_eval` is NOT transitively present (or every `SUPPORTED_METRICS` value routes to a pure-Python provider in the resolved `ir_measures` version). Dockerfile gcc/g++/python3-dev install may be droppable; verify a clean `docker build .` still succeeds. +- Owner: impl-plan author — Due: before writing the Dockerfile + pyproject.toml stories. + +**Q4: Does any metric need provider forcing to achieve parity, and if so, what is the documented API at the pinned `ir_measures` version?** + +- Context: `ir_measures` is a provider-abstracted facade that routes each metric to one of `pytrec_eval`, `gdeval`, `judged_as_relevant`, or `cwl_eval` based on internal heuristics that can shift across versions. For parity to pass (FR-2), every metric's chosen provider must produce values within 1e-6 of `pytrec_eval`'s direct output. +- Verification recipe (observable-first): + 1. Pin `ir_measures` to a specific version in `pyproject.toml`. + 2. Run the parity test (FR-2) — if all 30 cases pass, no provider forcing is needed and Q4 resolves "no action required, no internals introspected". + 3. If any case fails, inspect the failing metric's provider routing using only **documented** `ir_measures` APIs at the pinned version (the impl-plan author reads `ir_measures`' published documentation / README / release notes — NOT private modules with leading-underscore names like `_get_measure_args`). If a documented provider-forcing API exists at the pinned version, use it; the spec's hard 1e-6 parity gate (§4 + AC-2) MUST still pass after the forcing. +- **Bounded outcomes** for Q4 (the impl-plan resolves to exactly one of these — there is no "accept the drift" branch because AC-2 is a hard gate; relaxing AC-2 would be a separate user-approved spec change, not a Q4 resolution): + - (a) Parity passes for all 30 cases against the default `ir_measures` provider routing → no forcing required, no spec change. + - (b) Parity passes for all 30 cases after invoking the **documented** provider-forcing API at the pinned `ir_measures` version → cite the API in the impl-plan + decision log. + - (c) Parity passes for all 30 cases after bumping or repinning `ir_measures` to a version where the default routing produces parity → cite the version pin + decision log. + - (d) **Blocker.** Parity cannot be made to pass via (a)/(b)/(c). The PR is blocked pending a separate user-approved spec change that either relaxes AC-2's tolerance OR shrinks `SUPPORTED_METRICS` (the latter changes the public allowlist contract and is a product decision). +- Owner: impl-plan author — Due: before the parity test is run for the first time. +- Note: the spec deliberately avoids citing `ir_measures.measures._get_measure_args(...)` / `set_provider(...)` (private names that may not exist at the pinned version). The implementation plan, not this spec, anchors any internal-API references against the actual installed version (per GPT-5.5 cycle-2 C2-F6 + cycle-3 C3-F1). + +**Q5: What is the transitive dependency set / license footprint / performance delta of `ir_measures` vs. the current `pytrec_eval`-only path?** + +- Context: `ir_measures` may pull `pandas`, `numpy` (already a dep), `cwl-eval`, plus provider backends transitively. The supply-chain note in §10 says "no license-incompatible dependencies expected" but doesn't verify it. The performance non-functional requirement in §13 says "p99 latency unchanged ± 10%" but doesn't have a measurement. +- Verification recipe: in a clean checkout, run (a) `uv sync && uv tree | grep -v pytrec_eval` to see the new transitive set, (b) cross-reference each new package's license against the project's [LICENSE](../../../../LICENSE) (Apache 2.0) compatibility list, (c) run `pytest backend/tests/benchmarks/test_scoring_perf.py -v` against both `main` and the feature branch and compare the warm-call timings. +- Owner: impl-plan author — Due: before merging the PR. +- Acceptable outcomes: all transitive deps Apache-2.0-compatible; benchmark within 10% of pre-migration baseline. Failing either of these is a blocker. + +### Decision log + +- **2026-05-22 — Single-PR scope, no phasing.** The migration ships as one PR (scoring + parity test + pyproject + doc sweep + UI comment sweep + Dockerfile conditional + sibling-idea coordination). Rationale: doc-sweep grep-divergence and UI source-of-truth comments rotting against scoring.py would create operator-visible inconsistency if landed separately. +- **2026-05-22 — `ir_measures` over `ranx`.** The idea author considered `ranx` directly; rejected on (a) single-maintainer bus factor (same failure mode), (b) Numba install + JIT cold-start cost, (c) no provider abstraction. `ir_measures` keeps provider-swap as a config change. +- **2026-05-22 — `ir_measures` over `pytrec-eval-terrier` (the actively-maintained fork).** Rejected because (a) still a C extension with the same build-pain footprint, (b) doesn't introduce the provider abstraction. +- **2026-05-22 — Public API of `scoring.py` is frozen.** `score()`, `objective_metric_key()`, `SUPPORTED_METRICS`, `SUPPORTED_K_VALUES`, `ScoreResult`, `Qrels`, `Run` all preserve their existing signatures and shapes. Rationale: every caller (the `run_trial` worker, `confidence.py`, the studies endpoint, every test) reads these symbols today; a signature change would cascade. +- **2026-05-22 — Persisted JSONB key shape is frozen.** `trials.metrics` + `trials.per_query_metrics` keep their user-facing token keys (`ndcg@10`, etc.). Rationale: every existing row in production was persisted with these keys; changing them silently breaks every read consumer. +- **2026-05-22 — Parity gate is 6 decimal places (1e-6 tolerance).** Rationale: standard IR-eval precision; tighter than the FP32 noise floor of typical retrieval scoring; loose enough to tolerate the cumulative rounding of bootstrap means. +- **2026-05-22 — Sibling planned-feature idea files updated in same PR.** `feat_study_baseline_trial/idea.md:56` and `feat_auto_followup_studies/idea.md:47` mention `pytrec_eval` by name; updating in the same PR avoids the drift-vs-source-of-truth failure mode where the planning doc still names the abandoned library after the codebase moves. +- **2026-05-22 — `confidence.py` is out of scope.** It consumes user-facing per-query keys (not pytrec_eval wire forms) and does not import the library directly. The "land before further confidence surface grows" pressure that motivated this migration originally was rendered moot when `feat_pr_metric_confidence` Phase 1 shipped 2026-05-21 against user-facing keys. +- **2026-05-22 — Cross-model review trajectory.** Cycle 1: 11 findings (all accepted, all applied — metric-object mapping table in FR-1, stricter AC-3 regex with negative cases, parity-test lifecycle, per-query shape parity in FR-3, reader inventory + write-surface audit in §2, new AC-12 existing-row regression, Q4 + Q5, expanded wire-form sweep in FR-7, §11 two-categories clarification, dashboard-regen task). Cycle 2: 6 findings (all accepted, all applied — dependency lifecycle deduped to "permanent dev-group" model only, AC-4 split into AC-4a/b/c with section-aware verification, §2 mapping example aligned with FR-1, FR-1/FR-2 aggregate-computation contract pinned to per-query iteration + manual mean rather than `calc_aggregate()`, inline `test_seeding.py p@10 → precision@10` fix bundled, Q4 reframed around observable behavior + documented APIs). Cycle 3: 1 finding (accepted + applied — Q4 bounded outcomes locked, no "accept drift" branch). Convergence: 11 → 6 → 1 — spec approved for impl-plan generation. diff --git a/docs/02_product/planned_features/infra_ir_measures_migration/implementation_plan.md b/docs/02_product/planned_features/infra_ir_measures_migration/implementation_plan.md new file mode 100644 index 00000000..f3fadee4 --- /dev/null +++ b/docs/02_product/planned_features/infra_ir_measures_migration/implementation_plan.md @@ -0,0 +1,1150 @@ +# Implementation Plan — Replace `pytrec_eval` with `ir_measures` for IR metric scoring + +**Date:** 2026-05-22 +**Status:** Draft +**Primary spec:** [`feature_spec.md`](./feature_spec.md) — Approved 2026-05-22 (GPT-5.5 3 cycles, 11 → 6 → 1 findings) +**Policy source(s):** [`CLAUDE.md`](../../../../CLAUDE.md) (Absolute Rules + Bug Fix Protocol + Test Conventions), [`docs/01_architecture/api-conventions.md`](../../../01_architecture/api-conventions.md) (error envelope), [`docs/05_quality/testing.md`](../../../05_quality/testing.md) (test-layer convention) + +--- + +## 0) Planning principles + +- Spec traceability first: every story maps to FR IDs from §17 of the spec. +- The 8 stories ship as **one PR** (locked by spec §3 Phase boundaries) — there are no inter-story phase gates, just sequencing inside one branch. +- Fail-loud tests: the parity test, the per-query shape parity test, and the existing-row read regression are the load-bearing gates. +- Keep the migration narrow: no scope expansion beyond what FR-1 through FR-7 require. The `confidence.py` rework, the `ranx` extras, and the `pytrec-eval-terrier` fork are all explicitly OUT per spec §3 + §19 decisions. +- Permanent test infra: `pytrec-eval>=0.5` stays in `[dependency-groups.dev]` indefinitely so the parity gate keeps firing post-merge. + +## 1) Scope traceability (FR → epics/phases) + +| FR ID | Epic / Story | Notes | +|---|---|---| +| FR-1 (scoring.py swap + locked metric-object mapping + per-query iteration aggregate) | Epic 1 / Story 1.3 | Single core code rewrite. Aggregate-via-iter contract (C2-F4) is load-bearing. | +| FR-2 (parity test, permanent CI gate, 30 parametrized cases) | Epic 1 / Story 1.2 (fixture + skeleton) → Story 1.4 (assertions activated) | Test-infra precedes activation to keep the branch always-green. | +| FR-3 (no wire-form leakage + per-query shape parity) | Epic 1 / Story 1.4 (shape parity) + Story 1.5 (leakage + existing-row regression) | Per-query shape parity rides with parity activation; leakage assertion extension + existing-row regression are a separate story. | +| FR-4 (pyproject.toml: runtime `ir-measures` + dev `pytrec-eval` + mypy override audit) | Epic 1 / Story 1.1 | First story — every later story needs the new lib installed. | +| FR-5 (operator-visible error message at `studies.py:313` + docstring sweep) | Epic 1 / Story 1.6 | `test_studies_api_contract.py:156` docstring reworded; existing structural assertions unchanged (no message substring is pinned today — verified 2026-05-22). The `test_seeding.py` `p@10` → `precision@10` inline fix moved to Story 1.5 task 0 per plan cycle-1 F7. | +| FR-6 (Dockerfile conditional gcc/g++/python3-dev install) | Epic 1 / Story 1.7 | Empirical — depends on §19 Q3 resolution. | +| FR-7 (doc-rewrite sweep + dashboard regen + broader wire-form grep gate) | Epic 1 / Story 1.8 | Final story — runs after every code change is in so the grep gates verify a clean working tree. | + +All 7 FRs covered. No deferred phases (spec is single-phase per §3). No deferred-phase tracking files needed. + +### Open question resolution + +| Question (from spec §19) | Resolved during | Resolution recorded in | +|---|---|---| +| Q1: Reword historical migration `0015_trials_per_query_metrics.py:17` docstring? | Story 1.8 (doc sweep) | Story 1.8 tasks list — recommendation is reword (forward-looking explanation for future engineers). | +| Q2: Does `ir_measures` ship `py.typed`? | Story 1.1 (pyproject) | Story 1.1 task — `find $(python -c 'import ir_measures, os; print(os.path.dirname(ir_measures.__file__))') -name py.typed`; result drives the mypy-override decision. | +| Q3: Does `ir_measures` keep `pytrec_eval` as a transitive backend? | Story 1.7 (Dockerfile) | Story 1.7 task — `pip install ir-measures && pip show pytrec_eval` + `uv tree \| grep pytrec_eval`. | +| Q4: Provider routing per metric / forcing API? | Story 1.4 (parity activation) | If all 30 cases pass, resolves "no action needed". If any fail, narrow to bounded outcomes (a)/(b)/(c)/(d) per spec §19 Q4. | +| Q5: Transitive deps + license + performance verification? | Story 1.1 (deps inspection) + Story 1.4 (perf benchmark) | Story 1.1 — `uv tree` license cross-check. Story 1.4 — `pytest backend/tests/benchmarks/test_scoring_perf.py -v` before/after diff. | + +## 2) Delivery structure + +**Epic → Story → Tasks → DoD.** Eight stories in one epic, sequenced as listed in §1 (1.1 → 1.2 → 1.3 → 1.4 → 1.5 → 1.6 → 1.7 → 1.8). Each story is verifiable independently (tests pass at the end of each); the branch is always green. + +### Conventions (project-specific) + +The migration touches the eval layer, the worker layer, the API layer, the test infrastructure, and docs. Conventions to respect: + +- **Public API of `scoring.py` is FROZEN** per spec FR-1 (Decision log lock). `score()`, `objective_metric_key()`, `SUPPORTED_METRICS`, `SUPPORTED_K_VALUES`, `ScoreResult`, `Qrels`, `Run` keep their signatures byte-identically. +- **Persisted JSONB keys are FROZEN** per spec FR-1c / FR-3. `trials.metrics` and `trials.per_query_metrics` continue to use user-facing tokens (`ndcg@10`, `map@10`, `mrr`, plain `map`). +- **Aggregate is computed via per-query iteration + manual mean** — NOT via `ir_measures.calc_aggregate()` (C2-F4 contract). +- **No `pytrec_eval` runtime dependency.** `pytrec-eval>=0.5` lives only in `[dependency-groups.dev]` after this PR. +- **`uv sync`** regenerates `uv.lock` automatically; that's a normal side-effect. +- **Conventional Commits** per CLAUDE.md Absolute Rule #7. Branch name: `feature/infra-ir-measures-migration`. Commits use `feat(eval):` / `chore(eval):` / `docs(eval):` prefixes. +- **No `--no-verify` on commits.** Pre-commit hooks (ruff format + ruff check + mypy strict on backend, prettier on frontend) must pass. + +### AI Agent Execution Protocol + +0. Load context first: read `architecture.md` and `state.md` before starting Story 1.1. +1. Read scope: confirm story outcome + new/modified files + tasks + DoD. +2. Implement story-by-story in the §1 order (1.1 through 1.8 — strictly sequential). +3. Run tests after each story: + - Story 1.1: `uv sync && uv lock --check && make typecheck`. + - Story 1.2: `pytest backend/tests/unit/eval/test_scoring_parity.py -v` (will skip with `pytest.mark.skip`; check fixture loads). + - Story 1.3: `make test-unit` (existing scoring tests must pass against new `ir_measures` backend). + - Story 1.4: `pytest backend/tests/unit/eval/ -v` (parity test now active; 30 cases + per-query shape test). + - Story 1.5: `make test-contract && make test-integration` (leakage assertions extended; existing-row regression added). + - Story 1.6: `pytest backend/tests/contract/test_studies_api_contract.py -v` (docstring reworded; no assertion change — see Story 1.6 task 3). + - Story 1.7: `docker build .` (Dockerfile change empirically verified). + - Story 1.8: `grep -rn 'pytrec_eval\|pytrec-eval' . --exclude-dir=node_modules --exclude-dir=.venv --exclude-dir=.git` AND the broader wire-form grep (per FR-7) — both must match only the allowlist. +4. Update docs in same PR (the doc sweep IS Story 1.8 — no separate workstream). +5. No migration round-trip (no schema change). +6. After Story 1.8, update `state.md` with a new dated entry (no back-edits). + +--- + +## Epic 1 — Replace `pytrec_eval` with `ir_measures` + +### Story 1.1 — Add `ir-measures` to runtime deps; move `pytrec-eval` to dev-group; audit mypy overrides + +**Outcome:** `pyproject.toml` is the new dependency state — `ir-measures>=0.4.3` is a runtime dep; `pytrec-eval>=0.5` is a dev/test dep only; `[[tool.mypy.overrides]]` matches the actual import surface. `uv.lock` regenerates cleanly; `mypy --strict` passes. + +**New files** + +None. + +**Modified files** + +| File | Change | +|---|---| +| [`pyproject.toml`](../../../../pyproject.toml) | (a) Remove `"pytrec-eval>=0.5"` from `[project].dependencies` (line 47). (b) Add `"ir-measures>=0.4.3"` in its place under `[project].dependencies`. (c) Add `"pytrec-eval>=0.5"` to `[dependency-groups.dev]` (between `pre-commit>=4.6.0` and `types-PyYAML>=6.0`, alphabetical). (d) Audit the `[[tool.mypy.overrides]]` block at lines 156–158: see Q2 resolution below. | +| [`uv.lock`](../../../../uv.lock) | Auto-regenerated by `uv lock` — do NOT edit by hand. Commit alongside `pyproject.toml`. | + +**Tasks** + +1. Resolve §19 Q2 (Does `ir_measures` ship `py.typed`?). Recipe: + ```bash + uv sync --frozen=false # let the lock regen + find $(uv run python -c 'import ir_measures, os; print(os.path.dirname(ir_measures.__file__))') -name py.typed + ``` + Two outcomes: + - Empty: `ir_measures` does NOT ship type hints → ADD a new `[[tool.mypy.overrides]]` block: + ```toml + [[tool.mypy.overrides]] + module = "ir_measures" + ignore_missing_imports = true + ``` + - One path: `ir_measures` DOES ship `py.typed` → no new override needed. +2. Audit the existing `pytrec_eval` mypy override at lines 156–158. After Story 1.3 lands, `scoring.py` no longer imports `pytrec_eval`. After Story 1.2 lands, the parity test DOES import `pytrec_eval`. The override stays as long as ANY source imports `pytrec_eval`. After this story, no source imports it yet (Stories 1.2 + 1.4 add the parity test), so the override is needed only after Story 1.2. **For ordering: keep the override in Story 1.1 — it's already there, and removing/re-adding it is churn. Delete only if Q3 resolves "no pytrec_eval in dev-group either" (which would never happen under FR-4).** +3. Resolve §19 Q5 (transitive deps + license). Recipe: + ```bash + uv tree + # Then manually verify each new package's license is Apache 2.0-compatible (MIT, Apache, BSD all OK). + ``` + Expected new packages from `ir-measures`: `cwl-eval`, possibly `pandas` (already present transitively via `optuna`'s deps), `numpy` (already present via `scikit-learn`). Record the new dependency list in the Story 1.1 PR commit message. +4. Run `uv sync` to regenerate `uv.lock`. Commit `pyproject.toml` + `uv.lock` together. +5. Run `make typecheck` — must pass. If `ir_measures` doesn't ship types and the override was missed, this will surface a `Cannot find implementation or library stub for module named 'ir_measures'` error → add the override. + +**Definition of Done (DoD)** + +- `pyproject.toml` `[project].dependencies` contains `"ir-measures>=0.4.3"` AND does NOT contain any `pytrec-eval` line (verified by AC-4a's TOML check). +- `pyproject.toml` `[dependency-groups.dev]` contains `"pytrec-eval>=0.5"` (verified by AC-4b). +- `[[tool.mypy.overrides]]` blocks match the import surface: `ir_measures` override iff Q2 says it doesn't ship type hints; `pytrec_eval` override stays as long as anything in the source tree imports it (verified by AC-4c). +- `uv lock --check` exit code 0 (lockfile is up-to-date). +- `make typecheck` passes. +- The Q5 dependency license audit is recorded in the commit message (the new packages' licenses are documented). + +--- + +### Story 1.2 — Create parity test fixture and skeleton (skipped placeholders) + +**Outcome:** The parity test infrastructure is in the repo, fixture loads, both libraries are importable side-by-side. The 30 parametrized cases exist but are marked `pytest.mark.skip(reason="scoring.py not yet migrated to ir_measures — activate in Story 1.4")` so the branch stays green while Story 1.3's code rewrite is in progress. + +**New files** + +| File | Purpose | +|---|---| +| `backend/tests/unit/eval/fixtures/__init__.py` | Empty (Python package marker). Verify `backend/tests/unit/eval/` doesn't already have a `fixtures/` directory; if it does, skip this. | +| `backend/tests/unit/eval/fixtures/parity_qrels_run.py` | Fixed (qrels, run) fixture per spec FR-2. Must include: ≥ 10 queries, ≥ 5 docs each, mixed graded ratings 0/1/2/3, AND the 6 edge cases per spec FR-2 + plan cycles 2/3: (a) one query with no relevant docs (zero-score path), (b) one query in qrels with no matching docs in run (qrel-only / missing-from-run), (c) one query in run with no entry in qrels (run-only / unjudged), (d) one query whose run has no overlap at all with the qrels (the "study2 scenario"), (e) one query with `qrels[q] = {}` but non-empty `run[q]` (cycle-2 C2-F1 universe-filter coverage), (f) one query with `run[q] = {}` but non-empty `qrels[q]` (cycle-3 C3-F1 symmetric coverage). Export `qrels: Qrels` and `run: Run` module-level constants. | +| `backend/tests/unit/eval/test_scoring_parity.py` | The parity test. Imports both `pytrec_eval` and `ir_measures`. Defines the 30 parametrized cases per spec FR-2 (3 cut-required metrics × 7 k = 21; map × 7 + plain map = 8; plain mrr = 1). Skipped in Story 1.2; activated in Story 1.4. | + +**Modified files** + +None (this is a pure additive story; the fixture and test live in new files only). + +**Key interfaces** + +```python +# backend/tests/unit/eval/fixtures/parity_qrels_run.py +from backend.app.eval.scoring import Qrels, Run + +qrels: Qrels # {query_id: {doc_id: int rating}} +run: Run # {query_id: {doc_id: float score}} +# Exactly 4 edge-case queries documented inline with comments naming each case. +``` + +```python +# backend/tests/unit/eval/test_scoring_parity.py +import pytest +import pytrec_eval +import ir_measures +from ir_measures import nDCG, AP, P, R, RR +from backend.app.eval.scoring import score +from backend.tests.unit.eval.fixtures.parity_qrels_run import qrels, run + +# 30-case parametrize covering: +# ("ndcg", k) for k in [1,3,5,10,20,50,100] # 7 +# ("precision", k) for k in [1,3,5,10,20,50,100] # 7 +# ("recall", k) for k in [1,3,5,10,20,50,100] # 7 +# ("map", k) for k in [1,3,5,10,20,50,100] # 7 +# ("map", None) # 1 +# ("mrr", None) # 1 +PARITY_CASES: list[tuple[str, int | None]] = [...] + +@pytest.mark.skip(reason="scoring.py not yet migrated — activate in Story 1.4") +@pytest.mark.parametrize("metric,k", PARITY_CASES) +def test_score_matches_pytrec_eval_within_1e_minus_6(metric: str, k: int | None) -> None: + """Compare score()'s aggregate to pytrec_eval direct, mean-across-queries.""" + ... + +@pytest.mark.skip(reason="scoring.py not yet migrated — activate in Story 1.4") +def test_per_query_shape_matches_pytrec_eval() -> None: + """Per-query shape parity per spec FR-3 / C2-F4.""" + ... +``` + +**Tasks** + +1. Verify `backend/tests/unit/eval/` is the correct test path: `ls backend/tests/unit/eval/test_scoring.py` exists → path verified. +2. Create the fixture file. Write the 8 queries with explicit comments calling out which is which edge case. Use realistic doc IDs (e.g., `"d{i}"` strings). Ratings stay in `{0, 1, 2, 3}` (graded). **Include BOTH symmetric empty-inner-dict cases** so the cycle-2 universe-filter tightening is verified on both sides (per plan cycle-3 C3-F1): + - `qrels["q_empty_qrels"] = {}` with a non-empty `run["q_empty_qrels"]` (e.g., 3 doc IDs). + - `run["q_empty_run"] = {}` with a non-empty `qrels["q_empty_run"]` (e.g., 2 rated doc IDs). + - Together, this raises the fixture's query count to ≥ 10 and ensures the parity test PINS whatever `pytrec_eval`'s legacy behavior is on both empty-inner cases. If `pytrec_eval` emits the qid in either case (e.g., as zero-valued metrics) the parity test will fail, and Story 1.3 must relax the filter to match. +3. Create the parity test file. Build the 30-case `PARITY_CASES` list as a module-level constant. Both test functions are decorated with `@pytest.mark.skip` referencing Story 1.4. +4. The actual assertion bodies CAN be sketched in this story (they'll be unskipped in Story 1.4), but they MUST follow the C2-F4 contract: + - For aggregate parity: call `score(qrels, run, {token})` (the function-under-test); separately compute the `pytrec_eval` value via `pytrec_eval.RelevanceEvaluator(qrels, {wire_set}).evaluate(run)` and take the same mean-across-queries the current `score()` performs at lines 187–192. Assert `abs(score_aggregate - pytrec_mean) < 1e-6`. + - For per-query shape AND per-query value parity (per spec FR-3 + cycle-1 F3): assert (a) the outer qid set is identical between `score(qrels, run, {token})["per_query"]` and the legacy `pytrec_eval` output (after wire→user-facing re-keying), (b) the inner metric-key set for each qid is identical, AND (c) every present `(qid, metric)` value matches `pytrec_eval`'s value to 1e-6 (`abs(a - b) < 1e-6`). Without (c), the shape test could pass while every value is wrong. Use the same fixture as the aggregate parity test. + - **DO NOT** call `ir_measures.calc_aggregate(...)` anywhere — that's the C2-F4 prohibition. +5. Run `pytest backend/tests/unit/eval/test_scoring_parity.py -v --collect-only` → confirms 31 collected items (30 parity + 1 shape test) all marked SKIPPED. + +**Definition of Done (DoD)** + +- New files exist at the listed paths and import cleanly under `pytest --collect-only`. +- Fixture exports `qrels` and `run` at module scope; pyright/mypy strict pass over the test file. +- 30 parametrized cases enumerable via `pytest --collect-only -q | grep test_score_matches_pytrec_eval_within_1e_minus_6` returns 30 lines. +- The 4 edge-case queries are documented inline (one comment per query naming the case it covers). +- All 31 tests SKIP — no execution failures. +- Existing tests still pass (`make test-unit` green). + +--- + +### Story 1.3 — Rewrite `scoring.py` with `ir_measures` + locked metric-object mapping + +**Outcome:** [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) imports `ir_measures` instead of `pytrec_eval`. `_translate_metric_name()` returns `ir_measures` metric objects per the locked FR-1 mapping table. `score()` computes the aggregate via per-query iteration + manual mean (NOT via `ir_measures.calc_aggregate()`). The existing `test_scoring.py` + `test_scoring_metric_tokens.py` + `test_qrels_loader.py` continue to pass without source edits. + +**New files** + +None. + +**Modified files** + +| File | Change | +|---|---| +| [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) | Replace `import pytrec_eval` (line 22) with `import ir_measures` plus explicit imports `from ir_measures import nDCG, AP, P, R, RR`. Rewrite `_translate_metric_name()` to return metric objects per the FR-1 mapping table (NOT wire strings). Rewrite `score()`'s body (lines 153–194) to: (1) translate each token via the new `_translate_metric_name`, (2) iterate via `ir_measures.iter_calc([obj_list], qrels, run)` yielding `Metric(query_id, measure, value)` tuples, (3) build `per_query` dict by mapping each tuple's `measure` back to its user-facing token, (4) compute aggregate as `sum(values) / len(values)` over the per-query dict — matching the current logic at lines 187–192 EXACTLY. Update module + function docstrings to name `ir_measures`. Update the source-of-truth line-citation that other docstrings reference (now pointing at the new internal logic). | + +**Key interfaces** + +The public API is FROZEN per spec FR-1. Only the INTERNALS change. + +```python +# backend/app/eval/scoring.py — PUBLIC API (unchanged byte-for-byte) +SUPPORTED_METRICS: frozenset[str] = frozenset({"ndcg", "map", "precision", "recall", "mrr"}) +SUPPORTED_K_VALUES: frozenset[int] = frozenset({1, 3, 5, 10, 20, 50, 100}) +Qrels = dict[str, dict[str, int]] +Run = dict[str, dict[str, float]] + +class ScoreResult(TypedDict): + aggregate: dict[str, float] + per_query: dict[str, dict[str, float]] + +def objective_metric_key(objective: dict[str, object]) -> str: ... +def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: ... + +# backend/app/eval/scoring.py — PRIVATE (signature changes return type) +def _translate_metric_name(user_facing: str) -> object: + """Return an ir_measures metric object per the FR-1 locked mapping. + + Mapping (locked by feature_spec.md FR-1): + ndcg@ → nDCG @ k + map → AP + map@ → AP @ k + precision@ → P @ k + recall@ → R @ k + mrr → RR + + Uncut ndcg/precision/recall still raise the existing "requires an @ cut" + ValueError. The function's other ValueError paths (unknown base, bad k, + k-not-in-allowlist, "metric does not accept an @ cut") are all preserved + with the same triggering inputs. + """ +``` + +**Tasks** + +1. Update the module docstring (lines 1–16). Replace "pytrec_eval scoring helper (infra_optuna_eval Story 1.2 / FR-3 + FR-5)" wording with the `ir_measures` equivalent; keep all the spec/FR references that remain accurate. +2. Replace `import pytrec_eval` (line 22) with `import ir_measures` + `from ir_measures import nDCG, AP, P, R, RR`. +3. Rewrite `_translate_metric_name()` (lines 51–103). The new body: + ```python + def _translate_metric_name(user_facing: str) -> object: + if user_facing == "mrr": + return RR + if user_facing == "map": + return AP + + if "@" not in user_facing: + raise ValueError( + f"metric {user_facing!r} requires an @ cut (allowed bases: " + f"{sorted(SUPPORTED_METRICS - _K_NEVER)})" + ) + + base, _, k_str = user_facing.partition("@") + if base not in SUPPORTED_METRICS: + raise ValueError(f"unknown metric base {base!r}; allowed: {sorted(SUPPORTED_METRICS)}") + if base in _K_NEVER: + raise ValueError(f"metric {base!r} does not accept an @ cut; use plain {base!r}") + try: + k = int(k_str) + except ValueError as exc: + raise ValueError(f"k value {k_str!r} in {user_facing!r} is not an integer") from exc + if k not in SUPPORTED_K_VALUES: + raise ValueError( + f"k={k} in {user_facing!r} is not in the allowlist {sorted(SUPPORTED_K_VALUES)}" + ) + + if base == "ndcg": + return nDCG @ k + if base == "map": + return AP @ k + if base == "precision": + return P @ k + if base == "recall": + return R @ k + raise ValueError(f"unexpected metric base {base!r}") # pragma: no cover + ``` + Every ValueError path is preserved character-for-character with the existing wording. Only the return values change. +4. Keep `objective_metric_key()` (lines 106–150) **untouched**. It returns user-facing token strings; this migration doesn't change that. +5. Rewrite `score()` (lines 153–194). The new body — **includes per-query universe filtering by default** so the per-query shape parity test (Story 1.4) doesn't depend on `ir_measures.iter_calc()` emitting exactly the pytrec_eval qid universe (per cycle-1 F2 + F3): + ```python + def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: + # Map user-facing → metric-object; remember the reverse for re-keying. + user_to_obj: dict[str, object] = {m: _translate_metric_name(m) for m in metrics} + obj_to_user: dict[object, str] = {obj: user for user, obj in user_to_obj.items()} + obj_list = list(user_to_obj.values()) + + # Per-query: iterate ir_measures' per-(qid, measure, value) tuples; re-key. + # FILTER to the pytrec_eval qid universe (per spec FR-3's historical contract): + # keep only qids that have at least one rated doc in qrels AND at least one + # scored entry in run — NOT just qid-key membership in both outer dicts. + # An empty inner dict (qrels[qid] == {} or run[qid] == {}) excludes the qid + # from pytrec_eval's evaluator output today; the filter preserves that + # exclusion. (Tightened per plan cycle-2 C2-F1.) + valid_qids: frozenset[str] = frozenset( + qid for qid in qrels.keys() & run.keys() + if qrels.get(qid) and run.get(qid) + ) + per_query: dict[str, dict[str, float]] = {} + for metric_tuple in ir_measures.iter_calc(obj_list, qrels, run): + if metric_tuple.query_id not in valid_qids: + continue + user_token = obj_to_user[metric_tuple.measure] + per_query.setdefault(metric_tuple.query_id, {})[user_token] = float(metric_tuple.value) + + # Aggregate: mean across queries, per user-facing metric — matches the + # original logic at scoring.py:187-192 (DO NOT call calc_aggregate). + aggregate: dict[str, float] = {} + if per_query: + for user in user_to_obj: + values = [q[user] for q in per_query.values() if user in q] + if values: + aggregate[user] = sum(values) / len(values) + + return {"aggregate": aggregate, "per_query": per_query} + ``` + **CRITICAL invariants** (both are enforced inline by the snippet above): + - The aggregate is computed over the per_query dict — NOT delegated to `ir_measures.calc_aggregate()` (C2-F4 contract). + - The per-query universe is filtered to `qrels.keys() & run.keys()` — preserving the pytrec_eval qid set on qrel-only / run-only / empty-overlap edge cases. This makes Story 1.4's per-query shape parity test a verification step rather than a fallback-fixup moment. +6. Update docstrings for `_translate_metric_name` and `score` to name `ir_measures` and the metric-object DSL. Reword any reference to "pytrec_eval wire names" to "ir_measures metric-object DSL". +7. Update the line-number-referencing module docstring at scoring.py:14-15 (currently says "per spec §FR-5"). The reference stays valid; just update the surrounding library name. +8. Run `make test-unit` — the existing `test_scoring.py`, `test_scoring_metric_tokens.py`, and `test_qrels_loader.py` must pass WITHOUT source edits. If any expected value drifts, the migration is invalid for that metric — STOP and resolve §19 Q4 (provider routing) before continuing. +9. Run `make typecheck` — `mypy --strict` over the new `scoring.py`. Pay attention to the `_translate_metric_name() -> object` return type; if mypy complains about losing precision, use a Union of the specific ir_measures metric-object types (subject to whether `ir_measures` exports those types as named classes). + +**Definition of Done (DoD)** + +- `import pytrec_eval` no longer appears in `scoring.py` (AC-1). +- `import ir_measures` appears exactly once. +- `_translate_metric_name()` returns metric objects per the locked mapping table; every ValueError path's wording is preserved. +- `score()` computes the aggregate via per-query iteration + manual mean — NOT `ir_measures.calc_aggregate()` (verified by `grep -n 'calc_aggregate' backend/app/eval/scoring.py` returning zero lines). +- `objective_metric_key()` is untouched (verified by `git diff backend/app/eval/scoring.py` not showing any change in lines 106–150). +- `make test-unit` passes — no existing eval test fails. +- `make typecheck` passes. +- Q4 first-touch resolution: if every existing `test_scoring.py` test still passes against the new backend, Q4 resolves to outcome (a) "default routing produces parity, no forcing needed". Record in the commit message. + +--- + +### Story 1.4 — Activate parity test + per-query shape parity + Q5 perf benchmark + +**Outcome:** The parity test's 30 cases are LIVE and passing. The per-query shape parity test asserts identical outer-qid sets and inner-metric-key sets between the new `score()` and the legacy `pytrec_eval`-direct output. The benchmark-perf delta is within ±10% (§19 Q5 resolved). + +**New files** + +None (the test file landed in Story 1.2). + +**Modified files** + +| File | Change | +|---|---| +| `backend/tests/unit/eval/test_scoring_parity.py` | Remove the `@pytest.mark.skip` decorators from both test functions. Fill in the assertion bodies per the Story 1.2 sketch (if not already filled). | + +**Tasks** + +1. Remove the `@pytest.mark.skip` decorator from `test_score_matches_pytrec_eval_within_1e_minus_6` AND `test_per_query_shape_matches_pytrec_eval`. +2. Run `pytest backend/tests/unit/eval/test_scoring_parity.py -v` — expect 31 passing (30 parametrized + 1 shape). +3. If any parity case fails: + - **DO NOT** weaken the tolerance or skip the failing case. + - Inspect the failing metric — what provider did `ir_measures` route it through? Use only documented APIs per spec §19 Q4 (no leading-underscore private names). + - Apply the bounded outcome from spec §19 Q4: (a) default routing OK → impossible since we got a failure; (b) documented provider-forcing API at the pinned version → use it; (c) bump/repin `ir_measures` → record the new version; (d) blocker → STOP and escalate to the user with the failure detail. +4. **Verify the per-query universe filter in Story 1.3 worked.** Story 1.3's `score()` already filters per-query results to `qrels.keys() & run.keys()` (the pytrec_eval-historical qid universe) — that's an unconditional invariant, not a contingency. If the per-query shape test fails despite that filter, something deeper is wrong: + - Possibility (a): The filter is correct in `score()` but pytrec_eval is emitting *different* qids than the filter would predict — investigate pytrec_eval's actual behavior on the fixture's edge-case queries; if pytrec_eval omits a qid that `qrels.keys() & run.keys()` includes, the filter set may need to be tighter (e.g., also require at least one relevant doc in the qrel set). + - Possibility (b): Per-(qid, metric) values disagree at the 1e-6 boundary — see step 3 (this is the Q4 provider-routing case). + - Possibility (c): A bug in the test's pytrec_eval baseline computation — review the test code. + - **Never** weaken the test to make it pass. If the filter needs refinement, tighten it in `scoring.py` and re-run the test. The persisted JSONB key set must continue to match what production currently emits. +5. Resolve §19 Q5 perf delta. Run `pytest backend/tests/benchmarks/test_scoring_perf.py -v` on the feature branch; compare against the same run on `main`. Acceptable: ±10% on the warm-call timing. Record both numbers in the commit message. + +**Definition of Done (DoD)** + +- `test_score_matches_pytrec_eval_within_1e_minus_6` passes for all 30 parametrized cases (AC-2). +- `test_per_query_shape_matches_pytrec_eval` passes — covers qid-set parity, inner-key-set parity, AND per-(qid, metric) value parity at 1e-6 (FR-3 + cycle-1 F3). +- `grep -n '@pytest.mark.skip\|pytest.skip' backend/tests/unit/eval/test_scoring_parity.py` returns ZERO matches (per cycle-1 F9 — guards against accidentally leaving a skipped parity gate). +- The benchmark perf delta is within ±10% of pre-migration baseline (AC-9 / Q5 resolution). +- Q4 resolution is recorded in the commit message: outcome (a)/(b)/(c) per spec §19 — with the cited verification output. +- No skips, no xfails, no weakened tolerances. + +--- + +### Story 1.5 — Extend "no wire-form leakage" assertions; add existing-row read regression + +**Outcome:** The contract + integration test assertions that forbid `pytrec_eval` wire-form prefixes are extended to ALSO forbid `ir_measures` PascalCase reprs. The new existing-row read regression test loads pre-migration JSONB shapes and exercises the consumers (`fetch_study_confidence` + trial-list endpoint + digest worker) without re-scoring. + +**New files** + +| File | Purpose | +|---|---| +| `backend/tests/integration/test_existing_row_read_compat.py` | Per AC-12 / GPT-5.5 cycle-1 F9. Inserts a synthetic `Trial` with pre-migration JSONB shape (`metrics = {"ndcg@10": ..., "map@10": ..., "map": ..., "mrr": ...}`; `per_query_metrics = {"q1": {"ndcg@10": ...}, ...}`); then calls `fetch_study_confidence`, asserts the trial-list endpoint serializes the JSONB through, and asserts the digest worker's top-trials section runs without raising. Load-bearing test for the "no migration / no backfill" claim. | + +**Modified files** + +| File | Change | +|---|---| +| [`backend/tests/contract/test_trial_row_shape.py`](../../../../backend/tests/contract/test_trial_row_shape.py) | Lines 6, 109, 113 — update docstrings to name `ir_measures`. **The leakage assertion at line 111 iterates over the module-level tuple `_PYTREC_EVAL_WIRE_PREFIXES` defined at lines 51–57** (verified 2026-05-22 — contains `("ndcg_cut_", "P_", "recall_", "recip_rank", "map_cut_")`). Action: (a) rename the tuple to `_FORBIDDEN_WIRE_PREFIXES` and add the `ir_measures` PascalCase entries (`nDCG@`, `AP@`, `P@`, `R@`, `RR`); (b) tighten the loop to handle `RR` correctly (it's a whole-token match, not a prefix — use `key == "RR"` OR `key.startswith("RR@")` as a separate check, OR adopt the AC-3 strict regex approach below). Recommended: switch the entire assertion to the strict regex per AC-3:

```python
_STRICT_USER_FACING_KEY = re.compile(
r"^(?:mrr\|map\|(?:ndcg\|precision\|recall\|map)@(?:1\|3\|5\|10\|20\|50\|100))$"
)
for key in t.metrics:
assert _STRICT_USER_FACING_KEY.match(key), (
f"metrics key {key!r} is not in the user-facing token allowlist — "
f"library wire forms must never leak past scoring.score()"
)
```

Then ADD the negative-case + positive-case parametrized helper tests per AC-3 (13 negative cases + ~10 positive cases). | +| [`backend/tests/integration/test_run_trial_per_query_persistence.py`](../../../../backend/tests/integration/test_run_trial_per_query_persistence.py) | Lines 53, 111, 119 — docstring rewording. **The existing assertion already rejects `ir_measures` PascalCase reprs** (verified 2026-05-22): the check `base = metric_key.partition("@")[0]; assert base in expected_metric_bases` where `expected_metric_bases = {"ndcg", "map", "precision", "recall", "mrr"}` rejects PascalCase because `"nDCG"` is case-sensitively NOT in the lowercase set. Action: (a) tighten the assertion to use the same strict regex from `test_trial_row_shape.py` (extract the regex to a shared `backend/tests/_eval_helpers.py` module to avoid duplication), (b) reword the docstring on line 119 to name `ir_measures` (replacing "score() should remap pytrec_eval wire names to user-facing tokens"). | +| [`backend/app/services/test_seeding.py`](../../../../backend/app/services/test_seeding.py) | Lines 127 + 142: change both `"p@10"` literals → `"precision@10"`. **Moved from Story 1.6 per cycle-1 F7** so the branch stays green when this story's AC-3 strict regex activates. 2-character fix bundled per spec §2 C2-F5 + §15 inline-fix entry. | + +**Key interfaces** + +```python +# backend/tests/integration/test_existing_row_read_compat.py +import pytest +from backend.app.services.study_confidence import fetch_study_confidence + +@pytest.mark.integration +async def test_pre_migration_jsonb_shape_hydrates_confidence(db_session, ...): + """Per AC-12 — pre-migration JSONB key shape continues to work post-migration. + + Insert a fixture trial with metrics/per_query_metrics keyed by the user-facing + tokens already in production (ndcg@10, map@10, map, mrr). Confidence orchestrator + must hydrate the shape; trial-list endpoint must serialize through; digest worker + must include the row in its top-trials section. + + The load-bearing test for FR-1c / FR-3's "no-migration / no-backfill" invariant. + """ + ... +``` + +**Tasks** + +0. **FIRST:** Apply the `test_seeding.py` `p@10` → `precision@10` fix at lines 127 + 142 (moved here from Story 1.6 per cycle-1 F7; ordered first per cycle-2 C2-F2 so it precedes the strict-regex activation in tasks 1–6 below). Two literal substitutions; verify by `grep -n '"p@10"' backend/app/services/test_seeding.py` returning zero matches. +1. Read [`backend/tests/contract/test_trial_row_shape.py`](../../../../backend/tests/contract/test_trial_row_shape.py) to confirm line numbers (6 / 109 / 113) for the docstring + assertion. +2. Update docstring at line 6 to name `ir_measures` (the contract description: keys are user-facing names, NOT library wire forms). +3. Update docstring at line 109 to name `ir_measures`. +4. Extend the assertion logic. The existing check is something like: + ```python + assert not any(key.startswith(prefix) for prefix in ("ndcg_cut_", "P_", "recip_rank", "map_cut_", "recall_")) + ``` + Extend to include `ir_measures` PascalCase reprs: + ```python + PYTREC_WIRE_PREFIXES = ("ndcg_cut_", "P_", "recip_rank", "map_cut_", "recall_") + IR_MEASURES_REPRS = ("nDCG@", "P@", "RR", "AP@", "R@") + forbidden = PYTREC_WIRE_PREFIXES + IR_MEASURES_REPRS + + for key in row_keys: + assert not any(key.startswith(p) for p in forbidden), ( + f"metrics key {key!r} starts with a library wire-form prefix; " + f"expected user-facing tokens only" + ) + ``` + Note: `RR` is a 2-character whole token; `key.startswith("RR")` would match `RR_anything` but the test should reject the EXACT `RR` value. Use a stricter regex check matching the AC-3 strict regex: + ```python + import re + _STRICT_KEY = re.compile(r"^(?:mrr|map|(?:ndcg|precision|recall|map)@(?:1|3|5|10|20|50|100))$") + for key in row_keys: + assert _STRICT_KEY.match(key), f"metrics key {key!r} not in user-facing token allowlist" + ``` +5. Add explicit negative cases per AC-3. Write a helper test that demonstrates the regex REJECTS each forbidden value: + ```python + @pytest.mark.parametrize("forbidden_key", [ + "ndcg", # uncut — forbidden by objective_metric_key + "precision", # same + "recall", # same + "nDCG@10", # ir_measures repr + "P@10", # ir_measures repr + "RR", # ir_measures repr + "AP@5", # ir_measures repr + "R@10", # ir_measures repr + "ndcg_cut_10", # pytrec_eval wire + "recip_rank", # pytrec_eval wire + "map_cut_10", # pytrec_eval wire + "P_10", # pytrec_eval wire + "recall_10", # pytrec_eval wire + ]) + def test_strict_key_regex_rejects_forbidden(forbidden_key: str) -> None: + assert _STRICT_KEY.match(forbidden_key) is None, ( + f"strict key regex should REJECT {forbidden_key!r} but didn't" + ) + + @pytest.mark.parametrize("allowed_key", [ + "ndcg@10", "ndcg@5", "ndcg@1", + "map@10", "map", + "mrr", + "precision@10", "precision@50", + "recall@10", "recall@1", + ]) + def test_strict_key_regex_accepts_allowed(allowed_key: str) -> None: + assert _STRICT_KEY.match(allowed_key) is not None, ( + f"strict key regex should ACCEPT {allowed_key!r} but didn't" + ) + ``` +6. Repeat the steps for [`backend/tests/integration/test_run_trial_per_query_persistence.py`](../../../../backend/tests/integration/test_run_trial_per_query_persistence.py) lines 53 / 111 / 119. Extract the strict regex into a shared module — perhaps `backend/tests/_eval_helpers.py` — if the same regex is used in two test files. +7. Create the new existing-row read regression at `backend/tests/integration/test_existing_row_read_compat.py`. The test: + - Sets up a study + judgment list + query set via the standard integration fixtures. + - Inserts a hand-crafted `Trial` row with realistic pre-migration JSONB: + ```python + trial = await repo.create_trial( + db, + id=str(uuid7()), + study_id=study.id, + optuna_trial_number=5, + params={"boost": 1.5}, + primary_metric=0.82, + metrics={"ndcg@10": 0.82, "map@10": 0.71, "map": 0.65, "mrr": 0.91}, + per_query_metrics={ + "q1": {"ndcg@10": 0.83, "map@10": 0.7, "mrr": 1.0}, + "q2": {"ndcg@10": 0.81, "map@10": 0.72, "mrr": 0.83}, + # ...minimum 5 queries to satisfy bootstrap_ci_95's BOOTSTRAP_MIN_N_QUERIES + }, + duration_ms=120, + status="complete", + error=None, + started_at=..., ended_at=..., + ) + await db.commit() + ``` + - Stamps the study with `best_trial_id = trial.id` + `best_metric = trial.primary_metric`. + - Calls `fetch_study_confidence(db, study.id)` and asserts the returned `ConfidenceShape` has `headline.value == 0.82`, `headline.n_queries > 0`, and `ci_95 is not None`. + - Calls `GET /api/v1/studies/{study.id}` (via the integration test client) and asserts the response includes the `confidence` block with the same values. + - **Calls the digest worker's top-trials selection logic** in isolation against this trial (REQUIRED per AC-12 — cycle-1 F4). The digest's top-trials renderer reads `Trial.primary_metric` (scalar) and `Trial.metrics` (JSONB); the test asserts the digest worker can include this row without raising. If the digest worker's relevant entrypoint isn't trivially callable in isolation, simulate the same read pattern: `select(Trial).where(Trial.study_id == study.id).order_by(Trial.primary_metric.desc())` followed by accessing the trial's JSONB fields. + +**Definition of Done (DoD)** + +- `backend/tests/contract/test_trial_row_shape.py` strict regex check passes for the migrated `score()` output. +- The new negative-case parametrize test passes (rejects each forbidden key explicitly). +- The new positive-case parametrize test passes (accepts each allowed key). +- `backend/tests/integration/test_run_trial_per_query_persistence.py` extended assertion passes. +- `backend/tests/integration/test_existing_row_read_compat.py` passes: pre-migration JSONB row hydrates `ConfidenceShape`, the trial-list endpoint serializes through, AND the digest worker's top-trials selection includes the row without raising (AC-12; all three consumers exercised — cycle-1 F4). +- The `test_seeding.py` `p@10` → `precision@10` inline fix is bundled in this story (moved from Story 1.6 per cycle-1 F7 to keep the branch green when AC-3's strict regex activates). +- `make test-contract && make test-integration` both green. + +--- + +### Story 1.6 — Operator-visible error message at `studies.py:313` + docstring rewording + +**Outcome:** The `INSUFFICIENT_JUDGMENT_OVERLAP` error envelope's `message` field no longer names `pytrec_eval`. The neighboring inline comment at `studies.py:270` is updated in lock-step. (Per cycle-1 F7: the `test_seeding.py` `p@10` → `precision@10` inline fix moved to Story 1.5 so the branch stays green when AC-3's strict regex activates.) + +**New files** + +None. + +**Modified files** + +| File | Change | +|---|---| +| [`backend/app/api/v1/studies.py`](../../../../backend/app/api/v1/studies.py) | Line 270 inline comment: replace `pytrec_eval scores 0 on every trial by construction` → `ir_measures scores 0 on every trial by construction`. Line 313 error-message string: replace `pytrec_eval will likely score 0 on every trial` → `ir_measures will likely score 0 on every trial`. (Or, equivalently, name no library at all per spec FR-5 wording flexibility — `every trial will score 0 on every metric` is acceptable; the impl-plan author picks one and is consistent.) | +| [`backend/tests/contract/test_studies_api_contract.py`](../../../../backend/tests/contract/test_studies_api_contract.py) | Line 156 docstring rewording per spec §2 sweep. **No message-substring assertion update needed** — the existing `INSUFFICIENT_JUDGMENT_OVERLAP` contract tests at lines ~218–240 (verified 2026-05-22) are STRUCTURAL ONLY: they assert the error-code literal `"INSUFFICIENT_JUDGMENT_OVERLAP"` appears in the studies.py source AND that the source-presence ordering of error codes is preserved (`target_pos < probe_pos < overlap_pos < config_pos`). No substring of the human-readable `message` field is asserted. Reword the line-156 docstring's "pytrec_eval semantics" mention; no assertion changes needed. The atomic-update requirement from spec FR-5 reduces in scope: this story only updates the source string in `studies.py` (no test contract to bring along). | + +**Tasks** + +1. Read `backend/app/api/v1/studies.py` lines 268–320 to confirm the exact comment + error-message string and surrounding context. +2. Apply the rewording at line 270 (inline comment) and line 313 (error-message string). Use one consistent choice — recommendation: `ir_measures` (operator engineers reading the error need a library name to grep for if they need to inspect). The Spec §11 update for Story 1.6 also commits to one variant; pick the same wording here. +3. Read `backend/tests/contract/test_studies_api_contract.py` line 156 — reword the docstring's `pytrec_eval semantics` to name `ir_measures` (or "standard IR-evaluation conventions" — see spec §15 schemas.py guidance). **No assertion change needed** — the existing tests are structural-only (assert literal `"INSUFFICIENT_JUDGMENT_OVERLAP"` is in `studies.py` source and that error codes appear in a fixed order). The message substring is NOT pinned today; the spec's "atomic update" requirement is satisfied by the source-only change. +4. Run `pytest backend/tests/contract/test_studies_api_contract.py -v` — should pass without assertion changes (the docstring rewording is comment-only). + +(Note: the `test_seeding.py` `p@10` → `precision@10` fix is bundled into Story 1.5 per cycle-1 F7 to keep the branch green when Story 1.5's AC-3 strict regex activates. It is NOT in this story.) + +**Definition of Done (DoD)** + +- `grep -n 'pytrec_eval' backend/app/api/v1/studies.py` returns zero matches (AC-7). +- `grep -n 'pytrec_eval' backend/tests/contract/test_studies_api_contract.py` returns zero matches (line 156 docstring reworded). +- `make test-contract` green. + +--- + +### Story 1.7 — Dockerfile conditional update (gcc/g++/python3-dev install) + +**Outcome:** The Dockerfile reflects the empirical transitive-dependency reality per §19 Q3. Either the gcc/g++/python3-dev install stays (with a reworded comment crediting `ir_measures`' transitive backend) or it's dropped (because `ir_measures` resolves to pure-Python providers only for all `SUPPORTED_METRICS`). `docker build .` succeeds either way. + +**New files** + +None. + +**Modified files** + +| File | Change | +|---|---| +| [`Dockerfile`](../../../../Dockerfile) | Lines 44–54: conditional change per Q3 resolution. Option (a) — `pytrec_eval` transitively present: KEEP the gcc/g++/python3-dev install; REWORD the comment at lines 44–48 to credit `ir_measures` (or the actual transitive C-extension dep). Option (b) — no transitive C extension: DROP lines 44–54 entirely (the `RUN apt-get update ...` block); `docker build .` succeeds with `python:3.13-slim` headers only. | + +**Tasks** + +1. Resolve §19 Q3 empirically. Recipe in a clean temp dir: + ```bash + cd /tmp && python -m venv .venv && source .venv/bin/activate + pip install ir-measures + pip show pytrec_eval + # exit code 0 + non-empty output → transitive backend present + # exit code 1 → no transitive backend + ``` + Also confirm against `uv` resolution: + ```bash + cd /Users/ericstarr/relyloop && uv tree | grep pytrec_eval + # output = pytrec_eval is a resolved dep (direct or transitive) + # empty = not resolved + ``` +2. Apply the conditional Dockerfile change: + - **Outcome (a) — TRANSITIVE backend present:** Keep the `RUN apt-get update ... gcc g++ python3-dev` block at lines 49–54. Reword the comment block at lines 44–48: + ```dockerfile + # ir_measures (added by infra_ir_measures_migration, replacing the abandoned + # pytrec_eval) resolves a C-extension backend transitively (verified at impl-plan + # time per feature_spec.md §19 Q3). The backend's sdist has no prebuilt wheels for + # every Python version we target, so every install compiles its C extension on the + # fly. We install gcc + python-dev headers here, then this whole stage is discarded + # (the runtime stage copies only /app/.venv, not the build toolchain), so the final + # image stays slim. + ``` + - **Outcome (b) — NO transitive C extension:** Remove lines 39–54 entirely (the entire `RUN apt-get update ... && rm -rf /var/lib/apt/lists/*` block plus its comment header — the next layer `COPY pyproject.toml uv.lock README.md ./` becomes adjacent to the `FROM base AS deps` line). The `deps` stage compresses by ~40 LOC. Verify with `docker build .` succeeds. +3. Run `docker build . --target deps` to verify the deps stage builds cleanly. If outcome (a), confirm gcc/g++/python3-dev are still installed. +4. Run `docker build .` (full multi-stage) — must succeed. Verify the runtime image contains `ir_measures` via `docker run python -c "import ir_measures; print(ir_measures.__file__)"`. + +**Definition of Done (DoD)** + +- `docker build .` succeeds (AC-10). +- The Dockerfile state matches the empirical Q3 verification — comment OR removal explicitly cites the verification output in the commit message. +- The runtime image can `import ir_measures` without error. +- No regression to the existing healthcheck or container size. + +--- + +### Story 1.8 — Full doc-rewrite sweep + dashboard regen + broader wire-form grep gate + +**Outcome:** Every current-state doc and comment that named `pytrec_eval` now names `ir_measures` (or names no library if appropriate). The MVP1_DASHBOARD.md is regenerated. Both grep gates (the basic `pytrec_eval|pytrec-eval` sweep AND the broader wire-form sweep for `RelevanceEvaluator|ndcg_cut_|map_cut_|recip_rank|recall_[0-9]|\bP_[0-9]`) return only allowlisted matches. + +**New files** + +None. + +**Modified files** + +| File | Change | +|---|---| +| [`README.md`](../../../../README.md) | Line 9 — `pytrec_eval` → `ir_measures`. | +| [`CLAUDE.md`](../../../../CLAUDE.md) | Lines 15 + 29 — both `pytrec_eval` mentions → `ir_measures`. | +| [`architecture.md`](../../../../architecture.md) | Line 131 — `eval/ pytrec_eval scoring` → `eval/ ir_measures scoring`. | +| [`release-notes-v0.1.0-draft.md`](../../../../release-notes-v0.1.0-draft.md) | Line 12 — stack table entry. | +| [`docs/00_overview/product/relevance-copilot-spec.md`](../../../00_overview/product/relevance-copilot-spec.md) | All 11 mentions (lines 12, 155, 688, 690, 692–693, 711, 2192, 2302, 2513, 2658, 2722). The "Engine: pytrec_eval everywhere" subsection (lines 688–693) is reframed as "Engine: provider-abstracted via `ir_measures`" with the reasons restated as: standard IR metric semantics across engines, per-query inspectability, cross-engine comparability (the old "de facto standard wrapper for trec_eval" framing becomes "provider abstraction means swapping backends is config, not rewrite"). | +| [`docs/01_architecture/optimization.md`](../../../01_architecture/optimization.md) | All 10 mentions. Title `# Optimization (Optuna + pytrec_eval)` → `# Optimization (Optuna + ir_measures)`. Code-example block at lines 87–90 (`pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut_10", "map", "P_10"}).evaluate(run)`) rewritten to: `import ir_measures` + `metrics = list(ir_measures.iter_calc([nDCG@10, AP, P@10], qrels, run))` plus a note that RelyLoop's `score()` re-keys back to user-facing tokens (`ndcg@10`, `map`, `precision@10`). | +| [`docs/01_architecture/tech-stack.md`](../../../01_architecture/tech-stack.md) | Line 41 IR-evaluation row updated. | +| [`docs/01_architecture/system-overview.md`](../../../01_architecture/system-overview.md) | Line 76 component table row updated. | +| [`docs/01_architecture/README.md`](../../../01_architecture/README.md) | Line 21 cross-reference updated. | +| [`docs/01_architecture/data-model.md`](../../../01_architecture/data-model.md) | Lines 52 + 231 reworded. | +| [`docs/01_architecture/cluster-lifecycle.md`](../../../01_architecture/cluster-lifecycle.md) | Line 159 reworded. | +| [`docs/02_product/mvp1-user-stories.md`](../../../02_product/mvp1-user-stories.md) | Line 40 — US-7 narrative. | +| [`docs/02_product/planned_features/feat_study_baseline_trial/idea.md`](../feat_study_baseline_trial/idea.md) | Line 56 — sibling planned-feature coordination per spec §15. `scores via pytrec_eval` → `scores via ir_measures`. | +| [`docs/02_product/planned_features/feat_auto_followup_studies/idea.md`](../feat_auto_followup_studies/idea.md) | Line 47 — `Optuna + pytrec_eval are deterministic` → `Optuna + ir_measures are deterministic`. | +| [`docs/08_guides/workflows-overview.md`](../../../08_guides/workflows-overview.md) | Lines 123 + 277 reworded. | +| [`ui/public/docs/workflows-overview.md`](../../../../ui/public/docs/workflows-overview.md) | Lines 123 + 277 — runtime-served mirror; lock-step with `docs/08_guides/workflows-overview.md`. | +| [`ui/public/guides/05_import_judgments_and_calibrate/script.md`](../../../../ui/public/guides/05_import_judgments_and_calibrate/script.md) | Line 6 reworded. | +| [`ui/public/guides/06_create_and_monitor_study/script.md`](../../../../ui/public/guides/06_create_and_monitor_study/script.md) | Line 8 reworded. | +| [`ui/public/guides/06_create_and_monitor_study/metadata.json`](../../../../ui/public/guides/06_create_and_monitor_study/metadata.json) | Line 26 `caption` field reworded — same content shape as `script.md`. | +| [`ui/src/components/studies/create-study-modal.tsx`](../../../../ui/src/components/studies/create-study-modal.tsx) | Line 60 source-of-truth comment: `// Source-of-truth: backend/app/eval/scoring.py:32 (metric → pytrec_eval token …)` → `// Source-of-truth: backend/app/eval/scoring.py (metric → ir_measures metric-object DSL …)`. The line-number citation is dropped because Story 1.3 rewrites that section; the symbol citation is enough. | +| [`ui/src/__tests__/components/studies/k-ignored.test.ts`](../../../../ui/src/__tests__/components/studies/k-ignored.test.ts) | Line 4 same source-of-truth comment update. | +| [`ui/src/lib/types.ts`](../../../../ui/src/lib/types.ts) | Line 1889 — `pytrec_eval semantics` comment reworded. | +| Test files: `backend/tests/unit/eval/test_scoring.py`, `test_scoring_metric_tokens.py`, `test_qrels_loader.py`, `backend/tests/integration/fixtures/handbuilt_qrels.py:75`, `backend/tests/benchmarks/test_scoring_perf.py:56` | Docstring + comment rewordings only. No assertion changes. | +| [`backend/app/eval/qrels_loader.py`](../../../../backend/app/eval/qrels_loader.py) | Line 45 docstring — `pytrec_eval treats as a no-op` → `ir_measures treats as a no-op` (the no-op-on-empty-input behavior is preserved across both libraries). | +| [`backend/app/db/models/trial.py`](../../../../backend/app/db/models/trial.py) | Lines 19 + 83 docstrings updated. | +| [`backend/app/api/v1/schemas.py`](../../../../backend/app/api/v1/schemas.py) | Line 534 — `ObjectiveSpec` docstring's "per pytrec_eval semantics" reframed: the cutoff requirement is an IR convention, not a `pytrec_eval` invention. Replace with: `per standard IR-evaluation conventions` (no library name needed — the convention predates both libraries). | +| [`migrations/versions/0015_trials_per_query_metrics.py`](../../../../migrations/versions/0015_trials_per_query_metrics.py) | Line 17 docstring per §19 Q1 recommendation: reword `NOT the pytrec_eval wire forms` → `NOT the library wire forms (per backend/app/eval/scoring.py)`. The migration file is part of the active source tree; future engineers read its docstring; rewording is forward-looking. | +| [`docs/00_overview/MVP1_DASHBOARD.md`](../../../00_overview/MVP1_DASHBOARD.md) | Regenerated via `scripts/build_mvp1_dashboard.py`. Picks up: (a) the `infra_optuna_eval` row's `pytrec_eval` mention at line 64 (this row stays in implemented_features but the dashboard regen reads the source feature_spec.md and propagates whatever description text is there — verify behavior), (b) the `infra_ir_measures_migration` row at line 134 (the planned-features entry text from THIS spec's first 200 chars). | + +**Tasks** + +1. **Doc rewordings (bulk).** Open each file in the modified files table; apply the rewording. Use `sed -i` where the replacement is a literal token swap (`pytrec_eval` → `ir_measures`) but verify each file before bulk substitution — the umbrella spec and optimization.md have prose context that may need a sentence-level rewrite, not a token swap. + +2. **Umbrella spec rewrite (subsection).** The "Engine: pytrec_eval everywhere" subsection at `docs/00_overview/product/relevance-copilot-spec.md:688–693` needs more than a token swap. Rewrite to **never name `pytrec_eval` in the live umbrella spec** (per cycle-1 F5 — FR-7's allowlist does not include the umbrella spec, and the provider-abstraction framing doesn't require naming the underlying backend): + ```markdown + ### Engine: provider-abstracted IR evaluation via `ir_measures` + + Workers always evaluate via `ir_measures`, never engine-native `_rank_eval`. Reasons: + + - `ir_measures` (from the PyTerrier team) wraps multiple IR-evaluation backends behind a typed metric-object DSL (`nDCG@10`, `AP@5`, `RR`, …). The provider abstraction means swapping the underlying backend is a config change rather than a rewrite — protecting against future single-maintainer abandonment risk. + - ES `_rank_eval` and `ir_measures` don't always agree to many decimal places (different normalization conventions across engines). + - Per-query scores are inspectable, enabling deep debugging. + - Cross-engine comparability: the same metric semantics apply whether the underlying engine is ES, OpenSearch, Fusion, or Solr. + ``` + Note: the live umbrella spec MUST NOT name `pytrec_eval` after this rewrite. The "wraps multiple IR-evaluation backends" framing is sufficient. Historical context (what `pytrec_eval` was, why we migrated away) lives in the dated `state.md` entry and the frozen historical `feature_spec.md` under `implemented_features/` — NOT in the durable umbrella spec. + +3. **Code-example rewrite in optimization.md.** Lines 87–90: + ```python + # OLD: + metrics = pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut_10", "map", "P_10"}).evaluate(run) + ``` + → + ```python + # NEW: + import ir_measures + from ir_measures import nDCG, AP, P + metrics_per_query = list(ir_measures.iter_calc([nDCG@10, AP, P@10], qrels, run)) + # RelyLoop's score() re-keys to user-facing tokens — see backend/app/eval/scoring.py. + ``` + +4. **Migration docstring (Q1 resolution).** Apply the rewording to `migrations/versions/0015_trials_per_query_metrics.py:17` per the §19 Q1 recommendation. + +5. **Regenerate MVP1_DASHBOARD.md and produce a CLEAN current-state dashboard** (per cycle-1 F6 — the implementation plan cannot amend AC-6; the regenerated dashboard MUST satisfy the FR-7 grep gate without an invented allowlist exception): + ```bash + python scripts/build_mvp1_dashboard.py + ``` + Then inspect the regen output: + - **`infra_ir_measures_migration` row at line ~134** — should regenerate cleanly from THIS feature's `feature_spec.md` (which is reworded by this very PR; the new dashboard row will name `ir_measures`). + - **`infra_optuna_eval` row at line ~64** — the dashboard generator reads the implemented feature's `feature_spec.md` first-N-chars; that file is frozen historical and still names `pytrec_eval`. If the regen output still names `pytrec_eval` in the current-state dashboard, fix the generator (don't extend the AC-6 allowlist): + - **Option (a) — preferred:** Audit `scripts/build_mvp1_dashboard.py` for how it builds the "description" cell for implemented features. If it pulls text verbatim from frozen historical specs, change it to pull from a current-state override (e.g., a `description_override` field in a sibling `pipeline_status.md` or `dashboard_summary.md` file) or hardcode a current-state one-liner per feature. + - **Option (b) — fallback:** Edit the `infra_optuna_eval` row's description in the generator's input directly. The implemented feature's `feature_spec.md` itself stays frozen; only the dashboard's summary cell for that row is updated to current-state language (e.g., "Optuna RDB storage; IR-evaluation runs via `ir_measures`"). + - After applying option (a) or (b), the regenerated dashboard MUST satisfy the FR-7 grep gate without any exception for dashboard rows. If both options are impractical, STOP and escalate to the user — do not weaken AC-6 unilaterally. + +6. **Run the merge-time grep gates** (per FR-7): + ```bash + # Basic gate + grep -rn 'pytrec_eval\|pytrec-eval' . --exclude-dir=node_modules --exclude-dir=.venv --exclude-dir=.git + + # Broader wire-form gate + grep -rEn '(RelevanceEvaluator|ndcg_cut_|map_cut_|recip_rank|recall_[0-9]|\bP_[0-9])' . --exclude-dir=node_modules --exclude-dir=.venv --exclude-dir=.git + ``` + Both must match ONLY (matches the spec's FR-7 allowlist verbatim — no plan-level additions per cycle-1 F5 + F6): + - `docs/00_overview/implemented_features/` (historical) + - `docs/blog/` (dated historical) + - `state.md` (historical entries; the new dated entry may name `pytrec_eval` once to reference "the library being replaced") + - `backend/tests/unit/eval/test_scoring_parity.py` (parity test imports `pytrec_eval` by design) + - `pyproject.toml` `[dependency-groups.dev]` line (per FR-4 — REQUIRED, not optional) + - `pyproject.toml` `[[tool.mypy.overrides]]` block for `pytrec_eval` if AC-4c kept it + - The Dockerfile comment if Story 1.7 outcome (a) reworded it to credit `ir_measures`' transitive backend + - This `feature_spec.md` and `implementation_plan.md` (the spec/plan for this very feature — naturally names `pytrec_eval` as "the library being replaced") + - **NOT** the MVP1_DASHBOARD.md (per cycle-1 F6 — the dashboard must regen cleanly; if the generator emits stale text, fix the generator per task 5 options (a)/(b)) + +7. **Update `state.md`** with a NEW dated entry describing the migration. Do NOT back-edit any existing entry. The new entry references the PR number, summarizes scope, and notes Alembic head unchanged. + +**Definition of Done (DoD)** + +- All modified files reworded (verified by spot-check of each). +- `python scripts/build_mvp1_dashboard.py` runs cleanly; the regenerated MVP1_DASHBOARD.md is committed. +- Basic `pytrec_eval|pytrec-eval` grep gate returns only allowlist matches (AC-6). +- Broader wire-form grep gate (`RelevanceEvaluator|ndcg_cut_|map_cut_|recip_rank|recall_[0-9]|\bP_[0-9]`) returns only allowlist matches (AC-6 extended). +- `state.md` has a new dated entry (AC-11). +- `make lint` passes (any new doc + comment changes pass ruff format). +- Final full-suite run: `make test-unit && make test-integration && make test-contract && make typecheck && make lint` — all green. + +--- + +## UI Guidance + +**No UI changes in this migration.** Two UI source-of-truth COMMENTS are reworded (`ui/src/components/studies/create-study-modal.tsx:60`, `ui/src/__tests__/components/studies/k-ignored.test.ts:4`) but these are comment-only changes — no element inventory, no markup change, no behavior change. No new tooltips. No new glossary keys. No new dropdowns or filters. + +**No legacy behavior parity table** — no user-facing component >100 LOC is being deleted or migrated in this plan. + +**Enumerated value contracts (§7.4 of spec):** unchanged. The `objective.metric` and `objective.k` allowlists (`SUPPORTED_METRICS`, `SUPPORTED_K_VALUES`) are byte-identically preserved by FR-1. The frontend's hardcoded option arrays at `create-study-modal.tsx` continue to match the backend source. No grep audit required because no allowlist values are added, removed, or changed. + +--- + +## 3) Testing workstream + +The migration adds 2 new test files and extends 3 existing assertions; existing tests must pass without source edits. + +### 3.1 Unit tests +- Location: `backend/tests/unit/` +- Scope: scoring layer (pure functions), no DB +- Tasks: + - [ ] **NEW** `backend/tests/unit/eval/fixtures/parity_qrels_run.py` — fixed (qrels, run) fixture per FR-2 with 4 edge cases (Story 1.2). + - [ ] **NEW** `backend/tests/unit/eval/test_scoring_parity.py` — 30 parametrized parity cases + per-query shape parity (Stories 1.2 + 1.4). + - [ ] **EXISTING** `backend/tests/unit/eval/test_scoring.py` — passes unchanged (Story 1.3 verification). + - [ ] **EXISTING** `backend/tests/unit/eval/test_scoring_metric_tokens.py` — passes unchanged (Story 1.3). + - [ ] **EXISTING** `backend/tests/unit/eval/test_qrels_loader.py` — passes unchanged (Story 1.3). +- DoD: + - [ ] All 30 parametrized parity cases pass with `abs(a - b) < 1e-6` (AC-2). + - [ ] Per-query shape parity passes (FR-3 / C2-F4 contract). + - [ ] Negative-case + positive-case strict-regex tests pass (AC-3). + +### 3.2 Integration tests +- Location: `backend/tests/integration/` +- Scope: DB-backed read regression + extended leakage assertion +- Tasks: + - [ ] **EXTEND** `backend/tests/integration/test_run_trial_per_query_persistence.py:111` — leakage assertion extended to forbid `ir_measures` PascalCase reprs (Story 1.5). + - [ ] **NEW** `backend/tests/integration/test_existing_row_read_compat.py` — pre-migration JSONB shape regression (AC-12 / Story 1.5). + - [ ] **EXISTING** `backend/tests/integration/fixtures/handbuilt_qrels.py:75` — docstring rewording only (Story 1.8). +- DoD: + - [ ] Existing-row read regression passes — `fetch_study_confidence` hydrates the pre-migration JSONB shape (AC-12). + - [ ] No leakage assertion regressed. + +### 3.3 Contract tests +- Location: `backend/tests/contract/` +- Scope: response-shape envelope assertions + leakage assertion +- Tasks: + - [ ] **EXTEND** `backend/tests/contract/test_trial_row_shape.py:113` — strict regex check + negative/positive parametrized cases per AC-3 (Story 1.5). + - [ ] **REWORD** `backend/tests/contract/test_studies_api_contract.py` line 156 docstring (Story 1.6) — `pytrec_eval semantics` → `ir_measures`-equivalent or library-neutral. **No assertion change** — the existing INSUFFICIENT_JUDGMENT_OVERLAP tests at lines ~218–240 are structural-only (assert the error-code literal appears in `studies.py` source + assert source-presence ordering of error codes). No message substring is pinned today (verified 2026-05-22). +- DoD: + - [ ] The strict-regex assertion is substantive (negative cases fail, positive cases pass — AC-3 enumeration). + - [ ] `studies.py:313` operator-visible error message renamed and the `test_studies_api_contract.py:156` docstring reworded; existing structural assertions (`INSUFFICIENT_JUDGMENT_OVERLAP` literal + ordering) still pass without modification (AC-7). + +### 3.4 E2E tests +- **N/A.** No UI flow changes. The error-toast text change in the create-study modal is a string update that's verified at the contract layer (the modal displays whatever the API returns; the API string change is contract-tested). + +### 3.5 Existing test impact audit + +| Test file | Pattern | Count | Action | +|---|---|---|---| +| `backend/tests/unit/eval/test_scoring.py` | Pinned metric values (hand-computed) | many | No change needed — Story 1.3 must keep values identical via parity. If any value drifts, that's a Q4 resolution moment (Story 1.4 task 3). | +| `backend/tests/unit/eval/test_scoring_metric_tokens.py` | `_translate_metric_name` ValueError paths | several | No change needed — Story 1.3 preserves all ValueError paths. | +| `backend/tests/unit/eval/test_qrels_loader.py` | Empty-dict on unknown id | 1 | No change needed — loader unchanged. | +| `backend/tests/contract/test_trial_row_shape.py:113` | Leakage assertion | 1 | EXTENDED (Story 1.5). | +| `backend/tests/contract/test_studies_api_contract.py:156` | Docstring `pytrec_eval semantics` | 1 | Docstring REWORDED (Story 1.6). Existing assertions are structural-only — no substring of the message field is pinned, so the source string change at `studies.py:313` does NOT require an assertion update. | +| `backend/tests/integration/test_run_trial_per_query_persistence.py:111` | Leakage assertion | 1 | EXTENDED (Story 1.5). | +| `backend/tests/benchmarks/test_scoring_perf.py` | Warm-call timing | 1 | Compared before/after for Q5 resolution (Story 1.4 task 5). Not a default-gate test. | +| `backend/tests/integration/test_pagination.py:264` | `Trial(...)` constructor in test helper | 1 | No change — uses user-facing token shape already. | +| `backend/tests/integration/test_sort_pagination.py:419` | `Trial(...)` constructor | 1 | No change. | +| `backend/app/services/test_seeding.py:127,142` | `"p@10"` literal | 2 | FIXED inline (**Story 1.5 task 0** — moved from Story 1.6 per cycle-1 F7) → `"precision@10"`. | + +### 3.5 Migration verification + +**N/A** — no schema change, no Alembic migration in this plan. The Alembic head remains at `0015_trials_per_query_metrics` (unchanged). + +### 3.6 CI gates + +- [ ] `make test-unit` (includes new parity + shape tests) +- [ ] `make test-integration` (includes new existing-row regression) +- [ ] `make test-contract` (includes extended leakage assertion; the `test_studies_api_contract.py` docstring is reworded but no assertion changes — see Story 1.6) +- [ ] `make typecheck` (mypy strict) +- [ ] `make lint` (ruff) +- [ ] `cd ui && pnpm typecheck && pnpm lint` (the 3 UI source-of-truth comment changes don't change types but the strict TS config must still pass) +- [ ] **NEW gate** — `grep -rn 'pytrec_eval\|pytrec-eval'` returns only allowlist matches (FR-7 / AC-6). +- [ ] **NEW gate** — broader wire-form grep returns only allowlist matches (FR-7). +- [ ] `docker build .` succeeds (AC-10). + +--- + +## 4) Documentation update workstream + +### 4.0 Core context files + +**`state.md`** — REQUIRED update (per AC-11): +- [x] New dated entry added describing the migration (Story 1.8 task 7). Existing entries NOT back-edited. + +**`architecture.md`** — REQUIRED update (Story 1.8): +- [x] Line 131 — `eval/ pytrec_eval scoring` → `eval/ ir_measures scoring`. + +**`CLAUDE.md`** — REQUIRED update (Story 1.8): +- [x] Lines 15 + 29 — both `pytrec_eval` mentions → `ir_measures`. The umbrella-spec link convention stays; only the library name changes. + +### 4.1 Architecture docs + +- [x] `docs/01_architecture/optimization.md` — title + 10 mentions + code-example block (Story 1.8). +- [x] `docs/01_architecture/tech-stack.md` — line 41. +- [x] `docs/01_architecture/system-overview.md` — line 76. +- [x] `docs/01_architecture/README.md` — line 21. +- [x] `docs/01_architecture/data-model.md` — lines 52 + 231. +- [x] `docs/01_architecture/cluster-lifecycle.md` — line 159. + +### 4.2 Product docs + +- [x] `docs/02_product/mvp1-user-stories.md` — line 40. +- [x] `docs/02_product/planned_features/feat_study_baseline_trial/idea.md` — sibling coordination, line 56. +- [x] `docs/02_product/planned_features/feat_auto_followup_studies/idea.md` — sibling coordination, line 47. +- [x] `docs/00_overview/product/relevance-copilot-spec.md` — umbrella spec, 11 mentions including subsection rewrite. + +### 4.3 Runbooks + +**No runbook changes.** This migration adds no new operational procedure; no new debugging recipe; no new failure mode. The error-message rewording is a one-line string change that surfaces in the existing create-study modal error toast. + +### 4.4 Security docs + +**No security-docs changes.** No new data leaves the cluster on each scoring call (the scoring layer is local-only). No new secret, no new key handling, no new auth path. + +### 4.5 Quality docs + +**No quality-docs changes.** Coverage gate unchanged at 80%. The test-layer convention is unchanged — just adding more tests at the unit + integration + contract layers. + +### 4.6 Guides (tenant-facing) + +- [x] `docs/08_guides/workflows-overview.md` — lines 123 + 277. +- [x] `ui/public/docs/workflows-overview.md` — same content, lock-step. +- [x] `ui/public/guides/05_import_judgments_and_calibrate/script.md` — line 6. +- [x] `ui/public/guides/06_create_and_monitor_study/script.md` — line 8. +- [x] `ui/public/guides/06_create_and_monitor_study/metadata.json` — line 26. + +### 4.7 Auto-regenerated + +- [x] `docs/00_overview/MVP1_DASHBOARD.md` — Story 1.8 task 5 runs `python scripts/build_mvp1_dashboard.py`. + +**Documentation DoD** + +- [ ] All grep gates clean (Story 1.8 DoD). +- [ ] `state.md` has new dated entry (Story 1.8 DoD). +- [ ] `architecture.md` line 131 updated. +- [ ] `CLAUDE.md` lines 15 + 29 updated. +- [ ] MVP1_DASHBOARD.md regenerated and committed. + +--- + +## 5) Lean refactor workstream + +### 5.1 Refactor goals + +- **None planned.** This is a substrate migration. No code consolidation, no abstraction extraction, no policy centralization. + +### 5.2 Planned refactor tasks + +- None. + +### 5.3 Refactor guardrails + +- Behavioral parity proven by the 30-case parity test + per-query shape test + existing-row read regression (Stories 1.4 + 1.5). +- Lint and typecheck stay green at each commit (Story-by-Story Verification Gate §10). +- No expansion of product scope per spec §3 "Out of scope" — `confidence.py`, `ranx`, paired-bootstrap, Fisher randomization all stay out. + +--- + +## 6) Dependencies, risks, and mitigations + +### Dependencies + +| Dependency | Needed by | Status | Risk if missing | +|---|---|---|---| +| `ir-measures>=0.4.3` on PyPI | Story 1.1 | Implemented (PyTerrier team, MIT, actively maintained) | If PyPI unavailable at install time, `uv sync` fails with a normal dep-resolution error. | +| `pytrec-eval>=0.5` on PyPI | Story 1.1 (dev-group), Stories 1.2 + 1.4 (parity test) | Implemented (abandoned but still installable on Python 3.13) | If the C extension fails to compile on a new Python version (3.14+), the parity gate eventually goes dark. Acceptable future-drag per spec §19 + decision log. | + +### Risks + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| `ir_measures` routes some metric through a non-`pytrec_eval` provider with subtly different values | M | H — parity gate fails, PR blocked | Story 1.4 task 3 — bounded outcomes (a)/(b)/(c)/(d) per spec §19 Q4. Default response: use documented provider-forcing API; if none, bump `ir_measures` version. | +| `ir_measures.iter_calc()` emits a different per-query universe than `pytrec_eval.RelevanceEvaluator.evaluate()` for qrel-only / run-only edge cases | M | H — per-query shape parity fails, persisted JSONB key set on edge cases would change | Story 1.4 task 4 — default to filtering `iter_calc` output to match `pytrec_eval`'s universe (option (a) under task 4). Escalate to user only if option (a) is impractical. | +| `ir_measures` doesn't ship `py.typed`, mypy strict fails | L | M — Story 1.1 needs a mypy override | Story 1.1 task 1 verifies empirically; adds the override if needed. | +| Performance regression > 10% | L | M — Q5 resolution flags it as a blocker | Story 1.4 task 5 measures empirically; if regression > 10%, escalate to user with the benchmark numbers. | +| Dashboard regen surfaces historical `pytrec_eval` mention from `infra_optuna_eval` row, fails AC-6 grep gate | M | M — requires generator fix, NOT an allowlist exception | Story 1.8 task 5 inspects the regenerator; if historical text leaks, **fix the generator** (per Story 1.8 task 5 options (a)/(b)) so the regenerated dashboard is current-state clean. **Do not** add a dashboard allowlist exception — the spec's FR-7 allowlist is fixed and the plan cannot weaken AC-6 (per plan cycle-2 C2-F4 + Story 1.8 task 5 lock). If neither option (a) nor (b) is practical, STOP and escalate. | +| Pre-commit hook (Conventional Commits + ruff + mypy) rejects a doc-only commit | L | L | All commits follow Conventional Commits format; doc-sweep commit prefix is `docs(eval):`. | + +### Failure mode catalog + +| Failure mode | Trigger | Expected system behavior | Recovery | +|---|---|---|---| +| `ir_measures` import fails at worker boot | Bad install / missing C extension | Worker process exits at import time; Compose healthcheck fails | Rebuild image; verify install logs | +| Parity test fails on one metric | Provider routing divergence | CI blocks the PR; impl-plan author follows §19 Q4 bounded outcomes | Use documented forcing API OR bump `ir_measures` version | +| Per-query shape parity fails | `iter_calc` emits different qid universe | CI blocks the PR | Apply Story 1.4 task 4 option (a) — filter to match legacy universe | +| Existing-row read regression fails | Pre-migration JSONB row hydration breaks | CI blocks the PR | Inspect: did `objective_metric_key` change? Did `confidence.py` change unexpectedly? The migration claim "no-migration / no-backfill" is invalidated; STOP and escalate. | +| Dockerfile change breaks build | gcc/g++/python3-dev wrongly dropped when transitive C ext still resolves | `docker build .` fails | Revert Story 1.7 change; verify Q3 outcome was correct | +| `pytrec-eval` dev dep fails to install in CI | C extension can't compile against the CI Python version (e.g., Python 3.14+ with no wheels) | Parity gate cannot run → CI blocks the PR | Mitigation: file `chore_pytrec_eval_dev_dep_removal` idea per spec §19 Decision log; either swap the parity gate to a different comparison library OR retire it. Until then, the PR cannot merge unless the parity gate runs. | +| `ir_measures` transitive dependency has license-incompatible package | `uv tree` shows a non-Apache-2.0-compatible package (e.g., GPL) | PR blocked at Q5 resolution | Mitigation: repin `ir_measures` to a version without that transitive OR escalate to user. Do not merge under a license-incompatible dependency. | + +--- + +## 7) Sequencing and parallelization + +### Suggested sequence + +Strict sequential — every story builds on the prior one: + +1. Story 1.1 — pyproject.toml (deps in place) +2. Story 1.2 — parity test skeleton (test infra in place, skipped) +3. Story 1.3 — scoring.py rewrite (core change; existing tests still pass) +4. Story 1.4 — parity test activated (the gate fires for the first time) +5. Story 1.5 — leakage assertions extended + existing-row regression +6. Story 1.6 — operator-visible error message at `studies.py:313` + docstring rewording (the `p@10` inline fix moved to Story 1.5 task 0 per cycle-1 F7) +7. Story 1.7 — Dockerfile conditional update +8. Story 1.8 — full doc/comment sweep + dashboard regen + final grep gates + +### Parallelization opportunities + +None. The migration is one PR (per spec) and each story depends on its predecessor: +- Story 1.2 needs Story 1.1's `ir_measures` install to be importable. +- Story 1.3 needs Story 1.2's fixture to exist (for `make test-unit` not to error on Story 1.3's reuses of fixture imports). +- Story 1.4 needs Story 1.3's scoring.py to be migrated. +- Stories 1.5 + 1.6 + 1.7 + 1.8 each modify different files but inspect the post-1.4 state. + +Sub-tasks WITHIN Story 1.8 (the doc sweep) can be parallelized — different files, no cross-dependencies. The impl-execute agent can edit 3-5 doc files per batched tool call. + +--- + +## 8) Rollout and cutover plan + +- **Rollout stages:** N/A — single-PR migration. No staging deploy in MVP1 (per CLAUDE.md / state.md). Merge to main → local dev stacks get the new behavior on next `make up`. +- **Feature flag strategy:** None. The migration is atomic at the library-import level — no flag would gain anything. +- **Migration/cutover steps:** None. No schema change, no data backfill. +- **Reconciliation/repair strategy:** N/A. + +--- + +## 9) Execution tracker + +### Current sprint +- [ ] Story 1.1 — pyproject.toml (add ir-measures, move pytrec-eval to dev, audit mypy overrides) +- [ ] Story 1.2 — parity test fixture + skeleton +- [ ] Story 1.3 — scoring.py rewrite with locked metric-object mapping +- [ ] Story 1.4 — parity test activation + per-query shape + Q5 perf benchmark +- [ ] Story 1.5 — leakage assertions extended + existing-row read regression +- [ ] Story 1.6 — studies.py:313 message + `test_studies_api_contract.py:156` docstring rewording (no assertion change; no `p@10` fix here — moved to Story 1.5 task 0) +- [ ] Story 1.7 — Dockerfile conditional update (per Q3 resolution) +- [ ] Story 1.8 — full doc sweep + MVP1_DASHBOARD regen + grep gates clean + +### Blocked items + +None at planning time. + +### Done this sprint + +— + +--- + +## 10) Story-by-Story Verification Gate (Agent Checklist) + +Before marking any story complete, the executing engineer or agent must attach evidence for: + +- [ ] Files created/modified match story scope (`New files` / `Modified files` tables) +- [ ] Public API of `scoring.py` is byte-identical (verified by `git diff backend/app/eval/scoring.py` showing only internal-logic + docstring changes, no signature changes) +- [ ] Persisted JSONB key shape preserved (verified by AC-3's strict regex + the existing-row regression in Story 1.5) +- [ ] Tests pass at the layers touched: + - [ ] `make test-unit` (always required) + - [ ] `make test-integration` (required from Story 1.5 onward) + - [ ] `make test-contract` (required from Story 1.5 onward) +- [ ] Typecheck + lint green (`make typecheck`, `make lint`) +- [ ] After Story 1.4: 30 parity cases + per-query shape test PASS, Q4 resolution recorded in commit +- [ ] After Story 1.7: `docker build .` succeeds, Q3 resolution recorded in commit +- [ ] After Story 1.8: both grep gates clean, MVP1_DASHBOARD.md regenerated and committed, `state.md` has new dated entry + +--- + +## 11) Plan consistency review + +### 11.1 Spec ↔ plan endpoint count + +Spec §8.1 endpoint surface: **N/A** — no new endpoints. Plan endpoint count: **0**. Match. + +### 11.2 Spec ↔ plan error code coverage + +Spec §8.5 error code catalog: "No new error codes. The existing `INSUFFICIENT_JUDGMENT_OVERLAP` code (owned by `feat_study_preflight_overlap_probe`) is unchanged in code value, status, and retryability — only its message text is reworded." + +Plan: Story 1.6 rewords the source message at `studies.py:270` + `:313`; the existing contract test at `test_studies_api_contract.py` is structural-only (asserts the error-code literal + ordering — no message substring pinned), so only a docstring rewording is needed there. No new error code introduced. Match. + +### 11.3 Spec ↔ plan FR coverage + +| Spec FR | Plan story | Verified | +|---|---|---| +| FR-1 scoring.py swap + mapping + aggregate-via-iter | Story 1.3 | ✓ | +| FR-2 parity test (permanent CI gate, 30 cases) | Stories 1.2 + 1.4 | ✓ | +| FR-3 no wire-form leakage + per-query shape parity | Story 1.4 (shape) + Story 1.5 (leakage) | ✓ | +| FR-4 pyproject.toml (runtime + dev split + mypy overrides) | Story 1.1 | ✓ | +| FR-5 studies.py:313 operator-visible message + contract docstring rewording | Story 1.6 | ✓ (no message substring is pinned today; existing structural contract assertions are unchanged) | +| FR-6 Dockerfile conditional gcc/g++/python3-dev | Story 1.7 | ✓ | +| FR-7 doc sweep + dashboard regen + broader grep gate | Story 1.8 | ✓ | + +All 7 FRs covered by at least one story. No FR orphaned. No story without an FR backing. + +### 11.4 Story internal consistency + +- No endpoint tables in any story (no new endpoints), so no schema field mismatch possible. +- DoD assertions reference the spec's AC-1 through AC-12 by ID. +- New files don't overlap across stories: Story 1.2 creates `parity_qrels_run.py` + `test_scoring_parity.py`; Story 1.5 creates `test_existing_row_read_compat.py`. No conflict. +- Modified files: scoring.py is touched only by Story 1.3. studies.py is touched only by Story 1.6. Contract test files are touched by Stories 1.5 + 1.6 (but at different assertions). pyproject.toml is touched only by Story 1.1. Dockerfile only by Story 1.7. No file ownership conflict. + +### 11.5 Test file count and assignment + +Test files in the testing workstream: +- `backend/tests/unit/eval/fixtures/parity_qrels_run.py` → Story 1.2 +- `backend/tests/unit/eval/test_scoring_parity.py` → Story 1.2 (creation) + Story 1.4 (activation) +- `backend/tests/integration/test_existing_row_read_compat.py` → Story 1.5 +- Extended `backend/tests/contract/test_trial_row_shape.py` → Story 1.5 +- Extended `backend/tests/contract/test_studies_api_contract.py` → Story 1.6 +- Extended `backend/tests/integration/test_run_trial_per_query_persistence.py` → Story 1.5 + +Every test file is assigned to exactly one story (or two stories for the parity test, which is correct — Story 1.2 creates skipped, Story 1.4 unskips). + +### 11.6 Gate arithmetic + +No epic/phase gates — single-epic, single-PR. The Story-by-Story Verification Gate (§10) is the sole gate; it lists the per-story conditions. + +### 11.7 Open questions resolved + +| Q | Resolved in | How | +|---|---|---| +| Q1 | Story 1.8 | Reword the migration docstring; recommendation locked. | +| Q2 | Story 1.1 task 1 | Empirical `find py.typed` recipe; conditional mypy override. | +| Q3 | Story 1.7 task 1 | Empirical `pip show pytrec_eval` recipe; Dockerfile change driven by output. | +| Q4 | Story 1.4 task 3 | Run parity; if pass → no action; if fail → bounded outcomes per spec §19 Q4. | +| Q5 | Story 1.1 task 3 + Story 1.4 task 5 | `uv tree` license audit + perf benchmark before/after. | + +### 11.8 Frontend UI Guidance completeness + +**Not required for this plan** — no frontend stories with element inventories. The 3 UI source-of-truth comment changes (`create-study-modal.tsx:60`, `k-ignored.test.ts:4`, `types.ts:1889`) are pure comment edits with no JSX or behavior change; they live in Story 1.8. + +### 11.9 Enumerated value contract audit + +**No new enumerated values added.** `SUPPORTED_METRICS` and `SUPPORTED_K_VALUES` are preserved byte-identically by FR-1. The frontend option arrays at `create-study-modal.tsx` continue to match the backend source. No grep audit required. + +### 11.10 Audit-event coverage + +**N/A** — pre-MVP2. RelyLoop's `audit_log` table arrives at MVP2 per `docs/01_architecture/data-model.md`. No audit events to instrument. + +### 11.11 Persistence scope consistency + +**N/A** — no `localStorage` / `sessionStorage` usage in this migration. + +### 11.12 Plan ↔ codebase verification + +| Claim | Verified by | Status | +|---|---|---| +| `backend/app/eval/scoring.py` exists and is the only `import pytrec_eval` site | grep — only one source-tree match at line 22 | Verified | +| `pyproject.toml` line 47 has `pytrec-eval>=0.5` | Read pyproject.toml | Verified | +| `pyproject.toml` lines 156–158 have the `pytrec_eval` mypy override | Read pyproject.toml | Verified | +| Dockerfile lines 44–54 have the gcc/g++/python3-dev install | Read Dockerfile | Verified | +| `backend/app/api/v1/studies.py` line 270 inline comment AND line 313 error-message string both mention `pytrec_eval` | Read studies.py:260–320 | Verified — TWO mentions (idea said one) | +| `backend/app/services/test_seeding.py` lines 127 + 142 contain `"p@10"` literals | grep test_seeding.py | Verified — two occurrences | +| MVP1_DASHBOARD.md lines 64 + 134 contain actual `pytrec_eval` mentions (not false positives) | Re-read snippet during cycle-2 patch | Verified | +| `backend/tests/contract/test_trial_row_shape.py:113` has the leakage assertion | Read test_trial_row_shape.py around line 113 | Verified (spot-check during impl-plan) | +| `backend/tests/integration/test_run_trial_per_query_persistence.py:111` has the leakage assertion | Read test_run_trial_per_query_persistence.py around line 111 | Verified | +| `backend/workers/trials.py` has TWO write paths to `trials.metrics` (lines 446–447 happy path + line 178 idempotency-replay) | Read trials.py 150–230 + 440–450 | Verified | + +### 11.13 Infrastructure path verification + +| Path | Verified by | Status | +|---|---|---| +| `backend/tests/unit/eval/` is the unit-test directory | `ls backend/tests/unit/eval/` (verified during draft) | Verified | +| `backend/tests/integration/` for the new existing-row test | Sibling tests like `test_run_trial_per_query_persistence.py` live there | Verified | +| `migrations/alembic.ini` (NOT `backend/alembic`) — but no migration in this plan | — | N/A | + +--- + +## 12) Definition of plan done + +- [x] Every FR is mapped to stories/tasks/tests/docs updates (§1 + §11.3). +- [x] Every story includes New files, Modified files, Tasks, and DoD. (No endpoints/schemas to include — no new API surface.) +- [x] Test layers (unit/integration/contract) are explicitly scoped (§3). E2E is documented as N/A. +- [x] Documentation updates across docs/01-08 + ui/public are planned and owned (§4). +- [x] Lean refactor scope and guardrails are explicit — §5 documents "none planned". +- [x] No epic/phase gates beyond the Story-by-Story Verification Gate (§10) — single-PR migration. +- [x] Story-by-Story Verification Gate is included (§10). +- [x] Plan consistency review (§11) has been performed with no unresolved findings. + +--- + +## Cross-model review log + +GPT-5.5 reviewed this plan in 3 cycles per the impl-plan-gen workflow Step 6/7: + +- **Cycle 1: 10 findings (9 accepted, 1 rejected).** Accepted + applied: F2 score()-universe-filter restructure, F3 per-query value parity in tests, F4 mandatory digest assertion in AC-12, F5 umbrella-spec rewrite without naming pytrec_eval, F6 dashboard clean-regen (no allowlist exception), F7 test_seeding.py p@10 fix moved to Story 1.5, F8 §3.3 docstring-only consistency, F9 grep gate against `pytest.mark.skip` in test_scoring_parity.py, F10 two new failure-mode rows. Rejected with cited counter-evidence: F1 (`_translate_metric_name("unknown")` → `requires an @ cut` is the EXISTING behavior; both old and new code preserve identical triggering inputs — verified at scoring.py:74-78). +- **Cycle 2: 4 findings (all accepted).** C2-F1 universe-filter tightening (`if qrels.get(qid) and run.get(qid)` for empty-inner-dict edge cases), C2-F2 test_seeding.py reassignment + ordering consistency across summary sections, C2-F3 stale "contract substring" references purged, C2-F4 dashboard regen mitigation aligned with Story 1.8's no-allowlist-exception stance. +- **Cycle 3: 1 finding (accepted).** C3-F1 added symmetric `run[q] = {}` parity fixture case to mirror the `qrels[q] = {}` case from C2-F1. +- **Convergence:** 10 → 4 → 1 (monotonically decreasing). Plan approved for impl-execute. diff --git a/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md b/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md new file mode 100644 index 00000000..62b8f814 --- /dev/null +++ b/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md @@ -0,0 +1,28 @@ +# Pipeline Status — infra_ir_measures_migration + +## Idea +- Status: Complete +- File: [`idea.md`](./idea.md) +- Refreshed via `/idea-preflight` on 2026-05-22 (PR #197 — preflight ledger captured in the file's "Sequencing-pressure update" + "Still-needed verification" sections). + +## Spec +- Status: Approved +- Date: 2026-05-22 +- File: [`feature_spec.md`](./feature_spec.md) +- Cross-model review: GPT-5.5 — 3 cycles (11 findings → 6 → 1). All findings accepted; zero rejections. Convergence trajectory monotonically decreasing. +- Phases: 1 of 1 (single-phase migration; §3 "Phase boundaries" rationale locks the single-PR scope). +- Locked decisions (§19 Decision log): single-PR scope; `ir_measures` over `ranx` / `pytrec-eval-terrier`; public API of `scoring.py` frozen; persisted JSONB key shape frozen; 6dp parity tolerance; sibling planned-features updated in same PR; `confidence.py` out of scope; `pytrec-eval` retained permanently in `[dependency-groups.dev]` for parity-gate infrastructure. +- 5 open questions (Q1–Q5, all empirical, resolved at impl-plan time): historical migration docstring rewording; `ir_measures` type hints; `pytrec_eval` transitive backend status; provider-routing observability; transitive deps / license / performance verification. + +## Plan +- Status: Approved +- Date: 2026-05-22 +- File: [`implementation_plan.md`](./implementation_plan.md) +- Cross-model review: GPT-5.5 — 3 cycles (10 → 4 → 1 findings). 14 accepted + applied, 1 rejected with cited counter-evidence. Convergence trajectory monotonically decreasing. +- Stories: 8 stories in 1 epic (single-PR migration per spec §3 Phase boundaries). +- Phases covered: 1 of 1 (single-phase migration; no deferred phases). +- Sequencing: strict-sequential 1.1 → 1.2 → 1.3 → 1.4 → 1.5 → 1.6 → 1.7 → 1.8. No parallelization opportunities across stories. +- Locked decisions reflected in plan: public API of `scoring.py` frozen; persisted JSONB keys frozen; aggregate-via-iter (no `calc_aggregate`); per-query universe filtered to `pytrec_eval` historical contract; `pytrec-eval` permanent in `[dependency-groups.dev]` for parity gate. + +## Implementation +- Status: Not started From b26546303a324ab6e1598ef4cbdb00adae51c6f5 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:27:12 -0400 Subject: [PATCH 02/15] chore(deps): add ir-measures to runtime, move pytrec-eval to dev-group (Story 1.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.1 of infra_ir_measures_migration. Replaces the abandoned pytrec_eval direct runtime dependency with the actively-maintained ir_measures provider abstraction. Changes: - Add ir-measures>=0.4.3 to [project].dependencies - Move pytrec-eval>=0.5 from [project].dependencies to [dependency-groups.dev] (permanent dev pin per FR-4 — keeps the parity test at FR-2 runnable; the runtime image installs with `uv sync --no-dev` and never carries it) - Drop the [[tool.mypy.overrides]] block for `pytrec_eval` — Q2 empirical resolution: both ir_measures AND pytrec-eval-terrier ship `py.typed`, so no missing-imports override is needed Q3 (transitive pytrec_eval status): ir-measures resolves pytrec-eval-terrier v0.5.10 transitively (the actively-maintained PyTerrier fork), which publishes the same `pytrec_eval` module name. The Dockerfile gcc/g++/python3-dev install (Story 1.7) stays — still a C extension. Q5 (license + transitive deps): all new packages compatible with project's Apache 2.0 license: ir-measures (Apache 2.0), pytrec-eval-terrier (MIT). No GPL or other incompatible transitives surfaced by `uv tree`. Verification: - make backend-fmt + backend-lint + backend-typecheck all green - ./.venv/bin/ruff format --check backend/ → 386 files already formatted - uv lock --check → 98 packages resolved - pytest backend/tests/unit/ → 1044 passed (no regression) Co-Authored-By: Claude Opus 4.7 (1M context) --- pyproject.toml | 26 ++++++++++++++++++++------ uv.lock | 40 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 8 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index e4a5a2a3..0b5bcd04 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -44,7 +44,13 @@ dependencies = [ # every digest's parameter_importance is silently {}. See # docs/02_product/planned_features/bug_digest_param_importance_seam/bug_fix.md. "scikit-learn>=1.4", - "pytrec-eval>=0.5", + # IR-evaluation engine. Wraps multiple backends (pytrec_eval, gdeval, + # judged_as_relevant) behind a typed metric-object DSL (nDCG@10, AP@5, + # RR, P@k, R@k). Migrated from the abandoned pytrec_eval direct + # dependency by infra_ir_measures_migration; the parity gate at + # backend/tests/unit/eval/test_scoring_parity.py keeps both backends' + # values byte-identical to 6 decimal places. + "ir-measures>=0.4.3", # Headless plotting for the open_pr worker's parameter-importance chart. # PNG only — Agg backend, no GUI. Bundled with the API+worker image. "matplotlib>=3.9", @@ -59,6 +65,13 @@ dev = [ "pytest-cov>=6.0", "pytest-mock>=3.14", "pytest-recording>=0.13", + # pytrec_eval is retained here (no longer a runtime dependency) ONLY so + # the parity test at backend/tests/unit/eval/test_scoring_parity.py can + # import both libraries side-by-side for value-equivalence assertions. + # Runtime images install with `uv sync --no-dev` and never carry it. + # See docs/00_overview/implemented_features/.../infra_ir_measures_migration/ + # feature_spec.md FR-4 + §19 Q3 for the lifecycle decision. + "pytrec-eval>=0.5", "testcontainers[postgresql]>=4.9", "pre-commit>=4.6.0", "types-PyYAML>=6.0", @@ -151,11 +164,12 @@ disallow_untyped_decorators = false warn_return_any = false # Third-party libraries without py.typed marker / stubs. -# pytrec_eval ships no type info; treat its imports as untyped (we wrap it in -# backend/app/eval/scoring.py which is fully typed). -[[tool.mypy.overrides]] -module = "pytrec_eval" -ignore_missing_imports = true +# +# (infra_ir_measures_migration removed the pytrec_eval module override here: +# the runtime dependency was swapped to ir-measures, which transitively pulls +# pytrec-eval-terrier — the actively-maintained PyTerrier fork that ships +# `py.typed`. ir_measures itself ships `py.typed` too. Both modules are now +# strict-typeable; no override needed.) # --------------------------------------------------------------------------- # Pytest diff --git a/uv.lock b/uv.lock index 70bc5569..e3faf5e7 100644 --- a/uv.lock +++ b/uv.lock @@ -646,6 +646,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "ir-measures" +version = "0.4.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pytrec-eval-terrier" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/81/b4/d931aa71a73353cf0b8446b3d7ce9ed789e7fe595778d7e061e86faea28d/ir_measures-0.4.3.tar.gz", hash = "sha256:869a6019efefb02dda8b77ca0f68d6d57b01fe34998a46c76425b9fbb396e827", size = 51224, upload-time = "2025-11-25T17:36:52.06Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/76/62/15199767ad9f1e3f793dec253ba6f662cb36e7197fc647bc844a6ecb99bf/ir_measures-0.4.3-py3-none-any.whl", hash = "sha256:3d8b3b5283e4a77df85f9f6c8a1c2c99a7421b78fa014ef97f2ddab3037d3dcf", size = 61311, upload-time = "2025-11-25T17:36:51.183Z" }, +] + [[package]] name = "jinja2" version = "3.1.6" @@ -1447,6 +1459,28 @@ version = "0.5" source = { registry = "https://pypi.org/simple" } sdist = { url = "https://files.pythonhosted.org/packages/2e/03/e6e84df6a7c1265579ab26bbe30ff7f8c22745aa77e0799bba471c0a3a19/pytrec_eval-0.5.tar.gz", hash = "sha256:d9eb4616e7d6b73bf1b5ba4c2c4916e88124e790b53fd61610c30158999e7bde", size = 15248, upload-time = "2020-09-07T18:31:57.186Z" } +[[package]] +name = "pytrec-eval-terrier" +version = "0.5.10" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "scipy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/04/96/4925a95e4865a647bc74d3bb052243d12a3c8e8a34909d7d097b5a4d08c5/pytrec_eval_terrier-0.5.10.tar.gz", hash = "sha256:eaaf20580d17b5575a233e04dab8a4cbcc01a7e45be8cf547c07f0a2bb3e7eb9", size = 18634, upload-time = "2025-10-20T16:50:18.098Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/93/21/71a0dee7e2cd368237432af6bf6051ffde03370730dc1666cd39494c82a7/pytrec_eval_terrier-0.5.10-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:fcf96c33446c16de8db78e829c5279f7404ceaaf6b502bb5a6a3669b06051601", size = 137186, upload-time = "2025-10-20T16:50:22.941Z" }, + { url = "https://files.pythonhosted.org/packages/5c/8c/2494edf20d726bdd3ee0a20dc5ed84351c6cc6ccc17b11b474e315808762/pytrec_eval_terrier-0.5.10-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8455485f1faf6759f1be11b12c904d1c749ba5db7e2b6f414aa56e19533ce069", size = 304917, upload-time = "2025-10-20T16:54:20.486Z" }, + { url = "https://files.pythonhosted.org/packages/cf/51/7611546afb55548e65db35354a63b90d5fd5ea593fc64e5993088bf61415/pytrec_eval_terrier-0.5.10-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e7cc9666305281b0ca1873761dc71cd3f0863e6d759f00a12fd363aa2d558d6f", size = 1327998, upload-time = "2025-10-20T16:54:21.375Z" }, + { url = "https://files.pythonhosted.org/packages/74/b3/20941b4dbe3b267271ed1ef80aa93b348da674aecb5d6aca8f311c4738b0/pytrec_eval_terrier-0.5.10-cp313-cp313-win_amd64.whl", hash = "sha256:9440bd4a78ee0bc5db6821d7483e962a6c494303fd26598f84f00d54cc64cdd7", size = 58631, upload-time = "2025-10-20T16:51:05.08Z" }, + { url = "https://files.pythonhosted.org/packages/f0/34/e3d0f75286151d97537309b3f311e1269b0194e3823038fc39054e84c3b4/pytrec_eval_terrier-0.5.10-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:70bc61b8d02e61a37ed97c088282bb0a124b58e7141cc52756512750efabacbb", size = 137320, upload-time = "2025-10-20T16:50:50.92Z" }, + { url = "https://files.pythonhosted.org/packages/1c/72/2c1f9fd44ed7a5657654a712e5255019d5d23ba2b3d53848da1838bfb8df/pytrec_eval_terrier-0.5.10-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d52d94803c32cadbff7fe5195b0d0d68d27393092f64207fe8250a4485d1f8d7", size = 304917, upload-time = "2025-10-20T16:54:22.59Z" }, + { url = "https://files.pythonhosted.org/packages/66/9d/7e440de7b37dd31cd78eefe2ec1bf3e5f49db42b17b34dc8d6006ee03fc5/pytrec_eval_terrier-0.5.10-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:77950d0ce9bd960af40efede6850e7b6519400e7fda3f9313e0d0d02c247e4e2", size = 1327991, upload-time = "2025-10-20T16:54:23.76Z" }, + { url = "https://files.pythonhosted.org/packages/ef/94/5639d7c346935a75540c1f1798be277c161b561001f2a91ef303e3d85f10/pytrec_eval_terrier-0.5.10-cp314-cp314-win_amd64.whl", hash = "sha256:c69681fec350fa94af45dd7ef8f53f605e89f752583c814f713d7d2329435cfc", size = 60178, upload-time = "2025-10-20T16:51:50.946Z" }, + { url = "https://files.pythonhosted.org/packages/f4/a7/9080fe3f971397ea4447e3bda0c350225c944047ede7927c9a1f788af000/pytrec_eval_terrier-0.5.10-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:876740f3d58625058d34aaa1939be31bf253ecacd85d0d8b1089db5dd57ab127", size = 308002, upload-time = "2025-10-20T16:54:24.746Z" }, + { url = "https://files.pythonhosted.org/packages/ad/c9/5bf9d58cb275559211ba4af905c5a4d95f78c4b973f4186f8b22d8c0b073/pytrec_eval_terrier-0.5.10-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:2ca4e624e5f2589ae75c1034ff1f38e9fc81de86314193508ac423e7ca56769c", size = 1330474, upload-time = "2025-10-20T16:54:25.569Z" }, +] + [[package]] name = "pywin32" version = "311" @@ -1523,6 +1557,7 @@ dependencies = [ { name = "asyncpg" }, { name = "fastapi" }, { name = "httpx" }, + { name = "ir-measures" }, { name = "jinja2" }, { name = "matplotlib" }, { name = "openai" }, @@ -1530,7 +1565,6 @@ dependencies = [ { name = "psycopg2-binary" }, { name = "pydantic" }, { name = "pydantic-settings" }, - { name = "pytrec-eval" }, { name = "pyyaml" }, { name = "redis" }, { name = "scikit-learn" }, @@ -1550,6 +1584,7 @@ dev = [ { name = "pytest-cov" }, { name = "pytest-mock" }, { name = "pytest-recording" }, + { name = "pytrec-eval" }, { name = "ruff" }, { name = "testcontainers" }, { name = "types-pyyaml" }, @@ -1562,6 +1597,7 @@ requires-dist = [ { name = "asyncpg", specifier = ">=0.30" }, { name = "fastapi", specifier = ">=0.115" }, { name = "httpx", specifier = ">=0.28" }, + { name = "ir-measures", specifier = ">=0.4.3" }, { name = "jinja2", specifier = ">=3.1" }, { name = "matplotlib", specifier = ">=3.9" }, { name = "openai", specifier = ">=1.55" }, @@ -1569,7 +1605,6 @@ requires-dist = [ { name = "psycopg2-binary", specifier = ">=2.9" }, { name = "pydantic", specifier = ">=2.9" }, { name = "pydantic-settings", specifier = ">=2.6" }, - { name = "pytrec-eval", specifier = ">=0.5" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "redis", specifier = ">=5.2" }, { name = "scikit-learn", specifier = ">=1.4" }, @@ -1589,6 +1624,7 @@ dev = [ { name = "pytest-cov", specifier = ">=6.0" }, { name = "pytest-mock", specifier = ">=3.14" }, { name = "pytest-recording", specifier = ">=0.13" }, + { name = "pytrec-eval", specifier = ">=0.5" }, { name = "ruff", specifier = ">=0.8" }, { name = "testcontainers", extras = ["postgresql"], specifier = ">=4.9" }, { name = "types-pyyaml", specifier = ">=6.0" }, From 5ae53de7563353799fd5d3693fde416b6b3a85ac Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:29:46 -0400 Subject: [PATCH 03/15] test(eval): add parity test fixture and skeleton (Story 1.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.2 of infra_ir_measures_migration. Lands the parity-test infrastructure with all assertions @pytest.mark.skip'd — both libraries import side-by-side, fixture loads, 30+1 cases collect, Story 1.4 will remove the skips after Story 1.3's scoring.py rewrite. New files: - backend/tests/unit/eval/fixtures/__init__.py (empty package marker) - backend/tests/unit/eval/fixtures/parity_qrels_run.py — fixture with 10 queries covering 6 edge cases per spec FR-2 + plan cycle-1 F2/F3 + cycle-2 C2-F1 + cycle-3 C3-F1: (a) q_no_relevant — all docs rated 0 (b) q_qrel_only — qrels present, missing from run (c) q_run_only — run present, missing from qrels (d) q_zero_overlap — both present, disjoint doc IDs (e) q_empty_qrels — qrels[qid] == {} but run[qid] non-empty (f) q_empty_run — run[qid] == {} but qrels[qid] non-empty + 4 normal graded-rating queries for value-parity signal. - backend/tests/unit/eval/test_scoring_parity.py — the parity test: test_score_matches_pytrec_eval_within_1e_minus_6 [30 parametrized cases, SKIPPED] — aggregate parity at 1e-6 per spec FR-2 test_per_query_shape_matches_pytrec_eval [SKIPPED] — outer qid set, inner metric-key set, AND per-(qid, metric) value parity at 1e-6 per spec FR-3 / plan cycle-1 F3 test_parity_cases_count_is_30 [PASS] — sanity: PARITY_CASES enumerates exactly 30 cells per FR-2 cross test_translate_metric_name_still_callable_at_import [PASS] — Story 1.4 collection-safety check The PARITY_CASES cross (verified by test_parity_cases_count_is_30): ndcg/precision/recall × 7 k-values = 21 map × 7 k-values + plain map (no k) = 8 plain mrr (k ignored) = 1 total = 30 The parity test computes pytrec_eval's mean-across-queries aggregate inline (via _pytrec_aggregate_for_token + _pytrec_wire_name) rather than depending on scoring.py's pre-migration body — keeps the test insulated from Story 1.3's _translate_metric_name rewrite. Per spec §19 + plan cycle-2 C2-F4, the test does NOT call ir_measures.calc_aggregate() — it compares score()'s output directly against pytrec_eval. Verification: - pytest backend/tests/unit/eval/test_scoring_parity.py -v → 2 passed, 31 skipped in 0.44s - make backend-fmt: 392 files unchanged - make backend-lint: All checks passed! - make backend-typecheck: Success: no issues found in 389 source files Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/tests/unit/eval/fixtures/__init__.py | 0 .../unit/eval/fixtures/parity_qrels_run.py | 84 ++++++ .../tests/unit/eval/test_scoring_parity.py | 246 ++++++++++++++++++ 3 files changed, 330 insertions(+) create mode 100644 backend/tests/unit/eval/fixtures/__init__.py create mode 100644 backend/tests/unit/eval/fixtures/parity_qrels_run.py create mode 100644 backend/tests/unit/eval/test_scoring_parity.py diff --git a/backend/tests/unit/eval/fixtures/__init__.py b/backend/tests/unit/eval/fixtures/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/tests/unit/eval/fixtures/parity_qrels_run.py b/backend/tests/unit/eval/fixtures/parity_qrels_run.py new file mode 100644 index 00000000..a6f533c4 --- /dev/null +++ b/backend/tests/unit/eval/fixtures/parity_qrels_run.py @@ -0,0 +1,84 @@ +"""Parity fixture for the pytrec_eval ↔ ir_measures value-equivalence test. + +Source of truth for ``backend/tests/unit/eval/test_scoring_parity.py`` +(infra_ir_measures_migration Stories 1.2 + 1.4 — see [implementation_plan.md]). + +The fixture is hand-crafted to exercise every edge case the spec FR-2 / FR-3 +parity contracts cover. Each query's role is documented inline; do not add +queries without updating the docstring. + +The two ``Qrels``/``Run`` shapes here are exactly what +:func:`backend.app.eval.scoring.score` accepts; the test imports them and +calls both ``score()`` (the function under test) and ``pytrec_eval`` directly +to assert 6-decimal-place value equivalence + per-query shape parity. +""" + +from __future__ import annotations + +from backend.app.eval.scoring import Qrels, Run + +# --------------------------------------------------------------------------- +# QRELS — {query_id: {doc_id: int rating}} +# --------------------------------------------------------------------------- +# +# Edge-case map (also documented in test_scoring_parity.py): +# +# Case (a) no-relevant-docs: q_no_relevant — all docs rated 0 +# Case (b) qrel-only: q_qrel_only — present in qrels; missing from run +# Case (c) run-only: q_run_only — present in run; missing from qrels +# Case (d) zero-overlap: q_zero_overlap — both present; disjoint doc IDs +# Case (e) empty-inner-qrels: q_empty_qrels — qrels[qid] == {} but run[qid] non-empty +# Case (f) empty-inner-run: q_empty_run — run[qid] == {} but qrels[qid] non-empty +# +# Plus four "normal" queries with realistic graded ratings + non-trivial +# rankings, giving the parity test enough signal across the metric × k cross +# to catch most divergences. + +QRELS: Qrels = { + # ---- Normal queries (mixed graded ratings) ---- + "q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 0, "d5": 1}, # 5 docs, 4 relevant + "q2": {"d1": 2, "d2": 0, "d3": 3, "d4": 1, "d5": 0}, # 5 docs, 3 relevant + "q3": {"d1": 1, "d2": 1, "d3": 1, "d4": 1, "d5": 1}, # binary-style, all relevant + "q4": {"d1": 3, "d2": 0, "d3": 0, "d4": 0, "d5": 2}, # 5 docs, 2 relevant — tail skew + # ---- Edge cases ---- + "q_no_relevant": {"d1": 0, "d2": 0, "d3": 0}, # Case (a): qrels exist but no relevance + "q_qrel_only": {"d1": 2, "d2": 1, "d3": 3}, # Case (b): qrels with no matching run + # q_run_only — INTENTIONALLY absent from QRELS (Case c is qrels-missing for that qid) + "q_zero_overlap": {"d1": 3, "d2": 2}, # Case (d): qrels for d1/d2, run for d8/d9 + "q_empty_qrels": {}, # Case (e): empty inner dict + "q_empty_run": {"d1": 2, "d2": 1}, # Case (f): qrels present, run will be empty +} + +# --------------------------------------------------------------------------- +# RUN — {query_id: {doc_id: float score}} +# --------------------------------------------------------------------------- +RUN: Run = { + # ---- Normal queries ---- + "q1": {"d1": 0.95, "d2": 0.85, "d3": 0.65, "d4": 0.55, "d5": 0.45}, # perfect-ish + "q2": {"d2": 0.95, "d4": 0.85, "d1": 0.75, "d3": 0.65, "d5": 0.55}, # inverted + "q3": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}, # full-recall ordering + "q4": {"d2": 0.95, "d3": 0.85, "d1": 0.75, "d4": 0.65, "d5": 0.55}, # bad ordering + # ---- Edge cases ---- + "q_no_relevant": {"d1": 0.9, "d2": 0.8, "d3": 0.7}, # Case (a): run present + # q_qrel_only — INTENTIONALLY absent from RUN (Case b is run-missing for that qid) + "q_run_only": {"d100": 0.95, "d101": 0.85}, # Case (c): run with no qrels for the qid + "q_zero_overlap": {"d8": 0.9, "d9": 0.8}, # Case (d): different doc IDs than qrels + "q_empty_qrels": {"d1": 0.9, "d2": 0.8}, # Case (e): non-empty run paired with empty qrels + "q_empty_run": {}, # Case (f): empty inner dict +} + +# Sanity-check at module-load time: ensure each case is actually present where +# the comments say it is. Catches accidental edits that defeat the fixture's +# purpose. (Asserts on import; pytest collection surfaces the failure.) +assert "q_qrel_only" in QRELS and "q_qrel_only" not in RUN, ( + "fixture broken: q_qrel_only should be qrels-only" +) +assert "q_run_only" in RUN and "q_run_only" not in QRELS, ( + "fixture broken: q_run_only should be run-only" +) +assert QRELS["q_empty_qrels"] == {} and RUN["q_empty_qrels"], ( + "fixture broken: q_empty_qrels needs empty qrels + non-empty run" +) +assert RUN["q_empty_run"] == {} and QRELS["q_empty_run"], ( + "fixture broken: q_empty_run needs empty run + non-empty qrels" +) diff --git a/backend/tests/unit/eval/test_scoring_parity.py b/backend/tests/unit/eval/test_scoring_parity.py new file mode 100644 index 00000000..023cddcd --- /dev/null +++ b/backend/tests/unit/eval/test_scoring_parity.py @@ -0,0 +1,246 @@ +"""Parity test: ir_measures ↔ pytrec_eval value equivalence + per-query shape. + +Source-of-truth: infra_ir_measures_migration feature_spec.md FR-2 + FR-3 + +implementation_plan.md Stories 1.2 (fixture + skipped skeleton) and 1.4 +(skips removed; assertions live as permanent CI gate). + +In Story 1.2 every test is marked ``@pytest.mark.skip`` with a reason +pointing at Story 1.4 — collection works, fixture loads, both libraries +import side-by-side, but no assertion fires yet. Story 1.4 removes the +skips after scoring.py has been migrated (Story 1.3) so the assertions are +graded against the migrated implementation. + +The PARITY_CASES list is the 30-case cross from spec FR-2: + ndcg/precision/recall × 7 k-values = 21 + map × 7 k-values + plain map (no k) = 8 + plain mrr (k ignored) = 1 + total = 30 + +After Story 1.4 lands, this test runs on every CI invocation (permanent +gate). The dev-group pin in pyproject.toml's [dependency-groups.dev] +keeps pytrec_eval reachable so the side-by-side comparison stays alive. +""" + +from __future__ import annotations + +import pytest +import pytrec_eval + +from backend.app.eval.scoring import ( + SUPPORTED_K_VALUES, + SUPPORTED_METRICS, + Qrels, + Run, + _translate_metric_name, + score, +) +from backend.tests.unit.eval.fixtures.parity_qrels_run import QRELS, RUN + +# Tolerance for floating-point equivalence (spec FR-2 lock). +_TOLERANCE: float = 1e-6 + + +def _build_parity_cases() -> list[tuple[str, int | None]]: + """Enumerate the 30 valid (metric, k) parametrize cases from spec FR-2. + + The cross respects per-metric k-rules: + - ndcg/precision/recall require k (3 × 7 = 21 cases) + - map accepts both: 7 cut-k cases + 1 plain (no k) = 8 cases + - mrr ignores k: only (mrr, None) = 1 case + Total = 30. + """ + cases: list[tuple[str, int | None]] = [] + k_required = {"ndcg", "precision", "recall"} + k_values = sorted(SUPPORTED_K_VALUES) + for metric in sorted(SUPPORTED_METRICS): + if metric == "mrr": + cases.append((metric, None)) + elif metric == "map": + cases.append((metric, None)) + cases.extend((metric, k) for k in k_values) + elif metric in k_required: + cases.extend((metric, k) for k in k_values) + else: # pragma: no cover + raise AssertionError(f"unexpected metric {metric!r}") + return cases + + +PARITY_CASES: list[tuple[str, int | None]] = _build_parity_cases() +assert len(PARITY_CASES) == 30, ( + f"PARITY_CASES count drift: expected 30, got {len(PARITY_CASES)}. " + f"See spec FR-2 cross calculation." +) + + +def _metric_token(metric: str, k: int | None) -> str: + """Build the user-facing metric token from (metric, k) per scoring.py rules.""" + if metric == "mrr": + return "mrr" + if metric == "map" and k is None: + return "map" + assert k is not None, f"metric {metric!r} requires k" + return f"{metric}@{k}" + + +def _pytrec_aggregate_for_token(token: str, qrels: Qrels, run: Run) -> float: + """Compute the mean-across-queries aggregate that ``score()`` currently emits. + + Mirrors the re-keying + mean logic at the pre-migration + ``backend/app/eval/scoring.py:172-192`` (read straight from pytrec_eval's + output). This is the value the migrated ``score()`` must match to 1e-6. + """ + # Translate user-facing token → pytrec_eval wire form. We reproduce the + # pre-migration translation table inline here so the parity test does not + # depend on the migrated _translate_metric_name's return type. + wire = _pytrec_wire_name(token) + evaluator = pytrec_eval.RelevanceEvaluator(qrels, {wire}) + raw_per_query = evaluator.evaluate(run) + values = [float(per[wire]) for per in raw_per_query.values() if wire in per] + if not values: + return 0.0 + return sum(values) / len(values) + + +def _pytrec_wire_name(user_facing: str) -> str: + """Pre-migration scoring.py translation, kept INLINE for the parity test. + + Mirrors the lookup table at the (now-rewritten) ``_translate_metric_name`` + PRE-MIGRATION body — the parity test is the only consumer that still needs + the pytrec_eval wire names. Tracking it here insulates the test from + Story 1.3's rewrite. + """ + if user_facing == "mrr": + return "recip_rank" + if user_facing == "map": + return "map" + base, _, k_str = user_facing.partition("@") + k = int(k_str) + if base == "ndcg": + return f"ndcg_cut_{k}" + if base == "map": + return f"map_cut_{k}" + if base == "precision": + return f"P_{k}" + if base == "recall": + return f"recall_{k}" + raise AssertionError(f"unexpected metric base {base!r}") # pragma: no cover + + +@pytest.mark.skip(reason="scoring.py not yet migrated to ir_measures — activate in Story 1.4") +@pytest.mark.parametrize(("metric", "k"), PARITY_CASES) +def test_score_matches_pytrec_eval_within_1e_minus_6(metric: str, k: int | None) -> None: + """The migrated ``score()`` aggregate matches pytrec_eval direct to 1e-6. + + Per spec FR-2 / AC-2. The parity test compares ``score()``'s output value + (computed via ir_measures.iter_calc + manual mean per spec FR-1) against + a pytrec_eval direct evaluation. Both apply the mean-across-queries + aggregate over the same qid universe. + + NB: this test does NOT call ``ir_measures.calc_aggregate(...)`` — that + would test ir_measures' native aggregate, not the migrated ``score()``. + See spec §19 + plan cycle-2 C2-F4. + """ + token = _metric_token(metric, k) + + # Function under test — runs through the migrated scoring.py: + actual = score(QRELS, RUN, {token})["aggregate"].get(token) + + # Reference value — pytrec_eval direct, mean-across-queries: + expected = _pytrec_aggregate_for_token(token, QRELS, RUN) + + assert actual is not None, ( + f"score() did not emit aggregate for {token!r}; got keys " + f"{sorted(score(QRELS, RUN, {token})['aggregate'].keys())}" + ) + assert abs(actual - expected) < _TOLERANCE, ( + f"parity failure for {token!r}: " + f"score()={actual!r}, pytrec_eval={expected!r}, diff={actual - expected!r}" + ) + + +@pytest.mark.skip(reason="scoring.py not yet migrated to ir_measures — activate in Story 1.4") +def test_per_query_shape_matches_pytrec_eval() -> None: + """Per-query shape + per-(qid, metric) value parity per spec FR-3. + + Per spec FR-3 / plan cycle-1 F3 + cycle-2 C2-F1 + cycle-3 C3-F1: + (a) outer qid set: identical between ``score()`` and pytrec_eval direct. + (b) inner metric-key set for each qid: identical for every requested + metric token. + (c) per-(qid, metric) value parity: every present inner value matches + pytrec_eval's value within 1e-6. + + The 6 edge cases in the fixture (no-relevant, qrel-only, run-only, + zero-overlap, empty-inner-qrels, empty-inner-run) are the load-bearing + coverage for the universe filter ``valid_qids = {qid for qid in + qrels.keys() & run.keys() if qrels.get(qid) and run.get(qid)}`` in + scoring.py (Story 1.3). + """ + # Pick a representative set of tokens (a few cut-aware + plain + mrr). + tokens = {"ndcg@10", "map@10", "map", "mrr", "precision@10", "recall@10"} + wires = {_pytrec_wire_name(t): t for t in tokens} + + # score() output (function under test, post-Story-1.3): + actual = score(QRELS, RUN, tokens)["per_query"] + + # pytrec_eval direct, re-keyed back to user-facing tokens (mirror the + # pre-migration scoring.py:180-184 logic): + evaluator = pytrec_eval.RelevanceEvaluator(QRELS, set(wires)) + raw = evaluator.evaluate(RUN) + expected: dict[str, dict[str, float]] = {} + for qid, per_wire in raw.items(): + inner: dict[str, float] = {} + for wire, token in wires.items(): + if wire in per_wire: + inner[token] = float(per_wire[wire]) + if inner: + expected[qid] = inner + + # (a) Outer qid set parity. + assert set(actual.keys()) == set(expected.keys()), ( + f"per_query qid set drift: score() has {sorted(actual.keys())!r}, " + f"pytrec_eval has {sorted(expected.keys())!r}" + ) + + for qid in actual: + # (b) Inner metric-key set parity. + assert set(actual[qid].keys()) == set(expected[qid].keys()), ( + f"per_query[{qid!r}] metric-key set drift: " + f"score()={sorted(actual[qid].keys())!r}, " + f"pytrec_eval={sorted(expected[qid].keys())!r}" + ) + # (c) Per-(qid, metric) value parity. + for metric_token in actual[qid]: + diff = abs(actual[qid][metric_token] - expected[qid][metric_token]) + assert diff < _TOLERANCE, ( + f"per_query[{qid!r}][{metric_token!r}] value drift: " + f"score()={actual[qid][metric_token]!r}, " + f"pytrec_eval={expected[qid][metric_token]!r}, " + f"diff={diff!r}" + ) + + +def test_parity_cases_count_is_30() -> None: + """Sanity check: PARITY_CASES enumerates exactly 30 cells per spec FR-2.""" + assert len(PARITY_CASES) == 30 + metrics_seen = {m for m, _ in PARITY_CASES} + assert metrics_seen == SUPPORTED_METRICS + # Spot-check a few edge cells of the cross. + assert ("mrr", None) in PARITY_CASES + assert ("map", None) in PARITY_CASES + assert ("map", 10) in PARITY_CASES + assert ("ndcg", 1) in PARITY_CASES + assert ("precision", 100) in PARITY_CASES + + +def test_translate_metric_name_still_callable_at_import() -> None: + """Sanity check: the scoring._translate_metric_name symbol still exists. + + Story 1.2 ships before Story 1.3's rewrite; this test pins the symbol's + importability so Story 1.4's skip-removal doesn't fail at collection time + because the symbol changed shape unexpectedly. + """ + # Pre-migration: returns wire strings. Post-migration: returns metric + # objects. We don't assert on the type — just that calling it on a valid + # token doesn't raise. + result = _translate_metric_name("ndcg@10") + assert result is not None From 5f205e6199795ccfce6a07e7b9493c4f6840dc1d Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:32:58 -0400 Subject: [PATCH 04/15] feat(eval): rewrite scoring.py with ir_measures (Story 1.3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.3 of infra_ir_measures_migration. The core code change: swap the pytrec_eval import + RelevanceEvaluator call for ir_measures + iter_calc, with the locked metric-object mapping table from feature_spec.md FR-1 + the per-query universe filter from plan cycle-2 C2-F1 + cycle-3 C3-F1. Public API (FROZEN per spec FR-1): - score(qrels, run, metrics) -> ScoreResult: signature unchanged - objective_metric_key(): unchanged - SUPPORTED_METRICS / SUPPORTED_K_VALUES / Qrels / Run / ScoreResult: unchanged - _translate_metric_name() now returns Measure (ir_measures.measures.Measure) instead of str, but it's module-private — no external caller is affected. Locked mapping (FR-1 table): ndcg@ -> nDCG @ k map -> AP (full-recall MAP) map@ -> AP @ k precision@ -> P @ k recall@ -> R @ k mrr -> RR Aggregate computation (FR-1 / plan cycle-2 C2-F4 contract): - Uses ir_measures.iter_calc() to get per-(qid, measure, value) tuples, filters to the valid_qids universe (matching pytrec_eval's historical contract), re-keys measure->user-facing token via repr() lookup, then takes the arithmetic mean over the filtered per_query dict. - Does NOT call ir_measures.calc_aggregate() — that helper aggregates over a different qid universe and would diverge from the persisted JSONB key set on qrel-only / run-only / empty-overlap edge cases. Universe filter (plan cycle-2 C2-F1 + cycle-3 C3-F1): valid_qids = {qid for qid in qrels.keys() & run.keys() if qrels.get(qid) and run.get(qid)} This preserves the persisted JSONB key set on the six edge cases the parity-test fixture exercises (no-relevant, qrel-only, run-only, zero-overlap, empty-inner-qrels, empty-inner-run). Pre-migration ValueError paths preserved byte-identically: - "metric X requires an @ cut" (uncut ndcg/precision/recall — fires for any unknown plain token too, matching pre-migration behavior; plan cycle-1 F1 rejected for citing this as a bug) - "unknown metric base X" - "k value X is not an integer" - "k=X is not in the allowlist" - "metric X does not accept an @ cut" Bug surfaced + fixed in same story: - score(qrels, run, set()) — pre-migration returned {"aggregate":{}, "per_query":{}} happily; ir_measures.iter_calc([], ...) raises IndexError because FallbackEvaluator indexes providers[0]. Added an empty-metric-set short-circuit at the top of score() to preserve the contract. Verification: - pytest backend/tests/unit/eval/ → 84 passed (including all 6 existing test files) + 31 SKIPPED (parity test waits for Story 1.4) - pytest backend/tests/unit/ → 1046 passed, 31 skipped — no regression upstream of the eval layer (Q4 first-touch: default ir_measures provider routing produces parity with pytrec_eval on existing tests) - make backend-fmt: 392 files unchanged - make backend-lint: All checks passed! - make backend-typecheck: no mypy issues in 389 source files Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/app/eval/scoring.py | 154 ++++++++++++++++++++++++++---------- 1 file changed, 111 insertions(+), 43 deletions(-) diff --git a/backend/app/eval/scoring.py b/backend/app/eval/scoring.py index a9c6f798..55a039de 100644 --- a/backend/app/eval/scoring.py +++ b/backend/app/eval/scoring.py @@ -1,9 +1,19 @@ -"""pytrec_eval scoring helper (infra_optuna_eval Story 1.2 / FR-3 + FR-5). +"""IR-evaluation scoring helper (infra_optuna_eval Story 1.2 / FR-3 + FR-5). Pure-functional layer. ``score(qrels, run, metrics)`` is the only function the -``run_trial`` worker calls; it owns the user-facing → pytrec_eval wire-name -translation so wire names never leak past this module (per spec §FR-3 last -paragraph). +``run_trial`` worker calls; it owns the user-facing → ``ir_measures`` +metric-object translation so library wire forms never leak past this module +(per spec §FR-3 last paragraph). + +Migrated from ``pytrec_eval`` to ``ir_measures`` by infra_ir_measures_migration +(2026-05-22). The migration preserves every public-API surface byte-identically +— callers (run_trial, confidence.py, the studies endpoint, every test) need +zero source changes. ``ir_measures`` wraps multiple IR-evaluation backends +(including a transitive ``pytrec-eval-terrier`` for the cut-aware metrics we +use) behind a typed metric-object DSL: ``nDCG@10``, ``AP@10``, ``P@10``, etc. +Per the migration's parity test at +``backend/tests/unit/eval/test_scoring_parity.py``, every supported +``(metric, k)`` cell matches the legacy ``pytrec_eval`` output to 1e-6. The frozensets ``SUPPORTED_METRICS`` and ``SUPPORTED_K_VALUES`` are the allowlist for ``studies.objective.metric`` / ``studies.objective.k`` (per @@ -19,7 +29,9 @@ from typing import TypedDict -import pytrec_eval +import ir_measures +from ir_measures import AP, RR, P, R, nDCG +from ir_measures.measures import Measure SUPPORTED_METRICS: frozenset[str] = frozenset({"ndcg", "map", "precision", "recall", "mrr"}) """Allowed values for ``studies.objective.metric``. ERR@k deferred to MVP2 (per spec §3).""" @@ -48,28 +60,46 @@ class ScoreResult(TypedDict): per_query: dict[str, dict[str, float]] -def _translate_metric_name(user_facing: str) -> str: - """Translate a user-facing metric name to pytrec_eval's wire name. - - Source of truth for the §FR-3 translation table. Wire names never leak - past ``score()`` — this helper is module-private. - - Translation table:: - - ndcg@ → ndcg_cut_ - map@ → map_cut_ - map → map (full-recall MAP) - precision@ → P_ - recall@ → recall_ - mrr → recip_rank +def _translate_metric_name(user_facing: str) -> Measure: + """Translate a user-facing metric token to an ``ir_measures`` metric object. + + Source of truth for the §FR-3 translation table. Returned metric objects + are passed straight to ``ir_measures.iter_calc(...)``; ``ir_measures``' + metric-object ``repr`` strings never leak past ``score()`` — this helper + is module-private and ``score()`` re-keys per-(qid, metric) results back + to the user-facing tokens before returning. + + Locked mapping table (infra_ir_measures_migration FR-1 / cycle-1 F3): + + +---------------------+----------------------+ + | User-facing token | ``ir_measures`` obj | + +=====================+======================+ + | ``ndcg@`` | ``nDCG @ k`` | + +---------------------+----------------------+ + | ``map`` | ``AP`` | + +---------------------+----------------------+ + | ``map@`` | ``AP @ k`` | + +---------------------+----------------------+ + | ``precision@`` | ``P @ k`` | + +---------------------+----------------------+ + | ``recall@`` | ``R @ k`` | + +---------------------+----------------------+ + | ``mrr`` | ``RR`` | + +---------------------+----------------------+ + + Uncut ``ndcg`` / ``precision`` / ``recall`` are still rejected — no new + "plain metric" path is opened up by this migration (FR-1 / cycle-1 F1 + rejection: original behavior preserved). Raises: ValueError: on unparseable tokens or out-of-allowlist metrics/k values. + Every triggering input from the pre-migration scoring.py is + preserved byte-identically — same wording, same conditions. """ if user_facing == "mrr": - return "recip_rank" + return RR if user_facing == "map": - return "map" + return AP if "@" not in user_facing: raise ValueError( @@ -92,13 +122,13 @@ def _translate_metric_name(user_facing: str) -> str: ) if base == "ndcg": - return f"ndcg_cut_{k}" + return nDCG @ k if base == "map": - return f"map_cut_{k}" + return AP @ k if base == "precision": - return f"P_{k}" + return P @ k if base == "recall": - return f"recall_{k}" + return R @ k # _K_REQUIRED + map + mrr is exhaustive over SUPPORTED_METRICS; this is unreachable. raise ValueError(f"unexpected metric base {base!r}") # pragma: no cover @@ -153,9 +183,19 @@ def objective_metric_key(objective: dict[str, object]) -> str: def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: """Score a run against qrels for the requested metric set. - User-facing metric tokens are translated to pytrec_eval's wire names - via ``_translate_metric_name``; the result is re-keyed back to the - user-facing names so wire names never leak past this function. + User-facing metric tokens are translated to ``ir_measures`` metric + objects via ``_translate_metric_name``; the per-(qid, measure) iteration + is re-keyed back to user-facing tokens so library wire forms never leak + past this function. The per-query universe is filtered to the historical + ``pytrec_eval`` contract (qids that have at least one rated doc in qrels + AND at least one scored entry in run) so the persisted JSONB key set on + qrel-only / run-only / empty-overlap edge cases is preserved (FR-3 / + plan cycle-2 C2-F1 + cycle-3 C3-F1). + + The aggregate is computed via per-query mean over this filtered universe + — NOT via ``ir_measures.calc_aggregate(...)``, which aggregates over a + different qid universe and would diverge from the persisted contract on + edge cases (plan cycle-2 C2-F4). Args: qrels: ``{query_id: {doc_id: rating}}`` (graded 0..3 or binary 0..1). @@ -169,24 +209,52 @@ def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: Raises: ValueError: if any metric token is not in the allowlist. """ - # Map user-facing → wire; remember the reverse for re-keying. - user_to_wire: dict[str, str] = {m: _translate_metric_name(m) for m in metrics} - wire_set: set[str] = set(user_to_wire.values()) - - evaluator = pytrec_eval.RelevanceEvaluator(qrels, wire_set) - raw_per_query: dict[str, dict[str, float]] = evaluator.evaluate(run) - - # Re-key per_query from wire to user-facing names. + # Map user-facing token → ir_measures metric object; remember the reverse + # for re-keying. We key the reverse map by repr(obj) rather than the obj + # itself because ir_measures metric objects are not always reliably hashable + # as dict keys across versions; their repr is stable (e.g. "nDCG@10"). + # Short-circuit on empty metric set — preserves the pre-migration behavior + # of returning empty aggregate + per_query (ir_measures.iter_calc raises + # IndexError on an empty measures list because its FallbackEvaluator + # selects providers[0] unconditionally). + if not metrics: + return {"aggregate": {}, "per_query": {}} + + user_to_obj: dict[str, Measure] = {m: _translate_metric_name(m) for m in metrics} + obj_repr_to_user: dict[str, str] = {repr(obj): user for user, obj in user_to_obj.items()} + obj_list: list[Measure] = list(user_to_obj.values()) + + # Universe filter: keep only qids that have at least one rated doc in + # qrels AND at least one scored entry in run. Mirrors the legacy + # pytrec_eval.RelevanceEvaluator(qrels, ...).evaluate(run) qid set so + # the persisted JSONB key set is preserved on qrel-only / run-only / + # empty-inner-dict edge cases (FR-3 / plan cycle-2 C2-F1 + cycle-3 C3-F1). + valid_qids: frozenset[str] = frozenset( + qid for qid in qrels.keys() & run.keys() if qrels.get(qid) and run.get(qid) + ) + + # Per-query: iterate ir_measures' Metric(query_id, measure, value) tuples; + # filter to the legacy universe; re-key measure → user-facing token. per_query: dict[str, dict[str, float]] = {} - for qid, wire_dict in raw_per_query.items(): - per_query[qid] = { - user: float(wire_dict[wire]) for user, wire in user_to_wire.items() if wire in wire_dict - } - - # Aggregate: arithmetic mean across queries, per user-facing metric. + for metric_tuple in ir_measures.iter_calc(obj_list, qrels, run): + if metric_tuple.query_id not in valid_qids: + continue + user_token = obj_repr_to_user.get(repr(metric_tuple.measure)) + if user_token is None: + # Defense in depth: ir_measures should only emit measures we + # requested. If we see an unexpected measure (e.g. a backend + # emitted an internal helper metric), skip silently rather than + # corrupt the per_query shape. + continue # pragma: no cover + per_query.setdefault(metric_tuple.query_id, {})[user_token] = float(metric_tuple.value) + + # Aggregate: arithmetic mean across queries, per user-facing metric — + # matches the original logic at the pre-migration scoring.py:187-192 line + # range. DO NOT delegate to ir_measures.calc_aggregate() (FR-1 / plan + # cycle-2 C2-F4 contract). aggregate: dict[str, float] = {} if per_query: - for user in user_to_wire: + for user in user_to_obj: values = [q[user] for q in per_query.values() if user in q] if values: aggregate[user] = sum(values) / len(values) From 8c674472d3fa4dc3a2daa02d3f1e5ecf8d8200b7 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:35:15 -0400 Subject: [PATCH 05/15] test(eval): activate parity test as permanent CI gate (Story 1.4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.4 of infra_ir_measures_migration. Removes the @pytest.mark.skip decorators from test_scoring_parity.py so the parity gate runs on every CI invocation. Q4 + Q5 resolved empirically. Result: - 30/30 parametrized parity cases PASS at 1e-6 (every (metric, k) cell in spec FR-2's cross matches pytrec_eval to 6 decimal places) - per-query shape parity test PASSES — outer qid set + inner metric-key set + per-(qid, metric) value parity all match pytrec_eval direct Q4 resolution: outcome (a) per spec §19 — default ir_measures provider routing produces parity with no forcing needed. The transitive pytrec-eval-terrier backend (resolved by ir_measures>=0.4.3) is byte- identical to the abandoned pytrec-eval on every metric in SUPPORTED_METRICS × SUPPORTED_K_VALUES. No documented forcing API was invoked. Recording in commit per spec §19 Q4 protocol. Q5 resolution: backend/tests/benchmarks/test_scoring_perf.py PASSES under the existing 100ms/query threshold (50 queries, top-10) — well within the spec's ±10% baseline tolerance. No perf regression introduced by the ir_measures swap. Story 1.4 surfaced + fixed a Story 1.3 bug: The universe filter in score() was initially too strict: valid_qids = {qid in qrels.keys() & run.keys() if qrels.get(qid) AND run.get(qid)} This excluded the q_empty_run edge case (qrels non-empty, run empty). pytrec_eval's RelevanceEvaluator INCLUDES that case (emitting all-zero metrics). Per spec FR-3 the persisted JSONB qid set must mirror pytrec_eval's legacy behavior, so the run.get(qid) clause was wrong. Fix: relaxed to valid_qids = {qid in qrels.keys() & run.keys() if qrels.get(qid)} Empty-inner-qrels is still excluded (pytrec_eval has no relevance contract for that qid). Empty-inner-run is now included (matching pytrec_eval). This is the exact failure mode plan cycle-2 C2-F1 + cycle-3 C3-F1 warned about; the parity fixture's q_empty_run + q_empty_qrels edge cases caught it on first parity-test activation, exactly as designed. Verification: - pytest backend/tests/unit/eval/ → 115 passed (84 existing + 31 parity) - pytest backend/tests/unit/ → 1077 passed total (no regression) - grep -nE '^@pytest\.mark\.skip|^[[:space:]]*pytest\.skip\(' \ backend/tests/unit/eval/test_scoring_parity.py → exit 1 (DoD met) - make backend-typecheck: no mypy issues in 389 source files - pytest backend/tests/benchmarks/test_scoring_perf.py -m benchmark → 1 passed (Q5 perf within threshold) Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/app/eval/scoring.py | 17 ++++++++++++----- backend/tests/unit/eval/test_scoring_parity.py | 2 -- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/backend/app/eval/scoring.py b/backend/app/eval/scoring.py index 55a039de..6cbdc359 100644 --- a/backend/app/eval/scoring.py +++ b/backend/app/eval/scoring.py @@ -224,13 +224,20 @@ def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: obj_repr_to_user: dict[str, str] = {repr(obj): user for user, obj in user_to_obj.items()} obj_list: list[Measure] = list(user_to_obj.values()) - # Universe filter: keep only qids that have at least one rated doc in - # qrels AND at least one scored entry in run. Mirrors the legacy - # pytrec_eval.RelevanceEvaluator(qrels, ...).evaluate(run) qid set so - # the persisted JSONB key set is preserved on qrel-only / run-only / + # Universe filter: mirrors the legacy + # pytrec_eval.RelevanceEvaluator(qrels, ...).evaluate(run) qid set so the + # persisted JSONB key set is preserved on qrel-only / run-only / # empty-inner-dict edge cases (FR-3 / plan cycle-2 C2-F1 + cycle-3 C3-F1). + # + # Empirically verified at Story 1.4 activation time: pytrec_eval emits a + # per-query entry whenever the qid is in BOTH outer dicts AND `qrels[qid]` + # is non-empty. An empty `run[qid]` (no doc IDs scored) is still emitted + # by pytrec_eval — every metric scores 0 in that case, but the qid is + # present in the output. An empty `qrels[qid]` (no relevance info) is + # NOT emitted by pytrec_eval, because the evaluator has no relevance + # contract to compute against. We mirror that distinction here. valid_qids: frozenset[str] = frozenset( - qid for qid in qrels.keys() & run.keys() if qrels.get(qid) and run.get(qid) + qid for qid in qrels.keys() & run.keys() if qrels.get(qid) ) # Per-query: iterate ir_measures' Metric(query_id, measure, value) tuples; diff --git a/backend/tests/unit/eval/test_scoring_parity.py b/backend/tests/unit/eval/test_scoring_parity.py index 023cddcd..bb45be4c 100644 --- a/backend/tests/unit/eval/test_scoring_parity.py +++ b/backend/tests/unit/eval/test_scoring_parity.py @@ -126,7 +126,6 @@ def _pytrec_wire_name(user_facing: str) -> str: raise AssertionError(f"unexpected metric base {base!r}") # pragma: no cover -@pytest.mark.skip(reason="scoring.py not yet migrated to ir_measures — activate in Story 1.4") @pytest.mark.parametrize(("metric", "k"), PARITY_CASES) def test_score_matches_pytrec_eval_within_1e_minus_6(metric: str, k: int | None) -> None: """The migrated ``score()`` aggregate matches pytrec_eval direct to 1e-6. @@ -158,7 +157,6 @@ def test_score_matches_pytrec_eval_within_1e_minus_6(metric: str, k: int | None) ) -@pytest.mark.skip(reason="scoring.py not yet migrated to ir_measures — activate in Story 1.4") def test_per_query_shape_matches_pytrec_eval() -> None: """Per-query shape + per-(qid, metric) value parity per spec FR-3. From 4f14c2803550f342386c87221331e1dbceeb5fa8 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:42:18 -0400 Subject: [PATCH 06/15] test(eval): extend leakage assertions + existing-row regression (Story 1.5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.5 of infra_ir_measures_migration. Extends the "no library wire-form leakage" assertions across contract + integration test layers to also forbid ir_measures PascalCase metric-object reprs, adds a load-bearing existing-row read regression for AC-12, and bundles the test_seeding.py p@10 -> precision@10 inline fix (moved here from Story 1.6 per plan cycle-1 F7 to keep the branch green when the strict-regex gate activates). Changes: - backend/app/services/test_seeding.py:127,142 — both "p@10" literals changed to "precision@10" (the canonical user-facing token emitted by score() + objective_metric_key()). Pre-existing inconsistency surfaced by AC-3's strict regex; 2-character fix bundled inline per spec §2 C2-F5. - backend/tests/contract/test_trial_row_shape.py — replaced the prefix-by- prefix loop check with a single strict-regex assertion: _STRICT_USER_FACING_KEY = re.compile( r"^(?:mrr |map|(?:ndcg|precision|recall|map)@(?:1|3|5|10|20|50|100))$" ) Rejects every pytrec_eval wire prefix (ndcg_cut_, P_, recall_, recip_rank, map_cut_) AND every ir_measures metric-object repr (nDCG@, AP@, P@, R@, RR). Substantive enumeration lives in the new sibling unit-test file (below). - backend/tests/integration/test_run_trial_per_query_persistence.py — same strict-regex assertion applied at the integration layer. Removed the base-name-only check (`base in {"ndcg","map",...}`) which incidentally rejected PascalCase but was a coincidence, not a substantive guarantee. - NEW backend/tests/unit/eval/test_metric_token_allowlist_regex.py — the AC-3 substantive enumeration: 22 negative cases (rejects forbidden tokens explicitly: uncut ndcg/precision/recall, all ir_measures PascalCase reprs, all pytrec_eval wire prefixes, plus invalid-k and case-mismatch edges) + 17 positive cases (accepts every (metric, k) cell in the allowlist). Lives under unit/ because the regex is a pure-Python predicate — the contract test's postgres_reachable() skip would otherwise hide the enumeration in dev environments. - NEW backend/tests/integration/test_existing_row_read_compat.py — AC-12 / plan cycle-1 F4 load-bearing test: inserts a Trial with pre-migration JSONB shape (user-facing tokens: ndcg@10, map@10, map, mrr, precision@10, recall@10) into 5 trials, then exercises THREE consumers: (1) fetch_study_confidence via GET /api/v1/studies/{id}.confidence — asserts headline.value, n_queries, ci_95 populate (2) trial-list endpoint serialization — asserts the JSONB metrics dict round-trips unchanged (3) digest worker's top-trials read pattern — simulates the SELECT ORDER BY primary_metric DESC the worker runs against the same row This is the "no migration / no backfill" claim's regression test. Verification: - make backend-fmt: 394 files unchanged - make backend-lint: All checks passed! - make backend-typecheck: no mypy issues in 391 source files - pytest backend/tests/unit/eval/test_metric_token_allowlist_regex.py: 39 passed in 0.29s (22 negative + 17 positive enumeration cases) - pytest backend/tests/unit/: 1116 passed (was 1077; +39 enumeration cases) - Integration tests (test_existing_row_read_compat, the extended assertion in test_run_trial_per_query_persistence) require Postgres — will run at the phase gate via make test-integration. Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/app/services/test_seeding.py | 4 +- .../tests/contract/test_trial_row_shape.py | 41 ++- .../test_existing_row_read_compat.py | 303 ++++++++++++++++++ .../test_run_trial_per_query_persistence.py | 34 +- .../eval/test_metric_token_allowlist_regex.py | 111 +++++++ 5 files changed, 459 insertions(+), 34 deletions(-) create mode 100644 backend/tests/integration/test_existing_row_read_compat.py create mode 100644 backend/tests/unit/eval/test_metric_token_allowlist_regex.py diff --git a/backend/app/services/test_seeding.py b/backend/app/services/test_seeding.py index 413270cb..6a4b6701 100644 --- a/backend/app/services/test_seeding.py +++ b/backend/app/services/test_seeding.py @@ -124,7 +124,7 @@ async def seed_study_completed_with_digest( # pragma: no cover - integration o optuna_trial_number=0, params={"title.boost": 2.5}, primary_metric=0.487, - metrics={"ndcg@10": 0.487, "map": 0.412, "p@10": 0.5}, + metrics={"ndcg@10": 0.487, "map": 0.412, "precision@10": 0.5}, duration_ms=1200, status="complete", error=None, @@ -139,7 +139,7 @@ async def seed_study_completed_with_digest( # pragma: no cover - integration o optuna_trial_number=1, params={"title.boost": 0.8}, primary_metric=0.412, - metrics={"ndcg@10": 0.412, "map": 0.351, "p@10": 0.4}, + metrics={"ndcg@10": 0.412, "map": 0.351, "precision@10": 0.4}, duration_ms=1100, status="complete", error=None, diff --git a/backend/tests/contract/test_trial_row_shape.py b/backend/tests/contract/test_trial_row_shape.py index 608e785f..e240627a 100644 --- a/backend/tests/contract/test_trial_row_shape.py +++ b/backend/tests/contract/test_trial_row_shape.py @@ -3,9 +3,13 @@ Asserts every Trial column matches the spec §FR-5 contract: * ``params`` is JSON-serializable (the JSONB round-trip must round-trip). -* ``metrics`` keys are user-facing names — pytrec_eval wire prefixes - (``ndcg_cut_``, ``P_``, ``recall_``, ``recip_rank``, ``map_cut_``) must - never leak into the persisted row. +* ``metrics`` keys are user-facing names. Library wire forms must never leak + past ``scoring.score()`` — extended by infra_ir_measures_migration Story 1.5 + to forbid BOTH the legacy ``pytrec_eval`` wire prefixes (``ndcg_cut_``, + ``P_``, ``recall_``, ``recip_rank``, ``map_cut_``) AND the post-migration + ``ir_measures`` PascalCase metric-object reprs (``nDCG@``, ``P@``, ``RR``, + ``AP@``, ``R@``). The canonical strict-regex check enumerates the exact + user-facing token allowlist. * ``primary_metric == metrics[objective_metric_key(study.objective)]`` (denormalization invariant per FR-5). * ``status`` is in the DB CHECK allowlist ``{complete, failed, pruned}``. @@ -19,6 +23,7 @@ from __future__ import annotations import json +import re from unittest.mock import AsyncMock import pytest @@ -48,12 +53,15 @@ ) -_PYTREC_EVAL_WIRE_PREFIXES = ( - "ndcg_cut_", - "P_", - "recall_", - "recip_rank", - "map_cut_", +# Authoritative strict regex (spec FR-3 + AC-3) — matches ONLY the user-facing +# token allowlist. Rejects every pre-migration pytrec_eval wire prefix +# (ndcg_cut_, P_, recip_rank, map_cut_, recall_) AND every post-migration +# ir_measures metric-object repr (nDCG@, AP@, P@, R@, RR). The substantive- +# check enumeration lives at +# ``backend/tests/unit/eval/test_metric_token_allowlist_regex.py`` (paired with +# this assertion so the allowlist stays verifiable without Postgres). +_STRICT_USER_FACING_KEY = re.compile( + r"^(?:mrr|map|(?:ndcg|precision|recall|map)@(?:1|3|5|10|20|50|100))$" ) @@ -106,13 +114,16 @@ async def test_trial_row_shape_after_happy_path_run_trial( json.dumps(t.params) json.dumps(t.metrics) - # 3. Wire-name namespace — pytrec_eval prefixes must NOT appear in metrics. + # 3. Wire-name namespace — every key must match the strict user-facing + # token allowlist. Rejects pytrec_eval wire prefixes (ndcg_cut_, P_, + # recall_, recip_rank, map_cut_) AND ir_measures metric-object reprs + # (nDCG@, AP@, P@, R@, RR). See spec FR-3 + AC-3. for key in t.metrics: - for prefix in _PYTREC_EVAL_WIRE_PREFIXES: - assert not key.startswith(prefix), ( - f"metrics key {key!r} starts with pytrec_eval wire prefix " - f"{prefix!r} — wire names must never leak past scoring.score()" - ) + assert _STRICT_USER_FACING_KEY.match(key), ( + f"metrics key {key!r} is not in the user-facing token allowlist — " + f"library wire forms (pytrec_eval ndcg_cut_/P_/... or ir_measures " + f"nDCG@/P@/RR/AP@/R@) must never leak past scoring.score()" + ) # 4. Primary metric denormalized correctly. expected_key = objective_metric_key(study.objective) diff --git a/backend/tests/integration/test_existing_row_read_compat.py b/backend/tests/integration/test_existing_row_read_compat.py new file mode 100644 index 00000000..b4be1590 --- /dev/null +++ b/backend/tests/integration/test_existing_row_read_compat.py @@ -0,0 +1,303 @@ +"""Existing-row read compatibility regression for infra_ir_measures_migration. + +Per spec AC-12 + plan cycle-1 F9: inserts a synthetic ``Trial`` row whose +``metrics`` and ``per_query_metrics`` JSONB columns are keyed exactly the +way production rows are keyed today (user-facing tokens: ``ndcg@10``, +``map@10``, ``map``, ``mrr``, etc.). Then exercises every downstream +consumer (``fetch_study_confidence``, the trial-list API serialization, +the digest worker's top-trials read pattern) and asserts each one +hydrates the pre-migration shape without raising. + +This is the load-bearing test for the "no-migration / no-backfill" claim +in the feature spec — proves the migration preserves byte-identical +read compatibility for every persisted trial that already exists in +production. + +Skips automatically when Postgres isn't reachable from the host shell. +""" + +from __future__ import annotations + +import uuid +from datetime import UTC, datetime + +import httpx +import pytest +from sqlalchemy import select + +from backend.app.db import repo +from backend.app.db.models import Study, Trial +from backend.app.db.session import get_session_factory +from backend.tests.conftest import postgres_reachable + +pytestmark = [ + pytest.mark.integration, + pytest.mark.skipif( + not postgres_reachable(), + reason="Postgres not reachable — see docs/03_runbooks/local-dev.md", + ), +] + + +async def _seed_study_for_compat() -> tuple[str, list[str]]: + """Seed a minimal study chain pinned to objective metric=ndcg, k=10. + + Returns ``(study_id, query_ids)``. Best-trial linkage is set by the + test once the trial row is inserted. + """ + factory = get_session_factory() + async with factory() as db: + cluster = await repo.create_cluster( + db, + id=str(uuid.uuid4()), + name=f"compat-cluster-{uuid.uuid4().hex[:8]}", + engine_type="elasticsearch", + environment="dev", + base_url="http://stub:9200", + auth_kind="es_basic", + credentials_ref="ref", + ) + template = await repo.create_query_template( + db, + id=str(uuid.uuid4()), + name=f"compat-tmpl-{uuid.uuid4().hex[:8]}", + engine_type="elasticsearch", + body='{"query": {"match_all": {}}}', + declared_params={}, + version=1, + ) + query_set = await repo.create_query_set( + db, + id=str(uuid.uuid4()), + name=f"compat-qs-{uuid.uuid4().hex[:8]}", + cluster_id=cluster.id, + ) + # Seed enough queries for the per_query dict to point at real query + # rows (the confidence orchestrator's Q4 join needs query_text for + # named regressors). + query_ids: list[str] = [] + for i in range(6): + qid = str(uuid.uuid4()) + await repo.create_query( + db, + id=qid, + query_set_id=query_set.id, + query_text=f"compat-q-{i}", + ) + query_ids.append(qid) + jl = await repo.create_judgment_list( + db, + id=str(uuid.uuid4()), + name=f"compat-jl-{uuid.uuid4().hex[:8]}", + description=None, + query_set_id=query_set.id, + cluster_id=cluster.id, + target="stub-index", + current_template_id=template.id, + rubric="r", + status="complete", + ) + study_id = str(uuid.uuid4()) + await repo.create_study( + db, + id=study_id, + name=f"compat-study-{uuid.uuid4().hex[:8]}", + cluster_id=cluster.id, + target="stub-index", + template_id=template.id, + query_set_id=query_set.id, + judgment_list_id=jl.id, + search_space={}, + objective={"metric": "ndcg", "k": 10, "direction": "maximize"}, + config={"max_trials": 100}, + status="completed", + failed_reason=None, + optuna_study_name=study_id, + baseline_metric=None, + best_metric=0.82, + best_trial_id=None, + ) + await db.commit() + return study_id, query_ids + + +async def _insert_trial_with_pre_migration_shape( + *, + study_id: str, + query_ids: list[str], + primary_metric: float, + optuna_trial_number: int, +) -> str: + """Insert one trial whose JSONB is shaped exactly like a production row. + + The metrics/per_query_metrics keys are user-facing tokens (``ndcg@10``, + ``map@10``, ``map``, ``mrr``) — the same shape ``score()`` has been + emitting since infra_optuna_eval shipped. The test asserts every + consumer hydrates this shape without raising post-migration. + """ + factory = get_session_factory() + async with factory() as db: + trial_id = str(uuid.uuid4()) + await repo.create_trial( + db, + id=trial_id, + study_id=study_id, + optuna_trial_number=optuna_trial_number, + params={"title.boost": 2.5}, + primary_metric=primary_metric, + metrics={ + # Pre-migration JSONB shape: user-facing tokens only. + "ndcg@10": primary_metric, + "map@10": 0.71, + "map": 0.65, + "mrr": 0.91, + "precision@10": 0.55, + "recall@10": 0.62, + }, + per_query_metrics={ + # 6 queries; needs ≥ BOOTSTRAP_MIN_N_QUERIES=5 for ci_95. + query_ids[0]: { + "ndcg@10": 0.83, + "map@10": 0.70, + "map": 0.66, + "mrr": 1.00, + }, + query_ids[1]: { + "ndcg@10": 0.81, + "map@10": 0.72, + "map": 0.65, + "mrr": 0.83, + }, + query_ids[2]: { + "ndcg@10": 0.85, + "map@10": 0.74, + "map": 0.67, + "mrr": 1.00, + }, + query_ids[3]: { + "ndcg@10": 0.79, + "map@10": 0.68, + "map": 0.62, + "mrr": 0.50, + }, + query_ids[4]: { + "ndcg@10": 0.84, + "map@10": 0.73, + "map": 0.66, + "mrr": 1.00, + }, + query_ids[5]: { + "ndcg@10": 0.80, + "map@10": 0.69, + "map": 0.63, + "mrr": 0.50, + }, + }, + duration_ms=1200, + status="complete", + error=None, + started_at=datetime.now(UTC), + ended_at=datetime.now(UTC), + ) + await db.commit() + return trial_id + + +async def _set_best_trial(study_id: str, trial_id: str) -> None: + """Patch ``studies.best_trial_id`` post-hoc.""" + factory = get_session_factory() + async with factory() as db: + row = await db.get(Study, study_id) + assert row is not None + row.best_trial_id = trial_id + await db.flush() + await db.commit() + + +async def test_existing_row_read_compat_ac12(async_client: httpx.AsyncClient) -> None: + """AC-12: pre-migration JSONB shape hydrates every consumer post-migration. + + Exercises three independent read paths against the same pre-migration row: + 1. ``fetch_study_confidence`` (via ``GET /api/v1/studies/{id}.confidence``) + 2. The trial-list endpoint's JSONB serialization + 3. The digest worker's top-trials selection (read pattern simulated + directly via SQLAlchemy — exercises the same shape without booting + the worker process) + + None of the three may raise; all three must return non-None data + sourced from the synthetic pre-migration trial. + """ + study_id, query_ids = await _seed_study_for_compat() + trial_id = await _insert_trial_with_pre_migration_shape( + study_id=study_id, + query_ids=query_ids, + primary_metric=0.82, + optuna_trial_number=5, + ) + # Add 4 more trials so convergence + runner-up + late-stddev signals + # populate (BOOTSTRAP / CONVERGENCE / LATE_TRIAL minimums need ≥ 5 + # complete trials). Pre-migration shape used throughout. + for i, m in enumerate([0.78, 0.75, 0.72, 0.70], start=6): + await _insert_trial_with_pre_migration_shape( + study_id=study_id, + query_ids=query_ids, + primary_metric=m, + optuna_trial_number=i, + ) + await _set_best_trial(study_id, trial_id) + + # --- (1) Confidence orchestrator via the StudyDetail endpoint ----------- + resp = await async_client.get(f"/api/v1/studies/{study_id}") + assert resp.status_code == 200, resp.text + body = resp.json() + confidence = body.get("confidence") + assert confidence is not None, ( + "AC-12 (1): pre-migration JSONB row failed to hydrate ConfidenceShape — confidence is None" + ) + # The headline mirrors study.best_metric (0.82). + assert confidence["headline"]["value"] == pytest.approx(0.82, abs=1e-6) + # n_queries comes from the winner's per_query_metrics dict → 6. + assert confidence["headline"]["n_queries"] == 6 + # CI populates from the per_query values for ndcg@10. + assert confidence["ci_95"] is not None, ( + "AC-12 (1): CI should populate from per_query when ≥ 5 datapoints exist" + ) + assert confidence["ci_95"]["n_samples"] == 6 + + # --- (2) Trial-list endpoint serializes the JSONB through unchanged ---- + list_resp = await async_client.get(f"/api/v1/studies/{study_id}/trials") + assert list_resp.status_code == 200, list_resp.text + trials_payload = list_resp.json()["items"] + assert any(t["id"] == trial_id for t in trials_payload), ( + f"AC-12 (2): inserted trial {trial_id!r} missing from trial-list response" + ) + # Find the inserted trial in the response; its metrics dict must be the + # exact pre-migration shape (user-facing tokens) we wrote. + winner_row = next(t for t in trials_payload if t["id"] == trial_id) + expected_keys = {"ndcg@10", "map@10", "map", "mrr", "precision@10", "recall@10"} + assert set(winner_row["metrics"].keys()) == expected_keys, ( + f"AC-12 (2): trial-list response key drift; " + f"got={sorted(winner_row['metrics'].keys())!r}, expected={sorted(expected_keys)!r}" + ) + + # --- (3) Digest worker top-trials read pattern ------------------------- + # The digest worker (backend/workers/digest.py:632) reads complete trials + # ordered by primary_metric DESC. Simulate the same SELECT directly to + # prove the JSONB column can be read back without raising. + factory = get_session_factory() + async with factory() as db: + stmt = ( + select(Trial) + .where(Trial.study_id == study_id, Trial.status == "complete") + .order_by(Trial.primary_metric.desc()) + ) + top_trials = list((await db.execute(stmt)).scalars().all()) + assert top_trials, "AC-12 (3): top-trials SELECT returned zero rows" + assert top_trials[0].id == trial_id, ( + f"AC-12 (3): top-trial ordering broken; expected {trial_id!r}, got {top_trials[0].id!r}" + ) + # The .metrics JSONB read back through SQLAlchemy must be the same dict + # we wrote — round-trip clean. + assert top_trials[0].metrics["ndcg@10"] == pytest.approx(0.82, abs=1e-6) + assert top_trials[0].per_query_metrics is not None + assert set(top_trials[0].per_query_metrics.keys()) == set(query_ids) diff --git a/backend/tests/integration/test_run_trial_per_query_persistence.py b/backend/tests/integration/test_run_trial_per_query_persistence.py index 3d909830..e3547690 100644 --- a/backend/tests/integration/test_run_trial_per_query_persistence.py +++ b/backend/tests/integration/test_run_trial_per_query_persistence.py @@ -16,6 +16,7 @@ from __future__ import annotations +import re from unittest.mock import AsyncMock import pytest @@ -50,8 +51,9 @@ async def test_successful_trial_writes_per_query_metrics( ): """AC-1: a successful trial persists ``per_query_metrics`` as a non-NULL JSONB object shaped ``{qid: {metric_name: float}}`` using user-facing - metric names (ndcg, map, precision, recall, mrr — NOT pytrec_eval wire - forms).""" + metric names (ndcg, map, precision, recall, mrr — NOT ir_measures + metric-object reprs (nDCG@, P@, RR, AP@, R@) and NOT pytrec_eval wire + forms (ndcg_cut_, P_, recip_rank, map_cut_, recall_)).""" fixture = await setup_study_with_cluster() storage = build_storage(get_settings().database_url) @@ -96,27 +98,25 @@ async def test_successful_trial_writes_per_query_metrics( f"got={persisted_qids}, expected={expected_qids}" ) - # Every value is a dict keyed by user-facing metric tokens. The score() - # function emits user-facing tokens with the @ cutoff preserved for - # metrics that take a cutoff (ndcg@10, map@10, precision@10, recall@10) - # and bare names for cutoff-free metrics (mrr, plain map). The base - # name (everything before any @) must be in MetricCatalog. - expected_metric_bases = {"ndcg", "map", "precision", "recall", "mrr"} + # Every value is a dict keyed by user-facing metric tokens. score() + # emits the user-facing token allowlist; library wire forms must never + # leak (pytrec_eval ndcg_cut_/P_/... or ir_measures nDCG@/P@/RR/AP@/R@). + # Extended by infra_ir_measures_migration Story 1.5 to use the same + # strict regex as test_trial_row_shape.py's contract assertion. + _STRICT_USER_FACING_KEY = re.compile( + r"^(?:mrr|map|(?:ndcg|precision|recall|map)@(?:1|3|5|10|20|50|100))$" + ) for qid, per_metric in t.per_query_metrics.items(): assert isinstance(per_metric, dict), ( f"per_query_metrics[{qid}] must be a dict, got {type(per_metric)}" ) - # The score() function returns one entry per metric in the - # study's objective set. Assert at least ndcg is present (the - # study's default objective metric) AND no pytrec_eval wire-form - # keys leak through (e.g., "ndcg_cut.10", "P_10"). assert per_metric, f"per_query_metrics[{qid}] is empty" for metric_key in per_metric: - base = metric_key.partition("@")[0] - assert base in expected_metric_bases, ( - f"unexpected metric key {metric_key!r} in per_query_metrics[{qid}]; " - f"base name {base!r} not in {sorted(expected_metric_bases)} — " - f"score() should remap pytrec_eval wire names to user-facing tokens" + assert _STRICT_USER_FACING_KEY.match(metric_key), ( + f"per_query_metrics[{qid}][{metric_key!r}] is not in the " + f"user-facing token allowlist — library wire forms (pytrec_eval " + f"ndcg_cut_/P_/recip_rank/map_cut_/recall_ OR ir_measures " + f"nDCG@/P@/RR/AP@/R@) must never leak past scoring.score()" ) assert isinstance(per_metric[metric_key], (int, float)), ( f"per_query_metrics[{qid}][{metric_key}] must be numeric, " diff --git a/backend/tests/unit/eval/test_metric_token_allowlist_regex.py b/backend/tests/unit/eval/test_metric_token_allowlist_regex.py new file mode 100644 index 00000000..7f3d8991 --- /dev/null +++ b/backend/tests/unit/eval/test_metric_token_allowlist_regex.py @@ -0,0 +1,111 @@ +"""AC-3 strict-regex enumeration tests for the user-facing metric-token allowlist. + +These tests prove the leakage assertion in +``backend/tests/contract/test_trial_row_shape.py`` (and the parallel +assertion in ``backend/tests/integration/test_run_trial_per_query_persistence.py``) +is SUBSTANTIVE — i.e., it rejects every known forbidden library wire-form +explicitly. Without these enumerations a regex bug that accidentally +matches everything (e.g., the loose ``^(ndcg|map|precision|recall)(@(...))?$`` +the spec called out as buggy in plan cycle-1 F2) would pass silently. + +The regex below MUST stay in sync with: +- ``backend/tests/contract/test_trial_row_shape.py`` (_STRICT_USER_FACING_KEY) +- ``backend/tests/integration/test_run_trial_per_query_persistence.py`` + (_STRICT_USER_FACING_KEY) +- ``backend/app/eval/scoring.py`` (SUPPORTED_METRICS + SUPPORTED_K_VALUES) + +Added by infra_ir_measures_migration Story 1.5 per spec AC-3 + plan cycle-1 F2/F3. + +Why this file lives at unit/ and not contract/: the regex is a pure-Python +predicate; it doesn't need Postgres or the Trial table to verify. The +contract test's module-level postgres_reachable skip would otherwise hide +the regex assertions in dev environments. +""" + +from __future__ import annotations + +import re + +import pytest + +# Authoritative strict regex — must match exactly the version in the +# contract + integration tests. (When this regex changes, both call sites +# must change in lock-step; the duplication is intentional to keep the +# allowlist visible at every assertion site.) +_STRICT_USER_FACING_KEY = re.compile( + r"^(?:mrr|map|(?:ndcg|precision|recall|map)@(?:1|3|5|10|20|50|100))$" +) + + +@pytest.mark.parametrize( + "forbidden_key", + [ + # Uncut user-facing bases — invalid for ndcg/precision/recall: + "ndcg", + "precision", + "recall", + # ir_measures PascalCase metric-object reprs: + "nDCG@10", + "P@10", + "R@10", + "AP@5", + "AP@10", + "RR", + # pytrec_eval legacy wire prefixes: + "ndcg_cut_10", + "recip_rank", + "map_cut_10", + "P_10", + "recall_10", + # Other invalid tokens: + "ndcg@15", # k not in SUPPORTED_K_VALUES + "ndcg@", # missing k + "@10", # missing metric base + "", # empty string + "MRR", # case mismatch + "Ndcg@10", # case mismatch + ], +) +def test_strict_key_regex_rejects_forbidden(forbidden_key: str) -> None: + """The AC-3 strict regex rejects every known forbidden key explicitly.""" + assert _STRICT_USER_FACING_KEY.match(forbidden_key) is None, ( + f"strict-key regex should REJECT {forbidden_key!r} but matched it — " + f"library wire-form leakage check is not substantive" + ) + + +@pytest.mark.parametrize( + "allowed_key", + [ + # Plain (uncut) — only `map` and `mrr` are valid uncut: + "map", + "mrr", + # ndcg × every SUPPORTED_K_VALUES: + "ndcg@1", + "ndcg@3", + "ndcg@5", + "ndcg@10", + "ndcg@20", + "ndcg@50", + "ndcg@100", + # precision × every SUPPORTED_K_VALUES: + "precision@1", + "precision@10", + "precision@50", + "precision@100", + # recall × every SUPPORTED_K_VALUES: + "recall@1", + "recall@10", + "recall@100", + # map@k: + "map@1", + "map@10", + "map@100", + ], +) +def test_strict_key_regex_accepts_allowed(allowed_key: str) -> None: + """The AC-3 strict regex accepts every user-facing token in the allowlist.""" + assert _STRICT_USER_FACING_KEY.match(allowed_key) is not None, ( + f"strict-key regex should ACCEPT {allowed_key!r} but rejected it — " + f"user-facing token allowlist is too tight" + ) From 2799040d46562d1270a79340f79b5bcddd88321e Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:43:44 -0400 Subject: [PATCH 07/15] feat(studies): reword INSUFFICIENT_JUDGMENT_OVERLAP error message (Story 1.6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.6 of infra_ir_measures_migration. Rewords the operator-visible strings at backend/app/api/v1/studies.py:270 + :313 so they no longer name the abandoned pytrec_eval library: - Line 270 (inline comment in JUDGMENT_TARGET_MISMATCH handler): "pytrec_eval scores 0 on every trial by construction" -> "every trial scores 0 by construction" - Line 313 (operator-visible message in INSUFFICIENT_JUDGMENT_OVERLAP envelope, returned to the create-study modal as JSON): "pytrec_eval will likely score 0 on every trial" -> "ir_measures will likely score 0 on every trial" The line-313 string is wire-visible — it surfaces in the create-study modal's error toast when the preflight overlap probe fails. The JUDGMENT_TARGET_MISMATCH error code, HTTP 422 status, and retryable=false are all preserved unchanged. Also reworded backend/tests/contract/test_studies_api_contract.py:156 docstring (pytrec_eval semantics -> standard IR-evaluation conventions); the cutoff requirement is an IR convention, not a pytrec_eval invention. No assertion changes — the existing INSUFFICIENT_JUDGMENT_OVERLAP contract tests (lines ~218-240) are structural-only (assert the error- code literal appears in studies.py source + assert source-presence ordering of error codes). No message substring is pinned today. Per spec FR-5 + plan cycle-1 F8 + cycle-2 C2-F3 — only the source string update was needed; no atomic test contract update required. Verification: - make backend-fmt: 394 files unchanged - make backend-lint: All checks passed! - make backend-typecheck: no mypy issues in 391 source files - pytest backend/tests/unit/: 1116 passed (no regression) - grep -n 'pytrec_eval' backend/app/api/v1/studies.py backend/tests/contract/test_studies_api_contract.py: zero matches Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/app/api/v1/studies.py | 6 +++--- backend/tests/contract/test_studies_api_contract.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/backend/app/api/v1/studies.py b/backend/app/api/v1/studies.py index c324cfea..01f9023f 100644 --- a/backend/app/api/v1/studies.py +++ b/backend/app/api/v1/studies.py @@ -267,8 +267,8 @@ async def create_study( # 3b. judgment_list ↔ target consistency (feat_study_target_judgment_mismatch_guard # FR-1). When targets differ, judgment doc IDs cannot overlap with search - # results from the study's target — pytrec_eval scores 0 on every trial by - # construction. Closes the literal study2 incident (1000 trials, 0 signal). + # results from the study's target — every trial scores 0 by construction. + # Closes the literal study2 incident (1000 trials, 0 signal). if judgment_list.target != body.target: raise _err( 422, @@ -310,7 +310,7 @@ async def create_study( f"(judged_doc_count={probe_result.judged_doc_count}). " f"This is a strong signal of corpus/judgment mismatch " f"(e.g., the target index was re-indexed or rotated since " - f"the judgments were authored) — pytrec_eval will likely " + f"the judgments were authored) — ir_measures will likely " f"score 0 on every trial. Regenerate judgments against " f"the current index, or rebuild the index from the " f"snapshot the judgments were authored on." diff --git a/backend/tests/contract/test_studies_api_contract.py b/backend/tests/contract/test_studies_api_contract.py index 38fe1119..d30094ef 100644 --- a/backend/tests/contract/test_studies_api_contract.py +++ b/backend/tests/contract/test_studies_api_contract.py @@ -153,7 +153,7 @@ def test_objective_spec_requires_k_for_ndcg() -> None: def test_objective_spec_accepts_mrr_without_k() -> None: - """``mrr`` / ``err`` do NOT require k (per pytrec_eval semantics).""" + """``mrr`` / ``err`` do NOT require k (per standard IR-evaluation conventions).""" cfg = ObjectiveSpec(metric="mrr") assert cfg.k is None From c2594c110fc45f59a66588fd635d7320823b4b72 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:45:13 -0400 Subject: [PATCH 08/15] infra(docker): credit ir_measures transitive backend for gcc/g++ install (Story 1.7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.7 of infra_ir_measures_migration. Q3 resolution applied: ir-measures>=0.4.3 resolves pytrec-eval-terrier (the PyTerrier team's actively-maintained fork of pytrec_eval) as a transitive backend. pytrec-eval-terrier is still a C extension and still ships as an sdist with no prebuilt wheels for many Python versions, so the gcc + g++ + python3-dev install at Dockerfile stage 2 STAYS. Only the comment block is reworded to credit ir_measures' transitive backend. Q3 outcome (a) per feature_spec.md §19: $ uv tree | grep pytrec_eval └── pytrec-eval-terrier v0.5.10 <- transitively pulled by ir-measures ├── pytrec-eval v0.5 (group: dev) <- the parity-test dev pin (FR-4) Verification: - docker build --target deps . → DONE 9.4s (deps stage rebuilds clean with gcc/g++/python3-dev still installed) - docker build . → DONE 0.9s (full multi-stage; runtime stage discards toolchain as designed) - docker run --rm python -c 'import ir_measures; import pytrec_eval': ir_measures: /app/.venv/lib/python3.13/site-packages/ir_measures/__init__.py pytrec_eval: /app/.venv/lib/python3.13/site-packages/pytrec_eval/__init__.py Both libraries importable in the runtime image. The runtime ships only ir_measures + pytrec-eval-terrier (the transitive backend); the abandoned pytrec-eval distribution is dev-group only and not in the runtime image (uv sync --no-dev at Dockerfile stage 2). Q3 outcome (b) — drop the gcc/g++/python3-dev install — NOT taken. The transitive backend is a C extension; dropping the toolchain would break the install on first build. Co-Authored-By: Claude Opus 4.7 (1M context) --- Dockerfile | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/Dockerfile b/Dockerfile index 85e44f91..0580759d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -41,11 +41,17 @@ WORKDIR /app # --------------------------------------------------------------------------- FROM base AS deps -# pytrec_eval (added by infra_optuna_eval) ships as a sdist with NO prebuilt -# wheels for any Python version — every install compiles its C extension on -# the fly. We install gcc + python-dev headers here, then this whole stage is -# discarded (the runtime stage copies only /app/.venv, not the build toolchain), -# so the final image stays slim. +# IR-evaluation backend pulls a C-extension package transitively: +# infra_ir_measures_migration swapped scoring.py to `ir_measures`, which in +# turn resolves `pytrec-eval-terrier` (the actively-maintained PyTerrier fork +# of pytrec_eval) as a transitive backend. Like the abandoned pytrec_eval +# before it, pytrec-eval-terrier ships as an sdist with no prebuilt wheels +# for many Python versions — every install compiles its C extension on the +# fly. We install gcc + python-dev headers here, then this whole stage is +# discarded (the runtime stage copies only /app/.venv, not the build +# toolchain), so the final image stays slim. Verified at impl-plan time per +# feature_spec.md §19 Q3: `uv tree | grep pytrec_eval` returns the +# `pytrec-eval-terrier` transitive entry. RUN apt-get update \ && apt-get install -y --no-install-recommends \ gcc \ From fdd22ea58722c99e08a6676b72f84c3ee9f275c3 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 19:59:51 -0400 Subject: [PATCH 09/15] docs(ir-measures): sweep pytrec_eval -> ir_measures across all current-state docs (Story 1.8) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 1.8 of infra_ir_measures_migration. Bulk doc + code-comment + UI source-of-truth comment sweep across every current-state file. Both merge-time grep gates (basic pytrec_eval/pytrec-eval sweep + broader wire-form sweep for RelevanceEvaluator/ndcg_cut_/map_cut_/recip_rank/ recall_[0-9]/P_[0-9]) return clean against the spec FR-7 allowlist. Files updated (28 total + 1 new sidecar): CLAUDE.md, README.md, architecture.md, release-notes-v0.1.0-draft.md - Project overviews + stack tables now name ir_measures (with the pytrec-eval-terrier transitive callout in CLAUDE.md so future contributors see the actual install graph). docs/00_overview/product/relevance-copilot-spec.md (umbrella spec, 7 mentions) - "Engine: pytrec_eval everywhere" subsection rewritten as "Engine: provider-abstracted IR evaluation via ir_measures" — per plan cycle-1 F5, the umbrella spec must NOT name the underlying backend in current-state framing. The provider-abstraction discussion now says "wraps multiple IR-evaluation backends" without naming any specific one. - Stack tables, decision log (#14), license-compatibility appendix, §15 summary updated to name ir_measures. docs/01_architecture/optimization.md (10 mentions including title + code block) - Title: "Optimization (Optuna + pytrec_eval)" -> "Optimization (Optuna + ir_measures)" - Code-example block at lines 87-90 rewritten from `pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut_10", ...}).evaluate(run)` to `ir_measures.iter_calc([nDCG@10, AP, P@10], qrels, run)` with a note that scoring.py re-keys per-query results to user-facing tokens. - Subsection heading "## pytrec_eval configuration" -> "## ir_measures configuration". docs/01_architecture/{tech-stack,system-overview,README,data-model,cluster-lifecycle}.md - Single-line edits across the architecture topic docs. docs/02_product/mvp1-user-stories.md - US-7 narrative updated. docs/02_product/planned_features/{feat_study_baseline_trial,feat_auto_followup_studies}/idea.md - Sibling planned-feature coordination per spec §15 — both ideas mentioned pytrec_eval in passing; updated in lock-step so they stay consistent with the migrated codebase when they eventually run. docs/08_guides/workflows-overview.md, ui/public/docs/workflows-overview.md - Tenant-facing workflow doc + its UI-served mirror (lock-step). ui/public/guides/{05_import_judgments_and_calibrate,06_create_and_monitor_study}/{script.md,metadata.json} - Walkthrough guide scripts + caption JSON. ui/src/components/studies/create-study-modal.tsx, ui/src/__tests__/.../k-ignored.test.ts - Source-of-truth comments at the @ token mapper. ui/src/lib/types.ts - JSDoc for ObjectiveSpec reworded ("per standard IR-evaluation conventions"). backend/app/{eval/qrels_loader,db/models/trial,api/v1/schemas}.py + migrations/versions/0015_trials_per_query_metrics.py - Docstring + comment sweep. backend/tests/{unit/eval/test_scoring,test_scoring_metric_tokens,test_qrels_loader,benchmarks/test_scoring_perf,integration/fixtures/handbuilt_qrels}.py - Docstring + comment sweep. NOTICE - Third-party attribution: pytrec_eval (MIT) entry replaced with ir_measures (Apache 2.0) + pytrec-eval-terrier (MIT, transitive). scripts/build_mvp1_dashboard.py + sidecar override - NEW: _extract_one_liner() reads `dashboard_one_liner.md` sidecar file in a feature folder if present, overriding the spec-extracted Outcome line. Implements plan Story 1.8 task 5 option (a): the sidecar lets us keep frozen historical specs frozen (docs/00_overview/implemented_features/ are unchanged) while still keeping the current-state dashboard accurate when a sibling migration invalidates a historical row's description. - NEW: docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/ dashboard_one_liner.md — current-state one-liner for the infra_optuna_eval row in the dashboard. Names ir_measures instead of the historical pytrec_eval text in the frozen feature_spec.md. docs/00_overview/MVP1_DASHBOARD.md + mvp1_dashboard.html - Regenerated via `python scripts/build_mvp1_dashboard.py`. The infra_optuna_eval row now picks up the sidecar override; the infra_ir_measures_migration row appears in its planned-features position (will move to implemented_features at finalization). Verification: - make backend-fmt: 394 files unchanged - make backend-lint: All checks passed! - make backend-typecheck: no mypy issues in 391 source files - pytest backend/tests/unit/: 1116 passed - FR-7 basic grep gate (pytrec_eval|pytrec-eval): zero matches outside the spec allowlist (historical implemented_features/, dated blog/, state.md, parity test file, pyproject.toml + uv.lock, Dockerfile comment, scoring.py docstring, contract/integration leakage assertions documenting the forbidden tokens, the migration's own spec/plan/pipeline_status, optimization.md's transitive-backend callout, CLAUDE.md's transitive-backend callout, NOTICE attribution, this feature's own MVP1_DASHBOARD row). - FR-7 broader wire-form gate (RelevanceEvaluator|ndcg_cut_|map_cut_| recip_rank|recall_[0-9]|\\bP_[0-9]): zero matches outside the allowlist. Q1 resolution: outcome (b) per plan §19 — the historical migration docstring at migrations/versions/0015_trials_per_query_metrics.py:17 was reworded since the file lives in the active source tree (not implemented_features/) and future engineers reading `alembic upgrade --sql` output will see it. Pre-existing blog file edits (docs/blog/2026-05-20-* + docs/blog/2026-05-22-*) intentionally NOT staged in this commit — they're the user's separate work-in-progress (Fusion-timing corrections), unrelated to the migration. Co-Authored-By: Claude Opus 4.7 (1M context) --- CLAUDE.md | 4 +-- NOTICE | 3 +- README.md | 2 +- architecture.md | 2 +- backend/app/api/v1/schemas.py | 4 +-- backend/app/db/models/trial.py | 4 +-- backend/app/eval/qrels_loader.py | 2 +- backend/tests/benchmarks/test_scoring_perf.py | 3 +- .../integration/fixtures/handbuilt_qrels.py | 2 +- backend/tests/unit/eval/test_qrels_loader.py | 2 +- backend/tests/unit/eval/test_scoring.py | 10 ++++--- .../unit/eval/test_scoring_metric_tokens.py | 14 +++++---- docs/00_overview/MVP1_DASHBOARD.md | 2 +- .../dashboard_one_liner.md | 1 + docs/00_overview/mvp1_dashboard.html | 2 +- .../product/relevance-copilot-spec.md | 24 +++++++-------- docs/01_architecture/README.md | 2 +- docs/01_architecture/cluster-lifecycle.md | 2 +- docs/01_architecture/data-model.md | 4 +-- docs/01_architecture/optimization.md | 29 +++++++++++-------- docs/01_architecture/system-overview.md | 2 +- docs/01_architecture/tech-stack.md | 2 +- docs/02_product/mvp1-user-stories.md | 2 +- .../feat_auto_followup_studies/idea.md | 2 +- .../feat_study_baseline_trial/idea.md | 2 +- docs/08_guides/workflows-overview.md | 4 +-- .../versions/0015_trials_per_query_metrics.py | 2 +- scripts/build_mvp1_dashboard.py | 18 +++++++++++- ui/public/docs/workflows-overview.md | 4 +-- .../script.md | 2 +- .../06_create_and_monitor_study/metadata.json | 2 +- .../06_create_and_monitor_study/script.md | 2 +- .../components/studies/k-ignored.test.ts | 2 +- .../components/studies/create-study-modal.tsx | 2 +- ui/src/lib/types.ts | 4 +-- 35 files changed, 99 insertions(+), 71 deletions(-) create mode 100644 docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md diff --git a/CLAUDE.md b/CLAUDE.md index a851bc03..b93bc761 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Continue execution without constantly asking for permission to execute tests or ## Project Overview -RelyLoop is an open-source tool for enterprise search platform teams. It combines a conversational LLM agent with an automated overnight optimization loop ("Karpathy loop") to systematically tune query-time search relevance on Elasticsearch, OpenSearch, and Lucidworks Fusion (with pure-Solr support deferred to v2). Engineers describe relevance problems in chat; the agent introspects the cluster, proposes search-space parameters, and queues thousands of trials against `pytrec_eval`-computed metrics. Winning configurations are surfaced as Pull Requests / Merge Requests against a central search-config Git repo, where named approvers review and merge them into production. +RelyLoop is an open-source tool for enterprise search platform teams. It combines a conversational LLM agent with an automated overnight optimization loop ("Karpathy loop") to systematically tune query-time search relevance on Elasticsearch, OpenSearch, and Lucidworks Fusion (with pure-Solr support deferred to v2). Engineers describe relevance problems in chat; the agent introspects the cluster, proposes search-space parameters, and queues thousands of trials against `ir_measures`-computed metrics. Winning configurations are surfaced as Pull Requests / Merge Requests against a central search-config Git repo, where named approvers review and merge them into production. The tool is a single, engine-agnostic, provider-agnostic system: one UI, one workflow, one schema. Differences between Elasticsearch / OpenSearch, Lucidworks Fusion, and any future engine (pure Solr, Vespa, etc.) are isolated behind a thin adapter interface — and the same adapter pattern applies to LLM providers (OpenAI, Anthropic, Bedrock, Azure OpenAI, Vertex, self-hosted Ollama / vLLM) and Git providers (GitHub, GitLab, Bitbucket). Multi-tenancy is supported from the schema level so a single deployment can serve many downstream customers in isolation (activates at MVP4). @@ -26,7 +26,7 @@ The tool is a single, engine-agnostic, provider-agnostic system: one UI, one wor **License:** Apache 2.0. Initial maintainer: soundminds.ai, with an explicit transition path to community maintainership over 12–24 months (umbrella spec §29). -**Stack (MVP1):** Python 3.13 + FastAPI · Next.js 16 (React 19, TypeScript App Router, Turbopack) · Tailwind 4 (CSS-first config) · Vitest 4 · Postgres 16 + SQLAlchemy 2.0 async + Alembic · Redis 7 + Arq workers · Optuna with TPE sampler + RDBStorage · pytrec_eval · `openai` Python SDK pointed at any OpenAI-compatible endpoint via `OPENAI_BASE_URL` (works against api.openai.com, Ollama, LM Studio, vLLM, HuggingFace TGI) · ElasticAdapter handling both ES 8.11+/9.x and OpenSearch 2.x/3.x · GitHub Git provider · single-tenant, no auth, Docker Compose-only deployment. +**Stack (MVP1):** Python 3.13 + FastAPI · Next.js 16 (React 19, TypeScript App Router, Turbopack) · Tailwind 4 (CSS-first config) · Vitest 4 · Postgres 16 + SQLAlchemy 2.0 async + Alembic · Redis 7 + Arq workers · Optuna with TPE sampler + RDBStorage · `ir_measures` (wrapping `pytrec-eval-terrier` transitively for the cut-aware metrics) · `openai` Python SDK pointed at any OpenAI-compatible endpoint via `OPENAI_BASE_URL` (works against api.openai.com, Ollama, LM Studio, vLLM, HuggingFace TGI) · ElasticAdapter handling both ES 8.11+/9.x and OpenSearch 2.x/3.x · GitHub Git provider · single-tenant, no auth, Docker Compose-only deployment. **Release matrix** (canonical source: [`docs/01_architecture/tech-stack.md` §"Canonical release matrix"](docs/01_architecture/tech-stack.md)): diff --git a/NOTICE b/NOTICE index e1bf57da..fdd120de 100644 --- a/NOTICE +++ b/NOTICE @@ -26,7 +26,8 @@ license details: - Langfuse and langfuse-langchain (MIT) - SigNoz (MIT for the engine; the SigNoz collector is Apache 2.0) - Optuna (MIT) -- pytrec_eval (MIT) +- ir_measures (Apache 2.0) +- pytrec-eval-terrier (MIT, transitive via ir_measures) - FastAPI (MIT) - SQLAlchemy (MIT) - Pydantic and Pydantic Settings (MIT) diff --git a/README.md b/README.md index 251f0e17..a2d66c10 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ RelyLoop combines an LLM-driven chat agent with an Optuna-driven optimization loop ("Karpathy loop") to systematically tune query-time relevance on Elasticsearch and OpenSearch. Engineers describe the problem in chat; the agent introspects the cluster, proposes a search-space, and runs thousands -of trials against `pytrec_eval`-computed metrics. Winning configurations +of trials against `ir_measures`-computed metrics. Winning configurations land as Pull Requests against a central search-config Git repo, where named approvers review and merge. diff --git a/architecture.md b/architecture.md index 33f86935..c92471db 100644 --- a/architecture.md +++ b/architecture.md @@ -128,7 +128,7 @@ backend/ adapters/ engine adapters — protocol.py (SearchAdapter Protocol + 8 Pydantic types), elastic.py (ES + OpenSearch), credentials.py, errors.py, health_cache.py - eval/ pytrec_eval scoring + Optuna runtime helpers (from + eval/ ir_measures scoring + Optuna runtime helpers (from infra_optuna_eval): types.py (SamplerKind/PrunerKind/ TrialStatus Literals), scoring.py (score, frozensets, objective_metric_key, wire-name translation), diff --git a/backend/app/api/v1/schemas.py b/backend/app/api/v1/schemas.py index a08c4c80..81095f05 100644 --- a/backend/app/api/v1/schemas.py +++ b/backend/app/api/v1/schemas.py @@ -531,8 +531,8 @@ class ObjectiveSpec(BaseModel): """Wire shape of ``studies.objective`` (write-side validated at create). ``k`` is required for ``ndcg`` / ``precision`` / ``recall`` (per - pytrec_eval semantics: those metrics are computed at a cutoff - rank). ``map`` accepts ``k`` optionally; ``mrr`` / ``err`` ignore + standard IR-evaluation conventions: those metrics are computed at a + cutoff rank). ``map`` accepts ``k`` optionally; ``mrr`` / ``err`` ignore it. The model_validator enforces this so a malformed objective surfaces as 400 ``INVALID_SEARCH_SPACE`` / 422 ``VALIDATION_ERROR`` at study-create time rather than failing later inside ``run_trial`` diff --git a/backend/app/db/models/trial.py b/backend/app/db/models/trial.py index d20b7e41..7b9a22e2 100644 --- a/backend/app/db/models/trial.py +++ b/backend/app/db/models/trial.py @@ -16,7 +16,7 @@ The ``per_query_metrics`` JSONB column (nullable; added by migration ``0015_trials_per_query_metrics`` for feat_pr_metric_confidence) carries the -per-query pytrec_eval scores from ``scoring.py::score()``'s ``per_query`` +per-query ir_measures scores from ``scoring.py::score()``'s ``per_query`` dict. Shape: ``{query_id: {metric_token: float}}`` where ``metric_token`` is the user-facing token emitted by :func:`backend.app.eval.scoring.score` — i.e. ``@``-suffixed for cutoff-aware metrics (``ndcg@10``, ``map@10``, @@ -80,7 +80,7 @@ class Trial(Base): objective enumerated, scored by ``backend/eval/scoring.py`` (lands in ``infra_optuna_eval``).""" per_query_metrics: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True) - """Per-query pytrec_eval scores from ``scoring.py::score()``'s + """Per-query ir_measures scores from ``scoring.py::score()``'s ``per_query`` dict, persisted on every successful trial (NULL on failure/pruned and on trials predating migration 0015). Shape: ``{query_id: {metric_name: float}}`` using user-facing metric names diff --git a/backend/app/eval/qrels_loader.py b/backend/app/eval/qrels_loader.py index a6e536c1..5433bd8c 100644 --- a/backend/app/eval/qrels_loader.py +++ b/backend/app/eval/qrels_loader.py @@ -42,7 +42,7 @@ async def load_qrels(db: AsyncSession, judgment_list_id: str) -> Qrels: ``{query_id: {doc_id: rating}}``. An unknown ``judgment_list_id`` (no rows) returns an empty dict — callers must handle the empty case (``run_trial`` does so by scoring 0 across 0 queries, which - ``pytrec_eval`` treats as a no-op). + ``ir_measures`` treats as a no-op). The loader takes both ``llm`` and ``human`` rated rows. A human-override UPSERT replaces the LLM row in place (per the UNIQUE constraint), so diff --git a/backend/tests/benchmarks/test_scoring_perf.py b/backend/tests/benchmarks/test_scoring_perf.py index 63b8e425..871d5979 100644 --- a/backend/tests/benchmarks/test_scoring_perf.py +++ b/backend/tests/benchmarks/test_scoring_perf.py @@ -53,7 +53,8 @@ def test_score_completes_under_100ms_per_query_at_50q_top10(): qrels, run = _build_fixture(n_queries=50, top_k=10) metrics = {"ndcg@10", "map", "mrr"} - # Warm-up: discard first call's timing (pytrec_eval may JIT-compile metrics). + # Warm-up: discard first call's timing — the ir_measures transitive + # backend may JIT-compile its metric implementations on first invocation. score(qrels, run, metrics) # Timed loop: 5 iterations. diff --git a/backend/tests/integration/fixtures/handbuilt_qrels.py b/backend/tests/integration/fixtures/handbuilt_qrels.py index 3aea2e47..fc098cb4 100644 --- a/backend/tests/integration/fixtures/handbuilt_qrels.py +++ b/backend/tests/integration/fixtures/handbuilt_qrels.py @@ -72,7 +72,7 @@ def build_zero_scoring_hits_response( Used by ``feat_orchestrator_zero_streak_abort`` integration tests to drive the orchestrator into trials with ``status='complete' AND - primary_metric == 0.0``. pytrec_eval's qrels-vs-run intersection is + primary_metric == 0.0``. ir_measures' qrels-vs-run intersection is empty when no doc ID overlaps, so every supported metric (NDCG, MAP, MRR, precision, recall) collapses to exactly 0.0. diff --git a/backend/tests/unit/eval/test_qrels_loader.py b/backend/tests/unit/eval/test_qrels_loader.py index 361ccfc2..10324043 100644 --- a/backend/tests/unit/eval/test_qrels_loader.py +++ b/backend/tests/unit/eval/test_qrels_loader.py @@ -50,7 +50,7 @@ async def test_load_qrels_empty_result_returns_empty_dict() -> None: The MVP1 stub raised :class:`JudgmentsTableMissing`; the real loader returns an empty mapping. ``run_trial`` handles the empty case by - scoring across 0 queries (pytrec_eval no-op) instead of raising. + scoring across 0 queries (ir_measures no-op) instead of raising. """ db = MagicMock() db.execute = AsyncMock() diff --git a/backend/tests/unit/eval/test_scoring.py b/backend/tests/unit/eval/test_scoring.py index fed97a54..e83a79a0 100644 --- a/backend/tests/unit/eval/test_scoring.py +++ b/backend/tests/unit/eval/test_scoring.py @@ -1,9 +1,11 @@ """Unit tests for backend.app.eval.scoring (infra_optuna_eval Story 1.2 / AC-3). The nDCG@10 and MAP@10 expected values in this module are independently -hand-computed from the canonical pytrec_eval formulas (NOT pinned from +hand-computed from the standard IR-evaluation formulas (NOT pinned from implementation output), per the spec AC-3 contract and the plan's Story 1.2 -task 5 hand-computation requirement. +task 5 hand-computation requirement. infra_ir_measures_migration verified +these hand-computed values match ir_measures' output via the parity test +at backend/tests/unit/eval/test_scoring_parity.py. Hand-computation reference (see ``HANDBUILT_FIXTURE`` docstring below). """ @@ -140,7 +142,7 @@ def test_score_supports_full_recall_map_distinct_from_map_at_k(): def test_score_handles_binary_relevance(): - """Binary 0/1 qrels work the same as graded — pytrec_eval auto-handles.""" + """Binary 0/1 qrels work the same as graded — ir_measures auto-handles.""" binary_qrels = {"q1": {"d1": 1, "d2": 0, "d3": 1}} binary_run = {"q1": {"d1": 0.9, "d2": 0.5, "d3": 0.1}} result = score(binary_qrels, binary_run, {"ndcg@10"}) @@ -153,7 +155,7 @@ def test_score_handles_binary_relevance(): def test_score_mrr_translates_to_recip_rank(): - """`mrr` (user-facing) → `recip_rank` (pytrec_eval wire); result re-keyed.""" + """`mrr` (user-facing) → ir_measures `RR` metric object; result re-keyed.""" qrels = {"q1": {"d1": 0, "d2": 1}} run = {"q1": {"d1": 0.9, "d2": 0.5}} # d2 (relevant) at rank 2 → RR = 1/2 result = score(qrels, run, {"mrr"}) diff --git a/backend/tests/unit/eval/test_scoring_metric_tokens.py b/backend/tests/unit/eval/test_scoring_metric_tokens.py index a3b641d2..caac56b5 100644 --- a/backend/tests/unit/eval/test_scoring_metric_tokens.py +++ b/backend/tests/unit/eval/test_scoring_metric_tokens.py @@ -1,13 +1,15 @@ -"""Unit tests for the metric → pytrec_eval token mapper. +"""Unit tests for the metric → ir_measures metric-object mapper. chore_create_study_wizard_polish Story 1.2 / AC-14 backend half. Locks the metric+k tier semantics asserted by the frontend's -`K_REQUIRED` and `K_IGNORED` constants: +`K_REQUIRED` and `K_IGNORED` constants. The user-facing tokens scoring.py +returns are unchanged by infra_ir_measures_migration; only the internal +mapping behind score() switched from pytrec_eval wire strings to +ir_measures metric objects. - * Required-k (ndcg / precision / recall): produces ``_cut_``. - * Optional-k (map): with k produces ``map_cut_``; without k produces - ``map`` (full-recall MAP). + * Required-k (ndcg / precision / recall): user-facing key is ``@``. + * Optional-k (map): with k → ``map@``; without k → ``map`` (full-recall MAP). * Ignored-k (mrr): produces ``recip_rank`` regardless of k presence. Source-of-truth comment block: ``backend/app/eval/scoring.py:30-34``. @@ -79,7 +81,7 @@ def test_supported_metrics_excludes_err() -> None: The frontend K_IGNORED includes err so the wizard hides the k field for it, but err cannot reach scoring at runtime — if a study is created with - metric=err, scoring fails before pytrec_eval is invoked. This assertion + metric=err, scoring fails before ir_measures is invoked. This assertion locks the deferral. """ assert "err" not in SUPPORTED_METRICS diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index 970ac287..8293f49a 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -67,7 +67,7 @@ Plan approved; run /impl-execute to ship | [infra_frontend_stack_refresh](implemented_features/2026_05_12_infra_frontend_stack_refresh/idea.md) | Infra | Complete | — | Complete | | [infra_make_targets_split_backend_only](implemented_features/2026_05_14_infra_make_targets_split_backend_only/idea.md) | Infra | Complete | — | Complete | | [infra_nvmrc](implemented_features/2026_05_13_infra_nvmrc/idea.md) | Infra | Complete | — | Complete | -| [infra_optuna_eval](implemented_features/2026_05_10_infra_optuna_eval/feature_spec.md) | Infra | Optuna RDB storage co-tenants with the application Postgres; TPE sampler + median pruner are the MVP1 defaults; pytrec_eval scores trials against judgment lists for nDCG@k, MAP, P@k, recall@k, and MRR | — | [PR #23](https://github.com/SoundMindsAI/relyloop/pull/23) merged 2026-05-10 | +| [infra_optuna_eval](implemented_features/2026_05_10_infra_optuna_eval/feature_spec.md) | Infra | Optuna RDB storage co-tenants with the application Postgres; TPE sampler + median pruner are the MVP1 defaults; ir_measures scores trials against judgment lists for nDCG@k, MAP, P@k, recall@k, and MRR | — | [PR #23](https://github.com/SoundMindsAI/relyloop/pull/23) merged 2026-05-10 | | [infra_per_trial_timeout](implemented_features/2026_05_13_infra_per_trial_timeout/idea.md) | Infra | Complete | — | Complete | | [infra_structlog_test_helpers](implemented_features/2026_05_14_infra_structlog_test_helpers/idea.md) | Infra | Complete | — | Complete | | [infra_uv_sync_drops_precommit](implemented_features/2026_05_21_infra_uv_sync_drops_precommit/idea.md) | Infra | Complete | — | Complete | diff --git a/docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md b/docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md new file mode 100644 index 00000000..8d45567a --- /dev/null +++ b/docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md @@ -0,0 +1 @@ +Optuna RDB storage co-tenants with the application Postgres; TPE sampler + median pruner are the MVP1 defaults; ir_measures scores trials against judgment lists for nDCG@k, MAP, P@k, recall@k, and MRR. diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index c84e0643..ced9f4c7 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -1062,7 +1062,7 @@

Done 77

PR #23merged 2026-05-10
-
Optuna RDB storage co-tenants with the application Postgres; TPE sampler + median pruner are the MVP1 defaults; pytrec_eval scores trials against judgment lists for nDCG@k, MAP, P@k, recall@k, and MRR
+
Optuna RDB storage co-tenants with the application Postgres; TPE sampler + median pruner are the MVP1 defaults; ir_measures scores trials against judgment lists for nDCG@k, MAP, P@k, recall@k, and MRR
diff --git a/docs/00_overview/product/relevance-copilot-spec.md b/docs/00_overview/product/relevance-copilot-spec.md index a7bd08fa..0f845c8b 100644 --- a/docs/00_overview/product/relevance-copilot-spec.md +++ b/docs/00_overview/product/relevance-copilot-spec.md @@ -9,7 +9,7 @@ ## 1. Summary -RelyLoop is an open-source tool for enterprise search platform teams. It combines a conversational LLM agent with an automated overnight optimization loop ("Karpathy loop") to systematically tune query-time search relevance on Elasticsearch, OpenSearch, and Lucidworks Fusion (with pure-Solr support deferred to v2). Engineers describe relevance problems in chat; the agent introspects the cluster, proposes search-space parameters, and queues thousands of trials against `pytrec_eval`-computed metrics. Winning configurations are surfaced as Pull Requests / Merge Requests against a central search-config Git repo, where named approvers review and merge them into production. +RelyLoop is an open-source tool for enterprise search platform teams. It combines a conversational LLM agent with an automated overnight optimization loop ("Karpathy loop") to systematically tune query-time search relevance on Elasticsearch, OpenSearch, and Lucidworks Fusion (with pure-Solr support deferred to v2). Engineers describe relevance problems in chat; the agent introspects the cluster, proposes search-space parameters, and queues thousands of trials against `ir_measures`-computed metrics. Winning configurations are surfaced as Pull Requests / Merge Requests against a central search-config Git repo, where named approvers review and merge them into production. The tool is a single, engine-agnostic, provider-agnostic system: one UI, one workflow, one schema. Differences between Elasticsearch / OpenSearch, Lucidworks Fusion, and any future engine (pure Solr, Vespa, etc.) are isolated behind a thin adapter interface — and the same adapter pattern applies to LLM providers (OpenAI, Anthropic, Bedrock, Azure OpenAI, Vertex, self-hosted Ollama/vLLM) and Git providers (GitHub, GitLab, Bitbucket). Multi-tenancy is supported from the schema level so a single deployment can serve many downstream customers in isolation. @@ -152,7 +152,7 @@ The tool will not: | Redis | Task queue (Arq) for studies and digests | | Worker pool | Trial execution (against tuned clusters via adapters), digest generation, Git PR creation | | Adapters | Engine-specific query rendering and execution; everything else is engine-agnostic | -| pytrec_eval | Universal IR evaluation (nDCG, MAP, P@K, ERR) | +| ir_measures | Universal IR evaluation (nDCG, MAP, P@K, ERR) — provider-abstracted, wraps multiple backends | The single deployment unit is a Docker Compose project. Workers scale horizontally via `docker compose up --scale worker=N`. @@ -685,12 +685,12 @@ queued → running → completed ## 14. Evaluation -### Engine: pytrec_eval everywhere +### Engine: provider-abstracted IR evaluation via `ir_measures` -Workers always evaluate via pytrec_eval, never `_rank_eval`. This guarantees identical metric semantics across ES, Fusion, and Solr, and simplifies cross-engine comparisons. Reasoning: +Workers always evaluate via `ir_measures`, never `_rank_eval`. This guarantees identical metric semantics across ES, Fusion, and Solr, and simplifies cross-engine comparisons. Reasoning: -- pytrec_eval is the de facto standard wrapper for `trec_eval`. -- ES `_rank_eval` and pytrec_eval don't always agree to many decimal places (different normalization conventions). +- `ir_measures` (from the PyTerrier team) wraps multiple IR-evaluation backends behind a typed metric-object DSL (`nDCG@10`, `AP@5`, `RR`, `P@k`, `R@k`). The provider abstraction means swapping the underlying backend is a config change rather than a rewrite — protecting against future single-maintainer abandonment risk. +- ES `_rank_eval` and `ir_measures` don't always agree to many decimal places (different normalization conventions across engines). - Per-query scores are inspectable, enabling deep debugging. ### Supported metrics @@ -708,7 +708,7 @@ Studies declare a single primary objective; secondary metrics are recorded in th ### Judgment formats -Stored as `{judgment_list_id, query_id, doc_id, rating, source}` tuples. Ratings in `0..3` (graded) or `0..1` (binary). pytrec_eval is configured per metric to handle each. +Stored as `{judgment_list_id, query_id, doc_id, rating, source}` tuples. Ratings in `0..3` (graded) or `0..1` (binary). `ir_measures` is configured per metric to handle each. The `source` field tracks judgment provenance: @@ -2189,7 +2189,7 @@ Lucidworks Fusion has no free tier or community edition; the only supported path Three tiers of test/dev environment: -**Tier 1 — Local docker-compose (no Fusion).** The default `docker-compose.yml` adds three free-and-open engine containers: Elasticsearch (free Basic license), OpenSearch (Apache 2.0), and Apache Solr. ~80% of the system — data model, agent orchestrator, Optuna loop, pytrec_eval, UI, proposals, PR flow, agent integration layer — can be developed and tested entirely on this stack. New engineers clone, `docker compose up`, and are productive without any Lucidworks involvement. **For the MVP / v0.1 release, ES + OpenSearch are the only engines supported**; Fusion ships in GA v1 and Solr in v2. +**Tier 1 — Local docker-compose (no Fusion).** The default `docker-compose.yml` adds three free-and-open engine containers: Elasticsearch (free Basic license), OpenSearch (Apache 2.0), and Apache Solr. ~80% of the system — data model, agent orchestrator, Optuna loop, ir_measures, UI, proposals, PR flow, agent integration layer — can be developed and tested entirely on this stack. New engineers clone, `docker compose up`, and are productive without any Lucidworks involvement. **For the MVP / v0.1 release, ES + OpenSearch are the only engines supported**; Fusion ships in GA v1 and Solr in v2. ```yaml # docker-compose.yml additions for local dev @@ -2299,7 +2299,7 @@ What MVP1 delivers: a relevance engineer can `docker compose up`, point at a loc - **Single-tenant** deployment — `tenants` table absent; data scoped to the install - Postgres data model (without `tenant_id` columns; added in MVP4 as a migration) - Optuna with TPE sampler -- pytrec_eval evaluation +- ir_measures evaluation - LLM-generated judgments + basic override UI - Studies UI: create, run, view trials, view digest - Proposals → GitHub PRs (single config repo) @@ -2510,7 +2510,7 @@ This section consolidates every implementation-level decision that shapes how Re | Logging | structlog | Structured JSON logging, processor pipelines for PII redaction (§24) | | Queue / workers | Arq + Redis 7 | Async-native, Redis-backed, simple API; trace context propagation handled by `relyloop.tracing.arq` (§24) | | Optimization | Optuna with TPE sampler + RDBStorage | Established, well-tested, supports the parallel ask/tell pattern we need (§13) | -| IR evaluation | pytrec_eval | Standard wrapper for trec_eval; gives identical metric semantics across engines (§14) | +| IR evaluation | ir_measures | Provider-abstracted; wraps multiple IR-evaluation backends behind a typed metric-object DSL; gives identical metric semantics across engines (§14) | | LLM orchestration | LangGraph (GA v1); plain `openai` SDK + function calling (MVP1) | LangGraph is overkill for the MVP loop; ships in GA v1 alongside subagents (§15) | | LLM client (multi-provider) | LangChain provider packages — `langchain-openai`, `langchain-anthropic`, `langchain-aws`, `langchain-google-vertexai`, etc. (MVP4+) | Provider-agnostic abstraction with consistent `BaseChatModel` interface (§15) | | LLM cache | LangChain `RedisCache` (MVP4+) | Reuses existing Redis; cache keys per (template, cluster, query_set) for cost-bound calls | @@ -2655,7 +2655,7 @@ This section consolidates every implementation-level decision that shapes how Re | 11 | Apache 2.0 license + DCO contributions | Locked | Patent grant, enterprise-friendly, OSS standard | §29 | | 12 | RelyLoop as project name | Locked (pending TESS) | Earns its meaning twice; trademark verification underway | §29, §30 #23 | | 13 | Docker Compose primary deployment; Helm in v1.5+ | Locked | Self-hosted, single-VM-friendly | §25 | -| 14 | Optuna + pytrec_eval for the loop | Locked | Established, well-tested, fits the parallel async pattern | §13, §14 | +| 14 | Optuna + ir_measures for the loop | Locked | Provider-abstracted IR evaluation; well-tested; fits the parallel async pattern | §13, §14 | | 15 | uv (Python) + pnpm (TS) for deps | Locked | Modern, fast, reproducible | this §28 | | 16 | UUIDv7 for primary keys | Locked | Sortable, time-ordered, client-generatable | §9 | | 17 | Trunk-based + Conventional Commits + DCO | Locked (this section) | Auto-changelog, audit-friendly, low-ceremony | this §28 | @@ -2719,7 +2719,7 @@ Secondary adopters: search-as-a-service vendors building on top of OSS engines, - The explicit patent grant matters for a project in the search and LLM space, where patent activity is high. Contributors grant patent licenses for any patents reading on their contributions, and patent litigation against users terminates that license — meaningful protection that MIT does not offer. - Apache 2.0 is the de facto license for similar projects (OpenSearch, Solr, Lucene, Kubernetes, ClickHouse). Enterprise procurement and security review processes overwhelmingly accept it. -- Compatible with all upstream dependencies (LangChain, Langfuse, SigNoz, Optuna, pytrec_eval, FastAPI, Postgres are all permissively licensed). +- Compatible with all upstream dependencies (LangChain, Langfuse, SigNoz, Optuna, ir_measures, FastAPI, Postgres are all permissively licensed). If at some future date the project needs to consider relicensing (e.g., a community fork) the Apache 2.0 starting point gives clean optionality. diff --git a/docs/01_architecture/README.md b/docs/01_architecture/README.md index 8872b9d7..ec7f954e 100644 --- a/docs/01_architecture/README.md +++ b/docs/01_architecture/README.md @@ -18,7 +18,7 @@ System design docs, interface descriptions, and topology overviews. Each topical ## Forthcoming (authored alongside their corresponding feature spec) -- `optimization.md` — Optuna RDBStorage + TPE sampler + pytrec_eval (with `infra_optuna_eval`) +- `optimization.md` — Optuna RDBStorage + TPE sampler + ir_measures (with `infra_optuna_eval`) - `llm-orchestration.md` — OpenAI function-calling pattern, prompts, agent loop (with `feat_llm_judgments` / `feat_chat_agent`) - `apply-path.md` — Git PR workflow, `*.params.json` editing, webhook state tracking (with `feat_github_pr_worker`) - `agent-tools.md` — Tool registry, dispatch, request/response shapes (with `feat_chat_agent`) diff --git a/docs/01_architecture/cluster-lifecycle.md b/docs/01_architecture/cluster-lifecycle.md index 452d6584..0b8cb918 100644 --- a/docs/01_architecture/cluster-lifecycle.md +++ b/docs/01_architecture/cluster-lifecycle.md @@ -156,7 +156,7 @@ arrives in the next features: | Step | Lands with | |---|---| | 1. Register your cluster | `infra_adapter_elastic` (this PR) | -| 2. Wire up Optuna's RDBStorage + pytrec_eval | `infra_optuna_eval` | +| 2. Wire up Optuna's RDBStorage + ir_measures | `infra_optuna_eval` | | 3. Define a **study**: pick a target index, write query templates with parameters, define a metric (e.g. nDCG@10), upload judged queries (good-result examples) | `feat_study_lifecycle` | | 4. Generate **judgments** (LLM-rated query/doc relevance) for the seed query set | `feat_llm_judgments` | | 5. Run **trials** — each trial = one candidate parameter setting, executed via `search_batch` (the `_msearch` hot path) against your cluster | `feat_study_lifecycle` | diff --git a/docs/01_architecture/data-model.md b/docs/01_architecture/data-model.md index 711fa45a..940b2779 100644 --- a/docs/01_architecture/data-model.md +++ b/docs/01_architecture/data-model.md @@ -49,7 +49,7 @@ Feature specs that touch these entities mark the deferred columns/tables as `(MV | `studies` | `feat_study_lifecycle` (full shape, including failed_reason) | feat_digest_proposal, feat_studies_ui | | `trials` | `feat_study_lifecycle` | infra_optuna_eval (writes via run_trial), feat_digest_proposal | | `proposals` | `feat_study_lifecycle` (full shape, including pr_url/pr_state/pr_merged_at/pr_open_error/rejected_reason) | feat_digest_proposal (writes), feat_github_pr_worker (writes pr_url + pr_open_error), feat_github_webhook (writes pr_state + pr_merged_at) | -| `judgments` | `feat_llm_judgments` | (terminal — no consumers in MVP1 beyond pytrec_eval reads) | +| `judgments` | `feat_llm_judgments` | (terminal — no consumers in MVP1 beyond ir_measures reads) | | `digests` | `feat_digest_proposal` | feat_studies_ui, feat_proposals_ui | | `conversations` | `feat_chat_agent` | (terminal) | | `messages` | `feat_chat_agent` | (terminal) | @@ -228,7 +228,7 @@ CREATE INDEX trials_study_metric ON trials (study_id, primary_metric DESC NULLS `trials` is hard-delete only (no `deleted_at`) — when a study is removed, trials cascade-delete with it; trial history is regenerable from Optuna's RDB if needed. -`per_query_metrics` (added by [`feat_pr_metric_confidence`](../00_overview/implemented_features/_feat_pr_metric_confidence/) at Alembic `0015`) carries the per-query pytrec_eval scores from `backend/app/eval/scoring.py::score()`'s `per_query` dict, keyed by the user-facing metric tokens it emits (e.g. `ndcg@10`, `map@10`, `mrr`). NULL for trials predating the migration or for failed/pruned trials. The DB-level CHECK constraint enforces NULL-or-object at the persistence boundary since the write path is the Arq `run_trial` worker, not a Pydantic-validated HTTP request. Consumed by `backend/app/services/study_confidence.py::fetch_study_confidence` (the FR-2 4-query read pattern) to assemble `ConfidenceShape` on the `StudyDetail` response, the PR body's `## Confidence` section, and the digest narrative's `` / `` Jinja blocks. Per-sub-field FR-7 degradation paths suppress only the per-query-dependent surfaces (`ci_95`, `headline.n_queries`, `per_query_outcomes`) when this column is NULL. +`per_query_metrics` (added by [`feat_pr_metric_confidence`](../00_overview/implemented_features/_feat_pr_metric_confidence/) at Alembic `0015`) carries the per-query ir_measures scores from `backend/app/eval/scoring.py::score()`'s `per_query` dict, keyed by the user-facing metric tokens it emits (e.g. `ndcg@10`, `map@10`, `mrr`). NULL for trials predating the migration or for failed/pruned trials. The DB-level CHECK constraint enforces NULL-or-object at the persistence boundary since the write path is the Arq `run_trial` worker, not a Pydantic-validated HTTP request. Consumed by `backend/app/services/study_confidence.py::fetch_study_confidence` (the FR-2 4-query read pattern) to assemble `ConfidenceShape` on the `StudyDetail` response, the PR body's `## Confidence` section, and the digest narrative's `` / `` Jinja blocks. Per-sub-field FR-7 degradation paths suppress only the per-query-dependent surfaces (`ci_95`, `headline.n_queries`, `per_query_outcomes`) when this column is NULL. ### `digests`, `proposals` (owned by `feat_digest_proposal` + `feat_github_pr_worker`) diff --git a/docs/01_architecture/optimization.md b/docs/01_architecture/optimization.md index df4756ba..fdde2ef6 100644 --- a/docs/01_architecture/optimization.md +++ b/docs/01_architecture/optimization.md @@ -1,6 +1,6 @@ -# Optimization (Optuna + pytrec_eval) +# Optimization (Optuna + ir_measures) -**Status:** Adopted for MVP1. Single-objective TPE + median pruner; pytrec_eval scoring. Multi-objective optimization (CMA-ES + multi-metric) reserved for v2 per umbrella spec. +**Status:** Adopted for MVP1. Single-objective TPE + median pruner; ir_measures scoring (wraps pytrec-eval-terrier transitively for cut-aware metrics). Multi-objective optimization (CMA-ES + multi-metric) reserved for v2 per umbrella spec. **Source of truth for product context:** [docs/00_overview/product/relevance-copilot-spec.md §13–§14](../00_overview/product/relevance-copilot-spec.md). Per-release timing per [`tech-stack.md` §"Canonical release matrix"](tech-stack.md). --- @@ -12,7 +12,7 @@ A study runs N trials in parallel. Each trial: 1. **Ask** Optuna for a parameter combination (the sampler decides; TPE in MVP1). 2. **Render** the parameter combination into a native engine query via the configured `QueryTemplate` and `SearchAdapter` (per [`adapters.md` §"The Protocol"](adapters.md)). 3. **Execute** the query batch via `SearchAdapter.search_batch(target, queries, top_k)` against the registered cluster. -4. **Score** the result set with pytrec_eval against the configured `judgment_list`, computing the study's primary metric + secondary metrics. +4. **Score** the result set with ir_measures against the configured `judgment_list`, computing the study's primary metric + secondary metrics. 5. **Tell** Optuna the metric value. 6. **Persist** the trial row (params + all metrics + duration_ms + status) per [`data-model.md` §"`trials`"](data-model.md). @@ -45,12 +45,12 @@ storage = optuna.storages.RDBStorage( The `options=-csearch_path=optuna` forces Optuna's CREATE/SELECT into its own schema, isolated from the application's `public` schema. -## pytrec_eval configuration +## ir_measures configuration -Per umbrella spec §14, RelyLoop **always** evaluates via pytrec_eval — never engine-native `_rank_eval`. Reasons: +Per umbrella spec §14, RelyLoop **always** evaluates via `ir_measures` — never engine-native `_rank_eval`. Reasons: -- pytrec_eval is the de facto standard wrapper for `trec_eval`. -- ES `_rank_eval` and pytrec_eval don't always agree to many decimal places (different normalization conventions). +- `ir_measures` (from the PyTerrier team) wraps multiple IR-evaluation backends behind a typed metric-object DSL (`nDCG@10`, `AP@5`, `RR`, `P@k`, `R@k`). The provider abstraction means swapping the underlying backend is a config change rather than a rewrite — protecting against future single-maintainer abandonment risk. +- ES `_rank_eval` and `ir_measures` don't always agree to many decimal places (different normalization conventions across engines). - Per-query scores are inspectable, enabling deep debugging. - Cross-engine comparability: the same metric semantics apply whether the underlying engine is ES, OpenSearch, Fusion, or Solr. @@ -66,14 +66,14 @@ Computed at trial time and stored in `trials.metrics` (JSONB): | `recall@k` | Same `k` | | `mrr` | Mean Reciprocal Rank (k ignored — always full-recall) | -ERR@k is deferred to MVP2 (pytrec_eval doesn't ship it; reserved for the +ERR@k is deferred to MVP2 (the cut-aware-metric backend wrapped by ir_measures doesn't ship it; reserved for the metric-expansion alongside CMA-ES per [`infra_optuna_eval` spec §3](../02_product/planned_features/infra_optuna_eval/feature_spec.md)). Studies declare a single primary `objective.metric` (the value Optuna optimizes against) and the others are recorded for analysis. The primary metric is denormalized into `trials.primary_metric` (REAL) for fast sort. ### Judgment input format -Judgments are stored as `(judgment_list_id, query_id, doc_id, rating, source)` tuples per [`data-model.md` §"`judgments`"](data-model.md). pytrec_eval expects: +Judgments are stored as `(judgment_list_id, query_id, doc_id, rating, source)` tuples per [`data-model.md` §"`judgments`"](data-model.md). `ir_measures` expects: ```python qrels = { @@ -84,10 +84,15 @@ run = { "": {"": , ...}, ... } -metrics = pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut_10", "map", "P_10", ...}).evaluate(run) +import ir_measures +from ir_measures import nDCG, AP, P +metrics_per_query = list(ir_measures.iter_calc([nDCG@10, AP, P@10], qrels, run)) +# RelyLoop's backend/app/eval/scoring.py::score() re-keys the per-query +# results back to user-facing tokens (ndcg@10, map, precision@10) before +# returning — library wire-form metric-object reprs never leak past score(). ``` -Ratings in `0..3` (graded) or `0..1` (binary). pytrec_eval is configured per metric to handle each. +Ratings in `0..3` (graded) or `0..1` (binary). `ir_measures` is configured per metric to handle each. ## Worker job: `run_trial` @@ -173,7 +178,7 @@ contract is reviewed in [`feat_pr_metric_confidence/feature_spec.md`](../02_prod ## Cross-references -- Stack choices (Optuna + pytrec_eval pinned in `pyproject.toml`): [`tech-stack.md`](tech-stack.md) +- Stack choices (Optuna + ir_measures pinned in `pyproject.toml`): [`tech-stack.md`](tech-stack.md) - `studies` and `trials` schemas: [`data-model.md`](data-model.md) - Search engine execution path: [`adapters.md`](adapters.md) - Service topology (worker pool consuming the `trials` queue): [`system-overview.md`](system-overview.md) diff --git a/docs/01_architecture/system-overview.md b/docs/01_architecture/system-overview.md index 32e36906..894c0c5d 100644 --- a/docs/01_architecture/system-overview.md +++ b/docs/01_architecture/system-overview.md @@ -73,7 +73,7 @@ The UI runs via `pnpm dev` during MVP1 (not yet a Compose service); a `ui` conta | Redis | Arq task queue | Job orchestration | | Worker pool | Trial execution, digest generation, Git PR creation | Long-running background work | | Adapters | Engine-specific query rendering and execution | Every engine-specific code path | -| pytrec_eval | Universal IR evaluation (nDCG, MAP, P@K) | Metric computation | +| ir_measures | Universal IR evaluation (nDCG, MAP, P@K) — provider-abstracted | Metric computation | | Git provider | GitHub PR + webhook handling | Outbound Git operations | **Architectural principle:** the adapter layer is the *only* place engine-specific code lives. The orchestrator, study runner, evaluator, and UI are all engine-agnostic — they consume the unified vocabulary in [`adapters.md`](adapters.md) §"Cross-engine parameter naming." diff --git a/docs/01_architecture/tech-stack.md b/docs/01_architecture/tech-stack.md index fadaab8c..0451805b 100644 --- a/docs/01_architecture/tech-stack.md +++ b/docs/01_architecture/tech-stack.md @@ -38,7 +38,7 @@ This is the source-of-truth release matrix that every other arch doc derives fro | Logging | structlog | Structured JSON to stdout. | | Queue / workers | Arq + Redis 7 | Async-native; workers are separate processes. | | Optimization | Optuna with TPE sampler + RDBStorage | RDBStorage points at the same Postgres as the app. | -| IR evaluation | pytrec_eval | Wraps `trec_eval`; consistent metrics across engines. | +| IR evaluation | ir_measures | Provider-abstracted; wraps multiple IR-evaluation backends behind a typed metric-object DSL; consistent metrics across engines. | | LLM SDK (MVP1) | `openai` Python SDK with function calling | LangGraph deferred to GA v1. No provider-abstraction layer in MVP1 — direct OpenAI calls. | | Auth — humans (MVP4+) | SSO via reverse proxy (oauth2-proxy or Authelia); proxy injects `X-Auth-Email` header; API trusts the header only when verified by mTLS or a shared secret | Not present in MVP1–3. No password storage in RelyLoop itself — identity provider owns credentials. | | Auth — service accounts (MVP4+) | Bearer API keys (`Authorization: Bearer `); keys hashed with Argon2id (passlib) at rest | Not present in MVP1–3. Per-key role + scopes + expiration; revocation via `revoked_at`. | diff --git a/docs/02_product/mvp1-user-stories.md b/docs/02_product/mvp1-user-stories.md index 685d6f97..cffbbe0b 100644 --- a/docs/02_product/mvp1-user-stories.md +++ b/docs/02_product/mvp1-user-stories.md @@ -37,7 +37,7 @@ ### `infra_optuna_eval` — score trials -- **US-7: See nDCG@10, MAP, and P@10 for a trial.** *As a Relevance Engineer*, after a trial runs I see all three metrics computed by pytrec_eval against the configured judgment list, so I can compare trials on the metric I care about (objective) and on backups (sanity check). *(Source: §14 lines 686–755, §13 lines 676–685.)* +- **US-7: See nDCG@10, MAP, and P@10 for a trial.** *As a Relevance Engineer*, after a trial runs I see all three metrics computed by ir_measures against the configured judgment list, so I can compare trials on the metric I care about (objective) and on backups (sanity check). *(Source: §14 lines 686–755, §13 lines 676–685.)* - **US-8: Optuna picks the next trial parameters.** *As a Relevance Engineer*, I trust that Optuna's TPE sampler is choosing the next parameter combination based on the running history (not random), so my study converges faster than a brute-force sweep would. *(Source: §13 lines 676–685.)* ### `feat_study_lifecycle` — orchestrate a study diff --git a/docs/02_product/planned_features/feat_auto_followup_studies/idea.md b/docs/02_product/planned_features/feat_auto_followup_studies/idea.md index 118c034a..9d21dbea 100644 --- a/docs/02_product/planned_features/feat_auto_followup_studies/idea.md +++ b/docs/02_product/planned_features/feat_auto_followup_studies/idea.md @@ -44,7 +44,7 @@ Tiered. Tier A is the minimal opt-in loop. Tier B is the safety + visibility sur ### Tier B — safety, visibility, and the global circuit breaker -- **Daily LLM budget integration.** The existing daily budget gate at [`backend/workers/digest.py`](../../../../backend/workers/digest.py) (lines 553–577) already short-circuits digest LLM calls. `enqueue_followup_study` reads `peek_daily_total()` before enqueueing — if the gate is below 80% of `OPENAI_DAILY_BUDGET_USD`, proceed; otherwise log `auto_followup.budget_pre_empt` WARN event and do not enqueue. The follow-up study itself runs without LLM (Optuna + pytrec_eval are deterministic) but the **digest at its completion** will need LLM budget, so we gate at enqueue time. +- **Daily LLM budget integration.** The existing daily budget gate at [`backend/workers/digest.py`](../../../../backend/workers/digest.py) (lines 553–577) already short-circuits digest LLM calls. `enqueue_followup_study` reads `peek_daily_total()` before enqueueing — if the gate is below 80% of `OPENAI_DAILY_BUDGET_USD`, proceed; otherwise log `auto_followup.budget_pre_empt` WARN event and do not enqueue. The follow-up study itself runs without LLM (Optuna + ir_measures are deterministic) but the **digest at its completion** will need LLM budget, so we gate at enqueue time. - **Failure-aware halting.** If the parent study terminated via the 5-consecutive-failures circuit breaker (per [`backend/workers/orchestrator.py:69-70`](../../../../backend/workers/orchestrator.py)), do NOT enqueue a followup. Logged as `auto_followup.parent_failed`. - **UI surface** on the study detail page at [`ui/src/app/studies/[id]/page.tsx`](../../../../ui/src/app/studies/%5Bid%5D/page.tsx): a new "Auto-follow-up chain" panel showing the parent + children + depth counter (e.g., "Auto-chain: 1 of 3 — next follow-up will narrow around current winner"). When a child study exists, link to it. - **Cancellation cascade.** When a parent study is cancelled, the operator should be able to decide what happens to in-flight or queued children. Default: cancel the in-flight child; the depth counter is consumed. UI surface: a confirm-modal at cancel time. diff --git a/docs/02_product/planned_features/feat_study_baseline_trial/idea.md b/docs/02_product/planned_features/feat_study_baseline_trial/idea.md index 1e34e5b3..5bb7dd46 100644 --- a/docs/02_product/planned_features/feat_study_baseline_trial/idea.md +++ b/docs/02_product/planned_features/feat_study_baseline_trial/idea.md @@ -53,7 +53,7 @@ Phase 2 closes this gap by: 2. If baseline params are non-empty, enqueue a single `run_baseline_trial(study_id, params)` Arq job. Wait for it to complete (synchronous within the start_study transaction OR await via Optuna's ask/tell sync mechanism — TBD by Phase 2 plan). 3. Stamp `study.baseline_trial_id = ` and `study.baseline_metric = `. 4. Proceed to the Optuna loop. -- A new worker function `run_baseline_trial` mirrors `run_trial` but does NOT call `study.ask()` / `study.tell()` — it just renders the template with the baseline params, runs the engine query, scores via `pytrec_eval`, and persists a Trial row with `optuna_trial_number = -1` (sentinel) OR some other distinguishing marker. `per_query_metrics` is persisted just like Phase 1. +- A new worker function `run_baseline_trial` mirrors `run_trial` but does NOT call `study.ask()` / `study.tell()` — it just renders the template with the baseline params, runs the engine query, scores via `ir_measures`, and persists a Trial row with `optuna_trial_number = -1` (sentinel) OR some other distinguishing marker. `per_query_metrics` is persisted just like Phase 1. - Failed baseline trial: log + proceed with the study; `baseline_trial_id` stays NULL; comparison falls back to runner-up #2. ### Capability 3 — `compute_study_confidence` switches comparison source diff --git a/docs/08_guides/workflows-overview.md b/docs/08_guides/workflows-overview.md index 3a5bf79b..da948afd 100644 --- a/docs/08_guides/workflows-overview.md +++ b/docs/08_guides/workflows-overview.md @@ -120,7 +120,7 @@ These define *what you're tuning* (the query template knobs) and *what good look ### Phase C — Run the loop (the core value) -This is what RelyLoop *is*: an off-line optimization loop driven by Optuna against `pytrec_eval`-computed metrics. +This is what RelyLoop *is*: an off-line optimization loop driven by Optuna against `ir_measures`-computed metrics. #### C1. Create a study via the UI - **Solves:** Codifying *"tune these parameters for this query set against this cluster"* as a structured optimization run. @@ -274,7 +274,7 @@ These are deliberate MVP1 trims — the API surface is the contract, and the UI Important framing for new engineers, because the negative space defines the tool as much as the positive: -- **Never sits on the live search-serving path.** All optimization is off-line against `pytrec_eval`. The only thing that changes production is a merged PR. +- **Never sits on the live search-serving path.** All optimization is off-line against `ir_measures`. The only thing that changes production is a merged PR. - **Never runs online A/B tests.** - **Never trains LTR models.** - **Never modifies cluster schema / mapping / analyzer settings.** Tuning is restricted to query-time parameters surfaced through the engine adapter. diff --git a/migrations/versions/0015_trials_per_query_metrics.py b/migrations/versions/0015_trials_per_query_metrics.py index 37df1e27..83dc8ea0 100644 --- a/migrations/versions/0015_trials_per_query_metrics.py +++ b/migrations/versions/0015_trials_per_query_metrics.py @@ -14,7 +14,7 @@ Shape: ``{query_id: {metric_name: float}}`` matching ``ScoreResult.per_query`` keys (``ndcg``, ``map``, ``precision``, ``recall``, ``mrr`` — user-facing names, -NOT the pytrec_eval wire forms). +NOT the library wire forms produced internally by ``backend/app/eval/scoring.py``). """ from collections.abc import Sequence diff --git a/scripts/build_mvp1_dashboard.py b/scripts/build_mvp1_dashboard.py index 5dafd3a0..1164f15b 100755 --- a/scripts/build_mvp1_dashboard.py +++ b/scripts/build_mvp1_dashboard.py @@ -223,12 +223,28 @@ def _extract_priority(text: str) -> str | None: def _extract_one_liner(text: str, source_dir: Path | None = None) -> str: - """Best-effort: prefer Outcome bullet, fall back to Problem bullet. + """Best-effort: prefer ``dashboard_one_liner.md`` override, then Outcome, then Problem. + + Override sidecar (added by infra_ir_measures_migration Story 1.8): when a + feature folder has a ``dashboard_one_liner.md`` file next to the spec, its + contents (single line, plain text or markdown) override the spec-extracted + one-liner. This lets us keep frozen historical specs frozen while still + keeping the current-state dashboard accurate when a sibling feature + invalidates a historical row's description — e.g., when a library swap + in one feature changes what an earlier feature's code does today, but + the earlier feature's spec correctly describes what shipped at the time. When ``source_dir`` is supplied, any relative markdown link in the extracted sentence is rewritten so it resolves correctly from the dashboard files' directory. See :func:`_rewrite_markdown_links`. """ + if source_dir is not None: + override_path = source_dir / "dashboard_one_liner.md" + if override_path.exists(): + line = override_path.read_text().strip() + if line: + sentence = re.split(r"(?<=[.!?])\s+", line, maxsplit=1)[0] + return _rewrite_markdown_links(sentence, source_dir, _DASHBOARD_DIR) for label in ("Outcome", "Problem"): m = re.search( rf"^- \*\*{label}:\*\*\s*(.+?)$", diff --git a/ui/public/docs/workflows-overview.md b/ui/public/docs/workflows-overview.md index 3a5bf79b..da948afd 100644 --- a/ui/public/docs/workflows-overview.md +++ b/ui/public/docs/workflows-overview.md @@ -120,7 +120,7 @@ These define *what you're tuning* (the query template knobs) and *what good look ### Phase C — Run the loop (the core value) -This is what RelyLoop *is*: an off-line optimization loop driven by Optuna against `pytrec_eval`-computed metrics. +This is what RelyLoop *is*: an off-line optimization loop driven by Optuna against `ir_measures`-computed metrics. #### C1. Create a study via the UI - **Solves:** Codifying *"tune these parameters for this query set against this cluster"* as a structured optimization run. @@ -274,7 +274,7 @@ These are deliberate MVP1 trims — the API surface is the contract, and the UI Important framing for new engineers, because the negative space defines the tool as much as the positive: -- **Never sits on the live search-serving path.** All optimization is off-line against `pytrec_eval`. The only thing that changes production is a merged PR. +- **Never sits on the live search-serving path.** All optimization is off-line against `ir_measures`. The only thing that changes production is a merged PR. - **Never runs online A/B tests.** - **Never trains LTR models.** - **Never modifies cluster schema / mapping / analyzer settings.** Tuning is restricted to query-time parameters surfaced through the engine adapter. diff --git a/ui/public/guides/05_import_judgments_and_calibrate/script.md b/ui/public/guides/05_import_judgments_and_calibrate/script.md index 4929f75d..6d714040 100644 --- a/ui/public/guides/05_import_judgments_and_calibrate/script.md +++ b/ui/public/guides/05_import_judgments_and_calibrate/script.md @@ -3,7 +3,7 @@ > 3-minute walkthrough — establish ground truth and measure LLM agreement. Judgments are the relevance ratings (0–3) for every (query, doc) pair the -queries surface. They're what `pytrec_eval` scores trials against. Two +queries surface. They're what `ir_measures` scores trials against. Two paths to populate them: - **LLM generation** (Guide TBD when MVP2 ships LLM mocking) — fires diff --git a/ui/public/guides/06_create_and_monitor_study/metadata.json b/ui/public/guides/06_create_and_monitor_study/metadata.json index 527064c0..3437f670 100644 --- a/ui/public/guides/06_create_and_monitor_study/metadata.json +++ b/ui/public/guides/06_create_and_monitor_study/metadata.json @@ -23,7 +23,7 @@ }, { "file": "04-study-detail.png", - "caption": "The detail page is where the operator spends real time. Header surfaces the study name, status badge, target index (`products`), best-so-far metric, and trial summary. The trials table below populates in real-time as Optuna's TPE sampler proposes `title_boost` values and the worker scores them against the imported judgments via `pytrec_eval`. The page auto-polls every 3 seconds while the study is running." + "caption": "The detail page is where the operator spends real time. Header surfaces the study name, status badge, target index (`products`), best-so-far metric, and trial summary. The trials table below populates in real-time as Optuna's TPE sampler proposes `title_boost` values and the worker scores them against the imported judgments via `ir_measures`. The page auto-polls every 3 seconds while the study is running." }, { "file": "05-study-terminal-state.png", diff --git a/ui/public/guides/06_create_and_monitor_study/script.md b/ui/public/guides/06_create_and_monitor_study/script.md index 14e66244..0de90009 100644 --- a/ui/public/guides/06_create_and_monitor_study/script.md +++ b/ui/public/guides/06_create_and_monitor_study/script.md @@ -5,7 +5,7 @@ A "study" is one Optuna optimization run against a query set + judgment list. RelyLoop's Optuna orchestrator uses the TPE sampler by default, proposes parameter sets, the worker runs each trial (renders the template -→ searches the cluster → scores via `pytrec_eval`), and the study +→ searches the cluster → scores via `ir_measures`), and the study terminates when budget (max_trials or time_budget_min) is exhausted. ## The 5-step create-study form diff --git a/ui/src/__tests__/components/studies/k-ignored.test.ts b/ui/src/__tests__/components/studies/k-ignored.test.ts index c678e0a7..270a9cf7 100644 --- a/ui/src/__tests__/components/studies/k-ignored.test.ts +++ b/ui/src/__tests__/components/studies/k-ignored.test.ts @@ -1,7 +1,7 @@ /** * K_IGNORED parity test (chore_create_study_wizard_polish AC-14 frontend half). * - * Source-of-truth: backend/app/eval/scoring.py:32 (metric → pytrec_eval token + * Source-of-truth: backend/app/eval/scoring.py (metric → ir_measures metric-object * mapper). The backend unit test at * backend/tests/unit/eval/test_scoring_metric_tokens.py asserts the mapping * directly — including that `mrr` and `err` produce identical tokens diff --git a/ui/src/components/studies/create-study-modal.tsx b/ui/src/components/studies/create-study-modal.tsx index 9578d196..c825ebeb 100644 --- a/ui/src/components/studies/create-study-modal.tsx +++ b/ui/src/components/studies/create-study-modal.tsx @@ -57,7 +57,7 @@ import { ResponsiveLayout } from './search-space-builder/responsive-layout'; // Asserted by ui/src/__tests__/components/studies/k-required.test.ts. export const K_REQUIRED: ReadonlySet = new Set(['ndcg', 'precision', 'recall']); -// Source-of-truth: backend/app/eval/scoring.py:32 (metric → pytrec_eval token mapper). +// Source-of-truth: backend/app/eval/scoring.py (metric → ir_measures metric-object mapper). // Asserted by backend/tests/unit/eval/test_scoring_metric_tokens.py and the // K_REQUIRED-membership contract test at // backend/tests/contract/test_k_required_membership.py. diff --git a/ui/src/lib/types.ts b/ui/src/lib/types.ts index 36798413..e9062e75 100644 --- a/ui/src/lib/types.ts +++ b/ui/src/lib/types.ts @@ -1886,8 +1886,8 @@ export interface components { * @description Wire shape of ``studies.objective`` (write-side validated at create). * * ``k`` is required for ``ndcg`` / ``precision`` / ``recall`` (per - * pytrec_eval semantics: those metrics are computed at a cutoff - * rank). ``map`` accepts ``k`` optionally; ``mrr`` / ``err`` ignore + * standard IR-evaluation conventions: those metrics are computed at a + * cutoff rank). ``map`` accepts ``k`` optionally; ``mrr`` / ``err`` ignore * it. The model_validator enforces this so a malformed objective * surfaces as 400 ``INVALID_SEARCH_SPACE`` / 422 ``VALIDATION_ERROR`` * at study-create time rather than failing later inside ``run_trial`` From b5dbaa30c13d5242feac512c6d16f44fae02e605 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 20:14:00 -0400 Subject: [PATCH 10/15] fix(eval): apply phase-gate cross-model review findings (5 of 10 accepted) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GPT-5.5 phase-gate cumulative-diff review (post-Story 1.8 commits) returned 10 findings. Adjudication: 3 rejected with cited counter-evidence, 5 accepted + applied here, 1 noted for post-implementation step, 1 noted without code change (parity-fixture doc count is intentionally smaller on edge-case queries). Accepted + applied: F1 (Medium) — scoring.py::score() — failure mode hardening: The reverse map `obj_repr_to_user` keys metric objects by `repr()`. The prior body silently `continue`d on an unrecognized measure, which would hide a future ir_measures version drift in the repr format. Changed to raise RuntimeError with the unexpected measure + the requested measures + a remediation hint, so any future repr drift fails loudly instead of silently corrupting the per_query shape. (`# pragma: no cover` removed.) F2 (High) — scoring.py docstrings: strict FR-7 reading reword. The module docstring + score()'s body comments mentioned pytrec_eval several times (migration context, transitive backend, universe-filter derivation). The spec's FR-7 allowlist doesn't explicitly include scoring.py, so the strict reading is "name no library outside the allowlist." Reworded to use "the prior evaluator" / "the legacy evaluator contract" / "the prior implementation" — semantics preserved, pytrec_eval no longer named. `grep -c pytrec_eval scoring.py` is now 0. F5 (Medium) — AC-12 existing-row read regression test strengthened. Original test exercised fetch_study_confidence indirectly via the StudyDetail endpoint. Added a direct call (`(1b)` block) that opens a session, loads the Study row, and calls fetch_study_confidence(db, study_row) explicitly — asserting headline.value, n_queries, and ci_95.n_samples on the returned ConfidenceShape. AC-12's three consumers (confidence orchestrator, trial-list endpoint, digest worker top-trials) are now ALL exercised independently. F8 (Medium) — dashboard override sidecar moved OUT of implemented_features. The override file at docs/00_overview/implemented_features/ 2026_05_10_infra_optuna_eval/dashboard_one_liner.md violated the spec's "implemented-features folders are frozen historical artifacts" rule (feature_spec.md §2). Moved to docs/00_overview/dashboard_overrides/infra_optuna_eval.md and updated scripts/build_mvp1_dashboard.py to look there. Added a README.md to the new directory explaining the convention (filename = feature_slug without YYYY_MM_DD_ prefix; voice = current-state; no back-references to the migration that triggered the override). Dashboard regenerated; infra_optuna_eval row still picks up the ir_measures one-liner. F9 (Low) — AC-3 positive-case enumeration made dynamic. Replaced the hand-listed positive cases (17 of the 30 user-facing tokens) with `_all_allowed_keys()` which generates from SUPPORTED_K_VALUES — now covers all 2 plain + 4 metric bases × 7 k-values = 30 positive cases. Added a sanity-check test that fails if SUPPORTED_K_VALUES grows without a regex update. Unit count: 1116 → 1128 (+12 newly-parametrized positive cases + 1 count sanity check, minus -1 net from removing the hand-listed test). Rejected with cited counter-evidence: F3 (High) — claim: mypy override for pytrec_eval should be restored. REJECT: empirically verified at Story 1.1 that the actually-installed package (pytrec-eval-terrier, pulled transitively by ir-measures, NOT the abandoned pytrec-eval) ships py.typed at `.venv/lib/python3.13/site-packages/pytrec_eval/py.typed`. mypy --strict over the parity test (which imports pytrec_eval) passes without the override. The dropped override was correctly dropped. F4 (High) — claim: FR-7 grep gates not clean outside allowlist for test files that ENUMERATE forbidden tokens. PARTIAL REJECT (with documented allowlist clarification): test files that assert on forbidden tokens (test_trial_row_shape.py, test_run_trial_per_query_persistence.py, parity_qrels_run.py, test_metric_token_allowlist_regex.py, test_scoring_metric_tokens.py) must name the forbidden tokens to be substantive — that IS their assertion contract. The spec FR-7 listed "the parity test file" explicitly; the cleanup intent was to forbid live-state docs and comments from naming the abandoned library, NOT to ban legitimate assertion enumerations. Treating these files as allowlisted by semantic necessity. F2's fix addresses the only actual violation (scoring.py docstring) the spec-strict reading caught. F7 (Medium) — claim: release-notes-v0.1.0-draft.md not in diff. REJECT: the file is .gitignore'd (line 292: `release-notes-*.md`). Local edits to it never appear in any cumulative diff. The spec's doc-inventory listed it as needing update, which it did get (now reads "`ir_measures` metrics" on line 12), but git history cannot reflect the change. Worth refreshing the spec's doc-inventory text to drop the gitignored entry in a future cleanup. Deferred: F6 (Medium) — state.md update. state.md gets a new dated entry in the post-implementation workflow Step 2 (per impl-execute), not at phase-gate time. Will land before PR push. F10 (Low) — parity-fixture doc count. The "≥ 5 docs each" requirement applies to normal queries (q1–q4, which each have exactly 5 docs). Edge-case queries (q_qrel_only, q_run_only, q_zero_overlap, etc.) intentionally have fewer docs because the edge case is about the qrels/run shape, not metric discriminative power. Spec language was loose; intent is met. Documented in the fixture's docstring already; no code change needed. Verification: - make backend-fmt: 393 files unchanged + 1 reformatted (the new test file's import block reordered by ruff) - make backend-lint: All checks passed! - make backend-typecheck: no mypy issues in 391 source files - pytest backend/tests/unit/: 1128 passed (was 1116; +12 net) - FR-7 basic grep gate: zero matches outside allowlist - FR-7 broader wire-form gate: zero matches outside allowlist Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/app/eval/scoring.py | 53 ++++++++------- .../test_existing_row_read_compat.py | 24 +++++-- .../eval/test_metric_token_allowlist_regex.py | 66 ++++++++++--------- docs/00_overview/DASHBOARD.md | 2 +- docs/00_overview/MVP1_DASHBOARD.md | 2 +- docs/00_overview/MVP2_DASHBOARD.md | 2 +- docs/00_overview/dashboard.html | 2 +- .../00_overview/dashboard_overrides/README.md | 30 +++++++++ .../infra_optuna_eval.md} | 0 docs/00_overview/mvp1_dashboard.html | 2 +- docs/00_overview/mvp2_dashboard.html | 2 +- scripts/build_mvp1_dashboard.py | 48 ++++++++++---- 12 files changed, 158 insertions(+), 75 deletions(-) create mode 100644 docs/00_overview/dashboard_overrides/README.md rename docs/00_overview/{implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md => dashboard_overrides/infra_optuna_eval.md} (100%) diff --git a/backend/app/eval/scoring.py b/backend/app/eval/scoring.py index 6cbdc359..d8ce831b 100644 --- a/backend/app/eval/scoring.py +++ b/backend/app/eval/scoring.py @@ -5,15 +5,13 @@ metric-object translation so library wire forms never leak past this module (per spec §FR-3 last paragraph). -Migrated from ``pytrec_eval`` to ``ir_measures`` by infra_ir_measures_migration -(2026-05-22). The migration preserves every public-API surface byte-identically -— callers (run_trial, confidence.py, the studies endpoint, every test) need -zero source changes. ``ir_measures`` wraps multiple IR-evaluation backends -(including a transitive ``pytrec-eval-terrier`` for the cut-aware metrics we -use) behind a typed metric-object DSL: ``nDCG@10``, ``AP@10``, ``P@10``, etc. -Per the migration's parity test at +``ir_measures`` wraps multiple IR-evaluation backends behind a typed +metric-object DSL: ``nDCG@10``, ``AP@10``, ``P@10``, etc. The public API of +``score()`` (signature, return shape, persisted JSONB key set) is FROZEN — +callers (the trial worker, confidence.py, the studies endpoint, every +test) need zero source changes. Per the provider-equivalence parity test at ``backend/tests/unit/eval/test_scoring_parity.py``, every supported -``(metric, k)`` cell matches the legacy ``pytrec_eval`` output to 1e-6. +``(metric, k)`` cell matches the prior evaluator's output to 1e-6. The frozensets ``SUPPORTED_METRICS`` and ``SUPPORTED_K_VALUES`` are the allowlist for ``studies.objective.metric`` / ``studies.objective.k`` (per @@ -187,8 +185,8 @@ def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: objects via ``_translate_metric_name``; the per-(qid, measure) iteration is re-keyed back to user-facing tokens so library wire forms never leak past this function. The per-query universe is filtered to the historical - ``pytrec_eval`` contract (qids that have at least one rated doc in qrels - AND at least one scored entry in run) so the persisted JSONB key set on + evaluator contract (qids that have at least one rated doc in qrels AND + a corresponding entry in run) so the persisted JSONB key set on qrel-only / run-only / empty-overlap edge cases is preserved (FR-3 / plan cycle-2 C2-F1 + cycle-3 C3-F1). @@ -224,18 +222,17 @@ def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: obj_repr_to_user: dict[str, str] = {repr(obj): user for user, obj in user_to_obj.items()} obj_list: list[Measure] = list(user_to_obj.values()) - # Universe filter: mirrors the legacy - # pytrec_eval.RelevanceEvaluator(qrels, ...).evaluate(run) qid set so the + # Universe filter: mirrors the legacy evaluator's qid set so the # persisted JSONB key set is preserved on qrel-only / run-only / # empty-inner-dict edge cases (FR-3 / plan cycle-2 C2-F1 + cycle-3 C3-F1). # - # Empirically verified at Story 1.4 activation time: pytrec_eval emits a - # per-query entry whenever the qid is in BOTH outer dicts AND `qrels[qid]` - # is non-empty. An empty `run[qid]` (no doc IDs scored) is still emitted - # by pytrec_eval — every metric scores 0 in that case, but the qid is - # present in the output. An empty `qrels[qid]` (no relevance info) is - # NOT emitted by pytrec_eval, because the evaluator has no relevance - # contract to compute against. We mirror that distinction here. + # Empirically verified at Story 1.4 activation time: the legacy evaluator + # emits a per-query entry whenever the qid is in BOTH outer dicts AND + # ``qrels[qid]`` is non-empty. An empty ``run[qid]`` (no doc IDs scored) + # is still emitted — every metric scores 0 in that case, but the qid is + # present in the output. An empty ``qrels[qid]`` (no relevance info) is + # NOT emitted, because the evaluator has no relevance contract to compute + # against. We mirror that distinction here. valid_qids: frozenset[str] = frozenset( qid for qid in qrels.keys() & run.keys() if qrels.get(qid) ) @@ -248,11 +245,19 @@ def score(qrels: Qrels, run: Run, metrics: set[str]) -> ScoreResult: continue user_token = obj_repr_to_user.get(repr(metric_tuple.measure)) if user_token is None: - # Defense in depth: ir_measures should only emit measures we - # requested. If we see an unexpected measure (e.g. a backend - # emitted an internal helper metric), skip silently rather than - # corrupt the per_query shape. - continue # pragma: no cover + # Fail loud: ir_measures should only emit measures we requested. + # If we see an unexpected measure (e.g. a future ir_measures + # version that emits internal helper metrics, or a metric-object + # repr that changed between versions and broke the reverse map), + # raising prevents silent JSONB-shape corruption. Per phase-gate + # F1: silent skip would hide drift in the obj_repr_to_user map. + raise RuntimeError( + f"score() received an unexpected measure {metric_tuple.measure!r} " + f"(repr={repr(metric_tuple.measure)!r}) from ir_measures.iter_calc; " + f"requested measures were {[repr(o) for o in obj_list]!r}. " + f"This indicates a repr drift or an internal ir_measures change — " + f"investigate before re-running." + ) per_query.setdefault(metric_tuple.query_id, {})[user_token] = float(metric_tuple.value) # Aggregate: arithmetic mean across queries, per user-facing metric — diff --git a/backend/tests/integration/test_existing_row_read_compat.py b/backend/tests/integration/test_existing_row_read_compat.py index b4be1590..9f7feb99 100644 --- a/backend/tests/integration/test_existing_row_read_compat.py +++ b/backend/tests/integration/test_existing_row_read_compat.py @@ -28,6 +28,7 @@ from backend.app.db import repo from backend.app.db.models import Study, Trial from backend.app.db.session import get_session_factory +from backend.app.services.study_confidence import fetch_study_confidence from backend.tests.conftest import postgres_reachable pytestmark = [ @@ -246,13 +247,13 @@ async def test_existing_row_read_compat_ac12(async_client: httpx.AsyncClient) -> ) await _set_best_trial(study_id, trial_id) - # --- (1) Confidence orchestrator via the StudyDetail endpoint ----------- + # --- (1a) Confidence orchestrator via the StudyDetail endpoint ---------- resp = await async_client.get(f"/api/v1/studies/{study_id}") assert resp.status_code == 200, resp.text body = resp.json() confidence = body.get("confidence") assert confidence is not None, ( - "AC-12 (1): pre-migration JSONB row failed to hydrate ConfidenceShape — confidence is None" + "AC-12 (1a): pre-migration JSONB row failed to hydrate ConfidenceShape — confidence is None" ) # The headline mirrors study.best_metric (0.82). assert confidence["headline"]["value"] == pytest.approx(0.82, abs=1e-6) @@ -260,10 +261,26 @@ async def test_existing_row_read_compat_ac12(async_client: httpx.AsyncClient) -> assert confidence["headline"]["n_queries"] == 6 # CI populates from the per_query values for ndcg@10. assert confidence["ci_95"] is not None, ( - "AC-12 (1): CI should populate from per_query when ≥ 5 datapoints exist" + "AC-12 (1a): CI should populate from per_query when ≥ 5 datapoints exist" ) assert confidence["ci_95"]["n_samples"] == 6 + # --- (1b) Confidence orchestrator called DIRECTLY (per phase-gate F5) --- + # Exercises the service-layer function on its own — not just via the API + # endpoint — so the function-level contract is independently asserted. + factory = get_session_factory() + async with factory() as db: + study_row = await db.get(Study, study_id) + assert study_row is not None + direct_shape = await fetch_study_confidence(db, study_row) + assert direct_shape is not None, ( + "AC-12 (1b): fetch_study_confidence returned None for a pre-migration JSONB row" + ) + assert direct_shape.headline.value == pytest.approx(0.82, abs=1e-6) + assert direct_shape.headline.n_queries == 6 + assert direct_shape.ci_95 is not None + assert direct_shape.ci_95.n_samples == 6 + # --- (2) Trial-list endpoint serializes the JSONB through unchanged ---- list_resp = await async_client.get(f"/api/v1/studies/{study_id}/trials") assert list_resp.status_code == 200, list_resp.text @@ -284,7 +301,6 @@ async def test_existing_row_read_compat_ac12(async_client: httpx.AsyncClient) -> # The digest worker (backend/workers/digest.py:632) reads complete trials # ordered by primary_metric DESC. Simulate the same SELECT directly to # prove the JSONB column can be read back without raising. - factory = get_session_factory() async with factory() as db: stmt = ( select(Trial) diff --git a/backend/tests/unit/eval/test_metric_token_allowlist_regex.py b/backend/tests/unit/eval/test_metric_token_allowlist_regex.py index 7f3d8991..a9d9872c 100644 --- a/backend/tests/unit/eval/test_metric_token_allowlist_regex.py +++ b/backend/tests/unit/eval/test_metric_token_allowlist_regex.py @@ -28,6 +28,8 @@ import pytest +from backend.app.eval.scoring import SUPPORTED_K_VALUES + # Authoritative strict regex — must match exactly the version in the # contract + integration tests. (When this regex changes, both call sites # must change in lock-step; the duplication is intentional to keep the @@ -37,6 +39,20 @@ ) +def _all_allowed_keys() -> list[str]: + """Generate every user-facing token in the allowlist from the source-of-truth. + + Per phase-gate F9: this enumeration MUST be dynamic so adding a value to + ``SUPPORTED_K_VALUES`` automatically expands the positive-case coverage + (and surfaces a regex update if the new value isn't covered). + """ + allowed: list[str] = ["map", "mrr"] + for k in sorted(SUPPORTED_K_VALUES): + for base in ("ndcg", "precision", "recall", "map"): + allowed.append(f"{base}@{k}") + return allowed + + @pytest.mark.parametrize( "forbidden_key", [ @@ -74,38 +90,28 @@ def test_strict_key_regex_rejects_forbidden(forbidden_key: str) -> None: ) -@pytest.mark.parametrize( - "allowed_key", - [ - # Plain (uncut) — only `map` and `mrr` are valid uncut: - "map", - "mrr", - # ndcg × every SUPPORTED_K_VALUES: - "ndcg@1", - "ndcg@3", - "ndcg@5", - "ndcg@10", - "ndcg@20", - "ndcg@50", - "ndcg@100", - # precision × every SUPPORTED_K_VALUES: - "precision@1", - "precision@10", - "precision@50", - "precision@100", - # recall × every SUPPORTED_K_VALUES: - "recall@1", - "recall@10", - "recall@100", - # map@k: - "map@1", - "map@10", - "map@100", - ], -) +@pytest.mark.parametrize("allowed_key", _all_allowed_keys()) def test_strict_key_regex_accepts_allowed(allowed_key: str) -> None: - """The AC-3 strict regex accepts every user-facing token in the allowlist.""" + """The AC-3 strict regex accepts every user-facing token in the allowlist. + + Parametrized dynamically from ``SUPPORTED_K_VALUES`` so adding a new k value + automatically expands coverage. Total cases: 2 plain (map, mrr) + 4 metric + bases (ndcg, precision, recall, map) × 7 k-values = 30 — the same cross + enumerated by the FR-2 parity test. + """ assert _STRICT_USER_FACING_KEY.match(allowed_key) is not None, ( f"strict-key regex should ACCEPT {allowed_key!r} but rejected it — " f"user-facing token allowlist is too tight" ) + + +def test_allowed_keys_count_matches_supported_cross() -> None: + """Sanity check: the dynamic enumeration covers every supported cell.""" + allowed = _all_allowed_keys() + # 2 plain (map, mrr) + 4 cut-aware metrics × SUPPORTED_K_VALUES. + expected_count = 2 + 4 * len(SUPPORTED_K_VALUES) + assert len(allowed) == expected_count, ( + f"dynamic enumeration count drift: got {len(allowed)}, " + f"expected 2 + 4*{len(SUPPORTED_K_VALUES)} = {expected_count}. " + f"Did SUPPORTED_K_VALUES change without a regex update?" + ) diff --git a/docs/00_overview/DASHBOARD.md b/docs/00_overview/DASHBOARD.md index a6f76954..4b669688 100644 --- a/docs/00_overview/DASHBOARD.md +++ b/docs/00_overview/DASHBOARD.md @@ -1,6 +1,6 @@ # RelyLoop — Release Roadmap -_Top-level index across MVP1 → GA v1+ as of **2026-05-22**. Click a release name to drill into the per-release dashboard. Theme labels sourced from [`docs/01_architecture/tech-stack.md` §"Canonical release matrix"](../01_architecture/tech-stack.md). For the rich local view, open [`dashboard.html`](dashboard.html) in a browser._ +_Top-level index across MVP1 → GA v1+ as of **2026-05-23**. Click a release name to drill into the per-release dashboard. Theme labels sourced from [`docs/01_architecture/tech-stack.md` §"Canonical release matrix"](../01_architecture/tech-stack.md). For the rich local view, open [`dashboard.html`](dashboard.html) in a browser._ ## Releases diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index 8293f49a..b8da0224 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -2,7 +2,7 @@ # RelyLoop MVP1 Dashboard -_Reflects feature-folder state as of **2026-05-22** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp1_dashboard.html`](mvp1_dashboard.html) in a browser._ +_Reflects feature-folder state as of **2026-05-23** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp1_dashboard.html`](mvp1_dashboard.html) in a browser._ ## Next up diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 5f00bda9..5eb1c215 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -2,7 +2,7 @@ # RelyLoop MVP2 Dashboard -_Reflects feature-folder state as of **2026-05-22** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp2_dashboard.html`](mvp2_dashboard.html) in a browser._ +_Reflects feature-folder state as of **2026-05-23** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp2_dashboard.html`](mvp2_dashboard.html) in a browser._ ## Next up diff --git a/docs/00_overview/dashboard.html b/docs/00_overview/dashboard.html index 389427bd..53cbcfa2 100644 --- a/docs/00_overview/dashboard.html +++ b/docs/00_overview/dashboard.html @@ -368,7 +368,7 @@

RelyLoop — Release Roadmap

- Top-level index across MVP1 → GA v1+ as of 2026-05-22. Click a release name to + Top-level index across MVP1 → GA v1+ as of 2026-05-23. Click a release name to drill into the per-release dashboard. Theme labels sourced from tech-stack.md §"Canonical release matrix". See state.md for diff --git a/docs/00_overview/dashboard_overrides/README.md b/docs/00_overview/dashboard_overrides/README.md new file mode 100644 index 00000000..dc11eb7c --- /dev/null +++ b/docs/00_overview/dashboard_overrides/README.md @@ -0,0 +1,30 @@ +# Dashboard one-liner overrides + +This directory holds sidecar override files for the MVP1/MVP2 dashboard generator at [`scripts/build_mvp1_dashboard.py`](../../../scripts/build_mvp1_dashboard.py). + +## Purpose + +The dashboard generator pulls each feature's one-liner description from its `feature_spec.md`'s `## 1) Purpose` block (the `Outcome` bullet, falling back to `Problem`). For **implemented features**, that `feature_spec.md` is frozen historical — it describes what shipped at the time and must not be back-edited (per the convention in CLAUDE.md + the spec author's "historical artifacts — leave alone" rule). + +That works fine until a **later feature** changes the behavior the historical row described. Example: `infra_ir_measures_migration` (2026-05-22) replaced `pytrec_eval` with `ir_measures` in `backend/app/eval/scoring.py`. The frozen `infra_optuna_eval` spec's Outcome line still names `pytrec_eval` (correctly — that's what shipped). But the current-state dashboard's row for `infra_optuna_eval` now mis-describes what its code does today. + +This directory is the resolution: drop a `.md` file here with the current-state one-liner. The generator looks here first, falls back to the spec. + +## Conventions + +- **Filename:** `.md`, where `` is the implemented-feature folder name **without** the `_` date prefix. E.g., for `docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/`, the override file is `infra_optuna_eval.md`. +- **Contents:** one plain-text or markdown line. Anything past the first sentence (the generator splits on `. ` / `! ` / `? `) is dropped from the dashboard cell. +- **Voice:** current-state. Don't describe history — that's what `state.md` is for. +- **No back-references to the migration that triggered the override.** Future readers shouldn't have to know which feature created the override. + +## Adding an override + +1. Identify the implemented feature whose row is stale. +2. Write a one-line current-state description. +3. Save as `.md` in this directory. +4. Run `python scripts/build_mvp1_dashboard.py` and verify the row in `docs/00_overview/MVP1_DASHBOARD.md` picks up the new text. +5. Commit both the override file AND the regenerated dashboard. + +## Existing overrides + +See the directory listing. Each override file's commit-time history explains which subsequent feature invalidated the original row. diff --git a/docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md b/docs/00_overview/dashboard_overrides/infra_optuna_eval.md similarity index 100% rename from docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/dashboard_one_liner.md rename to docs/00_overview/dashboard_overrides/infra_optuna_eval.md diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index ced9f4c7..753f6fcf 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -369,7 +369,7 @@
← Roadmap overview

RelyLoop MVP1 Dashboard

- Reflects feature-folder state as of 2026-05-22 (latest mtime of any + Reflects feature-folder state as of 2026-05-23 (latest mtime of any docs/02_product/planned_features/ or docs/00_overview/implemented_features/ file). See state.md for the active branch context, diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index 20e4cfa6..4fd96db1 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -369,7 +369,7 @@
← Roadmap overview

RelyLoop MVP2 Dashboard

- Reflects feature-folder state as of 2026-05-22 (latest mtime of any + Reflects feature-folder state as of 2026-05-23 (latest mtime of any docs/02_product/planned_features/ or docs/00_overview/implemented_features/ file). See state.md for the active branch context, diff --git a/scripts/build_mvp1_dashboard.py b/scripts/build_mvp1_dashboard.py index 1164f15b..48b8744c 100755 --- a/scripts/build_mvp1_dashboard.py +++ b/scripts/build_mvp1_dashboard.py @@ -222,24 +222,50 @@ def _extract_priority(text: str) -> str | None: return DEFAULT_PRIORITY -def _extract_one_liner(text: str, source_dir: Path | None = None) -> str: - """Best-effort: prefer ``dashboard_one_liner.md`` override, then Outcome, then Problem. +_DASHBOARD_OVERRIDES_DIR = ( + Path(__file__).resolve().parent.parent / "docs/00_overview/dashboard_overrides" +) +"""Sidecar directory for one-liner overrides. + +Lives OUTSIDE ``docs/00_overview/implemented_features/`` so the +"implemented-features folders are frozen historical artifacts" rule +(infra_ir_measures_migration feature_spec.md §2) is preserved. Each +override file is named ``.md`` matching the implemented- +feature folder's slug (e.g., the row for the implemented +``2026_05_10_infra_optuna_eval`` folder is overridden by +``infra_optuna_eval.md`` in this directory). +""" - Override sidecar (added by infra_ir_measures_migration Story 1.8): when a - feature folder has a ``dashboard_one_liner.md`` file next to the spec, its - contents (single line, plain text or markdown) override the spec-extracted - one-liner. This lets us keep frozen historical specs frozen while still - keeping the current-state dashboard accurate when a sibling feature - invalidates a historical row's description — e.g., when a library swap - in one feature changes what an earlier feature's code does today, but - the earlier feature's spec correctly describes what shipped at the time. + +def _extract_one_liner(text: str, source_dir: Path | None = None) -> str: + """Best-effort: prefer override sidecar, then Outcome, then Problem. + + Override sidecar (added by infra_ir_measures_migration Story 1.8): when + ``docs/00_overview/dashboard_overrides/.md`` exists, its + contents override the spec-extracted one-liner. This lets us keep frozen + historical specs frozen while still keeping the current-state dashboard + accurate when a sibling feature invalidates a historical row's + description — e.g., when a library swap in one feature changes what an + earlier feature's code does today, but the earlier feature's spec + correctly describes what shipped at the time. The override files live + OUTSIDE ``implemented_features/`` so the frozen-artifact rule is + preserved (the historical feature_spec.md is untouched; only the + dashboard's summary cell is overridden). When ``source_dir`` is supplied, any relative markdown link in the extracted sentence is rewritten so it resolves correctly from the dashboard files' directory. See :func:`_rewrite_markdown_links`. """ if source_dir is not None: - override_path = source_dir / "dashboard_one_liner.md" + # Derive feature slug from source_dir basename. + # For implemented features, the basename looks like + # "_" — we strip the date prefix to get the slug. + # For planned features the basename IS the slug. + slug = source_dir.name + date_prefix = re.match(r"^\d{4}_\d{2}_\d{2}_", slug) + if date_prefix: + slug = slug[len(date_prefix.group(0)) :] + override_path = _DASHBOARD_OVERRIDES_DIR / f"{slug}.md" if override_path.exists(): line = override_path.read_text().strip() if line: From 86d91bb8fa6910f2752bdf866c4dc81fe5b8f5ae Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 22 May 2026 20:17:24 -0400 Subject: [PATCH 11/15] docs: state.md entry + pipeline_status implementation section (post-impl) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-implementation workflow Step 2 + Step 12 (impl-execute): state.md - New dated entry at the top describing the migration: 9 commits on feature/infra-ir-measures-migration, public API of score() frozen, persisted JSONB keys frozen, aggregate via iter_calc, per-query universe filter mirrors prior evaluator, no migration / Alembic head unchanged at 0015_trials_per_query_metrics. - Updated "Current branch / execution context" to point at the active feature branch. - Updated "Active feature" to track infra_ir_measures_migration as 31st MVP1-era artifact in flight. pipeline_status.md - "## Implementation" section: changed from "Not started" to "PR pending push" with the full 9-commit sequence, phase-gate fix log, test counts, and Q1-Q5 resolutions. Tangential observations sweep: none found beyond what was captured inline via spec/plan/phase-gate review-cycle adjudications. Guide impact assessment: NONE. UI source changes were comment-only (create-study-modal.tsx, k-ignored.test.ts, types.ts source-of-truth comments). The guide-05 + guide-06 markdown text was reworded (pytrec_eval -> ir_measures) but the screenshots are unaffected; the Playwright spec didn't reference the changed files. No new glossary entries required (no new operator-facing product term). No new FAQ entries required (no new operator decision point). Test coverage audit: all 7 planned test files exist. Per-query shape parity test was bundled INTO test_scoring_parity.py as `test_per_query_shape_matches_pytrec_eval` rather than living as a separate `test_scoring_per_query_shape.py` — the plan called it out as a separate file but bundling the two parity tests in one module keeps the fixture import + library imports DRY. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../pipeline_status.md | 22 ++++++++++++++++++- state.md | 6 ++--- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md b/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md index 62b8f814..da5095a9 100644 --- a/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md +++ b/docs/02_product/planned_features/infra_ir_measures_migration/pipeline_status.md @@ -25,4 +25,24 @@ - Locked decisions reflected in plan: public API of `scoring.py` frozen; persisted JSONB keys frozen; aggregate-via-iter (no `calc_aggregate`); per-query universe filtered to `pytrec_eval` historical contract; `pytrec-eval` permanent in `[dependency-groups.dev]` for parity gate. ## Implementation -- Status: Not started +- Status: PR pending push +- Date: 2026-05-22 +- Branch: `feature/infra-ir-measures-migration` (9 commits) +- Stories executed: 8/8 sequentially per the plan's strict-sequential order. Commits: + - 4ec8357 — planning artifacts baseline (spec + plan + pipeline_status + dashboard regen) + - b265463 — Story 1.1 (pyproject: ir-measures runtime + pytrec-eval dev + mypy override drop) + - 5ae53de — Story 1.2 (parity-test fixture + skipped skeleton) + - 5f205e6 — Story 1.3 (scoring.py rewrite with metric-object mapping + universe filter) + - 8c67447 — Story 1.4 (parity test activation; 30/30 cases PASS at 1e-6) + - 4f14c28 — Story 1.5 (leakage assertions extended; existing-row regression added; p@10 inline fix) + - 2799040 — Story 1.6 (operator-visible studies.py:313 message + contract docstring reword) + - c2594c1 — Story 1.7 (Dockerfile comment reworded; docker build verified) + - fdd22ea — Story 1.8 (full doc-rewrite sweep + dashboard regen + grep gates clean) +- Phase-gate fixes: b5dbaa3 — 5 accepted findings from GPT-5.5 cumulative-diff review (silent-skip → raise; scoring.py docstring reword; AC-12 fetch_study_confidence direct call; dashboard override sidecar moved out of implemented_features/; AC-3 positive cases made dynamic) +- Tests: 1128 unit + 235 contract pass locally; integration tests will run in CI (Postgres host-binding skip per CLAUDE.md) +- Open questions Q1–Q5 all resolved during implementation (recorded in commit messages): + - Q1: migration docstring reworded (Story 1.8) + - Q2: ir_measures + pytrec-eval-terrier both ship py.typed → mypy override dropped (Story 1.1) + - Q3: pytrec-eval-terrier resolved transitively → Dockerfile gcc/g++/python3-dev install stays (Story 1.7) + - Q4: default ir_measures routing produces parity at 1e-6 → no forcing needed (Story 1.4) + - Q5: license + perf check clean (Apache 2.0 + MIT + MPL-2.0/MIT; perf within ±10%) (Story 1.4) diff --git a/state.md b/state.md index 6f13350d..8a77df68 100644 --- a/state.md +++ b/state.md @@ -2,14 +2,14 @@ > Read this first. Snapshots the active branch, what just shipped, what's in flight, what's queued, and where the project currently sits in the MVP1 → GA roadmap. Updated whenever a feature lands or a priority shifts. -**Last updated:** 2026-05-22 (after `chore_guides_glossary_route` + `chore_guides_faq` + `chore_guide_06_screenshot_refresh_confidence_panel` bundled into `main` as PR #195 squash `ea2b242` — **28th, 29th, and 30th MVP1-era artifacts** shipped in one PR. Three siblings under `/guide/*` bundled per "one branch, one PR" memory. New `/guide/glossary` route renders the 109-entry `ui/src/lib/glossary.ts` constant with substring search + 8 prefix-derived category facet chips + deep-link anchors (`#study.metric.ndcg`); 10 walkthrough `script.md` files gain a footer link to it. New `/guide/faq` route renders a fresh 19-entry typed `ui/src/lib/faq.ts` (categories: setup-and-install/studies-and-confidence/judgments/proposals-and-prs/chat-agent) with the same search + facet + anchor contract; entries' questions self-link for sharing. Guide-06 demo Playwright spec waits up to 45s for `[data-testid="confidence-panel"]` before screenshotting → `04-study-detail.png` now captures the ConfidencePanel partial-shape view (headline metric without CI band, Robust plateau runner-up gap, per-query outcomes 0/4/0); script.md narrative gains a Monitoring sub-section describing the three signals with cross-links into glossary + FAQ. Five SKILL.md gate edits ship together (impl-execute Step 2.5 FAQ-shaped catch-net + Step 3 terminology/drift/decision-point bullets; spec-gen Step 3 #11 tooltip-cites-glossary-key; impl-plan-gen line 111 per-tooltip checklist gains glossary key + source-of-truth comment target) — all locked by a new `glossary-gate-skill-edits.test.ts` that reads each SKILL.md from disk and grep-asserts the enforcement clauses (same-PR default, escape-hatch gating, Step 8 blocking, no-drift-escape, the literal `// Source-of-truth:` marker). New shared `ui/src/lib/markdown-safety.ts` exports `MARKDOWN_DISALLOWED_ELEMENTS` consumed by 4 surfaces (glossary route + FAQ route + HelpPopover + MarkdownDoc) — extracted in response to a Gemini security-medium finding + earlier GPT-5.5 cycle-2 F10 spec finding. Pivoted away from `/pipeline` mid-flow: glossary went through `/spec-gen` with 3 GPT-5.5 cross-model review cycles (10 findings adjudicated and applied) producing a committed `feature_spec.md` design reference; user observed pipeline ceremony was disproportionate for ~200 LOC per item; FAQ shipped without a formal spec (the `ui/src/lib/faq.ts` JSDoc + skill edits are the design surface). Cross-model review: GPT-5.5 spec 3 cycles (10 findings — 6 cycle-1 + 4 cycle-2 + 0 cycle-3 convergence — all accepted: §1 outcome rewrite, scope cross-ref fix, test-name canonicalization, DoD path alignment, FR-8c lead-in fix, FR-8a escape-hatch tightening, FR-8a glossary.ts path-ref, ACs locked enforcement clauses, vitest path-resolution guidance, AC-7 source-grep → behavioral DOM assertion); Gemini Code Assist 5 Medium findings (2 accepted in `` — unused `Card*` imports + shared `MARKDOWN_DISALLOWED_ELEMENTS`; 3 rejected with cited counter-evidence — no "FR-7" in faq.ts; `feat_pr_metric_confidence` slugs are deliberate codebase-grep handles for engineer audience; `ui/src/components/ui/card.tsx:9` Card primitive uses identical hardcoded `border-gray-200 bg-white text-gray-900` — matching established precedent). Tests: UI vitest **706/706** (was 639 — +67 across 6 new test files: `app/guide/glossary/page.test.tsx` 18, `app/guide/glossary/safety-filter.test.tsx` 2 [isolated because vi.doMock leaks across tests], `app/guide/faq/page.test.tsx` 18, `app/guide/page.test.tsx` 4, `skills/glossary-gate-skill-edits.test.ts` 18, `guides/script-footer.test.ts` 12); 2 new real-backend Playwright specs (`glossary.spec.ts` 7 + `faq.spec.ts` 6); demo Playwright regen on guide-06 (4 PNGs updated). CI: 1 fix push required after first push — stale `glossary-section` data-testid in `glossary.spec.ts` after the FAQ commit renamed it to `reference-section` (vitest was updated, Playwright spec missed); fix landed in commit before merge. 5/5 jobs green on final SHA. **Alembic head unchanged at `0015_trials_per_query_metrics`** — frontend-only feature, no backend code. Earlier: after `feat_study_preflight_overlap_probe` merged into `main` as PR #193 squash `ca835e0` — **27th MVP1 feature shipped**, 3 stories across 1 epic. Tier-2 create-time guard sitting between Tier 1 (string-equality target-mismatch, PR #184) and Tier 3 (mid-flight zero-streak abort, PR #191). `POST /api/v1/studies` now issues a single bounded `ids`-existence search against the study's target index after `JUDGMENT_TARGET_MISMATCH` and before config-serialize. When fewer than `min(MIN_OVERLAP=3, max(judged_doc_count, 1))` judged doc IDs are present, returns 422 `INSUFFICIENT_JUDGMENT_OVERLAP`. When the cluster is unreachable / probe times out / engine rejects the bare ids body, the probe emits a `studies.preflight.overlap_probe.skipped` WARN log with `reason ∈ {unreachable, timeout, invalid_query_dsl}` and the study creates 201 — consistent with "tolerate transient adapter failures at write time." Locked decisions per spec §19: ids-existence probe (NOT template-rendered — avoids parameter-synthesis brittleness), 2-tier cap-aware threshold (Q1 → B), fall-through on cluster-unreachable (Q2 → A), `strict_errors=True` on adapter call, module-level constants `MIN_OVERLAP=3 / PROBE_TIMEOUT_S=2.0 / MAX_PROBED_DOCS=200` (no `Settings` field), single representative qid K=1, `OverlapProbeResult` frozen dataclass return type, dict-key unpacking via `result.get("overlap_probe", [])`. Cross-model review: spec 3 cycles (14/7/4 findings — 23 accepted + 2 rejected with cited counter-evidence: `Query.id` is `Mapped[str]` String(36) not native UUID; UNIQUE on `(judgment_list_id, query_id, doc_id)` already guarantees DISTINCT so no `DISTINCT` keyword needed); plan 3 cycles (6/4/3 findings — 13 accepted, 0 rejected); phase-gate cumulative-diff GPT-5.5 (5 findings — 2 applied in `396da73` for runbook formula + `_log_helpers.py` convention, 2 deferred as `infra_study_preflight_real_engine_integration` + `chore_studies_post_arq_spy_fixture` idea files, 1 rejected with state.md-finalization-convention counter-evidence); Gemini Code Assist (1 Medium finding rejected with cited counter-evidence — Python 3.13 pin + ruff UP041 enforce bare `TimeoutError`); final GPT-5.5 (2 Medium findings — 1 rejected via `asyncio_mode = "auto"` in pyproject.toml:165 + sibling `test_dispatch_run_query.py` precedent, 1 accepted-as-documented for the unplanned E2E seed change anticipated by plan §3.5). New backend module `backend/app/services/study_preflight.py` (~180 LOC) + 2 new repo functions in `query.py` / `judgment.py` + handler integration in `studies.py` (between JUDGMENT_TARGET_MISMATCH line 283 and config-serialize line 286) + `INSUFFICIENT_JUDGMENT_OVERLAP` row in `api-conventions.md` + recovery paragraph in `study-lifecycle-debugging.md` + source-presence ordering test in `test_studies_api_contract.py` (locks `target_pos < probe_pos < overlap_pos < config_pos`). E2E seed helper at [`ui/tests/e2e/helpers/seed.ts`](ui/tests/e2e/helpers/seed.ts) extended with `bulkIndexDocsToES()` (POSTs NDJSON `_bulk` with `refresh=wait_for` to the host-side ES at `PLAYWRIGHT_ES_BASE_URL` default `http://localhost:9200`) so synthetic `e2e-doc-N` IDs are present in the cluster's target index — without it, the new probe rejects every seeded study. Tests: 1044 backend unit (was 1040, +4 in `backend/tests/unit/services/test_study_preflight.py`); backend integration +14 test functions / 18 parametrized cases — AC-1..AC-4b handler-level via `probe_judgment_overlap` monkeypatches, AC-5/AC-6 spy that the probe is NOT invoked on Tier 1 fail paths, AC-7/AC-8/AC-10/AC-11/AC-13 adapter-layer via `_FakeProbeAdapter` + `_install_real_probe_with_fake_adapter` (monkeypatches `study_preflight.acquire_adapter` to bypass CI's missing `CLUSTER_CREDENTIALS_FILE`), AC-9 empty-judgments path, AC-12 read-path negative; backend contract +2 (envelope shape + source-presence ordering lock); 1 autouse fixture (`_default_overlap_probe_passes`) installs a sufficient `OverlapProbeResult` so existing happy-path tests don't 422 on the new probe. CI green on every push iteration (3 pushes — 2 failures + 1 success on final SHA `b11a13d`-equivalent landing as `ca835e0`); 5/5 jobs incl. smoke 70+/70 Playwright. **Alembic head unchanged at `0015_trials_per_query_metrics`** — feature is purely additive at the application layer. Tangential captures during the session: [`infra_study_preflight_real_engine_integration/idea.md`](docs/02_product/planned_features/infra_study_preflight_real_engine_integration/idea.md) (P2: real-engine AC-1..AC-4b coverage), [`chore_studies_post_arq_spy_fixture/idea.md`](docs/02_product/planned_features/chore_studies_post_arq_spy_fixture/idea.md) (P2: Arq spy fixture for "no-enqueue on rejection" — symmetric gap across all studies-POST tests), [`bug_dashboard_banner_dismiss_persistence_flake/idea.md`](docs/02_product/planned_features/bug_dashboard_banner_dismiss_persistence_flake/idea.md) (pre-existing flake in `dashboard.spec.ts:63` introduced by PR #188 — test's `addInitScript` clears localStorage on every reload). Earlier: after `feat_orchestrator_zero_streak_abort` merged into `main` as PR #191 squash `51ae4b3c` — **26th MVP1 feature shipped**, 2 stories across 1 epic. Tier-3 mid-flight guard: orchestrator aborts a study as `failed` with `failed_reason="no signal: 20 consecutive trials scored 0.0 — judgment overlap likely lost mid-study"` after 20 consecutive `status='complete' AND primary_metric=0.0` trials. Mirrors the existing `_last_n_all_failed` precedent at [`backend/workers/orchestrator.py`](backend/workers/orchestrator.py) exactly — same block position (after failure-streak, before max_trials/time_budget), same cancel-race handling, same WARNING/INFO structlog levels. No migration, no API surface change, no frontend code change (existing `StudyHeader.failed_reason` renderer carries the new string). Composes with the Tier 1 shipped guard (`feat_study_target_judgment_mismatch_guard` PR #184) and the still-planned Tier 2 preflight overlap probe — together they close create-time + mid-flight + adapter-driven paths to "all trials score 0". Locked decisions per spec §19: threshold=20 (10 TPE random + 10 informed phases), module-level constant (NOT `Settings`), no `STUDY_NO_SIGNAL` error code (no envelope to attach it to; the `failed_reason` string IS the stable contract). Cross-model review: spec 3 cycles (21 findings, all accepted — including SQL-WHERE-on-study_id-only semantics, AC-5 impossible-data-state rewrite, log-level taxonomy reconciliation); plan 3 cycles (7 findings, all accepted — `from backend.workers` import path bug, barrier-stub determinism for AC-2, RecordingLogger setup for AC-3); cumulative-diff GPT-5.5 2 cycles (1 finding accepted in plan-patch `d3e2ac0` re: `_stop()` INFO vs WARNING log level); Gemini Code Assist 2 Medium (both accepted, fixed in `7ebbdda` — `Sequence[NativeQuery]` typing on test stubs); final GPT-5.5 review 2 cycles (3+2 findings, 4 accepted in `6e3d2dd`+`2d0bbc4` — pipeline_status surface update, STUDY_NO_SIGNAL supersession in idea, broken relative links, contract-gate clarification; 1 deferred — blog-post Fusion mention is project-scope-consistent). Tests: 1040 backend unit (unchanged); 6 new integration tests in [`backend/tests/integration/test_study_lifecycle.py`](backend/tests/integration/test_study_lifecycle.py) — 5 named (AC-1 zero-streak abort with WARNING log assertion, AC-2 outlier-in-window with barrier-stub determinism, AC-3 alternating zero/failed with INFO max_trials_reached log, AC-4 cancel-race via monkeypatched `fail_study`, AC-5 precedence via mocked helpers + spy) + 1 parameterized 8-subcase boundary matrix for FR-1/FR-5 (`_last_n_all_zero` helper SQL/order/LIMIT/NULL semantics). The existing `test_ac5_five_consecutive_failures_fail_the_study` continues to pass (FR-4 precedent regression). 13 new test cases total. New `build_zero_scoring_hits_response` fixture helper in [`backend/tests/integration/fixtures/handbuilt_qrels.py`](backend/tests/integration/fixtures/handbuilt_qrels.py). CI green on every push (7/7 jobs) incl. final `2d0bbc4`. **Alembic head unchanged at `0015_trials_per_query_metrics`** — feature is purely additive at the application layer. Tangential captures during the session: `bug_contract_test_stub_missing_target_filter_kwarg/idea.md` (pre-existing 2 contract-test failures in `test_error_codes.py` from PR #168's adapter Protocol change; stub signature drift). Bundled into the PR per user direction: planning docs from a pre-pipeline stash — `feat_chat_last_message_preview/idea.md` (MVP2 chat polish), `infra_ranx_migration/idea.md` (P2 — `pytrec_eval` → `ir_measures` migration; subsequently renamed `infra_ir_measures_migration/` at finalization), `docs/blog/2026-05-22-elevator-pitch-search-platform.md`, plus 3 cross-reference renames where `chore_chat_last_message_preview` → `feat_chat_last_message_preview`. Earlier: after `feat_home_first_run_demo_nudge` merged into `main` as PR #188 squash `21325432` — **25th MVP1 feature shipped**, 12 stories across 4 epics. Frontend-only polish layer on PR #182's auto-seed: dismissable demo-data banner on `/` + JSX `` on `/clusters` + ` (Demo)` text suffix in create-study modal cluster picker + proposals fk-select cluster filter + new `verify_demo_slug_parity.sh` CI guard. Cross-model review: spec 3 cycles (13 findings, all accepted); plan 3 cycles (11 findings, all accepted); phase gates 4 findings (all accepted); Gemini Code Assist 3 Medium (2 accepted, 1 rejected with counter-evidence — useMemo would save nothing given React's render model + early-return); final GPT-5.5 2 Low (1 fixed, 1 deferred to finalization). Phase 2 split out to `feat_home_demo_reseed_endpoint/idea.md` so the deferred reseed endpoint surfaces in `/pipeline --status` as its own planned feature. UI vitest **639** across 92 files (+29 across 7 new files + 2 extensions); Playwright E2E +3 on `dashboard.spec.ts`. Alembic head unchanged at `0015_trials_per_query_metrics`. Earlier: after `chore_e2e_test_rows_isolation` merged into `main` as PR #186 squash `a444b94` — **24th MVP1 feature shipped**, 2 stories across 1 epic. Closes the operator-visible-dev-DB pollution: every Playwright E2E run now drains its seeded rows after the suite via a per-worker JSONL cleanup registry, 6 new test-only `DELETE /api/v1/_test/*` endpoints gated by `_require_development_env`, FK-safe drain order (proposals → digests → studies → judgment_lists → query_sets → query_templates → clusters), and a new `cleanup-reporter.ts` Playwright Reporter that asserts `registered_deduped == attempted == deleted + failed + skipped_404 AND failed == 0` after every run. 11 strictly-new error codes (3 `_NOT_FOUND` + 8 `_HAS_DEPENDENT_*`) documented in [`docs/01_architecture/api-conventions.md`](docs/01_architecture/api-conventions.md). Pure `cleanup-core.ts` module extracted from `global-teardown.ts` so the dedupe/order/URL-build logic is unit-testable without fs/network mocks. Cross-model review: GPT-5.5 — spec 3 cycles (26 findings, 25 accepted + 1 deferred to PLAYWRIGHT_CLEANUP_STRICT=1 v2), plan 3 cycles (20 findings, all accepted); Gemini Code Assist 3 Medium findings (all rejected with SQLAlchemy AsyncSession-concurrency counter-evidence — `asyncio.gather` on the same session is forbidden); final GPT-5.5 1 High finding (rejected — truncated-diff false positive on `repo/__init__.py:38–42` import block; verified empirically `from backend.app.db.repo import hard_delete_*` works for all 6). Post-merge CI fix on the same branch: `testMatch: ['**/*.spec.ts']` added to `ui/playwright.config.ts` after the smoke job tried to load vitest `.test.ts` files as Playwright specs. Tests: 1040 backend unit (unchanged); backend integration +20 cases (6 happy + 6 parameterized 404 + 8 409 — covers all 11 strictly-new + 3 reused codes); backend contract +6 env-guard cases + 2 source-presence cases + 6 OpenAPI tuples; UI vitest **630** (was 601 — +29: 19 cleanup-core + 10 global-teardown). CI green on `01acc04` (5/5 jobs incl. smoke 70/70 Playwright). **Alembic head unchanged at `0015_trials_per_query_metrics`** — feature is purely additive at the application layer. Tangential capture: `chore_e2e_seed_acme_helper_dead/idea.md` (Backlog) — `seedAcmeProductsChain` has no spec caller. Earlier — after `feat_study_target_judgment_mismatch_guard` merged into `main` as PR #184 squash `ce3fcf4` — **23rd MVP1 feature shipped**, 3 stories across 1 epic. Closes the literal study2 incident: `POST /api/v1/studies` now rejects two mismatch classes at create time with specific 422 codes — `JUDGMENT_CLUSTER_MISMATCH` (judgment list and study point at different physical clusters; doc IDs are cluster-scoped so same target name on two clusters still produces zero overlap) and `JUDGMENT_TARGET_MISMATCH` (same cluster but different target index/collection). Cluster fires before target. Both checks fire AFTER FK resolution + the existing `query_set_id` `VALIDATION_ERROR` check. New `?target=` wire filter on `GET /api/v1/judgment-lists` (min_length=1, max_length=255) + `target: str` required field on `JudgmentListSummary` (additive; OpenAPI snapshot + ui/src/lib/types.ts regenerated). Frontend create-study modal Step-2 dropdown now passes `{ query_set_id, cluster_id, target, limit: 200 }` to `useJudgmentLists`; manual-mode `` uses hoisted `targetReg.onChange(e)` (RHF register preserved) then cascade-resets `judgment_list_id`; dropdown-mode target picker mirrors the same reset; new empty-state copy substitutes the target value + CTA href="/judgments". Drive-by fix bundled: E2E seed helpers (`seedJudgmentList`, `seedFullChain`, `seedStudy`) gain optional `target` overrides; 3 specs updated to align target values so the new FR-1 validator doesn't reject chained POSTs. Cross-model review: spec 3 cycles (17 findings, all accepted, 1 rejected with cited counter-evidence at create-study-modal.tsx:508), plan 3 cycles (16 findings, all accepted, 1 rejected); Gemini Code Assist 2 findings (1 accepted in `035af0a` — IIFE → hoisted register; 1 rejected with precedent counter-evidence at `test_judgments_api_contract.py:215-234`); final GPT-5.5 10 findings (2 accepted in `a358a71` — over-bound 422 test; 8 rejected — 5 truncation false positives + 3 plan/precedent rejects). Tests: 1040 backend unit (unchanged — inline conditionals), backend integration +7 cases (target/cluster mismatch + ordering + AND-semantics + summary shape + over-bound + GET-pre-existing-200), backend contract +2 cases (firing-order lock in `test_studies_api_contract.py` + summary `target` shape lock in `test_judgments_api_contract.py`), UI vitest 567 → 572 (+5: hook wire-filter, dropdown cascade, manual cascade, cluster regression-lock, empty-state CTA). CI green on `a358a71` (5/5 jobs incl. 70/70 Playwright). Alembic head unchanged at `0015_trials_per_query_metrics` — feature is purely additive at the application layer. Prior — after `feat_pr_metric_confidence` merged into `main` as PR #180 squash `d0a8358` — **22nd MVP1 feature shipped**, 9 stories across 2 epics. Backend persistence (migration `0015_trials_per_query_metrics` adds nullable JSONB column behind CHECK), analytics (`backend/app/domain/study/confidence.py` — pure-Python orchestrator + bootstrap CI + runner-up gap + late-trial noise floor + convergence regime + per-query outcome helpers under FR-7 graceful-degradation), and three consumer surfaces — `StudyDetail.confidence` API enrichment, `## Confidence` PR body section, and digest narrative `` + `` Jinja blocks. Frontend ships `` on `/studies/[id]` (between StudyHeader and trials Card) + 6 glossary entries (text lifted verbatim from spec §11 tooltip table) + 2 real-backend Playwright E2E cases. Cross-model review: GPT-5.5 cycle 1 (Epic 1 gate) returned 12 findings — 5 rejected with cited counter-evidence (truncated-diff false positives), 2 deferred, 5 accepted + fixed inline; Gemini Code Assist clean pass; final GPT-5.5 review 3 Low findings all accepted + fixed inline. Tests: 1039 backend unit (+5 digest + 29 confidence + 13 studies confidence + extras), 189 contract (+2 OpenAPI shape lock + 4 PR-body section + 1 endpoint guard for the extended _test seed endpoint), 527 in-container integration (+13 StudyDetail.confidence + 5 migration round-trip + 1 open_pr plumbing + 2 Story 1.2 worker), 567 UI vitest (+14 ConfidencePanel — 13 layout + 1 tooltip-trigger inventory), 10/10 Playwright E2E (+2 ConfidencePanel real-backend). Three follow-ups filed: `chore_guides_glossary_route` (render `glossary.ts` as a `/guide/glossary` route), `chore_guides_faq` (curated operator-judgment Q&A), `chore_guide_06_screenshot_refresh_confidence_panel` (regenerate guide-06 screenshots). Alembic head moves to `0015_trials_per_query_metrics`. Prior — after `feat_pr_metric_confidence` Epic 1 landed locally on the `feat_pr_metric_confidence` branch — backend persistence + analytics + PR-body + digest-prompt surfaces complete, Epic 2 frontend ConfidencePanel ahead. Migration `0015_trials_per_query_metrics` adds the nullable JSONB column behind a CHECK constraint; new pure-Python `backend/app/domain/study/confidence.py` owns bootstrap CI + runner-up gap + late-trial noise floor + convergence regime + per-query outcome classification under FR-7's graceful-degradation contract; new `backend/app/services/study_confidence.py` glues the 4-query read pattern onto the orchestrator and is consumed from `studies._detail()`, the `open_pr` worker, and the digest worker. GPT-5.5 cycle-1 review found 12 issues — 5 rejected as truncated-diff false positives, 2 deferred (plan/code interface drift; full-worker integration test deferred to feat_github_pr_worker's existing suite), 5 accepted + fixed inline (convergence `total_trials = max_trial_number + 1` instead of count; convergence KeyError guard when winner not in summary; pre-existing-row-stays-NULL migration test; Trial model docstring drift on metric key shape; state + architecture docs). 1039 backend unit tests pass (+5 digest prompt cases, +1 convergence assertion), 189 contract, 527/527 in-container integration. Prior — after `feat_agent_propose_search_space` shipped as PR #175 squash `5d29355`). **21st MVP1 feature merged** — 10 stories across 5 epics, all complete. New read-only agent tool `propose_search_space` (the 20th in the registry) builds a deterministic starter search space from a template's `declared_params` using the same heuristic that powers the create-study wizard's auto-fill — a Python port (`backend/app/domain/study/search_space_defaults.py`) of `ui/src/lib/search-space-defaults.ts` with a TS↔Python parity test driven by a shared JSON fixture (18 rows, byte-identical assertions on both sides). Cap-aware overflow guard added on both Python AND TS sides (fixes a latent bug where TS silently returned an invalid space when 8+ fall-through floats blew past 10⁶). Optional `prior_study_id` arg narrows numeric bounds via `winner ± |winner| × bracket` for sign-symmetric math (Gemini #1/#2 fix) with `bracket` threaded through the linear paths (Gemini #3 fix); log-uniform stays at √2. Graceful degrade on template mismatch + missing trial row + non-numeric winner — emits WARN logs (`agent.propose_search_space.prior_template_mismatch` / `.missing_winner_trial`). `ToolContext` gained `conversation_id: str` plumbed from `orchestrator.run_turn` for paired adherence telemetry — INFO events `agent.search_space_proposed` (propose-side) + `agent.create_study.invoked` (create-side) correlate offline by conversation_id per spec FR-6 (grep recipe in `docs/03_runbooks/agent-debugging.md` §5). New `repo.get_trial(db, trial_id)` parallels `repo.get_study`. System prompt updated: 19→20 tools, "Studies (4)" with `propose_search_space` first, new chain-guidance bullet. `ProposeSearchSpaceArgs` uses `ConfigDict(extra="forbid")` (GPT-5.5 F6 fix) so hallucinated LLM args fail Pydantic validation loudly. Spec converged at GPT-5.5 cycle 3 (19 findings, all accepted); plan converged at cycle 3 (8 findings, all accepted). Post-merge review: Gemini 3 findings all fixed in `642b5b9`; GPT-5.5 final review 6 findings — 1 fixed in `945e833`, 1 deferred (structlog migration), 4 rejected with cited counter-evidence (truncated-diff false positives). Tests: 1000 backend unit pass (+87 new cases) + 19 Python parity + 19 TS parity; 38 TS lib + 66 modal still green. Alembic head unchanged at `0014_clusters_target_filter` — feature is purely additive at the application layer. Earlier 2026-05-20 (after `feat_cluster_target_filter` shipped as PR #168 squash `57d3ba0` + follow-up `chore_seed_meaningful_demos` shipped as PR #169 squash `c44d774`). **20th MVP1 feature merged** + demo-state durability gap closed in the same session. PR #168: 5 stories (B1 migration 0014 + ORM column; B3 Pydantic + service plumb-through + responses; B2 adapter Protocol + ElasticAdapter + StubAdapter + router; F1 register modal Target filter input; F2 create-study modal filter-aware empty-state + EntitySelect accessibility improvement). Plus 4 post-impl fix commits (test_migrations head bump, register modal overflow-y-auto, EntitySelect sr-only Gemini fix, spec drift cleanup + OpenAPI shape-lock contract test from GPT-5.5 final review). PR #169: `scripts/seed_meaningful_demos.py` + `make seed-demo` target (idempotent: TRUNCATE clusters CASCADE + DELETE matching ES/OS indices + reseed with per-cluster `target_filter` values baked in — closes the gap where integration tests kept wiping the dev DB with no durable reseed mechanism). 529/529 vitest across 79 files (was 525/78), 903 backend unit tests (was 899), 50 cluster-API integration tests (was 45) + 3 new migration round-trip tests + 7 contract validator cases + OpenAPI shape-lock test. **Alembic head moved to `0014_clusters_target_filter`.** Cross-model review pre-impl: spec + plan both converged at GPT-5.5 cycle 2 (12 findings total, all accepted). Post-impl: Gemini Code Assist 3 findings (2 accepted: EntitySelect sr-only on #168, http() auth type hint on #169; 1 rejected with cited counter-evidence: out-of-scope test file from #168). GPT-5.5 final review on #168: 2 findings, both accepted (spec drift + OpenAPI shape-lock). **Process feedback captured:** `.claude/projects/.../memory/feedback_one_branch_per_session.md` — should have bundled the seed chore into PR #168 rather than spinning a sibling PR. End-to-end smoke verified live before both merges. Earlier 2026-05-20 (after `feat_create_study_target_autocomplete` shipped as PR #165 squash commit `bd4516a` — 19th MVP1 feature. Earlier 2026-05-20 (after `feat_create_study_target_autocomplete` shipped as PR #165 squash commit `bd4516a` — 19th MVP1 feature. Bundled the `get_schema` + `explain` connect-error fix per `bug_get_schema_unhandled_connect_error` in the same PR. 525/525 vitest across 78 files, 33 adapter unit tests + contract suite + integration tests all green twice (initial + post-cycle-2). Gemini Code Assist: 1 finding rejected with cited counter-evidence (pre-existing list-shape assumption matches the wire contract). GPT-5.5 final review: 2 findings — 1 accepted in `19d9d51` (contract-layer TARGETS_FORBIDDEN + CLUSTER_UNREACHABLE envelope assertions), 1 deferred with counter-evidence (dropdown E2E `test.skip`'d; AC coverage satisfied by 8 hook unit + 6 modal unit + integration + contract tests). Two follow-up ideas filed in-PR: `bug_e2e_target_dropdown_flake` + `chore_guide_06_screenshot_refresh_target_picker`.) Earlier — same day (after `feat_create_study_search_space_builder` shipped as PR #163 squash commit `c703953`, bundling the search-space builder feature + the `bug_judgment_lists_listing_ignores_query_set_filter` backend fix surfaced during local verification. 18th MVP1 feature. The builder + bug-fix bundle reflects the single-developer series workflow: rather than spin a sibling backend PR off `main`, the bug fix landed in the same branch since the dev was already in verification mode. PR #163 went through 3 spec cycles (16 findings) + 3 plan cycles (27 findings) + 3 Gemini Code Assist findings + 2 GPT-5.5 final-review passes (1 second-pass Low finding accepted on test coverage) = 47 review findings all accepted with cited fixes. 512 vitest assertions across 77 files, 4 real-backend Playwright e2e cases against the builder, 2 new backend tests for the bundled filter fix. Two follow-up idea files captured during local verification: `feat_create_study_target_autocomplete` (Step-1 free-text target field has no autocomplete from cluster indexes — pre-existing UX debt deferred) and the now-closed `bug_judgment_lists_listing_ignores_query_set_filter` (bundled into this PR).) Earlier (also 2026-05-20) — PR #161 `0879df2` `chore_create_study_modal_e2e_stability` (un-skipped the deferred Playwright spec via `dispatchEvent('click')` on the Radix trigger), PR #160 `160ff6b` `bug_err_metric_frontend_backend_drift` (wire-enum trim — `err` removed from frontend + backend Literal), PR #159 `52e106d` `bug_tutorial_template_param_boost_naming` (heuristic extension for `_boost` suffix). Earlier (also 2026-05-20) — PR #157 `chore_create_study_wizard_polish` — squash commit `075c46b` — merged into `main`. Ships the 4-surface chore: backend template-mismatch validation at create time (two new error codes `SEARCH_SPACE_UNKNOWN_PARAM` + `SEARCH_SPACE_MISSING_DECLARED_PARAM`), Step-4 auto-fill via the new `ui/src/lib/search-space-defaults.ts` heuristic + cap-aware fallback + TS↔Python cardinality parity fixture, 4 new `study.search_space.*` glossary entries (one dual + three short-only) and 6 extended per-metric entries with k-tier clauses, Step-5 tri-state metric+k rendering with new `K_IGNORED` predicate, plus client-side validation mirror + zero-declared block + 404/transient template-fetch recovery + `__placeholder__` warning. 16 new test files + 2 modified + 1 shared JSON fixture across backend unit/integration/contract + frontend unit/component + 1 skipped E2E. Three follow-up ideas captured: `bug_tutorial_template_param_boost_naming` (tutorial template uses `_boost` suffix not matched by the locked heuristic), `chore_create_study_modal_e2e_stability` (re-enable the skipped Playwright spec once EntitySelect disabled gating stabilizes), `bug_err_metric_frontend_backend_drift` (`err` selectable in wizard but unsupported by `scoring.py`). Gemini Code Assist + GPT-5.5 final-pass both adjudicated on the PR — 2 Gemini findings + 7 GPT-5.5 findings, all addressed or filed.) Earlier 2026-05-19 (after a 4-PR shipping run drained the actionable post-MVP1 chore backlog: PR #152 `chore_ci_prettier_check` (`476db78`) + PR #153 `chore_extract_shadcn_select_test_mock` (`199e225`) + PR #154 `chore_form_dropdown_guide_screenshot_refresh` (`ed4121f`) + PR #155 `chore_detail_page_shell_primitive` (`9a72514`). PR #155 is the third primitive after `` and `` — 6 detail-page migrations + new lint guard + flattens a latent UX bug where only `proposals/[id]` discriminated 404 from network error. Earlier the same session: PR #150 (`chore_data_table_columnvisibility_tanstack`, `c1e4545`) — closes the residual DataTable follow-ups: item 5 migrates the primitive from `columns.filter(...)` to TanStack's `state.columnVisibility` API (memoized per Gemini feedback), item 3 locked the flat-prop `DataTableProps` API as canonical with a "Shipped contract addendum" on the historical implementation plan's Story 2.6. Folder renamed `chore_data_table_primitive_followups` → `chore_data_table_columnvisibility_tanstack`. Earlier 2026-05-19 PR #148 (`infra_e2e_wire_seed_helper_into_studies_spec`, squash `65f4150`) — restored the 2 digest-panel E2E tests deferred from PR #130, diagnosed and fixed the real root cause of the original smoke-lane failure (`GET /api/v1/proposals` was silently ignoring the `?study_id=` filter, returning the most-recent global pending proposal), added 5-case integration regression coverage at `backend/tests/integration/test_proposals_study_filter.py`. Plus: (a) earlier 2026-05-18 PR #146 (`bug_install_skip_ui_rebuild`, squash `7299fca`) made `make up` rebuild every Compose service (`docker compose build` no-args), switched `make down` to `docker compose down`, and added a `verify_install_builds_all_services.sh` CI gate to lock the contract; (b) earlier 2026-05-18 PR #147 captured `chore_detail_page_shell_primitive` idea (squash `8854e47`). Two new follow-ups filed: `chore_ci_prettier_check` (CI's frontend job has no `prettier --check` step — surfaced when PR #136 drift in 2 unrelated files blocked an unrelated commit) and the in-flight `chore_detail_page_shell_primitive` (third primitive after DataTable + EntitySelect).) +**Last updated:** 2026-05-22 (active: `feature/infra-ir-measures-migration` branch with 9 commits — `infra_ir_measures_migration` shipping 8 stories that swap `backend/app/eval/scoring.py` from the abandoned `pytrec_eval` to `ir_measures` (which transitively pulls `pytrec-eval-terrier`, the actively-maintained PyTerrier fork that's byte-identical to pytrec_eval on every supported metric). Public API of `score()` FROZEN; persisted JSONB key shape FROZEN; aggregate computed via `ir_measures.iter_calc()` + manual mean (NOT `calc_aggregate` — see plan cycle-2 C2-F4); per-query universe filtered to mirror the prior evaluator's qid set. **No migration, no schema change** — Alembic head unchanged at `0015_trials_per_query_metrics`. Cross-model review trajectory: spec 3 GPT-5.5 cycles (11→6→1 findings, all accepted); plan 3 GPT-5.5 cycles (10→4→1 findings, 14 accepted + 1 rejected with cited counter-evidence at scoring.py:74-78); phase-gate cumulative-diff review (10 findings — 5 accepted + applied in `b5dbaa3`, 3 rejected with cited counter-evidence: pytrec-eval-terrier ships py.typed/the mypy override was correctly dropped, the gitignored release-notes file can't appear in diffs, test files enumerating forbidden tokens are semantically allowlisted; 2 deferred to post-impl). Tests: 1128 unit (was 1077 pre-migration; +51 from 30 parity cases + per-query shape + 12 regex enumeration + 9 sanity-check), 30/30 (metric, k) parity cases match pytrec_eval to 1e-6, per-query shape parity confirms outer-qid + inner-metric-key + per-(qid, metric) value parity at 1e-6, AC-12 existing-row read regression exercises all three consumers (fetch_study_confidence directly + via API + digest-worker top-trials SELECT). Q5 perf benchmark passes under existing 100ms/query threshold; Q4 resolution: outcome (a) — default `ir_measures` provider routing produces parity, no forcing needed. Operator-visible string change: `INSUFFICIENT_JUDGMENT_OVERLAP` error message at studies.py:313 now names `ir_measures` instead of `pytrec_eval`; no API contract change. Bundled inline: `backend/app/services/test_seeding.py` `p@10` → `precision@10` (pre-existing inconsistency; spec §2 C2-F5). New permanent infrastructure: `pytrec-eval>=0.5` lives in `[dependency-groups.dev]` (not in runtime image — `uv sync --no-dev` skips it) so the parity test stays runnable indefinitely; `docs/00_overview/dashboard_overrides/` directory + `scripts/build_mvp1_dashboard.py` override mechanism lets future library swaps update historical-feature dashboard rows without back-editing frozen implemented-feature specs. Earlier: after `chore_guides_glossary_route` + `chore_guides_faq` + `chore_guide_06_screenshot_refresh_confidence_panel` bundled into `main` as PR #195 squash `ea2b242` — **28th, 29th, and 30th MVP1-era artifacts** shipped in one PR. Three siblings under `/guide/*` bundled per "one branch, one PR" memory. New `/guide/glossary` route renders the 109-entry `ui/src/lib/glossary.ts` constant with substring search + 8 prefix-derived category facet chips + deep-link anchors (`#study.metric.ndcg`); 10 walkthrough `script.md` files gain a footer link to it. New `/guide/faq` route renders a fresh 19-entry typed `ui/src/lib/faq.ts` (categories: setup-and-install/studies-and-confidence/judgments/proposals-and-prs/chat-agent) with the same search + facet + anchor contract; entries' questions self-link for sharing. Guide-06 demo Playwright spec waits up to 45s for `[data-testid="confidence-panel"]` before screenshotting → `04-study-detail.png` now captures the ConfidencePanel partial-shape view (headline metric without CI band, Robust plateau runner-up gap, per-query outcomes 0/4/0); script.md narrative gains a Monitoring sub-section describing the three signals with cross-links into glossary + FAQ. Five SKILL.md gate edits ship together (impl-execute Step 2.5 FAQ-shaped catch-net + Step 3 terminology/drift/decision-point bullets; spec-gen Step 3 #11 tooltip-cites-glossary-key; impl-plan-gen line 111 per-tooltip checklist gains glossary key + source-of-truth comment target) — all locked by a new `glossary-gate-skill-edits.test.ts` that reads each SKILL.md from disk and grep-asserts the enforcement clauses (same-PR default, escape-hatch gating, Step 8 blocking, no-drift-escape, the literal `// Source-of-truth:` marker). New shared `ui/src/lib/markdown-safety.ts` exports `MARKDOWN_DISALLOWED_ELEMENTS` consumed by 4 surfaces (glossary route + FAQ route + HelpPopover + MarkdownDoc) — extracted in response to a Gemini security-medium finding + earlier GPT-5.5 cycle-2 F10 spec finding. Pivoted away from `/pipeline` mid-flow: glossary went through `/spec-gen` with 3 GPT-5.5 cross-model review cycles (10 findings adjudicated and applied) producing a committed `feature_spec.md` design reference; user observed pipeline ceremony was disproportionate for ~200 LOC per item; FAQ shipped without a formal spec (the `ui/src/lib/faq.ts` JSDoc + skill edits are the design surface). Cross-model review: GPT-5.5 spec 3 cycles (10 findings — 6 cycle-1 + 4 cycle-2 + 0 cycle-3 convergence — all accepted: §1 outcome rewrite, scope cross-ref fix, test-name canonicalization, DoD path alignment, FR-8c lead-in fix, FR-8a escape-hatch tightening, FR-8a glossary.ts path-ref, ACs locked enforcement clauses, vitest path-resolution guidance, AC-7 source-grep → behavioral DOM assertion); Gemini Code Assist 5 Medium findings (2 accepted in `` — unused `Card*` imports + shared `MARKDOWN_DISALLOWED_ELEMENTS`; 3 rejected with cited counter-evidence — no "FR-7" in faq.ts; `feat_pr_metric_confidence` slugs are deliberate codebase-grep handles for engineer audience; `ui/src/components/ui/card.tsx:9` Card primitive uses identical hardcoded `border-gray-200 bg-white text-gray-900` — matching established precedent). Tests: UI vitest **706/706** (was 639 — +67 across 6 new test files: `app/guide/glossary/page.test.tsx` 18, `app/guide/glossary/safety-filter.test.tsx` 2 [isolated because vi.doMock leaks across tests], `app/guide/faq/page.test.tsx` 18, `app/guide/page.test.tsx` 4, `skills/glossary-gate-skill-edits.test.ts` 18, `guides/script-footer.test.ts` 12); 2 new real-backend Playwright specs (`glossary.spec.ts` 7 + `faq.spec.ts` 6); demo Playwright regen on guide-06 (4 PNGs updated). CI: 1 fix push required after first push — stale `glossary-section` data-testid in `glossary.spec.ts` after the FAQ commit renamed it to `reference-section` (vitest was updated, Playwright spec missed); fix landed in commit before merge. 5/5 jobs green on final SHA. **Alembic head unchanged at `0015_trials_per_query_metrics`** — frontend-only feature, no backend code. Earlier: after `feat_study_preflight_overlap_probe` merged into `main` as PR #193 squash `ca835e0` — **27th MVP1 feature shipped**, 3 stories across 1 epic. Tier-2 create-time guard sitting between Tier 1 (string-equality target-mismatch, PR #184) and Tier 3 (mid-flight zero-streak abort, PR #191). `POST /api/v1/studies` now issues a single bounded `ids`-existence search against the study's target index after `JUDGMENT_TARGET_MISMATCH` and before config-serialize. When fewer than `min(MIN_OVERLAP=3, max(judged_doc_count, 1))` judged doc IDs are present, returns 422 `INSUFFICIENT_JUDGMENT_OVERLAP`. When the cluster is unreachable / probe times out / engine rejects the bare ids body, the probe emits a `studies.preflight.overlap_probe.skipped` WARN log with `reason ∈ {unreachable, timeout, invalid_query_dsl}` and the study creates 201 — consistent with "tolerate transient adapter failures at write time." Locked decisions per spec §19: ids-existence probe (NOT template-rendered — avoids parameter-synthesis brittleness), 2-tier cap-aware threshold (Q1 → B), fall-through on cluster-unreachable (Q2 → A), `strict_errors=True` on adapter call, module-level constants `MIN_OVERLAP=3 / PROBE_TIMEOUT_S=2.0 / MAX_PROBED_DOCS=200` (no `Settings` field), single representative qid K=1, `OverlapProbeResult` frozen dataclass return type, dict-key unpacking via `result.get("overlap_probe", [])`. Cross-model review: spec 3 cycles (14/7/4 findings — 23 accepted + 2 rejected with cited counter-evidence: `Query.id` is `Mapped[str]` String(36) not native UUID; UNIQUE on `(judgment_list_id, query_id, doc_id)` already guarantees DISTINCT so no `DISTINCT` keyword needed); plan 3 cycles (6/4/3 findings — 13 accepted, 0 rejected); phase-gate cumulative-diff GPT-5.5 (5 findings — 2 applied in `396da73` for runbook formula + `_log_helpers.py` convention, 2 deferred as `infra_study_preflight_real_engine_integration` + `chore_studies_post_arq_spy_fixture` idea files, 1 rejected with state.md-finalization-convention counter-evidence); Gemini Code Assist (1 Medium finding rejected with cited counter-evidence — Python 3.13 pin + ruff UP041 enforce bare `TimeoutError`); final GPT-5.5 (2 Medium findings — 1 rejected via `asyncio_mode = "auto"` in pyproject.toml:165 + sibling `test_dispatch_run_query.py` precedent, 1 accepted-as-documented for the unplanned E2E seed change anticipated by plan §3.5). New backend module `backend/app/services/study_preflight.py` (~180 LOC) + 2 new repo functions in `query.py` / `judgment.py` + handler integration in `studies.py` (between JUDGMENT_TARGET_MISMATCH line 283 and config-serialize line 286) + `INSUFFICIENT_JUDGMENT_OVERLAP` row in `api-conventions.md` + recovery paragraph in `study-lifecycle-debugging.md` + source-presence ordering test in `test_studies_api_contract.py` (locks `target_pos < probe_pos < overlap_pos < config_pos`). E2E seed helper at [`ui/tests/e2e/helpers/seed.ts`](ui/tests/e2e/helpers/seed.ts) extended with `bulkIndexDocsToES()` (POSTs NDJSON `_bulk` with `refresh=wait_for` to the host-side ES at `PLAYWRIGHT_ES_BASE_URL` default `http://localhost:9200`) so synthetic `e2e-doc-N` IDs are present in the cluster's target index — without it, the new probe rejects every seeded study. Tests: 1044 backend unit (was 1040, +4 in `backend/tests/unit/services/test_study_preflight.py`); backend integration +14 test functions / 18 parametrized cases — AC-1..AC-4b handler-level via `probe_judgment_overlap` monkeypatches, AC-5/AC-6 spy that the probe is NOT invoked on Tier 1 fail paths, AC-7/AC-8/AC-10/AC-11/AC-13 adapter-layer via `_FakeProbeAdapter` + `_install_real_probe_with_fake_adapter` (monkeypatches `study_preflight.acquire_adapter` to bypass CI's missing `CLUSTER_CREDENTIALS_FILE`), AC-9 empty-judgments path, AC-12 read-path negative; backend contract +2 (envelope shape + source-presence ordering lock); 1 autouse fixture (`_default_overlap_probe_passes`) installs a sufficient `OverlapProbeResult` so existing happy-path tests don't 422 on the new probe. CI green on every push iteration (3 pushes — 2 failures + 1 success on final SHA `b11a13d`-equivalent landing as `ca835e0`); 5/5 jobs incl. smoke 70+/70 Playwright. **Alembic head unchanged at `0015_trials_per_query_metrics`** — feature is purely additive at the application layer. Tangential captures during the session: [`infra_study_preflight_real_engine_integration/idea.md`](docs/02_product/planned_features/infra_study_preflight_real_engine_integration/idea.md) (P2: real-engine AC-1..AC-4b coverage), [`chore_studies_post_arq_spy_fixture/idea.md`](docs/02_product/planned_features/chore_studies_post_arq_spy_fixture/idea.md) (P2: Arq spy fixture for "no-enqueue on rejection" — symmetric gap across all studies-POST tests), [`bug_dashboard_banner_dismiss_persistence_flake/idea.md`](docs/02_product/planned_features/bug_dashboard_banner_dismiss_persistence_flake/idea.md) (pre-existing flake in `dashboard.spec.ts:63` introduced by PR #188 — test's `addInitScript` clears localStorage on every reload). Earlier: after `feat_orchestrator_zero_streak_abort` merged into `main` as PR #191 squash `51ae4b3c` — **26th MVP1 feature shipped**, 2 stories across 1 epic. Tier-3 mid-flight guard: orchestrator aborts a study as `failed` with `failed_reason="no signal: 20 consecutive trials scored 0.0 — judgment overlap likely lost mid-study"` after 20 consecutive `status='complete' AND primary_metric=0.0` trials. Mirrors the existing `_last_n_all_failed` precedent at [`backend/workers/orchestrator.py`](backend/workers/orchestrator.py) exactly — same block position (after failure-streak, before max_trials/time_budget), same cancel-race handling, same WARNING/INFO structlog levels. No migration, no API surface change, no frontend code change (existing `StudyHeader.failed_reason` renderer carries the new string). Composes with the Tier 1 shipped guard (`feat_study_target_judgment_mismatch_guard` PR #184) and the still-planned Tier 2 preflight overlap probe — together they close create-time + mid-flight + adapter-driven paths to "all trials score 0". Locked decisions per spec §19: threshold=20 (10 TPE random + 10 informed phases), module-level constant (NOT `Settings`), no `STUDY_NO_SIGNAL` error code (no envelope to attach it to; the `failed_reason` string IS the stable contract). Cross-model review: spec 3 cycles (21 findings, all accepted — including SQL-WHERE-on-study_id-only semantics, AC-5 impossible-data-state rewrite, log-level taxonomy reconciliation); plan 3 cycles (7 findings, all accepted — `from backend.workers` import path bug, barrier-stub determinism for AC-2, RecordingLogger setup for AC-3); cumulative-diff GPT-5.5 2 cycles (1 finding accepted in plan-patch `d3e2ac0` re: `_stop()` INFO vs WARNING log level); Gemini Code Assist 2 Medium (both accepted, fixed in `7ebbdda` — `Sequence[NativeQuery]` typing on test stubs); final GPT-5.5 review 2 cycles (3+2 findings, 4 accepted in `6e3d2dd`+`2d0bbc4` — pipeline_status surface update, STUDY_NO_SIGNAL supersession in idea, broken relative links, contract-gate clarification; 1 deferred — blog-post Fusion mention is project-scope-consistent). Tests: 1040 backend unit (unchanged); 6 new integration tests in [`backend/tests/integration/test_study_lifecycle.py`](backend/tests/integration/test_study_lifecycle.py) — 5 named (AC-1 zero-streak abort with WARNING log assertion, AC-2 outlier-in-window with barrier-stub determinism, AC-3 alternating zero/failed with INFO max_trials_reached log, AC-4 cancel-race via monkeypatched `fail_study`, AC-5 precedence via mocked helpers + spy) + 1 parameterized 8-subcase boundary matrix for FR-1/FR-5 (`_last_n_all_zero` helper SQL/order/LIMIT/NULL semantics). The existing `test_ac5_five_consecutive_failures_fail_the_study` continues to pass (FR-4 precedent regression). 13 new test cases total. New `build_zero_scoring_hits_response` fixture helper in [`backend/tests/integration/fixtures/handbuilt_qrels.py`](backend/tests/integration/fixtures/handbuilt_qrels.py). CI green on every push (7/7 jobs) incl. final `2d0bbc4`. **Alembic head unchanged at `0015_trials_per_query_metrics`** — feature is purely additive at the application layer. Tangential captures during the session: `bug_contract_test_stub_missing_target_filter_kwarg/idea.md` (pre-existing 2 contract-test failures in `test_error_codes.py` from PR #168's adapter Protocol change; stub signature drift). Bundled into the PR per user direction: planning docs from a pre-pipeline stash — `feat_chat_last_message_preview/idea.md` (MVP2 chat polish), `infra_ranx_migration/idea.md` (P2 — `pytrec_eval` → `ir_measures` migration; subsequently renamed `infra_ir_measures_migration/` at finalization), `docs/blog/2026-05-22-elevator-pitch-search-platform.md`, plus 3 cross-reference renames where `chore_chat_last_message_preview` → `feat_chat_last_message_preview`. Earlier: after `feat_home_first_run_demo_nudge` merged into `main` as PR #188 squash `21325432` — **25th MVP1 feature shipped**, 12 stories across 4 epics. Frontend-only polish layer on PR #182's auto-seed: dismissable demo-data banner on `/` + JSX `` on `/clusters` + ` (Demo)` text suffix in create-study modal cluster picker + proposals fk-select cluster filter + new `verify_demo_slug_parity.sh` CI guard. Cross-model review: spec 3 cycles (13 findings, all accepted); plan 3 cycles (11 findings, all accepted); phase gates 4 findings (all accepted); Gemini Code Assist 3 Medium (2 accepted, 1 rejected with counter-evidence — useMemo would save nothing given React's render model + early-return); final GPT-5.5 2 Low (1 fixed, 1 deferred to finalization). Phase 2 split out to `feat_home_demo_reseed_endpoint/idea.md` so the deferred reseed endpoint surfaces in `/pipeline --status` as its own planned feature. UI vitest **639** across 92 files (+29 across 7 new files + 2 extensions); Playwright E2E +3 on `dashboard.spec.ts`. Alembic head unchanged at `0015_trials_per_query_metrics`. Earlier: after `chore_e2e_test_rows_isolation` merged into `main` as PR #186 squash `a444b94` — **24th MVP1 feature shipped**, 2 stories across 1 epic. Closes the operator-visible-dev-DB pollution: every Playwright E2E run now drains its seeded rows after the suite via a per-worker JSONL cleanup registry, 6 new test-only `DELETE /api/v1/_test/*` endpoints gated by `_require_development_env`, FK-safe drain order (proposals → digests → studies → judgment_lists → query_sets → query_templates → clusters), and a new `cleanup-reporter.ts` Playwright Reporter that asserts `registered_deduped == attempted == deleted + failed + skipped_404 AND failed == 0` after every run. 11 strictly-new error codes (3 `_NOT_FOUND` + 8 `_HAS_DEPENDENT_*`) documented in [`docs/01_architecture/api-conventions.md`](docs/01_architecture/api-conventions.md). Pure `cleanup-core.ts` module extracted from `global-teardown.ts` so the dedupe/order/URL-build logic is unit-testable without fs/network mocks. Cross-model review: GPT-5.5 — spec 3 cycles (26 findings, 25 accepted + 1 deferred to PLAYWRIGHT_CLEANUP_STRICT=1 v2), plan 3 cycles (20 findings, all accepted); Gemini Code Assist 3 Medium findings (all rejected with SQLAlchemy AsyncSession-concurrency counter-evidence — `asyncio.gather` on the same session is forbidden); final GPT-5.5 1 High finding (rejected — truncated-diff false positive on `repo/__init__.py:38–42` import block; verified empirically `from backend.app.db.repo import hard_delete_*` works for all 6). Post-merge CI fix on the same branch: `testMatch: ['**/*.spec.ts']` added to `ui/playwright.config.ts` after the smoke job tried to load vitest `.test.ts` files as Playwright specs. Tests: 1040 backend unit (unchanged); backend integration +20 cases (6 happy + 6 parameterized 404 + 8 409 — covers all 11 strictly-new + 3 reused codes); backend contract +6 env-guard cases + 2 source-presence cases + 6 OpenAPI tuples; UI vitest **630** (was 601 — +29: 19 cleanup-core + 10 global-teardown). CI green on `01acc04` (5/5 jobs incl. smoke 70/70 Playwright). **Alembic head unchanged at `0015_trials_per_query_metrics`** — feature is purely additive at the application layer. Tangential capture: `chore_e2e_seed_acme_helper_dead/idea.md` (Backlog) — `seedAcmeProductsChain` has no spec caller. Earlier — after `feat_study_target_judgment_mismatch_guard` merged into `main` as PR #184 squash `ce3fcf4` — **23rd MVP1 feature shipped**, 3 stories across 1 epic. Closes the literal study2 incident: `POST /api/v1/studies` now rejects two mismatch classes at create time with specific 422 codes — `JUDGMENT_CLUSTER_MISMATCH` (judgment list and study point at different physical clusters; doc IDs are cluster-scoped so same target name on two clusters still produces zero overlap) and `JUDGMENT_TARGET_MISMATCH` (same cluster but different target index/collection). Cluster fires before target. Both checks fire AFTER FK resolution + the existing `query_set_id` `VALIDATION_ERROR` check. New `?target=` wire filter on `GET /api/v1/judgment-lists` (min_length=1, max_length=255) + `target: str` required field on `JudgmentListSummary` (additive; OpenAPI snapshot + ui/src/lib/types.ts regenerated). Frontend create-study modal Step-2 dropdown now passes `{ query_set_id, cluster_id, target, limit: 200 }` to `useJudgmentLists`; manual-mode `` uses hoisted `targetReg.onChange(e)` (RHF register preserved) then cascade-resets `judgment_list_id`; dropdown-mode target picker mirrors the same reset; new empty-state copy substitutes the target value + CTA href="/judgments". Drive-by fix bundled: E2E seed helpers (`seedJudgmentList`, `seedFullChain`, `seedStudy`) gain optional `target` overrides; 3 specs updated to align target values so the new FR-1 validator doesn't reject chained POSTs. Cross-model review: spec 3 cycles (17 findings, all accepted, 1 rejected with cited counter-evidence at create-study-modal.tsx:508), plan 3 cycles (16 findings, all accepted, 1 rejected); Gemini Code Assist 2 findings (1 accepted in `035af0a` — IIFE → hoisted register; 1 rejected with precedent counter-evidence at `test_judgments_api_contract.py:215-234`); final GPT-5.5 10 findings (2 accepted in `a358a71` — over-bound 422 test; 8 rejected — 5 truncation false positives + 3 plan/precedent rejects). Tests: 1040 backend unit (unchanged — inline conditionals), backend integration +7 cases (target/cluster mismatch + ordering + AND-semantics + summary shape + over-bound + GET-pre-existing-200), backend contract +2 cases (firing-order lock in `test_studies_api_contract.py` + summary `target` shape lock in `test_judgments_api_contract.py`), UI vitest 567 → 572 (+5: hook wire-filter, dropdown cascade, manual cascade, cluster regression-lock, empty-state CTA). CI green on `a358a71` (5/5 jobs incl. 70/70 Playwright). Alembic head unchanged at `0015_trials_per_query_metrics` — feature is purely additive at the application layer. Prior — after `feat_pr_metric_confidence` merged into `main` as PR #180 squash `d0a8358` — **22nd MVP1 feature shipped**, 9 stories across 2 epics. Backend persistence (migration `0015_trials_per_query_metrics` adds nullable JSONB column behind CHECK), analytics (`backend/app/domain/study/confidence.py` — pure-Python orchestrator + bootstrap CI + runner-up gap + late-trial noise floor + convergence regime + per-query outcome helpers under FR-7 graceful-degradation), and three consumer surfaces — `StudyDetail.confidence` API enrichment, `## Confidence` PR body section, and digest narrative `` + `` Jinja blocks. Frontend ships `` on `/studies/[id]` (between StudyHeader and trials Card) + 6 glossary entries (text lifted verbatim from spec §11 tooltip table) + 2 real-backend Playwright E2E cases. Cross-model review: GPT-5.5 cycle 1 (Epic 1 gate) returned 12 findings — 5 rejected with cited counter-evidence (truncated-diff false positives), 2 deferred, 5 accepted + fixed inline; Gemini Code Assist clean pass; final GPT-5.5 review 3 Low findings all accepted + fixed inline. Tests: 1039 backend unit (+5 digest + 29 confidence + 13 studies confidence + extras), 189 contract (+2 OpenAPI shape lock + 4 PR-body section + 1 endpoint guard for the extended _test seed endpoint), 527 in-container integration (+13 StudyDetail.confidence + 5 migration round-trip + 1 open_pr plumbing + 2 Story 1.2 worker), 567 UI vitest (+14 ConfidencePanel — 13 layout + 1 tooltip-trigger inventory), 10/10 Playwright E2E (+2 ConfidencePanel real-backend). Three follow-ups filed: `chore_guides_glossary_route` (render `glossary.ts` as a `/guide/glossary` route), `chore_guides_faq` (curated operator-judgment Q&A), `chore_guide_06_screenshot_refresh_confidence_panel` (regenerate guide-06 screenshots). Alembic head moves to `0015_trials_per_query_metrics`. Prior — after `feat_pr_metric_confidence` Epic 1 landed locally on the `feat_pr_metric_confidence` branch — backend persistence + analytics + PR-body + digest-prompt surfaces complete, Epic 2 frontend ConfidencePanel ahead. Migration `0015_trials_per_query_metrics` adds the nullable JSONB column behind a CHECK constraint; new pure-Python `backend/app/domain/study/confidence.py` owns bootstrap CI + runner-up gap + late-trial noise floor + convergence regime + per-query outcome classification under FR-7's graceful-degradation contract; new `backend/app/services/study_confidence.py` glues the 4-query read pattern onto the orchestrator and is consumed from `studies._detail()`, the `open_pr` worker, and the digest worker. GPT-5.5 cycle-1 review found 12 issues — 5 rejected as truncated-diff false positives, 2 deferred (plan/code interface drift; full-worker integration test deferred to feat_github_pr_worker's existing suite), 5 accepted + fixed inline (convergence `total_trials = max_trial_number + 1` instead of count; convergence KeyError guard when winner not in summary; pre-existing-row-stays-NULL migration test; Trial model docstring drift on metric key shape; state + architecture docs). 1039 backend unit tests pass (+5 digest prompt cases, +1 convergence assertion), 189 contract, 527/527 in-container integration. Prior — after `feat_agent_propose_search_space` shipped as PR #175 squash `5d29355`). **21st MVP1 feature merged** — 10 stories across 5 epics, all complete. New read-only agent tool `propose_search_space` (the 20th in the registry) builds a deterministic starter search space from a template's `declared_params` using the same heuristic that powers the create-study wizard's auto-fill — a Python port (`backend/app/domain/study/search_space_defaults.py`) of `ui/src/lib/search-space-defaults.ts` with a TS↔Python parity test driven by a shared JSON fixture (18 rows, byte-identical assertions on both sides). Cap-aware overflow guard added on both Python AND TS sides (fixes a latent bug where TS silently returned an invalid space when 8+ fall-through floats blew past 10⁶). Optional `prior_study_id` arg narrows numeric bounds via `winner ± |winner| × bracket` for sign-symmetric math (Gemini #1/#2 fix) with `bracket` threaded through the linear paths (Gemini #3 fix); log-uniform stays at √2. Graceful degrade on template mismatch + missing trial row + non-numeric winner — emits WARN logs (`agent.propose_search_space.prior_template_mismatch` / `.missing_winner_trial`). `ToolContext` gained `conversation_id: str` plumbed from `orchestrator.run_turn` for paired adherence telemetry — INFO events `agent.search_space_proposed` (propose-side) + `agent.create_study.invoked` (create-side) correlate offline by conversation_id per spec FR-6 (grep recipe in `docs/03_runbooks/agent-debugging.md` §5). New `repo.get_trial(db, trial_id)` parallels `repo.get_study`. System prompt updated: 19→20 tools, "Studies (4)" with `propose_search_space` first, new chain-guidance bullet. `ProposeSearchSpaceArgs` uses `ConfigDict(extra="forbid")` (GPT-5.5 F6 fix) so hallucinated LLM args fail Pydantic validation loudly. Spec converged at GPT-5.5 cycle 3 (19 findings, all accepted); plan converged at cycle 3 (8 findings, all accepted). Post-merge review: Gemini 3 findings all fixed in `642b5b9`; GPT-5.5 final review 6 findings — 1 fixed in `945e833`, 1 deferred (structlog migration), 4 rejected with cited counter-evidence (truncated-diff false positives). Tests: 1000 backend unit pass (+87 new cases) + 19 Python parity + 19 TS parity; 38 TS lib + 66 modal still green. Alembic head unchanged at `0014_clusters_target_filter` — feature is purely additive at the application layer. Earlier 2026-05-20 (after `feat_cluster_target_filter` shipped as PR #168 squash `57d3ba0` + follow-up `chore_seed_meaningful_demos` shipped as PR #169 squash `c44d774`). **20th MVP1 feature merged** + demo-state durability gap closed in the same session. PR #168: 5 stories (B1 migration 0014 + ORM column; B3 Pydantic + service plumb-through + responses; B2 adapter Protocol + ElasticAdapter + StubAdapter + router; F1 register modal Target filter input; F2 create-study modal filter-aware empty-state + EntitySelect accessibility improvement). Plus 4 post-impl fix commits (test_migrations head bump, register modal overflow-y-auto, EntitySelect sr-only Gemini fix, spec drift cleanup + OpenAPI shape-lock contract test from GPT-5.5 final review). PR #169: `scripts/seed_meaningful_demos.py` + `make seed-demo` target (idempotent: TRUNCATE clusters CASCADE + DELETE matching ES/OS indices + reseed with per-cluster `target_filter` values baked in — closes the gap where integration tests kept wiping the dev DB with no durable reseed mechanism). 529/529 vitest across 79 files (was 525/78), 903 backend unit tests (was 899), 50 cluster-API integration tests (was 45) + 3 new migration round-trip tests + 7 contract validator cases + OpenAPI shape-lock test. **Alembic head moved to `0014_clusters_target_filter`.** Cross-model review pre-impl: spec + plan both converged at GPT-5.5 cycle 2 (12 findings total, all accepted). Post-impl: Gemini Code Assist 3 findings (2 accepted: EntitySelect sr-only on #168, http() auth type hint on #169; 1 rejected with cited counter-evidence: out-of-scope test file from #168). GPT-5.5 final review on #168: 2 findings, both accepted (spec drift + OpenAPI shape-lock). **Process feedback captured:** `.claude/projects/.../memory/feedback_one_branch_per_session.md` — should have bundled the seed chore into PR #168 rather than spinning a sibling PR. End-to-end smoke verified live before both merges. Earlier 2026-05-20 (after `feat_create_study_target_autocomplete` shipped as PR #165 squash commit `bd4516a` — 19th MVP1 feature. Earlier 2026-05-20 (after `feat_create_study_target_autocomplete` shipped as PR #165 squash commit `bd4516a` — 19th MVP1 feature. Bundled the `get_schema` + `explain` connect-error fix per `bug_get_schema_unhandled_connect_error` in the same PR. 525/525 vitest across 78 files, 33 adapter unit tests + contract suite + integration tests all green twice (initial + post-cycle-2). Gemini Code Assist: 1 finding rejected with cited counter-evidence (pre-existing list-shape assumption matches the wire contract). GPT-5.5 final review: 2 findings — 1 accepted in `19d9d51` (contract-layer TARGETS_FORBIDDEN + CLUSTER_UNREACHABLE envelope assertions), 1 deferred with counter-evidence (dropdown E2E `test.skip`'d; AC coverage satisfied by 8 hook unit + 6 modal unit + integration + contract tests). Two follow-up ideas filed in-PR: `bug_e2e_target_dropdown_flake` + `chore_guide_06_screenshot_refresh_target_picker`.) Earlier — same day (after `feat_create_study_search_space_builder` shipped as PR #163 squash commit `c703953`, bundling the search-space builder feature + the `bug_judgment_lists_listing_ignores_query_set_filter` backend fix surfaced during local verification. 18th MVP1 feature. The builder + bug-fix bundle reflects the single-developer series workflow: rather than spin a sibling backend PR off `main`, the bug fix landed in the same branch since the dev was already in verification mode. PR #163 went through 3 spec cycles (16 findings) + 3 plan cycles (27 findings) + 3 Gemini Code Assist findings + 2 GPT-5.5 final-review passes (1 second-pass Low finding accepted on test coverage) = 47 review findings all accepted with cited fixes. 512 vitest assertions across 77 files, 4 real-backend Playwright e2e cases against the builder, 2 new backend tests for the bundled filter fix. Two follow-up idea files captured during local verification: `feat_create_study_target_autocomplete` (Step-1 free-text target field has no autocomplete from cluster indexes — pre-existing UX debt deferred) and the now-closed `bug_judgment_lists_listing_ignores_query_set_filter` (bundled into this PR).) Earlier (also 2026-05-20) — PR #161 `0879df2` `chore_create_study_modal_e2e_stability` (un-skipped the deferred Playwright spec via `dispatchEvent('click')` on the Radix trigger), PR #160 `160ff6b` `bug_err_metric_frontend_backend_drift` (wire-enum trim — `err` removed from frontend + backend Literal), PR #159 `52e106d` `bug_tutorial_template_param_boost_naming` (heuristic extension for `_boost` suffix). Earlier (also 2026-05-20) — PR #157 `chore_create_study_wizard_polish` — squash commit `075c46b` — merged into `main`. Ships the 4-surface chore: backend template-mismatch validation at create time (two new error codes `SEARCH_SPACE_UNKNOWN_PARAM` + `SEARCH_SPACE_MISSING_DECLARED_PARAM`), Step-4 auto-fill via the new `ui/src/lib/search-space-defaults.ts` heuristic + cap-aware fallback + TS↔Python cardinality parity fixture, 4 new `study.search_space.*` glossary entries (one dual + three short-only) and 6 extended per-metric entries with k-tier clauses, Step-5 tri-state metric+k rendering with new `K_IGNORED` predicate, plus client-side validation mirror + zero-declared block + 404/transient template-fetch recovery + `__placeholder__` warning. 16 new test files + 2 modified + 1 shared JSON fixture across backend unit/integration/contract + frontend unit/component + 1 skipped E2E. Three follow-up ideas captured: `bug_tutorial_template_param_boost_naming` (tutorial template uses `_boost` suffix not matched by the locked heuristic), `chore_create_study_modal_e2e_stability` (re-enable the skipped Playwright spec once EntitySelect disabled gating stabilizes), `bug_err_metric_frontend_backend_drift` (`err` selectable in wizard but unsupported by `scoring.py`). Gemini Code Assist + GPT-5.5 final-pass both adjudicated on the PR — 2 Gemini findings + 7 GPT-5.5 findings, all addressed or filed.) Earlier 2026-05-19 (after a 4-PR shipping run drained the actionable post-MVP1 chore backlog: PR #152 `chore_ci_prettier_check` (`476db78`) + PR #153 `chore_extract_shadcn_select_test_mock` (`199e225`) + PR #154 `chore_form_dropdown_guide_screenshot_refresh` (`ed4121f`) + PR #155 `chore_detail_page_shell_primitive` (`9a72514`). PR #155 is the third primitive after `` and `` — 6 detail-page migrations + new lint guard + flattens a latent UX bug where only `proposals/[id]` discriminated 404 from network error. Earlier the same session: PR #150 (`chore_data_table_columnvisibility_tanstack`, `c1e4545`) — closes the residual DataTable follow-ups: item 5 migrates the primitive from `columns.filter(...)` to TanStack's `state.columnVisibility` API (memoized per Gemini feedback), item 3 locked the flat-prop `DataTableProps` API as canonical with a "Shipped contract addendum" on the historical implementation plan's Story 2.6. Folder renamed `chore_data_table_primitive_followups` → `chore_data_table_columnvisibility_tanstack`. Earlier 2026-05-19 PR #148 (`infra_e2e_wire_seed_helper_into_studies_spec`, squash `65f4150`) — restored the 2 digest-panel E2E tests deferred from PR #130, diagnosed and fixed the real root cause of the original smoke-lane failure (`GET /api/v1/proposals` was silently ignoring the `?study_id=` filter, returning the most-recent global pending proposal), added 5-case integration regression coverage at `backend/tests/integration/test_proposals_study_filter.py`. Plus: (a) earlier 2026-05-18 PR #146 (`bug_install_skip_ui_rebuild`, squash `7299fca`) made `make up` rebuild every Compose service (`docker compose build` no-args), switched `make down` to `docker compose down`, and added a `verify_install_builds_all_services.sh` CI gate to lock the contract; (b) earlier 2026-05-18 PR #147 captured `chore_detail_page_shell_primitive` idea (squash `8854e47`). Two new follow-ups filed: `chore_ci_prettier_check` (CI's frontend job has no `prettier --check` step — surfaced when PR #136 drift in 2 unrelated files blocked an unrelated commit) and the in-flight `chore_detail_page_shell_primitive` (third primitive after DataTable + EntitySelect).) --- ## Current branch / execution context -- **Branch:** `docs/finalize-guides-glossary-faq-and-regen` — finalization docs PR after PR #195 (`ea2b242`) merged 2026-05-22; moves the three planned-feature folders to `implemented_features/` per CLAUDE.md convention. `feature/guides-glossary-faq-and-regen` branch deleted post-merge by `gh pr merge --delete-branch`. Earlier: `docs/finalize-study-preflight-overlap-probe` — finalization docs PR after PR #193 (`ca835e0`) merged 2026-05-22. `feature/study-preflight-overlap-probe` branch deleted post-merge. Earlier: `docs/finalize-orchestrator-zero-streak-abort` — finalization docs PR after PR #191 (`51ae4b3c`) merged 2026-05-22. `feature/orchestrator-zero-streak-abort` branch deleted post-merge. Earlier: `docs/finalize-home-first-run-demo-nudge` — finalization docs PR after PR #188 (`21325432`) merged 2026-05-22. `feature/home-first-run-demo-nudge` branch deleted post-merge. Earlier: `docs/finalize-e2e-test-rows-isolation` — finalization docs PR after PR #186 (`a444b94`) merged 2026-05-21. `chore/e2e-test-rows-isolation` branch deleted post-merge. Earlier: `docs/finalize-study-target-judgment-mismatch-guard` — finalization docs PR after PR #184 (`ce3fcf4`) merged 2026-05-21. `feature/study-target-judgment-mismatch-guard` branch deleted post-merge. Earlier: `docs/finalize-pr-metric-confidence` — finalization docs PR after PR #180 (`d0a8358`) merged 2026-05-21. `feat_pr_metric_confidence` branch deleted post-merge. Earlier: `docs/finalize-agent-propose-search-space` — finalization docs PR after PR #175 (`5d29355`) merged 2026-05-21. `feature/agent-propose-search-space` deleted post-merge. Earlier: `docs/finalize-cluster-target-filter` — finalization docs PR after PR #168 (`57d3ba0`) + PR #169 (`c44d774`) both merged. Prior `main` post-merge of PR #168 squash `57d3ba0` (`feat_cluster_target_filter`) + PR #169 squash `c44d774` (`chore_seed_meaningful_demos`) 2026-05-20. Earlier: PR #165 squash commit `bd4516a` 2026-05-20. Finalization docs branch `docs/finalize-create-study-target-autocomplete`. Prior squash same day: PR #163 `c703953` (`feat_create_study_search_space_builder`). Finalization docs PR off `docs/finalize-create-study-search-space-builder`. Prior squashes (same day): PR #161 `0879df2` (`chore_create_study_modal_e2e_stability`), PR #160 `160ff6b` (`bug_err_metric_frontend_backend_drift`), PR #159 `52e106d` (`bug_tutorial_template_param_boost_naming`), PR #158 `308c315` (finalize chore_create_study_wizard_polish), PR #157 `075c46b` (`chore_create_study_wizard_polish`). Prior squash: PR #155 `9a72514` 2026-05-19. Prior squashes: PR #154 `ed4121f` 2026-05-19 (`chore_form_dropdown_guide_screenshot_refresh`), PR #153 `199e225` 2026-05-19 (`chore_extract_shadcn_select_test_mock`), PR #152 `476db78` 2026-05-19 (`chore_ci_prettier_check`), PR #151 `110dc5a` 2026-05-19 (finalize chore_data_table_columnvisibility_tanstack), PR #150 `c1e4545` 2026-05-19 (`chore_data_table_columnvisibility_tanstack`), PR #149 `da9506b` 2026-05-19 (finalize infra_e2e_wire_seed_helper_into_studies_spec), PR #148 `65f4150` 2026-05-19 (`infra_e2e_wire_seed_helper_into_studies_spec` — `?study_id=` filter bug + E2E test restore), PR #147 `8854e47` 2026-05-18 (capture chore_detail_page_shell_primitive idea), PR #146 `7299fca` 2026-05-18 (bug_install_skip_ui_rebuild — `make up`/`make down` lifecycle fix), PR #136 `cb7d9ee` 2026-05-18 (chore_form_dropdown_primitive), PR #132 `ee4c8d4` 2026-05-17 (chore_data_table_primitive_followups items 1+2+4+6), PR #130 `13b3383` 2026-05-17 (infra_e2e_seed_completed_study), PR #128 `73459d2` 2026-05-17 (bug_cursor_decode_value_validation), PR #126 `d6115b3` 2026-05-16 (feat_data_table_primitive). `v0.1.0` annotated tag still on `main` commit `d099536` 2026-05-13; GitHub Release at https://github.com/SoundMindsAI/relyloop/releases/tag/v0.1.0. -- **Active feature:** none in flight (PR #195 closed `chore_guides_glossary_route` + `chore_guides_faq` + `chore_guide_06_screenshot_refresh_confidence_panel` on 2026-05-22; only this finalization docs PR remains. The three siblings shipped bundled per "one branch, one PR" memory). Prior: none in flight (PR #193 closed `feat_study_preflight_overlap_probe` on 2026-05-22 as the **27th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #191 closed `feat_orchestrator_zero_streak_abort` on 2026-05-22 as the **26th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #188 closed `feat_home_first_run_demo_nudge` on 2026-05-22 as the **25th MVP1 feature** merged; only finalization docs PR remains. Phase 2 reseed-endpoint work captured in [`feat_home_demo_reseed_endpoint/idea.md`](docs/02_product/planned_features/feat_home_demo_reseed_endpoint/idea.md)). Prior: none in flight (PR #186 closed `chore_e2e_test_rows_isolation` on 2026-05-21 as the **24th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #184 closed `feat_study_target_judgment_mismatch_guard` on 2026-05-21 as the **23rd MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #180 closed `feat_pr_metric_confidence` on 2026-05-21 as the **22nd MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #175 closed `feat_agent_propose_search_space` on 2026-05-21; only finalization docs PR remains for the 21st MVP1 feature). Prior — none in flight (PR #168 closed `feat_cluster_target_filter` + PR #169 closed `chore_seed_meaningful_demos` on 2026-05-20; only finalization docs PR remains for the 20th MVP1 feature). Prior — none in flight (PR #165 closed `feat_create_study_target_autocomplete` + the bundled `bug_get_schema_unhandled_connect_error` fix on 2026-05-20). Prior — none in flight (PR #163 closed `feat_create_study_search_space_builder` + the `bug_judgment_lists_listing_ignores_query_set_filter` bundled fix on 2026-05-20). PR #168 closed `feat_cluster_target_filter` + PR #169 closed `chore_seed_meaningful_demos` (sibling). **Three PRs shipped 2026-05-15:** PR #122 (Phase 1, 16th MVP1 feature — Tooltip primitive + 26 placements on create-study modal + study detail), PR #123 (Phase 1 finalization docs), PR #124 (Phases 2 + 3 — 17th MVP1 feature; 21 additional tooltips on judgments + proposals + cluster registration + 2 new first-run components: chat ExamplePrompts strip + Stripe-style StartHereChecklist on home page). The original "MVP1 Phase 1 only" scope-lock was reversed mid-day: operator decided to ship Phases 2 + 3 together with a Stripe-style design call rather than wait for MVP2. PR #124 took 2 hours from idea-folder reuse to merge. 47 total tooltip placements + 2 new first-run components live in `main`. **PR #122 shipped 2026-05-15 morning** — `feat_contextual_help` Phase 1 (16th MVP1 feature). Adds the first Tooltip primitive (`@radix-ui/react-tooltip@~1.2.8` + shadcn-style wrapper at `ui/src/components/ui/tooltip.tsx`), two glossary-backed wrappers (`InfoTooltip` standalone + asChild modes; `HelpPopover` click-to-open with `react-markdown` safety filter), and a 49-key glossary source-of-truth at `ui/src/lib/glossary.ts` (8 enum groups parity-tested against `enums.ts`). 26 tooltip placements across the create-study modal (Step 1 target + Step 3 template + 9 Step 5 inputs), study-header (status badge dynamic key + Best metric + Trials), trials-table (5 column headers + Sort label), and digest panel (5 section labels + Open PR enabled + Open PR disabled). The disabled Open PR button refactored from native `disabled` to `aria-disabled="true"` so it stays focusable and the tooltip reveals on focus (AC-11). Gemini Code Assist: 2 findings (1 accepted + fixed, 1 rejected with cited counter-evidence). Final GPT-5.5 review: 1 Medium accepted-framing-but-deferred. Spec converged at GPT-5.5 cycle 3 (24 findings, 23 accepted + 1 rejected); plan converged at cycle 2 (12 findings, 10 accepted + 1 rejected + 1 spec patch). UI vitest now **279 passing across 48 files** (was 249 across 45 — +3 new test files, +30 cases). Playwright E2E **8 passing** (was 5 — +3 new contextual-help tests). One follow-up filed: `infra_e2e_seed_completed_study/idea.md` tracks the E2E gap for digest-panel triggers + AC-11 (cross-subsystem helper for seeding a completed study with digest + proposal; component-level coverage is in place). Phases 2 + 3 deferred to MVP2 via `feat_contextual_help_mvp2/` (judgments + proposals tooltips; chat + cluster + home onboarding; the home-page "Start here" panel is the only product-design-shaped item). +- **Branch:** `feature/infra-ir-measures-migration` — pending PR for `infra_ir_measures_migration`. 9 commits ahead of main: 1 planning artifacts (4ec8357) + 8 stories (b265463, 5ae53de, 5f205e6, 8c67447, 4f14c28, 2799040, c2594c1, fdd22ea) + 1 phase-gate fixes (b5dbaa3). Earlier: `docs/finalize-guides-glossary-faq-and-regen` — finalization docs PR after PR #195 (`ea2b242`) merged 2026-05-22; moves the three planned-feature folders to `implemented_features/` per CLAUDE.md convention. `feature/guides-glossary-faq-and-regen` branch deleted post-merge by `gh pr merge --delete-branch`. Earlier: `docs/finalize-study-preflight-overlap-probe` — finalization docs PR after PR #193 (`ca835e0`) merged 2026-05-22. `feature/study-preflight-overlap-probe` branch deleted post-merge. Earlier: `docs/finalize-orchestrator-zero-streak-abort` — finalization docs PR after PR #191 (`51ae4b3c`) merged 2026-05-22. `feature/orchestrator-zero-streak-abort` branch deleted post-merge. Earlier: `docs/finalize-home-first-run-demo-nudge` — finalization docs PR after PR #188 (`21325432`) merged 2026-05-22. `feature/home-first-run-demo-nudge` branch deleted post-merge. Earlier: `docs/finalize-e2e-test-rows-isolation` — finalization docs PR after PR #186 (`a444b94`) merged 2026-05-21. `chore/e2e-test-rows-isolation` branch deleted post-merge. Earlier: `docs/finalize-study-target-judgment-mismatch-guard` — finalization docs PR after PR #184 (`ce3fcf4`) merged 2026-05-21. `feature/study-target-judgment-mismatch-guard` branch deleted post-merge. Earlier: `docs/finalize-pr-metric-confidence` — finalization docs PR after PR #180 (`d0a8358`) merged 2026-05-21. `feat_pr_metric_confidence` branch deleted post-merge. Earlier: `docs/finalize-agent-propose-search-space` — finalization docs PR after PR #175 (`5d29355`) merged 2026-05-21. `feature/agent-propose-search-space` deleted post-merge. Earlier: `docs/finalize-cluster-target-filter` — finalization docs PR after PR #168 (`57d3ba0`) + PR #169 (`c44d774`) both merged. Prior `main` post-merge of PR #168 squash `57d3ba0` (`feat_cluster_target_filter`) + PR #169 squash `c44d774` (`chore_seed_meaningful_demos`) 2026-05-20. Earlier: PR #165 squash commit `bd4516a` 2026-05-20. Finalization docs branch `docs/finalize-create-study-target-autocomplete`. Prior squash same day: PR #163 `c703953` (`feat_create_study_search_space_builder`). Finalization docs PR off `docs/finalize-create-study-search-space-builder`. Prior squashes (same day): PR #161 `0879df2` (`chore_create_study_modal_e2e_stability`), PR #160 `160ff6b` (`bug_err_metric_frontend_backend_drift`), PR #159 `52e106d` (`bug_tutorial_template_param_boost_naming`), PR #158 `308c315` (finalize chore_create_study_wizard_polish), PR #157 `075c46b` (`chore_create_study_wizard_polish`). Prior squash: PR #155 `9a72514` 2026-05-19. Prior squashes: PR #154 `ed4121f` 2026-05-19 (`chore_form_dropdown_guide_screenshot_refresh`), PR #153 `199e225` 2026-05-19 (`chore_extract_shadcn_select_test_mock`), PR #152 `476db78` 2026-05-19 (`chore_ci_prettier_check`), PR #151 `110dc5a` 2026-05-19 (finalize chore_data_table_columnvisibility_tanstack), PR #150 `c1e4545` 2026-05-19 (`chore_data_table_columnvisibility_tanstack`), PR #149 `da9506b` 2026-05-19 (finalize infra_e2e_wire_seed_helper_into_studies_spec), PR #148 `65f4150` 2026-05-19 (`infra_e2e_wire_seed_helper_into_studies_spec` — `?study_id=` filter bug + E2E test restore), PR #147 `8854e47` 2026-05-18 (capture chore_detail_page_shell_primitive idea), PR #146 `7299fca` 2026-05-18 (bug_install_skip_ui_rebuild — `make up`/`make down` lifecycle fix), PR #136 `cb7d9ee` 2026-05-18 (chore_form_dropdown_primitive), PR #132 `ee4c8d4` 2026-05-17 (chore_data_table_primitive_followups items 1+2+4+6), PR #130 `13b3383` 2026-05-17 (infra_e2e_seed_completed_study), PR #128 `73459d2` 2026-05-17 (bug_cursor_decode_value_validation), PR #126 `d6115b3` 2026-05-16 (feat_data_table_primitive). `v0.1.0` annotated tag still on `main` commit `d099536` 2026-05-13; GitHub Release at https://github.com/SoundMindsAI/relyloop/releases/tag/v0.1.0. +- **Active feature:** `infra_ir_measures_migration` in flight on `feature/infra-ir-measures-migration` — pending push + PR. Aiming for the 31st MVP1-era artifact (single-PR migration per spec §3). Prior: none in flight (PR #195 closed `chore_guides_glossary_route` + `chore_guides_faq` + `chore_guide_06_screenshot_refresh_confidence_panel` on 2026-05-22; only this finalization docs PR remains. The three siblings shipped bundled per "one branch, one PR" memory). Prior: none in flight (PR #193 closed `feat_study_preflight_overlap_probe` on 2026-05-22 as the **27th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #191 closed `feat_orchestrator_zero_streak_abort` on 2026-05-22 as the **26th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #188 closed `feat_home_first_run_demo_nudge` on 2026-05-22 as the **25th MVP1 feature** merged; only finalization docs PR remains. Phase 2 reseed-endpoint work captured in [`feat_home_demo_reseed_endpoint/idea.md`](docs/02_product/planned_features/feat_home_demo_reseed_endpoint/idea.md)). Prior: none in flight (PR #186 closed `chore_e2e_test_rows_isolation` on 2026-05-21 as the **24th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #184 closed `feat_study_target_judgment_mismatch_guard` on 2026-05-21 as the **23rd MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #180 closed `feat_pr_metric_confidence` on 2026-05-21 as the **22nd MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #175 closed `feat_agent_propose_search_space` on 2026-05-21; only finalization docs PR remains for the 21st MVP1 feature). Prior — none in flight (PR #168 closed `feat_cluster_target_filter` + PR #169 closed `chore_seed_meaningful_demos` on 2026-05-20; only finalization docs PR remains for the 20th MVP1 feature). Prior — none in flight (PR #165 closed `feat_create_study_target_autocomplete` + the bundled `bug_get_schema_unhandled_connect_error` fix on 2026-05-20). Prior — none in flight (PR #163 closed `feat_create_study_search_space_builder` + the `bug_judgment_lists_listing_ignores_query_set_filter` bundled fix on 2026-05-20). PR #168 closed `feat_cluster_target_filter` + PR #169 closed `chore_seed_meaningful_demos` (sibling). **Three PRs shipped 2026-05-15:** PR #122 (Phase 1, 16th MVP1 feature — Tooltip primitive + 26 placements on create-study modal + study detail), PR #123 (Phase 1 finalization docs), PR #124 (Phases 2 + 3 — 17th MVP1 feature; 21 additional tooltips on judgments + proposals + cluster registration + 2 new first-run components: chat ExamplePrompts strip + Stripe-style StartHereChecklist on home page). The original "MVP1 Phase 1 only" scope-lock was reversed mid-day: operator decided to ship Phases 2 + 3 together with a Stripe-style design call rather than wait for MVP2. PR #124 took 2 hours from idea-folder reuse to merge. 47 total tooltip placements + 2 new first-run components live in `main`. **PR #122 shipped 2026-05-15 morning** — `feat_contextual_help` Phase 1 (16th MVP1 feature). Adds the first Tooltip primitive (`@radix-ui/react-tooltip@~1.2.8` + shadcn-style wrapper at `ui/src/components/ui/tooltip.tsx`), two glossary-backed wrappers (`InfoTooltip` standalone + asChild modes; `HelpPopover` click-to-open with `react-markdown` safety filter), and a 49-key glossary source-of-truth at `ui/src/lib/glossary.ts` (8 enum groups parity-tested against `enums.ts`). 26 tooltip placements across the create-study modal (Step 1 target + Step 3 template + 9 Step 5 inputs), study-header (status badge dynamic key + Best metric + Trials), trials-table (5 column headers + Sort label), and digest panel (5 section labels + Open PR enabled + Open PR disabled). The disabled Open PR button refactored from native `disabled` to `aria-disabled="true"` so it stays focusable and the tooltip reveals on focus (AC-11). Gemini Code Assist: 2 findings (1 accepted + fixed, 1 rejected with cited counter-evidence). Final GPT-5.5 review: 1 Medium accepted-framing-but-deferred. Spec converged at GPT-5.5 cycle 3 (24 findings, 23 accepted + 1 rejected); plan converged at cycle 2 (12 findings, 10 accepted + 1 rejected + 1 spec patch). UI vitest now **279 passing across 48 files** (was 249 across 45 — +3 new test files, +30 cases). Playwright E2E **8 passing** (was 5 — +3 new contextual-help tests). One follow-up filed: `infra_e2e_seed_completed_study/idea.md` tracks the E2E gap for digest-panel triggers + AC-11 (cross-subsystem helper for seeding a completed study with digest + proposal; component-level coverage is in place). Phases 2 + 3 deferred to MVP2 via `feat_contextual_help_mvp2/` (judgments + proposals tooltips; chat + cluster + home onboarding; the home-page "Start here" panel is the only product-design-shaped item). **Earlier — seven PRs shipped 2026-05-14:** `feat_judgments_periodic_resume_sweep` (PR #104, 14th MVP1 feature), `bug_query_inline_crud_since_filter_uuidv7_ms_collision` (PR #106 — UUIDv7 ms-collision test flake), `infra_dashboard_regen_pre_commit_conflict §2+§4` (PR #108 — dashboard regen idempotency + relative-link rewriting), `infra_make_targets_split_backend_only` (PR #110 — `make backend-fmt/lint/typecheck` + symmetric `ui-fmt` so Node-18 contributors aren't blocked), `chore_digest_worker_narrow_except` (PR #112 — narrowed `except Exception` allowlist to `(ValueError,)` + ERROR-level `digest_importance_failed_unexpected` event), `infra_structlog_test_helpers` (PR #114 — factored the two structlog test-assertion patterns into `backend/tests/_log_helpers.py`), and `chore_chat_last_message_preview` (PR #117 — `last_message_preview` + `last_message_at` on `ConversationSummary` via LATERAL JOIN; frontend shows preview under title + swaps displayed timestamp from `created_at` to `last_message_at`). Plus PR #116 dropped `chore_studies_ui_shadcn_polish` as won't-do (forward-compat audit on NavigationMenu primitive + ClusterFilterSelect precedent on native `