docs(planned): preflight-refresh infra_ir_measures_migration idea

docs/02_product/planned_features/infra_ir_measures_migration/idea.md

@@ -4,7 +4,7 @@
 **Status:** Idea — surfaced during blog drafting; noticed `pytrec_eval` is effectively abandoned (last commit 2020-09-07)
 **Priority:** P2
 **Origin:** Ad-hoc question in chat — "we are using pytrec_eval. This seems to be a very old codebase that is not being maintained. Do we have better options?" Confirmed: [github.com/cvangysel/pytrec_eval](https://github.com/cvangysel/pytrec_eval) last commit 2020-09-07, no GitHub releases. [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) is the only direct call site; pinned at `pytrec-eval>=0.5` in [`pyproject.toml:47`](../../../../pyproject.toml). An earlier draft of this idea proposed migrating to [`ranx`](https://github.com/AmenRa/ranx); rejected in favor of [`ir_measures`](https://github.com/terrierteam/ir_measures) — see "Alternatives considered" below.
-**Depends on:** None. Self-contained migration. Best sequenced **before** [`feat_pr_metric_confidence`](../../../00_overview/implemented_features/2026_05_21_feat_pr_metric_confidence/) gains its full statistical-confidence surface — the hand-rolled bootstrap there will sit on top of `ir_measures.iter_calc()` per-query output, and we don't want to throw away a `pytrec_eval`-based implementation the moment the rewrite lands.
+**Depends on:** None. Self-contained migration. **Sequencing-pressure update (preflight 2026-05-22):** [`feat_pr_metric_confidence`](../../../00_overview/implemented_features/2026_05_21_feat_pr_metric_confidence/) Phase 1 has already shipped, and inspection of [`backend/app/domain/study/confidence.py:247`](../../../../backend/app/domain/study/confidence.py) confirms `bootstrap_ci_95()` consumes the `per_query_metrics` dict (user-facing keys, not pytrec_eval wire forms) — it does NOT import pytrec_eval directly. So no rework of the shipped bootstrap is required by this migration. The "land before further confidence surface grows" pressure still applies for the **next** wave (paired-comparison + Fisher randomization), which has not shipped yet; this migration unblocks those features sitting cleanly on `ir_measures.iter_calc()` per-query output rather than the existing scoring re-keying.
 
 ## Problem
 
@@ -29,9 +29,24 @@
 
 ### Capability 3 — Refresh docs that name the library
 
-- Doc rewrites: [`architecture.md`](../../../../architecture.md), [`release-notes-v0.1.0-draft.md`](../../../../release-notes-v0.1.0-draft.md), and the tenant-facing tutorial/workflow material under [`ui/public/docs/workflows-overview.md`](../../../../ui/public/docs/workflows-overview.md) + [`ui/public/guides/05_import_judgments_and_calibrate/script.md`](../../../../ui/public/guides/05_import_judgments_and_calibrate/script.md) + [`ui/public/guides/06_create_and_monitor_study/script.md`](../../../../ui/public/guides/06_create_and_monitor_study/script.md). All `pytrec_eval` mentions become `ir_measures`.
-- The umbrella spec / CLAUDE.md "Stack (MVP1)" line should change `pytrec_eval` → `ir_measures`.
-- Code-comment sweep: docstrings and inline comments that name `pytrec_eval` to explain the wire-name translation contract — that contract is **simplified** (not eliminated) under `ir_measures`, so the references need rewording, not deletion. Files: [`backend/app/eval/qrels_loader.py`](../../../../backend/app/eval/qrels_loader.py), [`backend/app/db/models/trial.py`](../../../../backend/app/db/models/trial.py), [`backend/app/api/v1/schemas.py`](../../../../backend/app/api/v1/schemas.py), [`backend/app/api/v1/studies.py`](../../../../backend/app/api/v1/studies.py), [`migrations/versions/0015_trials_per_query_metrics.py`](../../../../migrations/versions/0015_trials_per_query_metrics.py), plus the test files under [`backend/tests/unit/eval/`](../../../../backend/tests/unit/eval/), [`backend/tests/contract/`](../../../../backend/tests/contract/), [`backend/tests/integration/`](../../../../backend/tests/integration/), and [`backend/tests/benchmarks/`](../../../../backend/tests/benchmarks/). Update to cite `ir_measures` and its metric-object DSL where the explanation still adds value.
+- Doc rewrites (preflight grep confirmed every named target contains a `pytrec_eval` mention as of 2026-05-22):
+  - [`architecture.md:131`](../../../../architecture.md) — "eval/  pytrec_eval scoring + Optuna runtime helpers"
+  - [`release-notes-v0.1.0-draft.md:12`](../../../../release-notes-v0.1.0-draft.md) — Stack mention in the release-notes table
+  - [`ui/public/docs/workflows-overview.md`](../../../../ui/public/docs/workflows-overview.md) — tenant-facing workflow doc
+  - [`ui/public/guides/05_import_judgments_and_calibrate/script.md`](../../../../ui/public/guides/05_import_judgments_and_calibrate/script.md)
+  - [`ui/public/guides/06_create_and_monitor_study/script.md`](../../../../ui/public/guides/06_create_and_monitor_study/script.md)
+  - [`ui/public/guides/06_create_and_monitor_study/metadata.json:26`](../../../../ui/public/guides/06_create_and_monitor_study/metadata.json) — caption field embeds the term ("scores them against the imported judgments via `pytrec_eval`"); same content shape as `script.md` so swap both in lock-step.
+
+  All `pytrec_eval` mentions become `ir_measures`.
+- **Two** `pytrec_eval` mentions in [`CLAUDE.md`](../../../../CLAUDE.md) need updating (preflight grep confirmed): line 15 in the project-overview paragraph ("thousands of trials against `pytrec_eval`-computed metrics") AND line 29 in the "Stack (MVP1)" dependency list. Update both.
+- Code-comment sweep: docstrings and inline comments that name `pytrec_eval` to explain the wire-name translation contract — that contract is **simplified** (not eliminated) under `ir_measures`, so the references need rewording, not deletion. Concrete files (preflight grep, 2026-05-22):
+  - [`backend/app/eval/qrels_loader.py:45`](../../../../backend/app/eval/qrels_loader.py) — "treats as a no-op" comment
+  - [`backend/app/eval/scoring.py`](../../../../backend/app/eval/scoring.py) — module docstring + `_translate_metric_name` notes
+  - [`backend/app/db/models/trial.py:19`](../../../../backend/app/db/models/trial.py) — "per-query pytrec_eval scores" docstring on `per_query_metrics`
+  - [`backend/app/api/v1/schemas.py:534`](../../../../backend/app/api/v1/schemas.py) — k-cutoff semantics comment
+  - [`backend/app/api/v1/studies.py:269`](../../../../backend/app/api/v1/studies.py) — "scores 0 on every trial" inline comment
+  - [`migrations/versions/0015_trials_per_query_metrics.py:17`](../../../../migrations/versions/0015_trials_per_query_metrics.py) — "NOT the pytrec_eval wire forms" docstring
+  - Test files: [`backend/tests/unit/eval/test_scoring.py`](../../../../backend/tests/unit/eval/test_scoring.py), [`test_scoring_metric_tokens.py`](../../../../backend/tests/unit/eval/test_scoring_metric_tokens.py), [`test_qrels_loader.py`](../../../../backend/tests/unit/eval/test_qrels_loader.py); [`backend/tests/contract/test_trial_row_shape.py`](../../../../backend/tests/contract/test_trial_row_shape.py), [`backend/tests/contract/test_studies_api_contract.py:156`](../../../../backend/tests/contract/test_studies_api_contract.py); [`backend/tests/integration/test_run_trial_per_query_persistence.py:53,111,119`](../../../../backend/tests/integration/test_run_trial_per_query_persistence.py), [`backend/tests/integration/fixtures/handbuilt_qrels.py:75`](../../../../backend/tests/integration/fixtures/handbuilt_qrels.py) (added 2026-05-22 by `feat_orchestrator_zero_streak_abort` — its `build_zero_scoring_hits_response` helper docstring); [`backend/tests/benchmarks/test_scoring_perf.py`](../../../../backend/tests/benchmarks/test_scoring_perf.py). Update each to cite `ir_measures` and its metric-object DSL where the explanation still adds value.
 - **Historical artifacts — leave alone:** [`state.md`](../../../../state.md), the [`infra_optuna_eval` implementation plan](../../../00_overview/implemented_features/2026_05_10_infra_optuna_eval/implementation_plan.md), and any other doc under `docs/00_overview/implemented_features/` describe what shipped at the time. Don't back-edit them. Add a new dated entry to `state.md` describing the migration when it lands, instead.
 
 ## Scope signals
@@ -45,10 +60,14 @@
 
 ## Why deferred
 
-Not blocking anything today: `pytrec_eval` still installs and produces correct numbers. The window to do this is **before** [`feat_pr_metric_confidence`](../../../00_overview/implemented_features/2026_05_21_feat_pr_metric_confidence/) grows its full statistical-test surface — the hand-rolled bootstrap there will sit on top of `ir_measures.iter_calc()` per-query output, and we don't want to throw away a `pytrec_eval`-based implementation when the rewrite lands. Sequencing: queue this ahead of any confidence-interval work that hasn't started yet; otherwise it's P2 cleanup.
+Not blocking anything today: `pytrec_eval` still installs and produces correct numbers. **The "land before feat_pr_metric_confidence ships" framing is obsolete** — that feature's Phase 1 already shipped 2026-05-21 (PR #180), and the shipped bootstrap CI at [`backend/app/domain/study/confidence.py:247`](../../../../backend/app/domain/study/confidence.py) consumes user-facing per-query metric keys, not pytrec_eval wire forms — so this migration won't have to rewrite Phase 1 code. The remaining sequencing pressure is the **next** confidence wave (paired-comparison + Fisher randomization, neither shipped yet); queueing this migration ahead of that wave keeps the new statistical code born on `ir_measures.iter_calc()` from day one. P2 cleanup otherwise.
 
 ## Relationship to other work
 
 - **Unblocks** statistical-significance and paired-comparison features in [`feat_pr_metric_confidence`](../../../00_overview/implemented_features/2026_05_21_feat_pr_metric_confidence/) — `ir_measures.iter_calc()` yields per-(query, metric) values that we hand-roll bootstrap + Fisher randomization on top of. If the hand-rolled stats grow uncomfortable, add `ir-measures[ranx]` as an extra and access `ranx`'s native paired-bootstrap / Fisher helpers through the `ir_measures` provider interface — no second migration required.
+- **Coordinated-with (now shipped)** — [`chore_guide_06_screenshot_refresh_confidence_panel`](../../../00_overview/implemented_features/2026_05_22_chore_guide_06_screenshot_refresh_confidence_panel/idea.md) already shipped 2026-05-22 (PR #195). The 06 script.md/metadata.json text rewrites this idea proposes will land on top of the post-#195 file state (which added a "Monitoring" sub-section but did not touch the existing `pytrec_eval` mentions). The next time guide-06 is regen'd via `/guide-gen 06 --regen`, the updated text will be picked up automatically. No file conflict, no sequencing constraint remaining.
+- **Coordinates with planned siblings that mention `pytrec_eval`** — two planned-features idea files reference the library by name today and will need their copy updated in the same PR as this migration (or as a fast-follow) to avoid drift between the renamed `scoring.py` and the planning docs that cite it:
+  - [`feat_study_baseline_trial`](../feat_study_baseline_trial/idea.md) — "scores via `pytrec_eval`" in the Capability 1 description. If that feature implements before this migration, its impl will use pytrec_eval and need a follow-up rewrite; if it implements after, it should land on `ir_measures` from the start.
+  - [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md) — "Optuna + pytrec_eval are deterministic" in the daily-budget integration note. Same coordination posture.
 - **Alternative considered: `ranx` as primary.** Earlier draft of this idea proposed `ranx` directly. Rejected on (a) **single-maintainer bus factor** — same failure mode that triggered this migration in the first place; swapping one abandoned-upstream risk for another doesn't improve anything, (b) **Numba install + cold-start cost** — ~30 MB wheel plus first-import JIT warm-up we'd be paying on every API container boot, (c) **no provider abstraction** — if `ranx` is ever abandoned we'd need another full rewrite, whereas with `ir_measures` swapping the backend is a config change. `ranx` remains available as an optional `ir-measures[ranx]` backend if its bootstrap / fusion features are needed later.
 - **Alternative considered: `pytrec-eval-terrier`** (the PyTerrier team's actively-maintained fork). One-line `import` rename, no semantic change. Rejected because (a) still a C extension with the same build-pain footprint, (b) doesn't introduce the provider abstraction that protects us from future abandonment, (c) `ir_measures` (same upstream team) supersedes it for new code.