From 84c810aac70cee42dd6a7d232e7be0f30b55a114 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 29 May 2026 14:02:17 -0400 Subject: [PATCH 01/21] =?UTF-8?q?docs(mvp2):=20feat=5Fubi=5Fjudgments=20?= =?UTF-8?q?=E2=80=94=20idea=20refresh=20+=20spec=20+=20plan=20(planning=20?= =?UTF-8?q?bundle)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Planning + spec + plan stage of /pipeline --auto for the engine-neutral UBI judgments feature. Bundles three sets of related doc state: (1) Operator-prep state from prior session (pre-existing in working tree): - feat_ubi_onramp folder merged back into feat_ubi_judgments (folder deletion + idea.md update explaining the merge) - Sibling MVP2 idea-file updates (infra_adapter_solr, feat_query_normalization_tuning) - bug_relyloop_spec_ubi_section_drift idea added (UBI section staleness) - MVP2 + Unsure dashboard regen - mvp2-overview.md update reflecting the merge (2) Feature spec (feature_spec.md, 11 FRs, 15 ACs, 1 additive migration): - Cross-model converged at 3-cycle cap (10 GPT-5.5 findings accepted) - Locks D-1..D-10 covering all idea-stage open questions + cycle-3 fixes - Decision D-1: _SourceBreakdown evolves to {llm, human, click} in place - Decision D-2: UI picker field is `method` (4 values), API request field is `converter` (3 values) — keeps llm-routing in the picker without polluting the UBI endpoint enum - Decision D-3: ?source= filter widens to accept click (3) Implementation plan (implementation_plan.md, 14 stories across 5 epics): - Cross-model converged at 3-cycle cap (3 GPT-5.5 findings accepted) - Cycle 2 fix: generation_params JSONB column persists generation_kind: 'ubi' discriminator for worker resume + value-delta card discrimination - Cycle 3 fix: dropped snapshot UbiRungBadge variant (spec FR-7 requires query_set_id + target which cluster pages don't have) - Pipeline status: ready for /impl-execute No code changes in this commit — implementation begins in subsequent per-story commits per the plan's execution tracker. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: SoundMindsAI --- docs/00_overview/DASHBOARD.md | 2 +- docs/00_overview/MVP2_DASHBOARD.md | 60 +- docs/00_overview/UNSURE_DASHBOARD.md | 8 +- docs/00_overview/dashboard.html | 2 +- docs/00_overview/mvp2_dashboard.html | 74 +- .../idea.md | 38 + .../feat_query_normalization_tuning/idea.md | 2 +- .../feat_ubi_judgments/feature_spec.md | 867 +++++++++ .../02_mvp2/feat_ubi_judgments/idea.md | 108 +- .../feat_ubi_judgments/implementation_plan.md | 1657 +++++++++++++++++ .../feat_ubi_judgments/pipeline_status.md | 38 + .../02_mvp2/feat_ubi_onramp/idea.md | 89 - .../02_mvp2/infra_adapter_solr/idea.md | 2 +- docs/00_overview/unsure_dashboard.html | 16 +- docs/01_architecture/mvp2-overview.md | 2 +- 15 files changed, 2779 insertions(+), 186 deletions(-) create mode 100644 docs/00_overview/planned_features/00_unsure/bug_relyloop_spec_ubi_section_drift/idea.md create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/pipeline_status.md delete mode 100644 docs/00_overview/planned_features/02_mvp2/feat_ubi_onramp/idea.md diff --git a/docs/00_overview/DASHBOARD.md b/docs/00_overview/DASHBOARD.md index 80a51aac..89c54d26 100644 --- a/docs/00_overview/DASHBOARD.md +++ b/docs/00_overview/DASHBOARD.md @@ -7,7 +7,7 @@ _Top-level index across MVP1 → GA v1+ as of **2026-05-29**. Click a release na | Release | Theme | Progress | Status | |---|---|---|---| | [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 95 / 95 scoped done | **Complete** | -| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 1 / 1 scoped done · 7 remaining | **In progress** | +| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 1 / 2 scoped done · 8 remaining | **In progress** | | MVP3 / v0.3 | Observable | — | **Not yet scoped** | | [GA v1 / v1.0](GA_DASHBOARD.md) | Production-ready | 1 item(s) queued | **Held / queued** | diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 0133e5cb..b0c83821 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -6,23 +6,29 @@ _Reflects feature-folder state as of **2026-05-29** (latest mtime of any planned ## Next up -All scoped MVP2 features shipped 🎉 +**[feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md)** — Feature, currently in **Plan** -Pull from the Idea backlog or capture a new feature spec. +> Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters + +Plan approved; run /impl-execute to ship + +```bash +/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md --all +``` ## MVP2 Progress | Metric | Value | |---|---| -| Scoped items done | **1 / 1** (100%) — feat_/infra_/chore_/epic_ past idea stage | -| Pending work | **16** items (every not-done feat/infra/chore/bug across all priorities) | +| Scoped items done | **1 / 2** (50%) — feat_/infra_/chore_/epic_ past idea stage | +| Pending work | **15** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | -| → P1 | **3** high-value, ready when P0 clears | +| → P1 | **2** high-value, ready when P0 clears | | → P2 (default) | 9 important to file, not blocking | | → Backlog | 4 captured for record, not planned | | Open bugs | 3 | -| Legacy "Path to MVP2" | 7 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 9 idea-only feat/infra (not yet scoped into MVP2) | +| Legacy "Path to MVP2" | 8 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | +| Backlog ideas | 7 idea-only feat/infra (not yet scoped into MVP2) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -37,34 +43,34 @@ Pull from the Idea backlog or capture a new feature spec. _None._ -### Plan (0) +### Plan (1) -_None._ +| # | Priority | Feature | Type | One-liner | Depends on | Status | +|---|---|---|---|---|---|---| +| 1 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md) | Feature | Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic | — | — | ### Spec (0) _None._ -### Idea (16) +### Idea (14) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/idea.md) | Feature | MVP1 ships with **LLM-as-judge** as the only authoritative judgment source. The architecture anticipated this would change — the `judgments.source` CHECK already accepts `click`… | — | Idea — bundled with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) into MVP2 / v0.2 "Three-Engine + Real Signals" | -| 2 | P1 | [feat_ubi_onramp](planned_features/02_mvp2/feat_ubi_onramp/idea.md) | Feature | `feat_ubi_judgments` adds click-derived judgments, but a UBI-centric release has a well-known failure mode: it makes the operators who *don't* have UBI feel like second-class citizens, when in reality | — | Idea — split out from [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) as a first-class feature (2026-05-29) so the no-signals-majority UX is dashboard-visible and not treated as optional UBI polish. | -| 3 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | -| 4 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../../feature_templates/README.md). | -| 5 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is already implemented and already autonomous, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | -| 6 | P2 | [feat_query_normalization_tuning](planned_features/02_mvp2/feat_query_normalization_tuning/idea.md) | Feature | A relevance pipeline runs in stages: (1) query understanding / normalization → (2) retrieval → (3) ranking / boosting → (4) re-ranking. RelyLoop tunes **stage 3 only**. But stage 1 is often where the | — | Idea — scoped to MVP2 (moved from `00_unsure/` 2026-05-29). The core capability is small and fits the existing parameter model. One **gating design fork remains** — the prod-reproducibility question (below) — which `/spec-gen` must resolve; if no clean deployment path exists, this idea may be deferred out of MVP2 at spec time. Tracked as a release item rather than parked because the operator has a clear MVP2 intent and the theme fits the "Real Signals" / relevance-quality story. | -| 7 | P2 | [feat_study_convergence_indicator](planned_features/02_mvp2/feat_study_convergence_indicator/idea.md) | Feature | After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. | -| 8 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | -| 9 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | -| 10 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| 11 | P2 | [bug_seed_meaningful_demos_silent_bulk_errors](planned_features/02_mvp2/bug_seed_meaningful_demos_silent_bulk_errors/idea.md) | Bug | [`scripts/seed_meaningful_demos.py:917-935`](../../scripts/seed_meaningful_demos.py#L917-L935) bulk-indexes 1000 Amazon ESCI products into a dedicated index per demo scenario: | — | Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep | -| 12 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | -| 13 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | -| 14 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | -| 15 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | -| 16 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | +| 1 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | +| 2 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../../feature_templates/README.md). | +| 3 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is already implemented and already autonomous, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | +| 4 | P2 | [feat_query_normalization_tuning](planned_features/02_mvp2/feat_query_normalization_tuning/idea.md) | Feature | A relevance pipeline runs in stages: (1) query understanding / normalization → (2) retrieval → (3) ranking / boosting → (4) re-ranking. RelyLoop tunes **stage 3 only**. But stage 1 is often where the | — | Idea — scoped to MVP2 (moved from `00_unsure/` 2026-05-29). The core capability is small and fits the existing parameter model. One **gating design fork remains** — the prod-reproducibility question (below) — which `/spec-gen` must resolve; if no clean deployment path exists, this idea may be deferred out of MVP2 at spec time. Tracked as a release item rather than parked because the operator has a clear MVP2 intent and the theme fits the "Real Signals" / relevance-quality story. | +| 5 | P2 | [feat_study_convergence_indicator](planned_features/02_mvp2/feat_study_convergence_indicator/idea.md) | Feature | After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. | +| 6 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | +| 7 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | +| 8 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | +| 9 | P2 | [bug_seed_meaningful_demos_silent_bulk_errors](planned_features/02_mvp2/bug_seed_meaningful_demos_silent_bulk_errors/idea.md) | Bug | [`scripts/seed_meaningful_demos.py:917-935`](../../scripts/seed_meaningful_demos.py#L917-L935) bulk-indexes 1000 Amazon ESCI products into a dedicated index per demo scenario: | — | Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep | +| 10 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | +| 11 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | +| 12 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | +| 13 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | +| 14 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | ## Dependency graph @@ -77,6 +83,8 @@ graph LR classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + feat_ubi_judgments["ubi judgments"] + class feat_ubi_judgments plan; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done; ``` diff --git a/docs/00_overview/UNSURE_DASHBOARD.md b/docs/00_overview/UNSURE_DASHBOARD.md index 829272b5..951009a9 100644 --- a/docs/00_overview/UNSURE_DASHBOARD.md +++ b/docs/00_overview/UNSURE_DASHBOARD.md @@ -20,9 +20,9 @@ Pull from the Idea backlog or capture a new feature spec. | → P1 | **0** high-value, ready when P0 clears | | → P2 (default) | 1 important to file, not blocking | | → Backlog | 0 captured for record, not planned | -| Open bugs | 0 | -| Legacy "Path to UNSURE" | 0 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 1 idea-only feat/infra (not yet scoped into UNSURE) | +| Open bugs | 1 | +| Legacy "Path to UNSURE" | 1 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | +| Backlog ideas | 0 idea-only feat/infra (not yet scoped into UNSURE) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -47,7 +47,7 @@ _None._ | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P2 | [feat_query_normalization_tuning](planned_features/00_unsure/feat_query_normalization_tuning/idea.md) | Feature | A relevance pipeline runs in stages: (1) query understanding / normalization → (2) retrieval → (3) ranking / boosting → (4) re-ranking. RelyLoop tunes **stage 3 only**. But stage 1 is often where the | — | Idea — exploratory. Placed in `00_unsure/` because the release target is genuinely unresolved: the core capability is small and fits the existing parameter model, but a prod-reproducibility question (below) must be answered before it can be committed to a release. | +| 1 | P2 | [bug_relyloop_spec_ubi_section_drift](planned_features/00_unsure/bug_relyloop_spec_ubi_section_drift/idea.md) | Bug | [`docs/00_overview/relyloop-spec.md`](relyloop-spec.md) §"Click-derived judgments — OpenSearch UBI as the engine-neutral primary path" (line ~706) carries two staleness bugs from the 2026-05-27 releas | — | Idea — captured during `feat_ubi_judgments` preflight (2026-05-29) | ## Dependency graph diff --git a/docs/00_overview/dashboard.html b/docs/00_overview/dashboard.html index a845e37a..fa3275f9 100644 --- a/docs/00_overview/dashboard.html +++ b/docs/00_overview/dashboard.html @@ -392,7 +392,7 @@

Releases

MVP2 / v0.2
Three-Engine + Real Signals
-
1 / 1 scoped done · 7 remaining
+
1 / 2 scoped done · 8 remaining
In progress
diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index 516fdd7e..f7e02c03 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -382,12 +382,12 @@

RelyLoop MVP2 Dashboard

-
-
Next up
-
All scoped MVP2 features shipped 🎉
-
- Pull from the Idea backlog or capture a new feature spec. -
+
+
Next up — Feature, currently in Plan
+
Ubi Judgments
+
Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters
+
Plan approved; run /impl-execute to ship
+ /impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md --all
@@ -395,15 +395,15 @@

RelyLoop MVP2 Dashboard

MVP2 Progress

-
+
Scoped items done
-
1 / 1
-
100% of feat_/infra_/chore_/epic_ items past idea stage
-
+
1 / 2
+
50% of feat_/infra_/chore_/epic_ items past idea stage
+
Pending work
-
16
+
15
every not-done feat/infra/chore/bug across all priorities
@@ -420,7 +420,7 @@

MVP2 Progress

P1
-
3
+
2
high-value, ready when P0 clears
@@ -435,14 +435,14 @@

MVP2 Progress

Legacy "Path to MVP2"
-
7
+
8
scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)
Backlog ideas: - 9 idea-only feat/infra folders (not yet scoped into MVP2) + 7 idea-only feat/infra folders (not yet scoped into MVP2) In flight: @@ -463,33 +463,7 @@

Pipeline

-

Idea 16

- -
-
Ubi Judgments
-
- Feature - P1 - -
-
MVP1 ships with **LLM-as-judge** as the only authoritative judgment source. The architecture anticipated this would change — the `judgments.source` CHECK already accepts `click`…
- - -
- - -
-
Ubi Onramp
-
- Feature - P1 - -
-
`feat_ubi_judgments` adds click-derived judgments, but a UBI-centric release has a well-known failure mode: it makes the operators who *don't* have UBI feel like second-class citizens, when in reality
- - -
- +

Idea 14

Adapter Solr
@@ -680,7 +654,19 @@

Spec 0

-

Plan 0

+

Plan 1

+ +
+
Ubi Judgments
+
+ Feature + P1 + +
+
Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic
+ + +
@@ -717,6 +703,8 @@

Dependency graph (feat_ + infra_)

classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + feat_ubi_judgments["ubi judgments"] + class feat_ubi_judgments plan; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done;
diff --git a/docs/00_overview/planned_features/00_unsure/bug_relyloop_spec_ubi_section_drift/idea.md b/docs/00_overview/planned_features/00_unsure/bug_relyloop_spec_ubi_section_drift/idea.md new file mode 100644 index 00000000..66e0f0c1 --- /dev/null +++ b/docs/00_overview/planned_features/00_unsure/bug_relyloop_spec_ubi_section_drift/idea.md @@ -0,0 +1,38 @@ +# bug — relyloop-spec.md §"Click-derived judgments" has stale title + broken sibling links + +**Date:** 2026-05-29 +**Status:** Idea — captured during `feat_ubi_judgments` preflight (2026-05-29) +**Priority:** P2 — doc-only; no operator impact today, but the broken links and stale title surface immediately to anyone who hits §706+ of the umbrella spec. +**Origin:** Surfaced during `/idea-preflight` of [`feat_ubi_judgments`](../../02_mvp2/feat_ubi_judgments/idea.md) on 2026-05-29 when verifying the idea's spec citations. + +## Problem + +[`docs/00_overview/relyloop-spec.md`](../../../relyloop-spec.md) §"Click-derived judgments — OpenSearch UBI as the engine-neutral primary path" (line ~706) carries two staleness bugs from the 2026-05-27 release-matrix reshuffle (which compressed MVP1.5 into MVP2 per [`state_history.md`](../../../../state_history.md) "Release-matrix reshuffle (2026-05-27)"): + +1. **Stale title:** The section header still reads `(MVP1.5)` — but MVP1.5 no longer exists as a release stop; the canonical matrix is MVP1 → MVP2 → MVP3 → GA v1. Should read `(MVP2)`. Concrete location: [`relyloop-spec.md:706`](../../../relyloop-spec.md). +2. **Broken sibling links:** The section's closing paragraph (line ~724) links to the two planned-features siblings via relative paths missing the `02_mvp2/` bucket directory AND using an extra `../00_overview/` prefix (wrong from inside `docs/00_overview/relyloop-spec.md`): + - Currently: `[feat_ubi_judgments/idea.md](../00_overview/planned_features/feat_ubi_judgments/idea.md)` — resolves to `docs/00_overview/00_overview/planned_features/feat_ubi_judgments/idea.md` (404). + - Correct: `[feat_ubi_judgments/idea.md](planned_features/02_mvp2/feat_ubi_judgments/idea.md)`. + - Same problem for the `infra_adapter_solr/idea.md` link on the same line. + +## Proposed fix + +Single targeted patch to `docs/00_overview/relyloop-spec.md` updating the §706 header from `(MVP1.5)` to `(MVP2)` and rewriting the two relative paths at line ~724 to include the `02_mvp2/` bucket and drop the stray `../00_overview/` prefix. No code change. No test change. + +## Scope signals + +- **Backend / Frontend / Migration / Config / Audit events:** all N/A. Pure doc edit. +- **Tests:** none required — this is a markdown link/title fix. + +## Why deferred / not inline + +`feat_ubi_judgments` preflight could have applied this patch in the same edit pass, but the spec staleness is bigger than just the UBI feature (the relyloop-spec.md edit would mix scopes: a UBI-feature idea patch + an unrelated spec-doc fix). Capturing as a standalone P2 chore so the spec fix lands in its own focused PR (or rolls into the next docs-sweep PR). The broken links don't block anything operational — they just surface to anyone reading the umbrella spec. + +## Relationship to other work + +- **Surfaced by:** [`feat_ubi_judgments`](../../02_mvp2/feat_ubi_judgments/idea.md) preflight 2026-05-29. +- **Adjacent staleness:** the 2026-05-27 release-matrix reshuffle (see [`state_history.md`](../../../../state_history.md) "Release-matrix reshuffle (2026-05-27)") updated 24 active-doc references but missed this spec section. Worth one grep pass for any other `MVP1.5` references in `docs/00_overview/relyloop-spec.md` while the section is open. + +## Open questions for /spec-gen + +None — this is a 3-line targeted fix; doesn't warrant a spec stage. Can ship via direct PR or as part of the next docs-only finalization sweep. diff --git a/docs/00_overview/planned_features/02_mvp2/feat_query_normalization_tuning/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_query_normalization_tuning/idea.md index 6ac29df4..1c3ff32b 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_query_normalization_tuning/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/feat_query_normalization_tuning/idea.md @@ -57,4 +57,4 @@ Normalizers are a small, **pure-domain, deterministic** library (`none` / `lower - **Documented in** [`docs/01_architecture/optimization.md` §"Where RelyLoop fits in your relevance pipeline"](../../../../01_architecture/optimization.md) (the Tier-1 boundary doc that landed with this idea) — that section is the operator-facing "normalize first; RelyLoop tunes ranking" guidance and points here for the tunable extension. - **Distinct from analyzer changes** — those stay a permanent non-goal (umbrella spec §4); this is strictly query-time string rewriting that never touches the cluster. - **Composes with the apply path** ([`apply-path.md`](../../../../01_architecture/apply-path.md)) — if resolved via option (a), the winning normalizer ships in the config-repo PR like any other tuned parameter. -- **Mirrors the UBI on-ramp opt-in philosophy** ([`feat_ubi_onramp`](../feat_ubi_onramp/idea.md)) — a capability for operators who *can* participate must not degrade the experience for those who can't; default behavior is unchanged. +- **Mirrors the UBI on-ramp opt-in philosophy** (the readiness-ladder / progressive-enhancement framing in [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md), which absorbed the former `feat_ubi_onramp` 2026-05-29) — a capability for operators who *can* participate must not degrade the experience for those who can't; default behavior is unchanged. diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md new file mode 100644 index 00000000..6bfabfad --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md @@ -0,0 +1,867 @@ +# Feature Specification — UBI Judgments (engine-neutral User Behavior Insights as an optional, first-class judgment source) + +**Date:** 2026-05-29 +**Status:** Approved (cross-model converged at cycle-3 cap with all findings accepted; see D-10) +**Owners:** RelyLoop maintainers (Product + Engineering) +**Related docs:** +- [`idea.md`](idea.md) +- [`infra_adapter_solr/idea.md`](../infra_adapter_solr/idea.md) (co-ships in MVP2 "Three-Engine + Real Signals") +- [`docs/01_architecture/api-conventions.md`](../../../../01_architecture/api-conventions.md) +- [`docs/01_architecture/adapters.md`](../../../../01_architecture/adapters.md) +- [`docs/01_architecture/data-model.md`](../../../../01_architecture/data-model.md) +- [`docs/00_overview/relyloop-spec.md`](../../../relyloop-spec.md) §14, §19, §27 (UBI patches) +- Reuses: [`feat_llm_judgments`](../../../implemented_features/2026_05_11_feat_llm_judgments/) (Arq worker shape, calibration helpers, `start_judgment_generation` dispatcher), [`feat_contextual_help`](../../../implemented_features/2026_05_15_feat_contextual_help/) (HelpPopover idiom), [`feat_chat_agent`](../../../implemented_features/2026_05_12_feat_chat_agent/) (tool-registry triad), [`chore_form_dropdown_primitive`](../../../implemented_features/) (form-select discipline) + +--- + +## 1) Purpose + +- **Problem:** MVP1 ships LLM-as-judge as the only authoritative judgment source. That is a weak trust anchor for any operator with production search traffic, the long tail of real queries never gets rated, and judgment lists go stale the moment they are snapshotted. The architecture has anticipated this — `judgments.source` CHECK already accepts `click` at [`backend/app/db/models/judgment.py:46-47`](../../../../../backend/app/db/models/judgment.py#L46-L47), `JudgmentSourceWire` already enumerates `('llm', 'human', 'click')` at [`backend/app/api/v1/schemas.py:827`](../../../../../backend/app/api/v1/schemas.py#L827), and the umbrella spec §14 calls out the mixed-source contract — but the actual UBI reader, signal converters, ingestion endpoint, and agent tool have never been built. +- **Outcome:** Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters (`ctr_threshold`, `dwell_time`, `hybrid_ubi_llm`) — the hybrid mode interleaves LLM-fill rows with `source='click'` rows in the same judgment list. Operators **without** UBI keep MVP1's LLM-as-judge experience unchanged, with a dismissible engine-aware nudge surfacing the on-ramp. Every UBI touchpoint is progressive enhancement — never a gate. The PR-body confidence claim grows from "scored against 500 LLM ratings against a snapshot query set" to "scored against 50,000 UBI-derived ratings covering 90% of last week's traffic." +- **Non-goal:** RelyLoop never installs the UBI plugin, never writes to the cluster, never modifies schema/mapping/analyzer settings (umbrella spec §4 non-goals). Detection is **read-only** via a `get_schema` probe for the `ubi_queries` index. RelyLoop never runs online A/B tests, never trains LTR models, never sits on the live serving path. Counterfactual click models (CCM / DBN) are out of scope — same `SignalsConverter` Protocol, additive in v1.5+ post-GA. + +## 2) Current state audit + +### Existing implementations + +| File / component | What it does | API used | Notes | +|---|---|---|---| +| [`backend/app/db/models/judgment.py`](../../../../../backend/app/db/models/judgment.py) | `Judgment` ORM with `source` CHECK `IN ('llm', 'human', 'click')` | — | `click` is reserved-but-unused per the module docstring (line 11–12). This spec lights it up. | +| [`backend/app/db/models/judgment_list.py`](../../../../../backend/app/db/models/judgment_list.py) | `JudgmentList` ORM with `calibration` JSONB (line 58) and `status IN ('generating', 'complete', 'failed')` CHECK | — | UBI worker writes calibration rows of a different shape (`coverage_pct`, `head_pairs`, `tail_pairs`, `position_bias_prior_id`, `ambiguous_query_skip_count`). Adds a new `generation_params` JSONB column (Alembic head `0021`) populated at INSERT for UBI lists so the boot-time resume sweep can reconstruct the worker call without depending on the Arq job payload (cycle-3 finding `ubi-generation-params-not-persisted`). LLM lists leave the new column NULL — the existing `current_template_id` + `rubric` already carry their resume state. | +| [`backend/app/api/v1/judgments.py:170-220`](../../../../../backend/app/api/v1/judgments.py#L170-L220) | `POST /api/v1/judgments/generate` — 202 + `GenerateJudgmentsResponse{judgment_list_id, status}` | Delegates to `start_judgment_generation` | The new UBI endpoint mirrors this shape exactly; same router, same prefix, same `generate-*` action verb. | +| [`backend/app/api/v1/judgments.py:126-147`](../../../../../backend/app/api/v1/judgments.py#L126-L147) | `_detail()` populates `_SourceBreakdown(llm=…, human=…)` from `repo.source_breakdown_for_list` | — | `_SourceBreakdown` currently locks `llm + human == judgment_count` per `feat_llm_judgments` cycle-2 F6. With UBI shipping mixed lists, this invariant becomes user-visible inaccuracy. This spec evolves the shape (D-1). | +| [`backend/app/api/v1/schemas.py:864-873`](../../../../../backend/app/api/v1/schemas.py#L864-L873) | `_SourceBreakdown(llm: int, human: int)` Pydantic model | — | Path A evolution: add `click: int`. The only consumers are the project's own UI + contract tests. | +| [`backend/app/api/v1/schemas.py:833`](../../../../../backend/app/api/v1/schemas.py#L833) | `JudgmentSourceFilterWire = Literal["llm", "human"]` | — | Spec §8.4 rejected `click` at the API filter boundary in MVP1 (cycle-1 F1) because no UBI rows existed yet. This spec promotes the filter to `Literal["llm", "human", "click"]` so operators can audit UBI-only or hybrid lists. | +| [`backend/app/services/agent_judgments_dispatch.py:69`](../../../../../backend/app/services/agent_judgments_dispatch.py#L69) | `start_judgment_generation` — shared dispatcher (preflight A–F + INSERT + Arq enqueue) used by router AND chat-agent tool | — | This spec adds a sibling `start_ubi_judgment_generation` in the same module, mirroring the seven-stage preflight (UBI-shaped: capability probe, query-mapping, coverage gate, FK, optional budget-peek for hybrid, INSERT, enqueue). | +| [`backend/workers/judgments.py:354`](../../../../../backend/workers/judgments.py#L354) | `generate_judgments_llm` Arq job — full pipeline with budget gate, per-query resume-skip, set-equal validation | — | UBI worker (`generate_judgments_from_ubi`) lives in the same module; shares `_safe_record_cost`, `_fail_list`, the `bulk_create_judgments` ON CONFLICT DO NOTHING pattern, and the `source='click'` / `rater_ref='ubi:{converter}'` row shape. Hybrid mode delegates LLM-fill calls to a new `_process_query_llm_fill` helper that wraps the existing `rate_query_batch` so the daily-budget gate at [`backend/app/llm/budget_gate.py`](../../../../../backend/app/llm/budget_gate.py) fires unchanged. | +| [`backend/app/agent/tools/judgments/generate_judgments_llm.py`](../../../../../backend/app/agent/tools/judgments/generate_judgments_llm.py) | LLM-judgment agent tool — `GenerateJudgmentsLLMArgs` + `generate_judgments_llm_impl` + `GENERATE_JUDGMENTS_LLM_TOOL` triad | — | New sibling `generate_judgments_from_ubi.py` follows the same triad pattern; registered in [`backend/app/agent/tools/__init__.py`](../../../../../backend/app/agent/tools/__init__.py) (the `TOOLS` / `TOOL_REGISTRY` / `TOOL_ARG_MODELS` registry with module-load drift assertion at line 232–236). | +| [`backend/app/adapters/protocol.py`](../../../../../backend/app/adapters/protocol.py) | `SearchAdapter` Protocol with `search_batch`, `get_schema`, `list_targets` | — | UBI reader uses only the existing surface — two scrolling `search_batch` calls (one against `ubi_queries`, one against `ubi_events`) plus a `get_schema('ubi_queries')` probe for the readiness ladder. No adapter Protocol changes. | +| [`backend/app/db/models/cluster.py:30`](../../../../../backend/app/db/models/cluster.py#L30) | `engine_type IN ('elasticsearch', 'opensearch')` CHECK | — | Solr arm of Capability B's engine-aware nudge is dark until `infra_adapter_solr` extends this CHECK. Spec accommodates by branching on `cluster.engine_type` with a `solr` arm that returns a placeholder runbook link guarded by a feature check. | +| [`ui/src/lib/enums.ts:111-120`](../../../../../ui/src/lib/enums.ts#L111-L120) | `JUDGMENT_SOURCE_VALUES`, `JUDGMENT_SOURCE_FILTER_VALUES` | — | Spec adds three new arrays, each with the `// Values must match backend/...` discipline comment: `JUDGMENT_GENERATION_METHOD_VALUES = ['llm', 'ctr_threshold', 'dwell_time', 'hybrid_ubi_llm']` (mirrors `JudgmentGenerationMethodWire`; consumed by the picker `` MUST follow the `chore_form_dropdown_primitive` discipline — `UBI_CONVERTER_VALUES.map(...)` imported from `@/lib/enums`. | +| [`ui/src/lib/glossary.ts:436-452`](../../../../../ui/src/lib/glossary.ts#L436-L452) | Existing `judgment.source.*` entries (definitional) | — | Spec adds 4 new keys: `judgment.converter`, `judgment.converter.ubi`, `judgment.converter.hybrid`, `cluster.ubi_readiness`. | +| [`ui/src/lib/faq.ts`](../../../../../ui/src/lib/faq.ts) | Operator-judgment-shaped Q&A | — | Spec adds 3 new entries under the `judgments` category (see §15). | +| [`ui/src/components/dashboard/demo-data-banner.tsx`](../../../../../ui/src/components/dashboard/demo-data-banner.tsx) | Dismissible card using `useSyncExternalStore` + `safeLocalStorageGet`/`safeLocalStorageSet` (SSR-safe pattern) | — | The Capability B nudge follows this exact pattern (NOT the `useLocalStorageSet` hook, which is Set-shaped for column visibility). Storage key shape: `relyloop.ubi-onramp-nudge.dismissed:{cluster_id}`. | +| [`ui/src/components/common/help-popover.tsx`](../../../../../ui/src/components/common/help-popover.tsx) | Glossary-backed popover (`feat_contextual_help`) | — | Inline helper text under each converter picker option uses `HelpPopover` keyed off the new glossary entries. | + +### Navigation and link impact + +| Source file | Current link target | New link target | +|---|---|---| +| [`docs/00_overview/relyloop-spec.md:706`](../../../relyloop-spec.md) | `### Click-derived judgments — OpenSearch UBI as the engine-neutral primary path (MVP1.5)` | `### Click-derived judgments — OpenSearch UBI as the engine-neutral primary path (MVP2)` — drift fix from the 2026-05-27 release-matrix compression | +| [`docs/00_overview/relyloop-spec.md:724`](../../../relyloop-spec.md) | Sibling planned-feature refs missing the `02_mvp2/` bucket | `planned_features/02_mvp2/feat_ubi_judgments/idea.md` and `planned_features/02_mvp2/infra_adapter_solr/idea.md` (correct relative path from inside `docs/00_overview/`) | +| [`docs/08_guides/tutorial-first-study.md`](../../../../08_guides/tutorial-first-study.md) | No "Step 7 — swap LLM list for UBI" yet | New optional Step 7 demonstrating the value-delta upgrade; tutorial must still complete fully on the LLM path for readers with no UBI cluster. | + +### Existing test impact + +| Test file | Pattern | Count | Required change | +|---|---|---|---| +| `backend/tests/contract/test_judgments_*.py` | Asserts `_SourceBreakdown` shape `{llm, human}` (no `click`) | TBD at impl time | Update to assert `{llm, human, click}` shape; assertions covering `source_breakdown.human` on LLM-only lists keep working (`click=0`); new assertions cover mixed-source lists. | +| `backend/tests/contract/test_judgments_*.py` | Asserts `?source=` filter rejects `click` with 422 (cycle-1 F1) | TBD | Update — `click` is now an accepted filter value (returns matching click-derived rows). Spec §8.4 in `feat_llm_judgments` was authored when no UBI rows existed; this feature is the moment to invert it. | +| `backend/tests/unit/eval/test_qrels_loader.py` | Loads qrels via `SELECT … FROM judgments WHERE judgment_list_id = :id` | — | No change — qrels_loader is source-agnostic (just reads `rating`). UBI and LLM rows blend identically for the optimizer. | +| `backend/tests/integration/agent/test_tool_registry.py` (or equivalent) | Enumerates tools in `TOOLS` / `TOOL_REGISTRY` / `TOOL_ARG_MODELS` | — | Add `generate_judgments_from_ubi` to all three; the module-load drift assertion at `backend/app/agent/tools/__init__.py:232-236` catches a missing-from-one regression at import time. | +| `ui/src/__tests__/components/query-sets/generate-judgments-dialog.test.tsx` | Submits the 4-field form | TBD | Add cases for: converter picker default selection (per rung), `hybrid_ubi_llm` showing the LLM-fill threshold field, `since`/`until` window controls, engine-aware nudge dismissal, sparse-data hybrid recommendation copy. | +| `ui/src/__tests__/components/common/form-select-discipline.test.tsx` | Lint guard rejects inline `` for wire-value enums in `ui/src/components/` form `.tsx` files | — | No change to the guard itself; the new converter `` in the generate-judgments dialog with four options (`llm`, `ctr_threshold`, `dwell_time`, `hybrid_ubi_llm`). Default selection follows the detected rung (rung 0 → `llm`; rung 1–2 → `hybrid_ubi_llm`; rung 3 → `ctr_threshold` if no dwell signal, else operator picks). `InfoTooltip` + `HelpPopover` glossary backing for each option. The picker MUST use the `UBI_CONVERTER_VALUES.map(...)` pattern per `chore_form_dropdown_primitive` lint guard. +- **Glossary + FAQ + runbook + tutorial.** 4 new glossary keys, 3 new FAQ entries, new runbook `docs/03_runbooks/ubi-judgment-generation.md`, optional Step 7 in `docs/08_guides/tutorial-first-study.md`, umbrella spec §14 + §706 + §724 patches. +- **Promote `JudgmentSourceFilterWire` from `Literal["llm", "human"]` to `Literal["llm", "human", "click"]`.** UI `JUDGMENT_SOURCE_FILTER_VALUES` widens to match. The `?source=` query on `GET /api/v1/judgment-lists/{id}/judgments` now accepts `click`. (D-3.) +- **Evolve `_SourceBreakdown` in-place to `{llm, human, click}` with the invariant `llm + human + click == judgment_count`.** No `V2` versioning — the only OpenAPI consumers today are the project's own UI + contract tests. (D-1.) + +### Out of scope + +- **Cluster writes / plugin installation.** RelyLoop never installs the UBI plugin, never modifies cluster schema/mapping/analyzer, never writes to `ubi_queries` or `ubi_events`. Detection is read-only via `get_schema` + aggregation searches. +- **Solr adapter.** Solr ships in the sibling `infra_adapter_solr` MVP2 feature. Capability B's `solr` arm is **dark code** in this PR — guarded by an `engine_type === 'solr'` branch that won't fire until `infra_adapter_solr` extends `clusters_engine_type_check`. When that lands, the Capability B copy switches on without code change here. +- **Counterfactual click models (CCM, DBN).** Same `SignalsConverter` Protocol, additive in v1.5+ post-GA when adopters have enough impressions per `(query, doc)` to be statistically valid. +- **Multi-tenancy.** RelyLoop is single-tenant through GA v1. No `tenant_id` column on new tables (there are no new tables — UBI rides existing `judgments` + `judgment_lists`). +- **`Idempotency-Key` header.** Scheduled for GA v1 per [`docs/01_architecture/api-conventions.md` line 179 + 225](../../../../01_architecture/api-conventions.md). The UBI endpoint follows the existing MVP2 pattern — name-uniqueness on `judgment_lists.name` catches duplicate-request collisions. +- **Online A/B testing, LTR training, query-time-config writes back to the cluster.** All explicitly excluded by umbrella spec §4. +- **Native non-OpenAI provider SDKs** for the hybrid LLM-fill path. Hybrid mode uses the existing `openai` SDK pointed at any OpenAI-compatible endpoint via `OPENAI_BASE_URL` (Absolute Rule #3 / #8 / #10). + +### API convention check + +Verified against [`docs/01_architecture/api-conventions.md`](../../../../01_architecture/api-conventions.md): + +- **Endpoint prefix:** `/api/v1/` for business endpoints. The new endpoint is `POST /api/v1/judgments/generate-from-ubi` — same `/judgments` prefix, same `generate-*` action verb as the existing `POST /api/v1/judgments/generate` ([`backend/app/api/v1/judgments.py:171`](../../../../../backend/app/api/v1/judgments.py#L171)). Readiness endpoint: `GET /api/v1/clusters/{id}/ubi-readiness` (per-cluster sub-resource pattern, mirrors `GET /api/v1/clusters/{id}/schema` shipped in `infra_adapter_elastic`). +- **Router namespace:** existing [`backend/app/api/v1/judgments.py`](../../../../../backend/app/api/v1/judgments.py) for the generate endpoint; existing [`backend/app/api/v1/clusters.py`](../../../../../backend/app/api/v1/clusters.py) for the readiness endpoint. +- **HTTP methods:** POST=create; GET=read. +- **Non-auth error envelope:** `{ "detail": { "error_code": "", "message": "", "retryable": } }` — exact shape from the existing `_err()` helper at [`backend/app/api/v1/judgments.py:86-90`](../../../../../backend/app/api/v1/judgments.py#L86-L90). +- **Auth error shape:** N/A — single-tenant, no auth in MVP2. + +### Phase boundaries + +**Default delivery: single phase.** All 10 capabilities (1, 2, 3, 4, 5, A, B, C, D, E) ship in one PR. The idea author explicitly merged the on-ramp folder (`feat_ubi_onramp`) back into this feature 2026-05-29 to make it impossible to ship half — the substrate without the on-ramp creates the exact "UBI is a wall" failure mode the on-ramp exists to prevent. + +**Contingency (decided at `/impl-plan-gen` time):** If the combined PR exceeds reviewability thresholds (~1500 LOC bundled diff is the historical pain point on this project; current scope estimate is ~1350 LOC bundled), the correct split is **by delivery phase within this folder**: + +- **Phase 1 — Substrate + always-LLM picker.** Capabilities 1, 2, 3, 4, 5, E with `converter='llm'` as the picker default for ALL rungs. Ships `UbiReader`, `SignalsConverter`, the endpoint, the worker, the agent tool, the converter picker, the in-place `_SourceBreakdown` evolution, the filter widening. Does NOT ship the on-ramp UX (Capabilities A, B, C, D); a rung-0 operator who picks UBI hits a bare 412 `UBI_NOT_ENABLED`. Acceptable for a 1–2-week gap because Phase 2 ships before the MVP2 release tag. +- **Phase 2 — Readiness + on-ramp UX.** Capabilities A, B, C, D. Wires the rung-detection probe, the dismissible nudge, the sparse-data guidance, the value-delta surface. Picker default per rung becomes active. + +A `phase2_idea.md` tracker file is **NOT created up front** because the default delivery is single-phase. If `/impl-plan-gen` elects to split, it creates `phase2_idea.md` at that time per the impl-plan-gen skill's Step 10 (deferred-phase tracking). + +## 4) Product principles and constraints + +- **UBI is opt-in progressive enhancement, never a gate.** Every UBI touchpoint MUST have a no-UBI fallback (or sparse-UBI hybrid recommendation). The 412 / 422 UBI error codes are structured renderable states the UI turns into on-ramps — never bare error pages. +- **No cluster writes ever.** RelyLoop reads `ubi_queries` + `ubi_events` via the existing `SearchAdapter.search_batch` surface. It NEVER calls `_index`, `_update`, `_bulk`, `PUT _mapping`, or any write API. The `get_schema` readiness probe is a read. +- **Engine-neutral by Protocol.** `UbiReader` consumes only `SearchAdapter` — no `ElasticAdapter`-specific code. When `infra_adapter_solr` ships, UBI on Solr works the same day, no UBI code change required. +- **Per-row source provenance.** Every persisted judgment carries the exact `source` (`llm` / `human` / `click`) and `rater_ref` (`openai:{model}` for LLM, `operator` for human, `ubi:{converter}` for click) so audit + calibration + delta-surfacing all trace back to the row's origin. +- **LLM-fill always routes through the existing budget gate.** Hybrid-mode LLM calls reuse `rate_query_batch` + `peek_daily_total` + `record_cost` from [`backend/app/llm/budget_gate.py`](../../../../../backend/app/llm/budget_gate.py). No new LLM client, no direct `openai.AsyncClient(...)` outside the shared helper (Absolute Rule #3 / #8 / #10). +- **Wire-value contracts grounded in backend Literals.** `converter` field values, readiness rung values, source filter values — all live as `Literal[...]` in `backend/app/api/v1/schemas.py` (or `backend/app/domain/ubi/converter.py` for the converter enum) and mirror in `ui/src/lib/enums.ts` with the `// Values must match backend/...` comment. Frontend `` with four options (`llm`, `ctr_threshold`, `dwell_time`, `hybrid_ubi_llm`) using `JUDGMENT_GENERATION_METHOD_VALUES.map(...)` imported from `@/lib/enums`. Labels per D-2. The picker drives endpoint routing: `llm` → `POST /api/v1/judgments/generate` (the existing endpoint); the other three → `POST /api/v1/judgments/generate-from-ubi` with the picker value passed as the request `converter` field. + - Default selection from `useUbiReadiness(cluster_id, query_set_id)` hook: rung_0 → `llm`; rung_1 / rung_2 → `hybrid_ubi_llm`; rung_3 → `ctr_threshold`. + - UBI window controls (`since` date-time picker + optional `until`); shown only when converter ≠ `llm`. Default `since = now - 30 days`. + - LLM-fill threshold input (default 20); shown only when converter == `hybrid_ubi_llm`. + - `InfoTooltip` + `HelpPopover` under each converter option backed by glossary keys (`judgment.converter.llm`, `judgment.converter.ubi`, `judgment.converter.hybrid`). + - The system **MUST** render the dismissible nudge (`ui/src/components/clusters/ubi-onramp-nudge.tsx`) above the dialog body when `rung === 'rung_0'`. Engine-aware copy switches on `cluster.engine_type`. + - The system **MUST** render the sparse-data recommendation card when the request would have returned 422 `UBI_INSUFFICIENT_DATA` (caught client-side by inspecting `useUbiReadiness().rung === 'rung_1'` before submit). Card includes a "Switch to hybrid" button that flips the picker to `hybrid_ubi_llm`. + - The system **MUST** display the value-delta on the judgment-list detail page (`ui/src/app/judgments/[id]/page.tsx`) for UBI/hybrid lists: `coverage_pct` always; `vs. previous LLM list X` delta only when a prior LLM list exists on the same `query_set_id` (resolved via a server-side helper `repo.list_judgment_lists(query_set_id=…, sort='created_at:desc')`). + +### FR-9: Wire-value contract — `converter` + `method` + `readiness rung` + +The picker's value space is intentionally larger than the new endpoint's accepted set: the picker chooses between *all four* judgment-generation methods (including `llm`, which routes to the existing `POST /api/v1/judgments/generate` endpoint), while the new `POST /api/v1/judgments/generate-from-ubi` endpoint accepts only the *three UBI converters*. Two distinct enums are required so the backend can reject `llm` at the new endpoint while the UI can still surface it as a first-class picker option. + +- **Requirement:** + - The system **MUST** define `UbiConverterKind = Literal["ctr_threshold", "dwell_time", "hybrid_ubi_llm"]` in [`backend/app/api/v1/schemas.py`](../../../../../backend/app/api/v1/schemas.py). This is the *request-side* enum — the value the new UBI endpoint accepts on `CreateJudgmentListFromUbiRequest.converter`. + - The system **MUST** define `JudgmentGenerationMethodWire = Literal["llm", "ctr_threshold", "dwell_time", "hybrid_ubi_llm"]` in the same file. This is the *picker-side* enum — the union of all 4 generation methods the UI surfaces. The `llm` value routes the submit to `POST /api/v1/judgments/generate`; the other three route to `POST /api/v1/judgments/generate-from-ubi`. The two are kept in sync structurally — `UbiConverterKind ⊂ JudgmentGenerationMethodWire` — but they are distinct types so static checks at the endpoint boundary correctly reject `llm`. + - The system **MUST** define `UbiReadinessRungWire = Literal["rung_0", "rung_1", "rung_2", "rung_3"]` in the same file. + - The system **MUST** add `JUDGMENT_GENERATION_METHOD_VALUES` (4 values; mirrors `JudgmentGenerationMethodWire`), `UBI_CONVERTER_VALUES` (3 values; mirrors `UbiConverterKind`), and `UBI_READINESS_RUNG_VALUES` (4 values) arrays to [`ui/src/lib/enums.ts`](../../../../../ui/src/lib/enums.ts), each with the `// Values must match backend/app/api/v1/schemas.py ` comment per the source-of-truth policy. The converter ``; `ui/src/lib/enums.ts` `JUDGMENT_GENERATION_METHOD_VALUES` (4 values). `llm` selection routes the submit to `POST /api/v1/judgments/generate`; the other three route to `POST /api/v1/judgments/generate-from-ubi`. | +| `mapping_strategy` (request body, optional) | `reject` (default), `first_match`, `most_recent` | `backend/app/api/v1/schemas.py` `UbiMappingStrategyWire` | Future advanced-settings panel; not exposed in MVP2 dialog default form | +| `rung` (response — `GET /clusters/{id}/ubi-readiness`) | `rung_0`, `rung_1`, `rung_2`, `rung_3` | `backend/app/api/v1/schemas.py` `UbiReadinessRungWire` | `ui/src/lib/enums.ts` `UBI_READINESS_RUNG_VALUES`; `ui/src/components/clusters/ubi-rung-badge.tsx` | +| `?source` (existing filter on `GET /judgment-lists/{id}/judgments`) | `llm`, `human`, `click` (widened from `{llm, human}` per D-3) | `backend/app/api/v1/schemas.py` `JudgmentSourceFilterWire` | `ui/src/lib/enums.ts` `JUDGMENT_SOURCE_FILTER_VALUES` | +| `source_breakdown` keys (response on `GET /judgment-lists/{id}`) | `llm`, `human`, `click` (evolved from `{llm, human}` per D-1) | `backend/app/api/v1/schemas.py` `_SourceBreakdown` | `ui/src/app/judgments/[id]/page.tsx` breakdown rendering | +| `judgments.source` (per-row, persisted) | `llm`, `human`, `click` | `backend/app/db/models/judgment.py` `judgments_source_check` CHECK + `JudgmentSourceWire = Literal["llm", "human", "click"]` (unchanged) | `ui/src/lib/enums.ts` `JUDGMENT_SOURCE_VALUES` (unchanged) | + +UI labels for the converter picker (per D-2): +- `llm` → "LLM-as-judge" +- `ctr_threshold` → "UBI (click-through)" +- `dwell_time` → "UBI (dwell-time)" +- `hybrid_ubi_llm` → "Hybrid UBI + LLM" + +### 8.5 Error code catalog + +| Code | HTTP Status | Meaning | +|---|---|---| +| `UBI_NOT_ENABLED` | 412 | `ubi_queries` index does not exist on the cluster. Operator must install the UBI plugin + enable event capture. Capability B nudge surfaces this state inline; the 412 envelope is the structured form. | +| `UBI_INSUFFICIENT_DATA` | 422 | Fewer than `min_impressions_threshold` (default 100) UBI events match the `(since, until, target)` window. Returned **sync from preflight U-D2**; Capability C surfaces a "Switch to hybrid" recommendation card; the 422 envelope is the structured form. The worker also carries a terminal `failed_reason='UBI_INSUFFICIENT_DATA'` path as a race-condition fallback for the (essentially impossible) case where data disappears between preflight and execution. | +| `UBI_WINDOW_TOO_LARGE` | 422 | `(until - since) > 90 days`. Cost guardrail — operators run multiple narrower windows and merge via separate lists. | +| `VALIDATION_ERROR` | 422 | Generic request-shape failures (FK consistency, oversize query set, etc.) — reuses the existing envelope from `feat_llm_judgments`. | +| `CLUSTER_NOT_FOUND` | 404 | `cluster_id` does not resolve. Existing code, unchanged. | +| `QUERY_SET_NOT_FOUND` | 404 | `query_set_id` does not resolve. Existing code, unchanged. | +| `TEMPLATE_NOT_FOUND` | 404 | Hybrid mode only — `current_template_id` does not resolve. Reused from `start_judgment_generation` preflight D (see [`backend/app/services/agent_judgments_dispatch.py:146-152`](../../../../../backend/app/services/agent_judgments_dispatch.py#L146-L152)); inherited verbatim by the shared `_resolve_cluster_query_set` helper in FR-4. | +| `JUDGMENT_LIST_NAME_TAKEN` | 409 | `name` collides with an existing `judgment_lists.name`. Existing code, unchanged. | +| `CLUSTER_UNREACHABLE` | 503 | Engine reachability failure during the readiness probe. Retryable. Existing code, unchanged. | +| `OPENAI_NOT_CONFIGURED` | 503 | Hybrid-mode only — operator hasn't configured a key. Existing code (Absolute Rule #2), unchanged. | +| `LLM_PROVIDER_INCAPABLE` | 503 | Hybrid-mode only — capability cache miss or structured-output unsupported. Existing code, unchanged. | +| `UNKNOWN_MODEL_PRICING` | 503 | Hybrid-mode only — `OPENAI_MODEL` not in `cost_model`. Existing code, unchanged. | +| `OPENAI_BUDGET_EXCEEDED` | 503 | Hybrid-mode only — daily-budget peek already at cap. Existing code, unchanged. Retryable. | + +## 9) Data model and state transitions + +### One additive migration: `0021_judgment_lists_generation_params.py` + +**Modified table: `judgment_lists`** — adds one nullable JSONB column to support UBI worker resume: + +- `generation_params` (`JSONB`, nullable) — UBI lists populate this at INSERT with the JSON-serialized request shape so the boot-time resume sweep can reconstruct the worker call without depending on the Arq job payload (cycle-3 finding `ubi-generation-params-not-persisted`). LLM lists leave it NULL — `current_template_id` + `rubric` already carry LLM resume state. + +Migration `0021_judgment_lists_generation_params.py`: + +```python +def upgrade() -> None: + op.add_column( + "judgment_lists", + sa.Column("generation_params", postgresql.JSONB, nullable=True), + ) + +def downgrade() -> None: + op.drop_column("judgment_lists", "generation_params") +``` + +Round-trip verified via `alembic upgrade head && alembic downgrade -1 && alembic upgrade head` per Absolute Rule #5. Pre-existing LLM judgment lists pass through both directions cleanly because the column is nullable and never read on the LLM path. + +**Modified columns / responses (not migrations — Pydantic-level evolutions):** + +- **`_SourceBreakdown` (Pydantic, [`backend/app/api/v1/schemas.py:864-873`](../../../../../backend/app/api/v1/schemas.py#L864-L873))** — adds `click: int` field. Invariant evolves to `llm + human + click == judgment_count`. (D-1.) +- **`JudgmentSourceFilterWire` (Pydantic, [`backend/app/api/v1/schemas.py:833`](../../../../../backend/app/api/v1/schemas.py#L833))** — widens from `Literal["llm", "human"]` to `Literal["llm", "human", "click"]`. (D-3.) +- **`JudgmentList.calibration` (JSONB column, existing)** — adds new content shape for UBI lists: `{coverage_pct, head_pairs, tail_pairs, position_bias_prior_id, llm_fill_calls?, ambiguous_query_skip_count, sparse_query_skip_count}`. The column is free-form JSONB; no schema change. LLM and UBI lists are distinguished by the presence of `cohens_kappa` (LLM/calibrated) vs `coverage_pct` (UBI). When `POST /judgment-lists/{id}/calibration` runs against a UBI list later, the merge appends `cohens_kappa` + sibling keys to the existing UBI-shaped object. + +### Required invariants + +- **UNIQUE `(judgment_list_id, query_id, doc_id)`** on `judgments` — enforced by existing `judgments_unique_key` constraint. UBI worker's `bulk_create_judgments` ON CONFLICT DO NOTHING relies on it for resume idempotency. +- **`source IN ('llm', 'human', 'click')`** on `judgments` — enforced by existing `judgments_source_check` CHECK. No change. +- **`status IN ('generating', 'complete', 'failed')`** on `judgment_lists` — enforced by existing `judgment_lists_status_check` CHECK. No change. UBI worker terminal flips use the same three states. +- **`rating BETWEEN 0 AND 3`** on `judgments` — enforced by existing `judgments_rating_check` CHECK. Each `SignalsConverter` MUST return ratings in `{0, 1, 2, 3}`. +- **`name`** uniqueness on `judgment_lists` (existing UNIQUE on the column) — UBI `POST` collides with 409 `JUDGMENT_LIST_NAME_TAKEN` on conflict. +- **`llm + human + click == judgment_count`** on `_SourceBreakdown` — invariant evolved from the cycle-2 F6 two-term form. + +### State transitions + +`judgment_lists.status` for a UBI-generated list: + +``` +(create) → generating +generating → complete [worker: clean loop, all pairs persisted] +generating → failed [worker: UbiInsufficientDataError | UbiQueryMappingAmbiguousError | + BudgetExceededError (hybrid) | UnknownModelPricingError (hybrid) | + UNEXPECTED:] +``` + +Identical lifecycle to `generate_judgments_llm`. The boot-time resume sweep at `backend/workers/all.py:148-161` re-enqueues any `status='generating'` row (UBI or LLM) when the worker boots — UBI worker MUST also handle the resume-skip per query (`count_judgments_for_list_and_query > 0` ⇒ skip). + +### Idempotency / replay + +- The Arq job is keyed by `_job_id=f"generate_judgments_from_ubi:{judgment_list_id}"` — duplicate enqueues collapse to one running job. +- Per-query resume-skip via `count_judgments_for_list_and_query` (same pattern as `generate_judgments_llm`). +- The worker's `bulk_create_judgments` uses ON CONFLICT DO NOTHING — re-runs after partial crashes don't double-insert. + +## 10) Security, privacy, and compliance + +**Threats:** + +1. **Leaked operator query text through the LLM-fill path.** Hybrid-mode LLM calls send `query_text` and (truncated, ≤500 char) document bodies to whatever endpoint `OPENAI_BASE_URL` points at. Mitigation: same data-flow document as `feat_llm_judgments` ([`docs/04_security/llm-data-flow.md`](../../../../04_security/llm-data-flow.md)) — the runbook (§15) extends with a "Hybrid-mode UBI fill" subsection. The capability check is the operator's signal that LLM-dependent paths will fire. +2. **Inadvertent cluster writes.** Bug class: somebody adds `_index` / `_bulk` / `_update` to `UbiReader`. Mitigation: code-review checklist + an integration test asserting that `UbiReader.read_features` never issues a write-shaped HTTP call (mock the adapter, fail the test if `client.send` is called with an HTTP method other than GET / POST `_search`). +3. **PII in UBI events.** UBI events may carry `client_id`, `session_id`, `user_query` text — operator-emitted. RelyLoop never persists raw events; it persists per-(query, doc) ratings derived from them. The original event data stays in the cluster. Mitigation: persist only ratings + `rater_ref='ubi:{converter}'` — no per-user identifiers in `judgments` rows. +4. **Budget-gate bypass on hybrid mode.** A naive implementation might call `openai.AsyncClient(...)` directly in the converter for LLM-fill, bypassing `peek_daily_total` / `record_cost`. Mitigation: the worker MUST inject `rate_query_batch` (which wraps the budget gate) as the converter's `llm_rate` callback. Code review + an integration test stubbing `rate_query_batch` and asserting it's invoked is the gate. +5. **Position-bias prior file path traversal.** Operator-supplied `UBI_POSITION_BIAS_PRIOR_FILE` env var — could point at arbitrary file. Mitigation: file is read via the existing `_FILE` mount-secret pattern (Absolute Rule #2 / [`backend/app/core/settings.py`](../../../../../backend/app/core/settings.py)); JSON parse failure logs WARN and falls back to uninformed prior (no error propagation). The container's mount surface is the operator's own infra. + +**Controls:** + +- **Secrets:** `UBI_POSITION_BIAS_PRIOR_FILE` follows the `_FILE`-mounted pattern. No new secret types. +- **Audit:** N/A in MVP2 — see §6 audit-event matrix for MVP3 activation. +- **Data retention/deletion:** UBI judgments persist on the existing `judgments` table — same CASCADE behavior on `judgment_lists` delete. +- **Logging:** Every UBI worker log line MUST include `judgment_list_id`, `event_type`, and (where relevant) `query_id` for traceability. NEVER log raw UBI event bodies, query text, or doc bodies (only counts + IDs). + +## 11) UX flows and edge cases + +### Information architecture + +- **Navigation placement:** the converter picker lives inside the existing "Generate judgments" dialog opened from the query-sets detail page (`ui/src/app/query-sets/[id]/page.tsx`). The dialog gains a top-bar "Method:" select; the rest of the dialog (name / target / template / rubric) stays in place. +- **Cluster rung badge:** small text-only badge on cluster cards (`ui/src/app/clusters/page.tsx`) + on the cluster detail page (`ui/src/app/clusters/[id]/page.tsx`). Reads from a per-cluster `useUbiReadiness(cluster_id)` hook with a 60s React Query stale time + the server-side 60s Redis cache. +- **Labeling taxonomy:** + - "Generate judgments" — dialog title (unchanged) + - "Method" — converter `` has `llm` pre-selected +- And at rung_1 it has `hybrid_ubi_llm` pre-selected +- And at rung_3 it has `ctr_threshold` pre-selected + +### AC-12: Worker resume-skip after crash + +- Given a hybrid generation interrupted after 500/847 queries persisted +- When the worker reboots and the resume sweep re-enqueues the row +- Then the worker skips the 500 already-persisted queries (resume-skip on `count_judgments_for_list_and_query > 0`) +- And processes the remaining 347 queries +- And total cost-recorded amount equals the second-run cost only (not the first run again) + +### AC-13: Tool registry drift assertion fails on missing registration + +- Given the `generate_judgments_from_ubi_impl` is added to `backend/app/agent/tools/judgments/generate_judgments_from_ubi.py` +- And the tool is added to `TOOLS` but NOT to `TOOL_REGISTRY` or `TOOL_ARG_MODELS` in `backend/app/agent/tools/__init__.py` +- When the module loads +- Then `RuntimeError("TOOLS / TOOL_REGISTRY / TOOL_ARG_MODELS drift: ...")` is raised at import time +- Example: every PR-CI suite that imports `agent.tools` catches the regression before any test runs + +### AC-14: `UBI_WINDOW_TOO_LARGE` caps long windows + +- Given a request with `since=2026-01-01T00:00:00Z, until=2026-06-01T00:00:00Z` (151 days) +- When the request reaches preflight U-D +- Then HTTP 422 `UBI_WINDOW_TOO_LARGE` is returned with `detail.message` citing the 90-day cap + +### AC-15: No cluster writes during readiness probe or generation + +- Given an integration test that mocks the `ElasticAdapter` HTTP client +- When `GET /api/v1/clusters/{id}/ubi-readiness` runs AND when the UBI worker runs end-to-end +- Then zero HTTP requests are issued with a write-shaped method (`PUT`, `DELETE`) or path (`_bulk`, `_update`, `_doc`, `_create`) +- And only `GET` + `POST _search` calls fire + +## 13) Non-functional requirements + +- **Performance:** + - `GET /api/v1/clusters/{cluster_id}/ubi-readiness` p99 ≤ 2 s (server-side 60s Redis cache + cluster `_count` aggregation). + - UBI worker throughput for `ctr_threshold` on a 1000-pair query set: ≤ 60 s (dominated by the two `search_batch` scrolls + client-side join — no per-query LLM cost). + - Hybrid-mode throughput: same as `generate_judgments_llm` for the LLM-fill subset (~1–2 s per query when LLM-fill fires; ~50 ms when UBI alone covers). +- **Reliability:** + - Per-query failures (one ambiguous mapping, one sparse query) are isolated — log WARN, skip, continue. + - Boot-time resume sweep at `backend/workers/all.py` re-enqueues any `generating` UBI list when the worker boots. + - Terminal `failed` only on global failures. +- **Operability:** + - Every worker log line includes `judgment_list_id`, `event_type` (one of: `ubi_read_complete`, `ubi_converter_complete`, `ubi_persist_complete`, `ubi_resume_skip`, `ubi_per_query_skipped`, `ubi_list_complete`, `ubi_list_failed`, `ubi_budget_exceeded`). + - `/healthz` is unaffected — UBI doesn't introduce a startup-time check (the readiness probe is per-request, not boot-time). + - Metrics surfaced on the judgment-list detail page: `judgment_count`, `source_breakdown`, `calibration.coverage_pct`, `calibration.llm_fill_calls` (hybrid only). +- **Accessibility / usability:** + - Engine-aware nudge follows the dismiss-button + `aria-labelledby` pattern from `demo-data-banner.tsx`. + - Sparse-data recommendation card uses `role='region'` + `aria-labelledby` for the heading, matching the existing card pattern. + - Rung badge color contrast ≥ 4.5:1 (WCAG AA) — text-only badge, no color-only meaning. + - Converter `` MUST use the `*_VALUES.map(...)` pattern + import from `@/lib/enums` per `chore_form_dropdown_primitive` (the form-select discipline lint guard at [`ui/src/__tests__/components/common/form-select-discipline.test.tsx`](../../../../../ui/src/__tests__/components/common/form-select-discipline.test.tsx) rejects inline `` for wire-value enums otherwise). - **Migration:** **none.** UBI rides the existing `judgments` table; the `source IN ('llm', 'human', 'click')` CHECK already accepts the new value. Alembic head unchanged at whatever MVP1 ships. - **Config:** one new optional env var `UBI_POSITION_BIAS_PRIOR_FILE` for operators who want to override the default Wang-Bendersky prior with a learned table. Default behaves like an uninformed prior. - **Audit events:** N/A (MVP2 still pre-`audit_log`; that surface activates at MVP3). @@ -77,9 +137,33 @@ Single-tier — small, additive, no schema migration. Five capability blocks bel ## Relationship to other work -- **Cleans up [`docs/00_overview/relyloop-spec.md`](../../../relyloop-spec.md) §14 + §19 + §27** — the spec previously framed click data as a per-engine adapter concern with engine-specific timelines. The §14 patch (landing with this idea) re-anchors the architecture around the engine-neutral UBI schema (which works across all three OSS engines via their respective UBI implementations), with engine-native readers (Elastic Behavioral Analytics, etc.) as thin extensions feeding the same `SignalsConverter` Protocol when an adopter needs them. +- **Cleans up [`docs/00_overview/relyloop-spec.md`](../../../relyloop-spec.md) §14 + §19 + §27** — the spec previously framed click data as a per-engine adapter concern with engine-specific timelines. The §14 patch (landing with this idea) re-anchors the architecture around the engine-neutral UBI schema (which works across all three OSS engines via their respective UBI implementations), with engine-native readers (Elastic Behavioral Analytics, etc.) as thin extensions feeding the same `SignalsConverter` Protocol when an adopter needs them. **Also clean up two preflight-discovered staleness bugs in the spec while the section is open:** (a) §706 header still says `(MVP1.5)` after the 2026-05-27 reshuffle compressed MVP1.5 into MVP2 — re-title to `(MVP2)`; (b) §724 relative paths to the two sibling planned-features (`feat_ubi_judgments/idea.md` and `infra_adapter_solr/idea.md`) are missing the `02_mvp2/` bucket directory + prefix the path with `../00_overview/planned_features/` (wrong from inside `docs/00_overview/`) — should be `planned_features/02_mvp2//idea.md`. +- **`feat_ubi_onramp` was merged back into this feature 2026-05-29** (see "How this feature stays a single coherent unit" below). The no-UBI / partial-UBI operator experience — readiness ladder, optional-by-design framing, engine-aware enablement nudge, sparse-data guidance, value-delta — is now Capabilities A–E above, not a separate folder. - **Composes with [`feat_auto_followup_studies`](../../../implemented_features/2026_05_24_feat_auto_followup_studies/idea.md)** — auto-chained follow-up studies become dramatically more useful with a continuously-refreshed UBI judgment list than with a snapshot LLM-as-judge list. The two features are complementary; UBI ships first. - **Composes with [`feat_pr_metric_confidence`](../../../implemented_features/2026_05_21_feat_pr_metric_confidence/)** (shipped 2026-05-21) — the confidence framing in the PR body becomes meaningfully stronger when "the metric was scored against 50,000 UBI-derived ratings covering 90% of last week's traffic" replaces "the metric was scored against 500 LLM ratings against a snapshot query set." - **Composes with [`feat_study_baseline_trial`](../../../implemented_features/2026_05_25_feat_study_baseline_trial/idea.md) + [`feat_config_repo_baseline_tracking`](../../../implemented_features/2026_05_23_feat_config_repo_baseline_tracking/idea.md)** — once UBI is the judgment source, "the baseline metric on the live config" becomes a meaningful absolute number rather than a synthetic LLM-rated approximation. Materially raises the credibility of every winning trial. - **Does NOT block MVP2 "Observable"** — Langfuse and SigNoz instrumentation can layer on top of `generate_judgments_from_ubi` exactly as it would on top of `generate_judgments_llm`. The `langfuse_trace_id` lineage column landing at MVP2 will be NULL for `source='click'` rows (which never invoke an LLM) and populated for `source='llm'` rows in the hybrid case — same column, source-dependent fill. - **Does NOT block later engine work** — the MVP2 `SignalsConverter` Protocol is engine-agnostic. New adapters added in later releases contribute their own engine-native reader (where they have one) feeding the same Protocol; the converter library and the API surface are unchanged regardless of which engines ship. + +## How this feature stays a single coherent unit (why the on-ramp is folded in, not split) + +The feature was briefly split into `feat_ubi_judgments` (machinery) + `feat_ubi_onramp` (no-UBI UX) on 2026-05-29 for dashboard visibility, then re-merged the same day per an explicit operator decision. The merge rationale: + +- **The machinery is a dead-end without the UX.** Shipping the substrate alone delivers a bare converter picker + bare `UBI_NOT_ENABLED` (412) / `UBI_INSUFFICIENT_DATA` (422) error codes. An operator who picks UBI without the plugin hits an error wall — the exact failure mode the on-ramp exists to prevent. Merging makes it **impossible to ship half**: the "UBI is optional / here's how it helps" framing ships in the same PR as the machinery that needs it. +- **One detection mechanism, one mental model.** All five operator-facing capabilities (A–E) share the single read-only `ubi_queries` `get_schema` probe and the single readiness-ladder mental model. They are one system, reviewed as one unit. +- **The dashboard-visibility concern that motivated the split is satisfied by this section + the capability blocks** — the no-signals-majority UX is explicit, sequenced, and dashboard-visible inside this idea, not buried as story stubs. +- **Phase boundary (for `/spec-gen`):** if the combined feature proves too large for one PR at plan time, the correct split is **by delivery phase within this one feature** (Phase 1 = substrate + converter picker + LLM-stays-default; Phase 2 = readiness ladder + nudge + value-delta), tracked via `phase2_idea.md` in THIS folder — NOT by re-spawning a separate `feat_ubi_onramp` folder. The phases ship in the same MVP2 release regardless. + +## Open questions for /spec-gen + +1. **`_SourceBreakdown` response invariant (design fork — preflight 2026-05-29).** The existing `JudgmentListDetail` response model at [`backend/app/api/v1/schemas.py:864-872`](../../../../../backend/app/api/v1/schemas.py#L864-L872) ships with a `_SourceBreakdown` sub-shape that locks the invariant `llm + human == judgment_count`, with `source='click'` rows **folded into `human`** per the `feat_llm_judgments` GPT-5.5 cycle 2 F6 decision. With UBI shipping hybrid lists that contain explicit `source='click'` rows alongside `source='llm'` rows, that invariant becomes user-visible inaccuracy ("3000 human ratings" when really "200 human + 2800 click"). Two paths: + - **Path A (locked, recommended):** evolve `_SourceBreakdown` to `llm + human + click == judgment_count` with a third `click: int` field. Breaks the cycle-2 F6 invariant by design; OpenAPI consumers who pinned to the two-field shape get a 3-field surface. Aligns the breakdown with the underlying CHECK constraint values. Slot for `/spec-gen` to decide whether to ship as a versioned breakdown shape (`SourceBreakdownV2`) or in-place. + - **Path B:** keep folding `click → human`, document the "non-LLM rows = human OR click" convention in the field docstring, update the create-judgment-list-detail UI copy to say "non-LLM" instead of "human" when the list has UBI lineage. Conserves the OpenAPI shape; trades accuracy for backwards compatibility. + - **Recommendation: Path A.** The cycle-2 F6 decision was made in a click-rows-don't-yet-exist world; shipping UBI is the moment to update the invariant. Lock at `/spec-gen` time once an OpenAPI-consumer impact audit confirms no external consumers exist (the only consumers today are the project's own UI + contract tests). +2. **Query-text → query_set mapping disambiguation strategy.** The `UBI_QUERY_MAPPING_AMBIGUOUS` (422) error path triggers when a UBI `user_query` matches more than one `query_set.queries.query_text`. The request body needs an optional tiebreaker field (`mapping_strategy: 'reject' | 'first_match' | 'most_recent'` or similar) — `/spec-gen` decides the wire enum + default. **Recommendation:** default to `'reject'` (most operator-control-preserving, matches the existing 422-on-ambiguity pattern); `first_match` and `most_recent` as opt-in escape hatches. +3. **Operator-facing `min_impressions_threshold` default.** The `UBI_INSUFFICIENT_DATA` error references this threshold but the idea doesn't pin a default. Two anchors: (a) the `HybridUbiLlmConverter`'s `llm_fill_threshold` default is 20 impressions — using the same default for the global "insufficient data" gate keeps the two thresholds symmetric; (b) different surfaces — `min_impressions_threshold` gates "should this judgment-list even be generated?", `llm_fill_threshold` gates "which (query, doc) pairs in this list get the LLM fill vs the UBI converter?". **Recommendation:** keep them separate but set the `min_impressions_threshold` default at 100 events total (not per-pair), which is well below the `llm_fill_threshold * typical_query_set_size` floor and lets even small UBI deployments generate a useful list. +4. **`SourceBreakdown` shape — affected contract tests.** Whichever Path A/B is taken, the contract test at `backend/tests/contract/test_judgment_lists_*.py` (verify exact path at impl time) needs to be updated to assert the new shape. List the affected test files in the `/impl-plan-gen` deliverable. +5. **Converter-picker field name + label taxonomy (folded-in Capability E).** Lock the request field name (`converter` vs `judgment_mode`) and the exact UI labels for the four options. **Recommendation:** field `converter`; labels "LLM-as-judge" / "UBI (click-through)" / "UBI (dwell-time)" / "Hybrid UBI + LLM". Critically — do NOT name the field `source` (it is not the per-row `judgments.source` value, which stays `llm | human | click`). The naming-contract resolution in Capability E is the locked decision; this question is only the final label wording. +6. **Nudge persistence + cadence (folded-in from `feat_ubi_onramp` OQ1).** Store the "dismissed" state client-side (localStorage, like `demo-data-banner` via `use-local-storage-set.ts`) or per-cluster server-side? Re-surface on a schedule or only while the readiness rung is unchanged? **Recommendation:** client-side localStorage keyed by `cluster_id` + re-surface only when the rung is still 0 (matches the shipped contextual-help dismissal idiom; no new server state). +7. **Readiness-rung thresholds (folded-in from `feat_ubi_onramp` OQ2).** What impression counts define the rung 1→2→3 boundaries — operator-configurable or fixed MVP2 defaults? **Recommendation:** fixed defaults for MVP2 (rung 1 = `ubi_queries` present but <`min_impressions_threshold` on >50% of pairs; rung 2 = head covered; rung 3 = ≥`min_impressions_threshold` across the query set), revisit configurability if adopter feedback wants it. Ties to OQ3's `min_impressions_threshold` default. +8. **Value-delta baseline when no prior LLM list exists (folded-in from `feat_ubi_onramp` OQ3).** Capability D shows an LLM→UBI delta when a prior LLM list exists on the same query set — but what does it show on a first-ever UBI list with no prior? **Recommendation:** coverage-only ("covered N queries / X% of traffic in the window") with no delta; do NOT spend an LLM spot-rating call just to manufacture a comparison baseline (cost + latency for marginal value). `/spec-gen` confirms. diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md new file mode 100644 index 00000000..e41be5d0 --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md @@ -0,0 +1,1657 @@ +# Implementation Plan — UBI Judgments (engine-neutral User Behavior Insights) + +**Date:** 2026-05-29 +**Status:** Approved (cross-model converged at 3-cycle cap; all 3 GPT-5.5 findings accepted — see footer) +**Primary spec:** [feature_spec.md](feature_spec.md) (Approved 2026-05-29; cross-model converged at cycle-3 cap with all 10 findings accepted — see spec §19 D-10) +**Policy source(s):** +- [docs/01_architecture/api-conventions.md](../../../../01_architecture/api-conventions.md) — `/api/v1/` prefix, error envelope `{"detail":{"error_code","message","retryable"}}`, cursor pagination, `X-Total-Count` +- [docs/01_architecture/adapters.md](../../../../01_architecture/adapters.md) — `SearchAdapter` Protocol; UBI uses `search_batch` + `get_schema` only +- [docs/01_architecture/llm-orchestration.md](../../../../01_architecture/llm-orchestration.md) — capability cache, daily-budget gate, `rate_query_batch` is the only LLM entry +- [docs/01_architecture/data-model.md](../../../../01_architecture/data-model.md) — `judgments` + `judgment_lists` shapes; CHECK constraints +- [CLAUDE.md](../../../../../CLAUDE.md) — Absolute Rules #2 (mounted secrets), #3/#8/#10 (LLM via shared client + Settings.openai_model), #4 (engine code in adapters), #5 (migration downgrade) + +--- + +## 0) Planning principles + +- Spec traceability first: every story/task maps to FR IDs from the spec. +- The migration is the single hard gate (Story 1.1) — it lands first because every later story reads/writes `judgment_lists.generation_params`. +- The `SignalsConverter` Protocol is **async** (cycle-3 fix D-10e); all three concrete converters await uniformly even when the pure ones don't actually await anything. One Protocol shape for the worker to consume. +- LLM calls in hybrid mode go through `rate_query_batch` + `peek_daily_total` + `record_cost` from [`backend/app/llm/budget_gate.py`](../../../../../backend/app/llm/budget_gate.py). No new LLM client. (Absolute Rules #3/#8/#10.) +- Engine I/O is the existing `SearchAdapter.search_batch` + `get_schema` surface — no new adapter method (Absolute Rule #4). +- `UBI_INSUFFICIENT_DATA` is **sync 422 from preflight U-D2**; the worker terminal `failed` path is the race-condition fallback only (cycle-3 fix D-10d). +- Per-query ambiguous mapping under `mapping_strategy='reject'` is a **skip + counter** (calibration JSONB), not a 422 (cycle-3 fix D-10f). The endpoint catalog removed `UBI_QUERY_MAPPING_AMBIGUOUS`. +- Wire enums live in `backend/app/api/v1/schemas.py` and mirror in `ui/src/lib/enums.ts` with the `// Values must match backend/...` comment. Form `` consumes `*_VALUES.map(...)`. + +### AI Agent Execution Protocol + +0. Load context: read `CLAUDE.md`, `architecture.md`, `state.md`, this plan, and the spec top-to-bottom before the first story. +1. Read scope of the current story (Outcome, New/Modified files, Endpoints, Key interfaces, DoD). +2. Backend-first per story: migration → model → repo → domain → service → worker (when applicable) → router → schemas → agent-tool. +3. Run unit + integration + contract tests after each story; if migration touched, also `alembic upgrade head && alembic downgrade -1 && alembic upgrade head`. +4. Frontend (Epic 4): enums first → hook → primitive (``) → integration into existing dialog/detail pages. +5. E2E (Epic 5): real-backend Playwright against the existing OpenSearch service container + the new `tests/e2e/helpers/seed_ubi.ts` helper. +6. Update docs/checklists in the same PR when behavior/contract changed. +7. Migration round-trip verified before merging Story 1.1. +8. Attach evidence (commands run, pass/fail) in the PR description. +9. After the final story, update `state.md` + `architecture.md` per §4.0. + +--- + +## Epic 1 — Foundations (migration + pure-domain UBI library) + +### Story 1.1 — Migration `0021_judgment_lists_generation_params` (FR-4 + FR-5 backing) + +**Outcome:** `judgment_lists.generation_params` JSONB column exists in Postgres at Alembic head `0021_judgment_lists_generation_params`; round-trips cleanly. + +**New files** + +| File | Purpose | +|---|---| +| `migrations/versions/0021_judgment_lists_generation_params.py` | Add nullable JSONB column `generation_params` to `judgment_lists`. `downgrade()` drops the column. | + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/db/models/judgment_list.py` | Add `generation_params: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)` with docstring noting UBI-only population (LLM lists stay NULL). | + +**Tasks** + +1. Create `migrations/versions/0021_judgment_lists_generation_params.py` with `revision="0021"` and `down_revision="0020"` matching the `0020_studies_baseline_trial.py` style. +2. `upgrade()`: `op.add_column("judgment_lists", sa.Column("generation_params", postgresql.JSONB, nullable=True))`. No CHECK constraint (free-form JSONB). +3. `downgrade()`: `op.drop_column("judgment_lists", "generation_params")`. +4. Add `generation_params` to `JudgmentList` ORM model in `backend/app/db/models/judgment_list.py` after the `calibration` field (line 58). Update the module docstring's "MVP1 shape" reference to note the column is `feat_ubi_judgments` MVP2 additive. +5. Run `alembic upgrade head && alembic downgrade -1 && alembic upgrade head` and capture output for the PR description. + +**Definition of Done (DoD)** + +- [ ] `migrations/versions/0021_judgment_lists_generation_params.py` exists; `alembic upgrade head` succeeds locally and in CI. +- [ ] Round-trip works: `alembic downgrade -1` (back to `0020`) and `alembic upgrade head` both succeed. +- [ ] Pre-existing `judgment_lists` rows (LLM lists) survive both directions cleanly (column is nullable; never read on LLM path). +- [ ] Integration test at `backend/tests/integration/db/test_migration_0021_generation_params.py` introspects `information_schema.columns` and asserts column type `jsonb` + nullable. +- [ ] `state.md` bump to `0021_judgment_lists_generation_params` happens in the finalization step (NOT this story). + +--- + +### Story 1.2 — `domain/ubi/` package (FR-2 + FR-11) + +**Outcome:** Pure-domain UBI library with feature vectors, async `SignalsConverter` Protocol + 3 concrete converters, and the position-bias prior loader. No I/O, no DB, no LLM client construction. + +**New files** + +| File | Purpose | +|---|---| +| `backend/app/domain/ubi/__init__.py` | Exports `FeatureVec`, `SignalsConverter`, `CtrThresholdConverter`, `DwellTimeThresholdConverter`, `HybridUbiLlmConverter`, `ConverterConfig`, `load_position_bias_prior`. | +| `backend/app/domain/ubi/features.py` | `FeatureVec` Pydantic model + `aggregate_features(events_by_pair)` pure aggregation (sums clicks, impressions, computes corrected CTR, dwell mean). | +| `backend/app/domain/ubi/converter.py` | `SignalsConverter` async Protocol + 3 concrete impls + `ConverterConfig` Pydantic model + per-converter config sub-models. | +| `backend/app/domain/ubi/position_bias_prior.py` | `load_position_bias_prior(path: Path \| None) -> dict[int, float]`; returns `{}` (uninformed) on missing/empty; WARN-logs on malformed JSON. | + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/core/settings.py` | Add optional `ubi_position_bias_prior_file: Path \| None` field + `@cached_property ubi_position_bias_prior` accessor (delegates to `load_position_bias_prior`). | + +**Key interfaces** + +```python +# domain/ubi/features.py +class FeatureVec(BaseModel): + click_count: int + impression_count: int + corrected_ctr: float # Wang-Bendersky-corrected + dwell_mean_seconds: float | None + conversion_rate: float | None + refinement_rate: float | None + +def aggregate_features( + events_by_pair: dict[tuple[str, str], list["UbiEvent"]], + position_bias_prior: dict[int, float], +) -> dict[tuple[str, str], FeatureVec]: ... + +# domain/ubi/converter.py +class ConverterConfig(BaseModel): + """Discriminated by converter kind in the serializer; opaque to the Protocol.""" + extra: dict[str, Any] = Field(default_factory=dict) + +class SignalsConverter(Protocol): + async def convert( + self, + features: dict[tuple[str, str], FeatureVec], + config: ConverterConfig, + ) -> dict[tuple[str, str], int]: ... + +class CtrThresholdConverter: + """Pure UBI — no LLM I/O. Default thresholds: {1: 0.05, 2: 0.15, 3: 0.30}.""" + async def convert(self, features, config) -> dict[tuple[str, str], int]: ... + +class DwellTimeThresholdConverter: + """Pure UBI — no LLM I/O. Default thresholds (seconds): {1: 10.0, 2: 30.0, 3: 90.0}.""" + async def convert(self, features, config) -> dict[tuple[str, str], int]: ... + +LlmRateCallback = Callable[ + [list[tuple[str, str, str]]], # [(query_id, doc_id, query_text), ...] + Awaitable[dict[tuple[str, str], int]], +] + +class HybridUbiLlmConverter: + """UBI-head + LLM-tail. inner = CtrThresholdConverter or DwellTimeThresholdConverter. + + Pairs with impression_count >= llm_fill_threshold (default 20) → inner.convert(...). + Pairs below threshold → llm_rate(...) callback (worker-supplied; routes through + rate_query_batch + budget gate). + """ + def __init__(self, inner: SignalsConverter, llm_rate: LlmRateCallback): ... + async def convert(self, features, config) -> dict[tuple[str, str], int]: ... + +# domain/ubi/position_bias_prior.py +def load_position_bias_prior(path: Path | None) -> dict[int, float]: + """Return {rank: weight} or {} for uninformed default. WARN-log on malformed JSON.""" +``` + +**Tasks** + +1. Implement `FeatureVec` and `aggregate_features` per FR-1 feature shape; position-bias correction: `corrected_ctr = clicks / sum(impressions[r] * prior.get(r, 1.0))`. +2. Implement `CtrThresholdConverter` + `DwellTimeThresholdConverter` as async classes (their body doesn't `await` anything — trivially async to satisfy the Protocol). +3. Implement `HybridUbiLlmConverter` with the splitter: partition `features` into head (`impression_count >= llm_fill_threshold`) → call `inner.convert`; tail → build the `(query_id, doc_id, query_text)` payload and await `llm_rate(...)`. Merge dicts; `head` wins on collision (impossible by construction, but explicit). +4. Implement `load_position_bias_prior(path)`: read JSON `{positions: {1: 1.0, 2: 0.65, ...}}`; on `FileNotFoundError` / `OSError` / `json.JSONDecodeError` / unexpected shape → `logger.warning(event_type='ubi_position_bias_prior_malformed', error=...)` and return `{}` (uninformed). +5. Wire `Settings.ubi_position_bias_prior_file` + `@cached_property ubi_position_bias_prior` (calls `load_position_bias_prior` once per process via `@cached_property`). +6. Update `backend/app/domain/__init__.py` (if present) and `backend/app/core/settings.py` `Settings` fields. +7. Write unit tests in `backend/tests/unit/domain/ubi/test_features.py`, `test_converter.py`, `test_position_bias_prior.py` per §3.1. + +**Definition of Done (DoD)** + +- [ ] `FeatureVec.click_count == sum(events.click_count_for_pair)`; corrected CTR matches a hand-computed example with informed prior `{1: 1.0, 2: 0.5, 3: 0.25}` (verified in `test_features.py`). +- [ ] `CtrThresholdConverter` maps `corrected_ctr ∈ {0.04, 0.10, 0.20, 0.40}` → `{0, 1, 2, 3}` respectively with default thresholds. +- [ ] `DwellTimeThresholdConverter` maps `dwell_mean ∈ {5, 15, 60, 120}` → `{0, 1, 2, 3}` respectively. +- [ ] `HybridUbiLlmConverter` invokes the inner converter for above-threshold pairs and the `llm_rate` callback for below-threshold pairs; merged dict size matches input size (no dropped pairs). +- [ ] `HybridUbiLlmConverter` does NOT import `openai` or construct an `AsyncOpenAI` instance (lint guard: `ast`-based test asserts neither import nor `AsyncOpenAI(` token in `converter.py`). +- [ ] `load_position_bias_prior(None)` returns `{}`; malformed JSON returns `{}` + WARN log captured by `caplog`. +- [ ] `Settings().ubi_position_bias_prior` returns `{}` when the env var is unset. + +--- + +## Epic 2 — Reader + dispatcher + breakdown evolution + +### Story 2.1 — `UbiReader` service (FR-1) + +**Outcome:** `UbiReader.read_features(...)` reads `ubi_queries` + `ubi_events` via `SearchAdapter.search_batch`, performs the `query_id` join client-side, returns `dict[tuple[str, str], FeatureVec]`. Raises `UbiNotEnabledError` when the schema probe fails. + +**New files** + +| File | Purpose | +|---|---| +| `backend/app/services/ubi_reader.py` | `UbiReader` class + `UbiNotEnabledError` + `_probe_enabled(adapter)` helper. Issues two scrolling `search_batch` calls; joins on `query_id` in Python. Disambiguates per-application emissions by `target`. | +| `backend/app/services/ubi_errors.py` | `UbiNotEnabledError`, `UbiInsufficientDataError` (raised by `read_features` on empty post-filter; race-condition fallback only). | + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/services/__init__.py` | Re-export `UbiReader`, `UbiNotEnabledError`, `UbiInsufficientDataError` if `__all__` is used. | + +**Key interfaces** + +```python +# services/ubi_reader.py +class UbiReader: + def __init__(self, adapter: SearchAdapter, position_bias_prior: dict[int, float]): + ... + + async def read_features( + self, + *, + target: str, + since: datetime, + until: datetime | None = None, + query_filter: str | None = None, + max_queries: int = 5000, + ) -> dict[tuple[str, str], FeatureVec]: + """Two-index scan + client-side join. Raises UbiNotEnabledError on rung_0.""" + + async def _probe_enabled(self) -> None: + """get_schema('ubi_queries') — raises UbiNotEnabledError on TargetNotFoundError.""" + +# services/ubi_errors.py +class UbiNotEnabledError(RuntimeError): ... +class UbiInsufficientDataError(RuntimeError): ... +``` + +**Tasks** + +1. Implement `UbiReader._probe_enabled` calling `adapter.get_schema('ubi_queries')`; catch `TargetNotFoundError` → raise `UbiNotEnabledError(f"ubi_queries index not found on engine {adapter.engine_type}")`. +2. Implement `read_features`: + - Build the `ubi_queries` scroll query: filter `timestamp >= since AND timestamp < (until or now) AND application = target`, paginate via `search_batch` with `top_k=max_queries`. Optional `query_filter` adds an `AND user_query ~ ` clause when present. + - Build the `ubi_events` scroll query keyed by the `query_id` set from step 1's result. Same `search_batch` mechanism. + - Join in Python: bucket events by `(query_id, doc_id)`; pass to `aggregate_features(...)` from Story 1.2. +3. Treat empty post-filter results as the race-condition fallback path: log `event_type='ubi_reader_empty_features'` and return `{}` (worker terminal `failed` path picks this up — preflight U-D2 covers the sync case). +4. Write integration test `backend/tests/integration/services/test_ubi_reader.py` that mocks `adapter.search_batch` to return canned `ubi_queries` + `ubi_events` payloads; asserts the joined `FeatureVec` shape. +5. Write integration test `backend/tests/integration/services/test_ubi_reader_no_writes.py` per §10 threat #2 — mock the underlying `httpx.AsyncClient.send`; run `read_features` end-to-end; assert zero requests with HTTP method `PUT`/`DELETE` or path containing `_bulk`/`_update`/`_doc`/`_create`. + +**Definition of Done (DoD)** + +- [ ] `UbiReader.read_features` returns the expected `FeatureVec` map against stubbed canned data. +- [ ] `UbiNotEnabledError` raised when `get_schema('ubi_queries')` raises `TargetNotFoundError`. +- [ ] `test_ubi_reader_no_writes.py` asserts zero write-shaped HTTP calls. +- [ ] Empty-features path returns `{}` without raising (race fallback per FR-1). +- [ ] No new method added to `SearchAdapter` Protocol — verified by re-running `backend/tests/unit/adapters/test_protocol.py` shape assertions. + +--- + +### Story 2.2 — Readiness service + `start_ubi_judgment_generation` dispatcher (FR-4 + FR-7) + +**Outcome:** A new `ubi_readiness` service classifies a `(cluster, query_set, target)` tuple on the rung 0–3 ladder. A new `start_ubi_judgment_generation` dispatcher runs the full UBI preflight + INSERT + Arq enqueue. The existing `start_judgment_generation` is refactored to share helpers with the new function (no copy-pasted body). + +**New files** + +| File | Purpose | +|---|---| +| `backend/app/services/ubi_readiness.py` | `classify_rung(adapter, query_set, target) -> UbiReadiness` + `UbiReadiness` dataclass. Uses `_probe_enabled` from `UbiReader` + a single `_count` aggregation on `ubi_events` filtered by `(application=target, timestamp >= now-30d, query_id IN )`. Redis-cached 60s per scope. | + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/services/agent_judgments_dispatch.py` | Add `start_ubi_judgment_generation(...)` alongside the existing `start_judgment_generation`. Factor shared helpers: `_resolve_cluster_query_set`, `_check_consistency`, `_check_llm_preflight` (A+B+B.1+C from existing dispatcher), `_check_oversized_query_set`, `_insert_generating_list_and_enqueue(kind: Literal["llm","ubi"], **fields)`. Existing `start_judgment_generation` body is rewritten to call the shared helpers. | + +**Key interfaces** + +```python +# services/ubi_readiness.py +@dataclass(frozen=True, slots=True) +class UbiReadiness: + rung: Literal["rung_0", "rung_1", "rung_2", "rung_3"] + covered_pairs_pct: float | None + head_covered: bool | None + checked_at: datetime + +async def classify_rung( + *, + adapter: SearchAdapter, + cluster_id: str, + query_set_id: str, + query_set_query_ids: list[str], + target: str, + redis: Redis, + min_impressions_threshold: int = 100, +) -> UbiReadiness: + """Probe + classify; 60s Redis cache per (cluster_id, query_set_id, target).""" + +# services/agent_judgments_dispatch.py +@dataclass(frozen=True, slots=True) +class UbiJudgmentGenerationRequest: + name: str + description: str | None + query_set_id: str + cluster_id: str + target: str + since: datetime + until: datetime | None + converter: Literal["ctr_threshold", "dwell_time", "hybrid_ubi_llm"] + converter_config: dict[str, Any] | None + llm_fill_threshold: int | None + min_impressions_threshold: int | None + mapping_strategy: Literal["reject", "first_match", "most_recent"] + current_template_id: str | None # required for hybrid + rubric: str | None # required for hybrid + +async def start_ubi_judgment_generation( + *, + db: AsyncSession, + redis: Redis, + arq_pool: ArqRedis | None, + settings: Settings, + req: UbiJudgmentGenerationRequest, +) -> JudgmentGenerationResult: + """Full preflight U-A..U-H + INSERT + best-effort Arq enqueue. + + Sync 422 on UBI_INSUFFICIENT_DATA per FR-4 U-D2 (cycle-3 D-10d). + """ +``` + +**Tasks** + +1. Implement `classify_rung`: + - Cache key `ubi-readiness:{cluster_id}:{query_set_id}:{target}` (60s TTL). + - On cache miss: call `UbiReader._probe_enabled(adapter)` (catch `UbiNotEnabledError` → rung_0); otherwise run a single `_count` aggregation. Apply the FR-7 rung rules. Cache + return. +2. Factor shared helpers in `agent_judgments_dispatch.py`: + - Extract `_resolve_cluster_query_set(db, cluster_id, query_set_id, template_id=None) -> (Cluster, QuerySet, QueryTemplate|None)` raising `_err(404, ...)` per failure. + - Extract `_check_consistency(query_set, cluster, template=None)` raising `_err(422, "VALIDATION_ERROR", ...)`. + - Extract `_check_llm_preflight(settings, redis)` raising the existing OPENAI_NOT_CONFIGURED / LLM_PROVIDER_INCAPABLE / UNKNOWN_MODEL_PRICING / OPENAI_BUDGET_EXCEEDED codes. + - Extract `_check_oversized_query_set(db, query_set_id)` raising 422. + - Extract `_insert_generating_list_and_enqueue(db, arq_pool, kind, fields, enqueue_job_name)` with the existing IntegrityError → 409 JUDGMENT_LIST_NAME_TAKEN handling; commit; best-effort enqueue. +3. Rewrite `start_judgment_generation` body to call the helpers (behavioral parity — existing contract tests at `backend/tests/contract/test_judgments_generate*.py` must still pass with no assertion changes). +4. Implement `start_ubi_judgment_generation`: + - U-A: `_resolve_cluster_query_set(db, ..., template_id=req.current_template_id if hybrid)`. + - U-B: `_check_consistency(...)`. + - U-C: `await UbiReader._probe_enabled` (build adapter once via `build_adapter(cluster)`); raise 412 UBI_NOT_ENABLED. + - U-D: window validity + 90-day cap → 422 UBI_WINDOW_TOO_LARGE. + - U-D2 (NEW): issue one `_count` aggregation; if `count < min_impressions_threshold` → 422 UBI_INSUFFICIENT_DATA with message per spec §8.5. + - U-E (hybrid only): `_check_llm_preflight(settings, redis)`. + - U-F: `_check_oversized_query_set`. + - U-G: build `generation_params` via a dedicated helper that **injects `generation_kind: 'ubi'` server-side** (Spec FR-4 U-G mandates this discriminator; without it Story 3.3's resume reconstruction loses its kind hint and Story 4.3's `` can't discriminate UBI/hybrid from LLM lists). The helper signature: `_build_ubi_generation_params(req: UbiJudgmentGenerationRequest) -> dict[str, Any]`; body: `return {"generation_kind": "ubi", **req.model_dump(mode="json")}`. Pass the resulting dict to `_insert_generating_list_and_enqueue(kind='ubi', fields={..., generation_params=_build_ubi_generation_params(req)}, enqueue_job_name='generate_judgments_from_ubi')`. + - Return `JudgmentGenerationResult(judgment_list_id, status='generating')`. +5. Write integration test `backend/tests/integration/services/test_agent_judgments_dispatch_ubi.py` covering all preflight branches; verify the shared `start_judgment_generation` still returns the same shapes/codes for the LLM path (parity). **Additional assertion**: the persisted `judgment_lists.generation_params` JSONB MUST contain `generation_kind == 'ubi'` AND the round-trip via `UbiJudgmentGenerationRequest(**{k: v for k, v in persisted.items() if k != 'generation_kind'})` succeeds (i.e., the discriminator is additive, not field-replacing). + +**Definition of Done (DoD)** + +- [ ] `classify_rung` returns the right rung for canned `_count` results (rung_0 / rung_1 / rung_2 / rung_3); Redis caching round-trip verified (second call within 60s does not re-probe). +- [ ] `start_ubi_judgment_generation` raises the right `HTTPException` for each preflight step with the spec §8.5 envelope shape. +- [ ] `start_judgment_generation` (LLM path) behavior unchanged — existing contract tests pass with no modification. +- [ ] Hybrid-mode rejection when `current_template_id` or `rubric` missing surfaces as 422 (Pydantic validator at the request schema layer — Story 3.2 owns the validator; this dispatcher merely consumes a validated `req`). +- [ ] Persisted `judgment_lists.generation_params` JSONB contains `generation_kind == 'ubi'` (spec FR-4 U-G discriminator) + the request fields; round-trip via `UbiJudgmentGenerationRequest(**{k: v for k, v in persisted.items() if k != 'generation_kind'})` succeeds. + +--- + +### Story 2.3 — `_SourceBreakdown` evolution + filter widening + Literals (FR-9 + FR-10) + +**Outcome:** `_SourceBreakdown` returns `{llm, human, click}` with invariant `llm + human + click == judgment_count`. `?source=` filter accepts `click`. New backend Literals (`UbiConverterKind`, `JudgmentGenerationMethodWire`, `UbiReadinessRungWire`, `UbiMappingStrategyWire`) live in `schemas.py`. + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/api/v1/schemas.py` | Evolve `_SourceBreakdown` to add `click: int`; widen `JudgmentSourceFilterWire` from `Literal["llm", "human"]` to `Literal["llm", "human", "click"]`; add `UbiConverterKind`, `JudgmentGenerationMethodWire`, `UbiReadinessRungWire`, `UbiMappingStrategyWire` (all per FR-9). Update the `_SourceBreakdown` docstring to remove the cycle-2 F6 fold-into-human contract; cite cycle-3 D-10 + FR-10 as the new contract. | +| `backend/app/db/repo/judgment.py` | Evolve `source_breakdown_for_list` from `{'llm': 0, 'human': 0}` start dict to `{'llm': 0, 'human': 0, 'click': 0}`; route `click` to the `click` key (was folding into `human`). Update the function docstring to cite FR-10 and remove the cycle-2 F6 "click folds into human" contract. | +| `backend/app/api/v1/judgments.py` | `_detail()` populates `_SourceBreakdown(llm=..., human=..., click=breakdown.get("click", 0))`. | + +**Key interfaces** + +```python +# api/v1/schemas.py +class _SourceBreakdown(BaseModel): + """{llm + human + click == judgment_count} per FR-10 (evolved 2026-05-29 + from the cycle-2 F6 two-term shape now that UBI ships click rows).""" + llm: int + human: int + click: int + +JudgmentSourceFilterWire = Literal["llm", "human", "click"] # widened from {llm, human} + +UbiConverterKind = Literal["ctr_threshold", "dwell_time", "hybrid_ubi_llm"] +JudgmentGenerationMethodWire = Literal["llm", "ctr_threshold", "dwell_time", "hybrid_ubi_llm"] +UbiReadinessRungWire = Literal["rung_0", "rung_1", "rung_2", "rung_3"] +UbiMappingStrategyWire = Literal["reject", "first_match", "most_recent"] + +# db/repo/judgment.py +async def source_breakdown_for_list(db: AsyncSession, judgment_list_id: str) -> dict[str, int]: + """{'llm': N, 'human': M, 'click': K} — invariant llm + human + click == judgment_count. + Evolved from the cycle-2 F6 two-term contract now that UBI ships click rows (FR-10).""" +``` + +**Tasks** + +1. Add new Literals to `schemas.py` per FR-9. Group them under a `# UBI wire-value contracts (feat_ubi_judgments FR-9)` section comment. +2. Evolve `_SourceBreakdown` class definition + docstring. +3. Widen `JudgmentSourceFilterWire` and remove the "click reserved, rejected at API filter" comment. +4. Update `backend/app/db/repo/judgment.py` `source_breakdown_for_list` to count `click` separately; update the function docstring + the module docstring's "Source breakdown folds click into human" decision note (mark as superseded by FR-10). +5. Update `backend/app/api/v1/judgments.py` `_detail()` populator to pass `click`. +6. Update existing contract test at `backend/tests/contract/test_judgments_list_detail.py` (or equivalent — verify path at impl time) to assert all 3 keys on the source_breakdown shape. New assertion: on LLM-only lists `click == 0`. +7. Update existing contract test at `backend/tests/contract/test_judgments_filter.py` (or equivalent) — remove the assertion that `?source=click` returns 422 VALIDATION_ERROR; add assertion that `?source=click` returns 200 with rows where `source == 'click'`. +8. Write unit test `backend/tests/unit/api/test_source_breakdown_evolution.py` directly constructing `_SourceBreakdown(llm=10, human=5, click=20)` and asserting field access + JSON serialization shape. + +**Definition of Done (DoD)** + +- [ ] `_SourceBreakdown` has all 3 fields; existing LLM-list contract tests pass with `click=0`. +- [ ] `?source=click` returns 200 with matching rows; `?source=human` and `?source=llm` unchanged. +- [ ] Repo function returns all 3 keys; `click` is no longer aggregated into `human`. +- [ ] Mypy `--strict` clean across `schemas.py`, `judgment.py`, `judgments.py`. + +--- + +## Epic 3 — API + worker + agent tool + +### Story 3.1 — `GET /api/v1/clusters/{cluster_id}/ubi-readiness` endpoint (FR-7) + +**Outcome:** The readiness endpoint returns `{rung, covered_pairs_pct, head_covered, checked_at}` for a `(cluster, query_set, target)` tuple. Cached 60s server-side per scope. + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/api/v1/clusters.py` | Add `GET /clusters/{cluster_id}/ubi-readiness` handler. Required `?query_set_id=&target=` query params (else 422 VALIDATION_ERROR per cycle-3 D-10c). Calls `classify_rung` from Story 2.2. | +| `backend/app/api/v1/schemas.py` | Add `UbiReadinessResponse(BaseModel)` with the 4 fields. | + +**Endpoints** + +| Method | Path | Request body | Success response | Error codes | +|---|---|---|---|---| +| `GET` | `/api/v1/clusters/{cluster_id}/ubi-readiness?query_set_id=&target=` | — (query params required) | `200` `{ rung: UbiReadinessRungWire, covered_pairs_pct: float \| null, head_covered: bool \| null, checked_at: datetime }` | `404 CLUSTER_NOT_FOUND`, `404 QUERY_SET_NOT_FOUND`, `422 VALIDATION_ERROR` (missing query params), `503 CLUSTER_UNREACHABLE` | + +**Pydantic schemas** + +```python +class UbiReadinessResponse(BaseModel): + rung: UbiReadinessRungWire + covered_pairs_pct: float | None + head_covered: bool | None + checked_at: datetime +``` + +**Tasks** + +1. Add the handler `get_cluster_ubi_readiness` in `backend/app/api/v1/clusters.py` after `get_cluster_schema` (line ~317). FastAPI signature mirrors `get_cluster_schema` for cluster resolution + builds the adapter via `build_adapter(cluster)`. +2. Resolve `query_set_id` → `repo.get_query_set` (404 QUERY_SET_NOT_FOUND); pull query IDs via `repo.list_queries_for_set` (Story 2.2's `classify_rung` needs the id list for the `query_id IN` filter). +3. Open Redis client per request (use the same `_open_redis` helper pattern from `judgments.py:107`). +4. Call `classify_rung(...)` and return `UbiReadinessResponse(**asdict(reading))`. +5. Catch `ClusterUnreachableError` from the adapter → 503 CLUSTER_UNREACHABLE; catch generic exception → 500 with `event_type='ubi_readiness_unexpected_error'` log. +6. Write contract test `backend/tests/contract/test_clusters_ubi_readiness_shape.py` asserting all 4 error envelopes + 200 shape. +7. Write integration test `backend/tests/integration/api/test_clusters_ubi_readiness.py` with the 4-rung paths (mock the adapter); Redis cache hit verified on second call. + +**Definition of Done (DoD)** + +- [ ] All 4 error envelopes (404 ×2, 422, 503) return the structured `{"detail": {"error_code", "message", "retryable"}}` shape. +- [ ] 200 response shape matches `UbiReadinessResponse` exactly. +- [ ] Cache hit within 60s returns identical response without re-running the cluster aggregation (verified via spy on `adapter.search_batch`). +- [ ] Endpoint registered in `backend/app/main.py` via the existing `clusters_router.router` mount (no new mount required). + +--- + +### Story 3.2 — `POST /api/v1/judgments/generate-from-ubi` endpoint (FR-3) + +**Outcome:** The UBI generate endpoint returns 202 with `GenerateJudgmentsResponse{judgment_list_id, status}`. Delegates to `start_ubi_judgment_generation`. Conditional `current_template_id`/`rubric` validator on the request body. + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/api/v1/judgments.py` | Add `POST /judgments/generate-from-ubi` handler `generate_judgments_from_ubi` after the existing `generate_judgments` (line ~170). | +| `backend/app/api/v1/schemas.py` | Add `CreateJudgmentListFromUbiRequest(BaseModel)` with the 12 fields per FR-3 + a `model_validator(mode='after')` for the hybrid conditional. | + +**Endpoints** + +| Method | Path | Request body | Success response | Error codes | +|---|---|---|---|---| +| `POST` | `/api/v1/judgments/generate-from-ubi` | `CreateJudgmentListFromUbiRequest` (12 fields, see schema below) | `202` `GenerateJudgmentsResponse{ judgment_list_id, status: "generating" }` | `412 UBI_NOT_ENABLED`, `422 UBI_INSUFFICIENT_DATA`, `422 UBI_WINDOW_TOO_LARGE`, `422 VALIDATION_ERROR`, `404 CLUSTER_NOT_FOUND`, `404 QUERY_SET_NOT_FOUND`, `404 TEMPLATE_NOT_FOUND` (hybrid only), `409 JUDGMENT_LIST_NAME_TAKEN`, `503 OPENAI_NOT_CONFIGURED` (hybrid only), `503 LLM_PROVIDER_INCAPABLE` (hybrid only), `503 UNKNOWN_MODEL_PRICING` (hybrid only), `503 OPENAI_BUDGET_EXCEEDED` (hybrid only) | + +**Pydantic schemas** + +```python +class CreateJudgmentListFromUbiRequest(BaseModel): + name: str = Field(min_length=1, max_length=256) + description: str | None = Field(default=None, max_length=2000) + query_set_id: str = Field(min_length=1, max_length=36) + cluster_id: str = Field(min_length=1, max_length=36) + target: str = Field(min_length=1, max_length=256) + since: datetime + until: datetime | None = None + converter: UbiConverterKind + converter_config: dict[str, Any] | None = None + llm_fill_threshold: int | None = Field(default=20, ge=1) + min_impressions_threshold: int | None = Field(default=100, ge=1) + mapping_strategy: UbiMappingStrategyWire = "reject" + current_template_id: str | None = Field(default=None, min_length=36, max_length=36) + rubric: str | None = Field(default=None, min_length=1) + + @model_validator(mode="after") + def _validate_hybrid_conditional(self) -> "CreateJudgmentListFromUbiRequest": + is_hybrid = self.converter == "hybrid_ubi_llm" + has_template = self.current_template_id is not None + has_rubric = self.rubric is not None + if is_hybrid and not (has_template and has_rubric): + raise ValueError( + "current_template_id and rubric are REQUIRED when converter == 'hybrid_ubi_llm'" + ) + if not is_hybrid and (has_template or has_rubric): + raise ValueError( + "current_template_id and rubric MUST be null for non-hybrid converters" + ) + return self +``` + +**Tasks** + +1. Add `CreateJudgmentListFromUbiRequest` to `schemas.py` after the existing `CreateJudgmentListGenerateRequest` (line ~840). +2. Add the handler in `judgments.py`: + ```python + @router.post( + "/judgments/generate-from-ubi", + response_model=GenerateJudgmentsResponse, + status_code=status.HTTP_202_ACCEPTED, + tags=["judgments"], + ) + async def generate_judgments_from_ubi( + body: CreateJudgmentListFromUbiRequest, + request: Request, + db: Annotated[AsyncSession, Depends(get_db)], + ) -> GenerateJudgmentsResponse: + """Start a UBI-derived judgment generation job.""" + settings = get_settings() + arq_pool = getattr(request.app.state, "arq_pool", None) + redis_client: Redis | None = None + try: + redis_client = await _open_redis() + result = await start_ubi_judgment_generation( + db=db, redis=redis_client, arq_pool=arq_pool, settings=settings, + req=UbiJudgmentGenerationRequest(**body.model_dump()), + ) + return GenerateJudgmentsResponse( + judgment_list_id=result.judgment_list_id, + status=result.status, + ) + finally: + if redis_client is not None: + try: + await redis_client.aclose() + except Exception as exc: + logger.debug("redis close raised", error=str(exc)) + ``` +3. Write contract test `backend/tests/contract/test_judgments_generate_from_ubi_shape.py` asserting the 202 response shape + each of the 13 error envelopes documented in the endpoint table. +4. Write integration test `backend/tests/integration/api/test_judgments_generate_from_ubi.py` exercising every preflight branch end-to-end against a stubbed `UbiReader` + adapter; verify the `judgment_lists.generation_params` JSONB column is populated correctly on success. +5. Update OpenAPI schema lock test (if any) to include the new endpoint in the inventory. + +**Definition of Done (DoD)** + +- [ ] All 13 error envelopes match the spec §8.5 catalog + spec §8.3 example bodies. +- [ ] 202 success returns `GenerateJudgmentsResponse` with `judgment_list_id` matching the row's `id` and `generation_params` populated for inspection in the integration test. +- [ ] Hybrid-mode missing `current_template_id` returns 422 with detail mentioning the field name. +- [ ] Non-hybrid with `rubric` present returns 422 with the conditional-validator message. +- [ ] `model_validator` test asserts both branches of the conditional via FastAPI's `RequestValidationError` envelope. + +--- + +### Story 3.3 — `generate_judgments_from_ubi` Arq worker (FR-5) + +**Outcome:** The UBI worker pipeline reads UBI features → applies the chosen converter (awaiting LLM-fill for hybrid pairs) → bulk-inserts judgments with the correct `source`/`rater_ref` → writes calibration JSONB → terminal-status flip. + +**Modified files** + +| File | Change | +|---|---| +| `backend/workers/judgments.py` | Add `generate_judgments_from_ubi(ctx, judgment_list_id)` after the existing `generate_judgments_llm` (line ~354). Reuse `_safe_record_cost`, `_fail_list`. Add a worker-local `_make_llm_rate_callback(...)` helper that wraps `rate_query_batch` for the hybrid converter's `llm_rate` parameter. | +| `backend/workers/all.py` | Register `generate_judgments_from_ubi` in `WorkerSettings.functions`. Extend the boot-time resume sweep at `backend/workers/all.py:148-161` to enqueue both `generate_judgments_llm` (for LLM lists where `generation_params IS NULL`) AND `generate_judgments_from_ubi` (for UBI lists where `generation_params IS NOT NULL`) on `status='generating'` rows. | + +**Key interfaces** + +```python +# workers/judgments.py +async def generate_judgments_from_ubi(ctx: dict[str, Any], judgment_list_id: str) -> None: + """Arq entry point — run the UBI judge pipeline for one judgment list. + + Contract (per FR-5): + 1. Load judgment_list row + generation_params; bail if missing/terminal/missing params. + 2. Build adapter, UbiReader, position-bias prior. + 3. Read features → apply converter (awaits LLM-fill via injected callback for hybrid). + 4. Apply mapping_strategy per query; count per-query skips. + 5. Bulk-insert judgments (source='click' for UBI rows, 'llm' for hybrid LLM-fill). + 6. Write calibration JSONB with coverage + skip counts. + 7. Terminal status flip. + """ + +def _make_llm_rate_callback( + *, + openai_client: AsyncOpenAI, + model: str, + rubric: str, + bundle_system: str, + query_id_to_text: dict[str, str], + redis: Redis, + budget_usd: float, +) -> LlmRateCallback: + """Worker-local: build the LlmRateCallback for HybridUbiLlmConverter. + + Each call routes through rate_query_batch + _safe_record_cost so the + daily-budget gate + capability cache fire unchanged. + """ +``` + +**Tasks** + +1. Implement `generate_judgments_from_ubi`: + - Load row → bail if missing or `status != 'generating'`. + - Read `generation_params`; bail with `_fail_list(..., 'MISSING_GENERATION_PARAMS')` if NULL (race guard). + - Build adapter via `build_adapter(cluster)`; build `UbiReader` with `Settings.ubi_position_bias_prior`. + - `features = await ubi_reader.read_features(target, since, until, ...)`. Catch `UbiNotEnabledError` → `_fail_list(..., 'UBI_NOT_ENABLED')`; empty `features` → `_fail_list(..., 'UBI_INSUFFICIENT_DATA')` (race fallback per FR-1). + - Construct the converter: + - `'ctr_threshold'` → `CtrThresholdConverter()` + - `'dwell_time'` → `DwellTimeThresholdConverter()` + - `'hybrid_ubi_llm'` → `HybridUbiLlmConverter(inner=..., llm_rate=_make_llm_rate_callback(...))`. Pull `inner` per `converter_config.inner` (default `'ctr_threshold'`). + - `ratings = await converter.convert(features, ConverterConfig(extra=generation_params.get('converter_config') or {}))`. + - Apply `mapping_strategy` per query (Python-side dict resolution; track `ambiguous_query_skip_count`). + - Build the rows: UBI pairs → `{source: 'click', rater_ref: f'ubi:{converter_kind}'}`; hybrid LLM-fill pairs (carried through via the callback's persistence pattern — see step 2) → `{source: 'llm', rater_ref: f'openai:{model}'}`. + - `await repo.bulk_create_judgments(db, rows)`; `await db.commit()`. + - Write calibration JSONB: `{coverage_pct, head_pairs, tail_pairs, position_bias_prior_id, llm_fill_calls?, ambiguous_query_skip_count, sparse_query_skip_count}`. + - Terminal flip to `complete` (or `failed` on caught error per FR-5 step 8). +2. Implement `_make_llm_rate_callback`: closure over openai_client/model/rubric/redis/budget; per call computes `expected_doc_ids`, runs `rate_query_batch`, persists rows immediately (NOT deferred to caller — matches the LLM-worker persist-first / record-cost-second pattern in `_process_query`), calls `_safe_record_cost`. Returns `{(query_id, doc_id): rating}`. +3. Register the job in `backend/workers/all.py` `WorkerSettings.functions = [..., generate_judgments_from_ubi]`. +4. Extend the boot-time resume sweep at `backend/workers/all.py:148-161` per the spec FR-5 + the `feat_judgments_periodic_resume_sweep` pattern. Use `generation_params IS NOT NULL` as the UBI discriminator (LLM lists have NULL). +5. Write integration test `backend/tests/integration/workers/test_generate_judgments_from_ubi.py` covering: clean-loop completes with all-`click` rows; hybrid produces mixed list; resume after crash skips already-rated queries; ambiguous-mapping per-query skip; `UbiInsufficientDataError` mid-loop → `failed_reason='UBI_INSUFFICIENT_DATA'`; `BudgetExceededError` mid-hybrid → `failed_reason='OPENAI_BUDGET_EXCEEDED'`. + +**Definition of Done (DoD)** + +- [ ] Clean loop on a rung_3 fixture completes with `judgment_count > 0`, all rows `source='click'`, `calibration.coverage_pct >= 0.5`. +- [ ] Hybrid loop produces both `source='click'` and `source='llm'` rows; `calibration.llm_fill_calls == count(source='llm')`. +- [ ] Ambiguous-mapping per-query skip increments `calibration.ambiguous_query_skip_count` without flipping status to `failed`. +- [ ] Race-fallback `UbiInsufficientDataError` mid-loop → terminal `status='failed'`, `failed_reason='UBI_INSUFFICIENT_DATA'`. +- [ ] Resume sweep re-enqueues a UBI list with `status='generating'` and per-query resume-skip prevents re-spending LLM calls. +- [ ] Worker file does NOT instantiate `openai.AsyncClient(...)` outside `_make_llm_rate_callback`'s parameter binding (lint guard: ast scan). + +--- + +### Story 3.4 — `generate_judgments_from_ubi` agent tool + orchestrator system-prompt update (FR-6) + +**Outcome:** The chat agent exposes a new MUTATING tool `generate_judgments_from_ubi`. The orchestrator's system prompt prefers UBI when the cluster's readiness rung ≥ 1. + +**New files** + +| File | Purpose | +|---|---| +| `backend/app/agent/tools/judgments/generate_judgments_from_ubi.py` | `GenerateJudgmentsFromUbiArgs` + `generate_judgments_from_ubi_impl` + `GENERATE_JUDGMENTS_FROM_UBI_TOOL` triad. | + +**Modified files** + +| File | Change | +|---|---| +| `backend/app/agent/tools/__init__.py` | Import + add to `TOOLS`, `TOOL_REGISTRY`, `TOOL_ARG_MODELS`. Module-load drift assertion at line 232–236 catches a missing registration. | +| `backend/app/agent/tools/judgments/__init__.py` | Re-export the new tool symbols. | +| `prompts/orchestrator.system.md` (verify path at impl time; if `_load_system_prompt` in `orchestrator.py` reads from a different path, adjust) | Update the "generate a judgment list" tool selection guidance: prefer `generate_judgments_from_ubi` when the cluster has UBI rung ≥ 1; default to `generate_judgments_llm` otherwise. Provide one-line examples for both. | + +**Key interfaces** + +```python +# agent/tools/judgments/generate_judgments_from_ubi.py +class GenerateJudgmentsFromUbiArgs(BaseModel): + """Mirrors CreateJudgmentListFromUbiRequest (the router's request body).""" + name: str = Field(min_length=1, max_length=256) + description: str | None = Field(default=None, max_length=2000) + query_set_id: UUID + cluster_id: UUID + target: str = Field(min_length=1, max_length=256) + since: datetime + until: datetime | None = None + converter: UbiConverterKind + converter_config: dict[str, Any] | None = None + llm_fill_threshold: int | None = Field(default=20, ge=1) + min_impressions_threshold: int | None = Field(default=100, ge=1) + mapping_strategy: UbiMappingStrategyWire = "reject" + current_template_id: UUID | None = None + rubric: str | None = None + +async def generate_judgments_from_ubi_impl( + args: GenerateJudgmentsFromUbiArgs, ctx: ToolContext, +) -> dict[str, Any]: + """Start a UBI-judgment job. MUTATING — confirmation guard required.""" + +GENERATE_JUDGMENTS_FROM_UBI_TOOL: ChatCompletionToolParam = {...} +``` + +**Tasks** + +1. Implement the triad following the exact pattern from `backend/app/agent/tools/judgments/generate_judgments_llm.py`. +2. Add `model_validator(mode='after')` mirroring the API schema's hybrid conditional so the agent-tool dispatch rejects the same bad shape locally (before hitting the dispatcher) for cleaner error messages in the chat stream. +3. Register in `TOOLS` / `TOOL_REGISTRY` / `TOOL_ARG_MODELS` — the module-load assertion at `__init__.py:232-236` will fail if any of the three is missed. +4. Update `prompts/orchestrator.system.md` (or wherever `_load_system_prompt` reads) — add a "Choosing between LLM and UBI judgment generation" section. Reference both tool names. Note the rung-detection heuristic (probe via `get_schema` if the operator's intent is judgment-generation and the rung is unknown). +5. Write integration test `backend/tests/integration/agent/test_generate_judgments_from_ubi_tool.py`: tool dispatch returns the expected shape on a stubbed `start_ubi_judgment_generation`; confirmation guard fires; tool registry drift assertion catches a forced mis-registration (negative test). +6. Add the tool to the contract-test agent-tool inventory at `backend/tests/contract/test_agent_tool_inventory.py` (or equivalent). + +**Definition of Done (DoD)** + +- [ ] Tool dispatch round-trips through the confirmation guard → `start_ubi_judgment_generation` → returns `{judgment_list_id, status}`. +- [ ] Removing the tool from any one of `TOOLS` / `TOOL_REGISTRY` / `TOOL_ARG_MODELS` causes a `RuntimeError` at module import time (negative test). +- [ ] Updated system prompt mentions both tool names; the orchestrator system-prompt unit test (if present) verifies the names appear. + +--- + +## Epic 4 — Frontend (method picker + on-ramp UX + value-delta) + +### Story 4.1 — Wire enums + `useUbiReadiness` hook + `` primitive (FR-7 + FR-8 + FR-9 mirror) + +**Outcome:** UI mirrors of the four backend Literals exist with the `// Values must match backend/...` discipline comment. `useUbiReadiness(clusterId, querySetId, target)` hook hits the new endpoint with 60s React Query stale time. `` renders the rung as a text-only badge with a glossary-keyed tooltip. + +**New files** + +| File | Purpose | +|---|---| +| `ui/src/lib/api/ubi.ts` | TanStack Query hooks: `useUbiReadiness(clusterId, querySetId, target)`, `useGenerateJudgmentsFromUbi()`. | +| `ui/src/components/clusters/ubi-rung-badge.tsx` | `` text-only badge with glossary-keyed tooltip (per rung label per spec §11). **Single variant** — renders one of the 4 `UBI_READINESS_RUNG_VALUES` labels. Only consumed inside the generate-judgments dialog (Story 4.2) where the parent has `clusterId` + `querySetId` + `target` to call `useUbiReadiness(...)`. NOT rendered on cluster cards or cluster detail (those pages lack query_set/target context — spec FR-7 requires both as query params, so calling the endpoint without them returns 422). | + +**Modified files** + +| File | Change | +|---|---| +| `ui/src/lib/enums.ts` | Add `JUDGMENT_GENERATION_METHOD_VALUES` (4 values, mirrors `JudgmentGenerationMethodWire`), `UBI_CONVERTER_VALUES` (3 values, mirrors `UbiConverterKind`), `UBI_READINESS_RUNG_VALUES` (4 values), `UBI_MAPPING_STRATEGY_VALUES` (3 values). Widen `JUDGMENT_SOURCE_FILTER_VALUES` from `['llm', 'human']` to `['llm', 'human', 'click']`. Each new array carries the `// Values must match backend/app/api/v1/schemas.py ` comment per the source-of-truth policy. | +| `ui/src/lib/glossary.ts` | Add 5 new entries: `judgment.converter`, `judgment.converter.llm`, `judgment.converter.ubi`, `judgment.converter.hybrid`, `cluster.ubi_readiness` (per spec §11 + §15). Each carries a `// Source-of-truth: backend/app/api/v1/schemas.py ` comment. | +| (none) | **No changes to `ui/src/app/clusters/page.tsx` or `ui/src/app/clusters/[id]/page.tsx`.** Cycle-3 plan review: the spec FR-7 readiness endpoint requires `?query_set_id` + `?target` as query params (returns 422 without). Cluster cards / cluster detail pages don't have a query_set/target in their data flow, so calling the endpoint there would either always 422 or require inventing a separate cluster-level capability endpoint outside spec scope. Decision: render `` ONLY inside the generate-judgments dialog (Story 4.2), where the parent supplies all three context fields. Operators discover UBI when they open the dialog — the same surface where they choose the converter. If a future operator-feedback signal asks for a cluster-card-level UBI indicator, capture as `chore_cluster_card_ubi_indicator` idea file with a proposed `GET /clusters/{id}/ubi-enabled` endpoint scoped to a simple plugin-present/absent boolean. | + +**UI element inventory** + +- `` — text-only badge. Single variant. Renders one of 4 labels per rung: "rung_0: UBI not enabled", "rung_1: UBI sparse", "rung_2: UBI dense head", "rung_3: UBI full coverage" (final wording verify against spec §11 IA labels at impl time). +- Color: text-only with subtle muted-background, no color-only meaning. WCAG AA contrast. +- Tooltip: `HelpPopover` keyed off `cluster.ubi_readiness` glossary entry. +- **Consumption surface**: ONLY inside the generate-judgments dialog (Story 4.2). NOT rendered on cluster list/detail (cycle-3 review: spec FR-7 requires `?query_set_id`+`?target`; those pages don't have it). + +**Wire value enumeration table** (per CLAUDE.md "Enumerated Value Contract Discipline"): + +| Frontend array | Backend source-of-truth | Values | +|---|---|---| +| `JUDGMENT_GENERATION_METHOD_VALUES` | `JudgmentGenerationMethodWire` | `'llm', 'ctr_threshold', 'dwell_time', 'hybrid_ubi_llm'` | +| `UBI_CONVERTER_VALUES` | `UbiConverterKind` | `'ctr_threshold', 'dwell_time', 'hybrid_ubi_llm'` | +| `UBI_READINESS_RUNG_VALUES` | `UbiReadinessRungWire` | `'rung_0', 'rung_1', 'rung_2', 'rung_3'` | +| `UBI_MAPPING_STRATEGY_VALUES` | `UbiMappingStrategyWire` | `'reject', 'first_match', 'most_recent'` | +| `JUDGMENT_SOURCE_FILTER_VALUES` (widened) | `JudgmentSourceFilterWire` (widened in Story 2.3) | `'llm', 'human', 'click'` | + +**Tasks** + +1. Add the 4 new arrays + the widening to `enums.ts`. Run the existing enum-source-of-truth grep gate (CI gate at `scripts/ci/verify_enum_source_of_truth.sh` — verify path during impl) on a local branch to confirm parity. +2. Add the 5 glossary entries to `glossary.ts` with the source-of-truth comment. +3. Implement `useUbiReadiness` with React Query: queryKey `['ubi-readiness', clusterId, querySetId, target]`, 60s `staleTime`, gracefully degrades to `{rung: 'rung_0', covered_pairs_pct: null, head_covered: null}` on 503/404 (operator gets the LLM-default behavior). +4. Implement `useGenerateJudgmentsFromUbi` as a TanStack mutation hitting `POST /api/v1/judgments/generate-from-ubi`. +5. Implement `` (single variant). Tooltip uses `` keyed off `cluster.ubi_readiness`. Per-rung labels per the UI element inventory above. +6. (Removed at cycle-3 plan review per finding `readiness-snapshot-badge-contract-drift` — no cluster-card / cluster-detail badge render. The component is solely consumed inside Story 4.2's dialog.) +7. Write component tests `ui/src/__tests__/components/clusters/ubi-rung-badge.test.tsx` for all 4 rungs (single variant, no snapshot mode). +8. Write enum lint guard pass: the new arrays MUST appear in `enums.ts` with the discipline comment; the lint guard at `ui/src/__tests__/lib/enums-source-of-truth.test.ts` (verify path at impl time — may need to add to the regex if not generic). + +**Definition of Done (DoD)** + +- [ ] All 4 new enum arrays + the widened `JUDGMENT_SOURCE_FILTER_VALUES` in `enums.ts` with the discipline comment. +- [ ] All 5 new glossary entries with source-of-truth comments. +- [ ] `useUbiReadiness` returns the expected shape against a mocked API response; falls back gracefully on 503/404. +- [ ] `` component tests pass; all 4 rung labels render correctly; tooltip wired to glossary. +- [ ] Cluster list + detail pages unchanged (no badge added there per cycle-3 plan-review fix `readiness-snapshot-badge-contract-drift`). + +--- + +### Story 4.2 — Generate-judgments dialog: method picker + UBI window controls + engine-aware nudge + sparse-data card (FR-8) + +**Outcome:** The existing `` extends with a 4-option **method** picker, conditional UBI window controls, conditional LLM-fill threshold, an engine-aware dismissible nudge above the dialog body when rung_0, and a sparse-data recommendation card when rung_1. + +**New files** + +| File | Purpose | +|---|---| +| `ui/src/components/clusters/ubi-onramp-nudge.tsx` | Dismissible nudge card. Engine-aware copy via switch on `cluster.engine_type`. Persists dismissal in localStorage keyed by `cluster_id` (per D-7). | +| `ui/src/components/query-sets/ubi-sparse-data-card.tsx` | Inline recommendation card with "Switch to hybrid" affordance. Visible when `useUbiReadiness().rung === 'rung_1'` AND the picker currently selects a pure UBI converter. | + +**Modified files** + +| File | Change | +|---|---| +| `ui/src/components/query-sets/generate-judgments-dialog.tsx` | (i) Render `` above the dialog body when `useUbiReadiness().rung === 'rung_0'`. (ii) Add **Method** ``** — 4 options from `JUDGMENT_GENERATION_METHOD_VALUES`. Labels per D-2: "LLM-as-judge", "UBI (click-through)", "UBI (dwell-time)", "Hybrid UBI + LLM". Default selection from rung (rung_0 → `llm`; rung_1/rung_2 → `hybrid_ubi_llm`; rung_3 → `ctr_threshold`). Each option has an inline `` keyed off the matching `judgment.converter.*` glossary entry (per spec §11 tooltip table). +- **Window controls** — `` + ``. Default `since = now - 30 days`. Both visible when method ≠ `llm`. +- **LLM-fill threshold** — ``. Visible when method == `hybrid_ubi_llm`. +- **Rubric** — existing `