From 8ff7add9a762a2e2355db6ecff21c0d000903a91 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 29 May 2026 07:55:54 -0400 Subject: [PATCH 1/5] docs(mvp2): regenerate dashboards for 3 new 02_mvp2 idea folders Lockstep dashboard regen (mvp1-dashboard-regen pre-commit hook output) for feat_study_budget_presets + feat_overnight_autopilot + feat_study_convergence_indicator added in the prior commit. Signed-off-by: SoundMindsAI Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/00_overview/MVP2_DASHBOARD.md | 35 ++- docs/00_overview/mvp2_dashboard.html | 49 ++- .../02_mvp2/feat_overnight_autopilot/idea.md | 69 +++++ .../02_mvp2/feat_study_budget_presets/idea.md | 70 +++++ .../feat_study_convergence_indicator/idea.md | 60 ++++ docs/01_architecture/mvp2-overview.md | 285 ++++++++++++++++++ 6 files changed, 547 insertions(+), 21 deletions(-) create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_study_budget_presets/idea.md create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_study_convergence_indicator/idea.md create mode 100644 docs/01_architecture/mvp2-overview.md diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 71c345e2..978ed6ac 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -15,14 +15,14 @@ Pull from the Idea backlog or capture a new feature spec. | Metric | Value | |---|---| | Scoped items done | **1 / 1** (100%) — feat_/infra_/chore_/epic_ past idea stage | -| Pending work | **11** items (every not-done feat/infra/chore/bug across all priorities) | +| Pending work | **14** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | -| → P1 | **2** high-value, ready when P0 clears | -| → P2 (default) | 5 important to file, not blocking | +| → P1 | **3** high-value, ready when P0 clears | +| → P2 (default) | 7 important to file, not blocking | | → Backlog | 4 captured for record, not planned | | Open bugs | 2 | | Legacy "Path to MVP2" | 6 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 5 idea-only feat/infra (not yet scoped into MVP2) | +| Backlog ideas | 8 idea-only feat/infra (not yet scoped into MVP2) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -45,21 +45,24 @@ _None._ _None._ -### Idea (11) +### Idea (14) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/idea.md) | Feature | MVP1 ships with **LLM-as-judge** as the only authoritative judgment source. The architecture anticipated this would change — the `judgments.source` CHECK already accepts `click`… | — | Idea — bundled with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) into MVP2 / v0.2 "Three-Engine + Real Signals" | -| 2 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | -| 3 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../feature_templates/README.md). | -| 4 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | -| 5 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | -| 6 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| 7 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | -| 8 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | -| 9 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | -| 10 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | -| 11 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | +| 1 | P1 | [feat_study_budget_presets](planned_features/02_mvp2/feat_study_budget_presets/idea.md) | Feature | Three connected gaps make under-budgeting the default outcome: | — | Idea — surfaced from an operator dogfooding review of MVP1 studies (2026-05-29). Highest-leverage item in the "overnight autopilot ergonomics" theme. | +| 2 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/idea.md) | Feature | MVP1 ships with **LLM-as-judge** as the only authoritative judgment source. The architecture anticipated this would change — the `judgments.source` CHECK already accepts `click`… | — | Idea — bundled with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) into MVP2 / v0.2 "Three-Engine + Real Signals" | +| 3 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | +| 4 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../feature_templates/README.md). | +| 5 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | +| 6 | P2 | [feat_study_convergence_indicator](planned_features/02_mvp2/feat_study_convergence_indicator/idea.md) | Feature | After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. | +| 7 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | +| 8 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | +| 9 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | +| 10 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | +| 11 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | +| 12 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | +| 13 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | +| 14 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | ## Dependency graph diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index cec0d272..e7b96810 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -403,7 +403,7 @@

MVP2 Progress

Pending work
-
11
+
14
every not-done feat/infra/chore/bug across all priorities
@@ -420,12 +420,12 @@

MVP2 Progress

P1
-
2
+
3
high-value, ready when P0 clears
P2 (default)
-
5
+
7
important to file, not blocking
@@ -442,7 +442,7 @@

MVP2 Progress

Backlog ideas: - 5 idea-only feat/infra folders (not yet scoped into MVP2) + 8 idea-only feat/infra folders (not yet scoped into MVP2) In flight: @@ -463,7 +463,20 @@

Pipeline

-

Idea 11

+

Idea 14

+ +
+
Study Budget Presets
+
+ Feature + P1 + +
+
Three connected gaps make under-budgeting the default outcome:
+ + +
+
Ubi Judgments
@@ -504,6 +517,32 @@

Idea 11

+
+
Overnight Autopilot
+
+ Feature + P2 + +
+
The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it:
+ + +
+ + +
+
Study Convergence Indicator
+
+ Feature + P2 + +
+
After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the
+ + +
+ +
Demo Seeding Integration Tests Rewrite
diff --git a/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md new file mode 100644 index 00000000..157a632c --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md @@ -0,0 +1,69 @@ +# Overnight autopilot — surface autonomous study chaining as a first-class "set it and wake up to results" path + +**Date:** 2026-05-29 +**Status:** Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. +**Priority:** P2 — high operator value, but the underlying capability already works; this is surfacing + a morning summary, not new core machinery. Pairs with [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md) (the P1 of the theme). +**Origin:** Operator's stated goal: "set a study in motion, come back in a few hours (overnight), and wake up to a few results I could review and potentially turn into a PR." Tracing the live DB (2026-05-29) showed **zero** studies have ever used `auto_followup_depth` and zero are chain children — the feature that delivers exactly this goal is shipped but invisible. +**Depends on:** [`feat_auto_followup_studies`](../../../implemented_features/2026_05_24_feat_auto_followup_studies/) (shipped 2026-05-24, PR #223) — the autonomous chaining engine. This idea is purely the surfacing + summary layer on top of it. + +## Problem + +The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: + +1. **`auto_followup_depth` is a hidden config key.** When set (1–5, [`schemas.py:645`](../../../../../backend/app/api/v1/schemas.py#L645)), a completed study automatically narrows the search space around its winner, decrements the depth, and spawns a child study — **zero human intervention between iterations** ([`backend/workers/auto_followup.py`](../../../../../backend/workers/auto_followup.py)). The chain self-terminates on depth exhaustion, sub-epsilon lift (<0.5%), budget at 80%, or parent failure. This is exactly the operator's "wake up to a few results" ask. But it is not exposed as a first-class control in the create-study wizard — the operator never knew it existed, so all 7 studies ran one-shot. + +2. **No "what happened overnight" surface.** Even with chaining on, there is no single place that says "here are the 3 studies that ran while you slept, here's the best config each found, here's the cumulative lift, here's the one that's ready to become a PR." The operator would have to piece it together from the studies list + individual proposals. + +3. **The human-approval boundary is correct but undescribed.** PR-opening is deliberately a manual click (production config changes require human approval — umbrella spec §6 hard constraint). That's right, but the operator doesn't have a framing that says "the loop runs autonomously up to the PR; the PR is your one decision." Without that framing, "overnight autopilot" feels either impossible (it's not) or unsafe (it isn't — nothing reaches production without a human merge). + +## Proposed capabilities + +### First-class "Run overnight (compound automatically)" toggle in the create-study wizard + +Promote `auto_followup_depth` from a hidden config key to a labeled wizard control with plain-language copy: + +> **🌙 Run overnight (compound automatically)** — When this study finishes, automatically start a follow-up that narrows in on the best result, and repeat. Stops on its own when it stops improving, runs out of depth, or hits the daily budget. No production changes happen without your review — you still open every PR by hand. +> +> Compound up to **[3]** times. (1–5) + +Pairs naturally with the "Thorough (overnight)" budget preset from [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md): selecting the overnight preset could default the chain depth on. Sets the existing `config.auto_followup_depth` field — no schema change. + +### Morning results summary ("the overnight digest") + +A surface — a chain-view panel on the study detail page and/or a top-of-`/studies` "ran while you were away" card — that, for a completed chain, shows: + +- The chain as an ordered list (parent → child → grandchild), each with its best metric and the delta from the prior link. +- Cumulative lift from the chain's start to its best link. +- A clear "best config across the whole chain" and a one-click path to the proposal that carries it (which is then one more click from a PR). +- Why the chain stopped (depth exhausted / no further lift / budget) — reusing the telemetry events the worker already emits (`auto_followup_depth_exhausted`, `auto_followup_skipped_no_lift`, `auto_followup_skipped_budget`). + +This is the "wake up to a few results you could review and potentially turn into a PR" deliverable, made concrete. + +### Tutorial + docs: name the autopilot path + +A short tutorial section ("Run the loop overnight") that walks: pick the overnight budget preset → enable compounding depth 3 → start before you log off → review the chain summary in the morning → open the winning PR. Make the human-approval boundary explicit and reassuring. + +## Scope signals + +- **Backend:** small-to-moderate. Mostly read-side: a chain-summary aggregation (walk `parent_study_id` links, roll up best metrics + deltas + stop reason). The write path (`auto_followup_depth`) already exists and is validated. No migration. +- **Frontend:** moderate. Wizard toggle + the chain-summary panel/card. The chain-summary panel is the bulk of the work. +- **Migration:** none. +- **Config:** none (uses existing `config.auto_followup_depth` and existing budget settings). +- **Audit events:** N/A (pre-`audit_log`). + +## Why this isn't just "add a tooltip" + +The capability is real but the *trust and discoverability* gap is the whole barrier. An operator will not hand a tool an unattended overnight run unless (a) they can find the switch, (b) they understand it can't touch production without them, and (c) there's a clean morning surface that makes the results reviewable in minutes. All three are missing today, which is why a shipped feature has zero usage. + +## Relationship to other work + +- **Surfaces** [`feat_auto_followup_studies`](../../../implemented_features/2026_05_24_feat_auto_followup_studies/) (the engine). +- **Sibling theme:** [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md) (the overnight preset feeds this) + [`feat_study_convergence_indicator`](../feat_study_convergence_indicator/idea.md) (each chain link's convergence display). +- **Composes with the MVP2 UBI anchor** ([`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)): overnight compounding is dramatically more valuable against a continuously-fresh UBI judgment list than a static LLM snapshot — the `feat_ubi_judgments` idea already notes this composition. This is the operator-facing payoff of that pairing. +- **Respects** the human-merge invariant (umbrella spec §6) — autopilot runs the *exploration* side unattended; the *deployment* side stays a deliberate human click. + +## Open questions for /spec-gen + +1. Where the morning summary lives: study-detail chain panel, a `/studies` "ran while away" card, or both. +2. Whether selecting the "Thorough (overnight)" budget preset auto-enables compounding (and at what default depth), or whether they stay independent toggles. +3. Whether to add an optional notification hook (the spec backlog already lists "outgoing webhooks for resource lifecycle events" — a chain-complete webhook would be the real "wake up to results" trigger, but that's likely a separate backlog item, not MVP2). diff --git a/docs/00_overview/planned_features/02_mvp2/feat_study_budget_presets/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_study_budget_presets/idea.md new file mode 100644 index 00000000..759e0448 --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/feat_study_budget_presets/idea.md @@ -0,0 +1,70 @@ +# Study budget presets + sub-warmup guard ("don't ship a study the optimizer never woke up for") + +**Date:** 2026-05-29 +**Status:** Idea — surfaced from an operator dogfooding review of MVP1 studies (2026-05-29). Highest-leverage item in the "overnight autopilot ergonomics" theme. +**Priority:** P1 (for the theme) — directly fixes the root cause of the "studies feel like they need follow-ups" friction. Small, high-leverage, no new optimizer machinery. +**Origin:** Operator ran 7 studies through the full loop and felt follow-ups were near-mandatory. Tracing the actual `studies.config` values (2026-05-29) showed the cause: 6 of 7 studies ran `max_trials` of **12–15**. With Optuna TPE's ~10-trial random warmup ([`optimization.md`](../../../../01_architecture/optimization.md) §"Optuna configuration"), those studies did ~10 random samples + 2 TPE-guided trials + 1 baseline — the Bayesian optimizer barely engaged before the stop condition fired. The unconverged result is what made the digest emit "narrow/widen" follow-ups, which *felt* like a required second pass but was really compensation for under-budgeting. +**Depends on:** MVP1 study lifecycle (shipped). No dependency on the MVP2 anchors; composes with them. + +## Problem + +Three connected gaps make under-budgeting the default outcome: + +1. **No trial-budget default.** `StudyConfigSpec` requires the operator to supply `max_trials` or `time_budget_min` ([`backend/app/api/v1/schemas.py:629`](../../../../../backend/app/api/v1/schemas.py#L629)) — there is no preset, no recommended value, no wizard guidance. An operator with no intuition for "how many trials is enough" picks a small round number. The observed real-world picks were 12 and 15. + +2. **No floor at the TPE warmup threshold.** Optuna's `TPESampler` runs its first ~10 trials as random search before the estimator guides sampling; `MedianPruner` is disabled under 50 trials ([`optimization.md`](../../../../01_architecture/optimization.md), [`backend/app/eval/optuna_runtime.py`](../../../../../backend/app/eval/optuna_runtime.py)). A study with `max_trials < ~20` therefore barely exercises the Bayesian optimizer at all — it is effectively random search. Nothing warns the operator that they are about to run a study that won't converge. + +3. **The cost of under-budgeting is invisible and misattributed.** The unconverged study produces a digest with "narrow"/"widen" follow-ups. The operator reasonably reads that as "the tool needs me to iterate," when the real signal is "this study stopped before the optimizer learned anything." The friction the operator feels (too much manual follow-up work) is a defaults problem masquerading as a workflow problem. + +**Evidence (traced 2026-05-29 against the live DB):** 7 studies total; `max_trials` distribution = `12 (×4), 15 (×2), 200 (×1)`; zero used `auto_followup_depth`; zero chain children. The single 200-trial study is the only one that would have converged. + +## Proposed capabilities + +### Budget presets in the create-study wizard + +Replace the bare `max_trials` number input with a preset selector + an "advanced / custom" escape hatch. Recommended initial presets (exact numbers are a `/spec-gen` decision): + +| Preset | `max_trials` | Intended use | Rough wall-clock at parallelism 8 | +|---|---|---|---| +| **Quick look** | ~30 | Smoke-test the search space / template wiring; not for decisions | minutes | +| **Standard** (default) | ~200 | The everyday "tune this and give me a real answer" run | tens of minutes | +| **Thorough (overnight)** | ~1000 | Wide space, many params, converge hard while you sleep | hours | +| **Custom** | operator-set | Power users; current behavior | — | + +The defaults should be grounded, not arbitrary: Standard must comfortably clear the TPE warmup (≥ ~10× the warmup count) so the optimizer is the thing doing the work. Presets set `max_trials` (and optionally a sensible `parallelism` bump for the overnight preset) in `studies.config` — no schema change, just wizard ergonomics over the existing fields. + +### Sub-warmup guard (the important half) + +When an operator chooses Custom and enters a `max_trials` below a warmup-derived floor (e.g. `< 2 × n_startup_trials`, so ~20), surface a non-blocking inline warning at create time: + +> "TPE runs its first ~10 trials as random search before it starts optimizing. At 12 trials this study is essentially random search and is unlikely to converge — consider ≥ 50 (Standard) for a result worth turning into a PR." + +Non-blocking (the operator can still proceed — quick smoke tests are legitimate), but it makes the cost legible at the moment of the decision. This is the single change that would most have changed the operator's experience. + +### Optional: surface the warmup boundary in the digest + +When a completed study ran fewer trials than the warmup floor, the digest's narrative (or a small banner on the proposal) notes that the result is pre-convergence and that the right next step is *re-running with a larger budget*, not necessarily accepting a narrow/widen follow-up. This stops the misattribution at the point where the operator reads the result. + +## Scope signals + +- **Backend:** small. A warmup-floor constant + a create-time validation warning surfaced through the existing study-create response (warning, not error — does not block). Optionally a digest-narrative note when `trials_run < floor`. No migration (presets write existing `config` keys). +- **Frontend:** moderate. Preset selector component in the create-study wizard; inline sub-warmup warning; "advanced/custom" disclosure. Grounds the preset values in a backend-exposed constant per the Enumerated Value Contract Discipline so the wizard and backend can't drift. +- **Migration:** none. +- **Config:** none required; the warmup floor could be an optional `STUDIES_TPE_WARMUP_FLOOR` setting with a sane default. +- **Audit events:** N/A (MVP2 is pre-`audit_log`). + +## Why not just tell people to set max_trials higher? + +Because the data shows they won't, and the tool gives them no reason to. The whole point of the Karpathy-loop framing is that the operator shouldn't have to be a Bayesian-optimization expert to get a converged result — "tireless and structured" is the *tool's* job ([blog: haystack-to-relyloop](../../../../blog/2026-05-20-haystack-to-relyloop.md)). A defaulted, warmup-aware budget is the cheapest way to make the loop deliver on that promise. + +## Relationship to other work + +- **Sibling in the "overnight autopilot ergonomics" theme:** [`feat_overnight_autopilot`](../feat_overnight_autopilot/idea.md) (surfaces the already-shipped `auto_followup_depth` autonomous chaining as a first-class wizard toggle + morning results summary) and [`feat_study_convergence_indicator`](../feat_study_convergence_indicator/idea.md) (shows when a study actually plateaued, so the operator can tell whether the budget was enough). The three compose: presets prevent under-budgeting, the convergence indicator confirms it worked, and overnight-autopilot makes the whole thing unattended. +- **Composes with the MVP2 UBI anchor** ([`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)): a converged overnight study is far more valuable scored against a fresh UBI judgment list than a 12-trial random sample against a static LLM snapshot. +- **Adjacent to the shipped** [`feat_auto_followup_studies`](../../../implemented_features/2026_05_24_feat_auto_followup_studies/) and [`feat_create_study_search_space_builder`](../../../implemented_features/2026_05_20_feat_create_study_search_space_builder/) — this idea changes the *budget* surface of the same wizard, not the search-space surface. + +## Open questions for /spec-gen + +1. Exact preset trial counts + whether the overnight preset also bumps `parallelism`. +2. Warmup floor: fixed constant vs derived from the configured `n_startup_trials` (if it's configurable per study). +3. Whether the digest convergence note belongs here or in [`feat_study_convergence_indicator`](../feat_study_convergence_indicator/idea.md) (avoid double-owning it). diff --git a/docs/00_overview/planned_features/02_mvp2/feat_study_convergence_indicator/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_study_convergence_indicator/idea.md new file mode 100644 index 00000000..31e3ee5d --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/feat_study_convergence_indicator/idea.md @@ -0,0 +1,60 @@ +# Study convergence indicator — "did this study actually finish learning, or did I stop it too early?" + +**Date:** 2026-05-29 +**Status:** Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. +**Priority:** P2 — valuable feedback loop, but lower-leverage than fixing the defaults. Best landed alongside [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md). +**Origin:** Same dogfooding trace that found 6 of 7 studies ran 12–15 trials (well under the TPE ~10-trial warmup). The operator had no on-screen signal that those studies stopped before the optimizer converged — so "should I have run more trials?" was unanswerable from the UI, and the digest's narrow/widen follow-ups filled the void as the apparent next step. +**Depends on:** MVP1 study lifecycle + trials persistence (shipped). Independent of the MVP2 anchors. The `trials` table already stores per-trial metric + `optuna_trial_number`, which is all the raw material a convergence view needs. + +## Problem + +After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the difference between "this is a real answer" and "I stopped the optimizer mid-climb." Concretely: + +- A study that ran 12 trials and whose best-so-far metric was still improving at trial 11 is almost certainly under-budgeted — more trials would help. The operator should re-run with a larger budget, not accept a narrow/widen follow-up. +- A study whose best-so-far metric flattened 200 trials ago genuinely converged — a follow-up that *narrows* might find a little more, but the big win is banked. + +Today these two cases look identical in the UI. The operator can't distinguish "the optimizer is done" from "the optimizer was just getting started," so they can't tell whether the friction they feel (needing follow-ups) is real or self-inflicted. This is the feedback gap that makes [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md) hard to reason about without — presets prevent under-budgeting, this indicator *confirms* the budget was enough. + +## Proposed capabilities + +### Best-so-far convergence curve on the study detail page + +Plot best-metric-so-far against `optuna_trial_number` (a monotonic non-decreasing curve for a maximize study). The shape tells the story at a glance: still-rising tail = under-budgeted; long flat tail = converged. The raw data is already in the `trials` table; this is a read-side aggregation + a Recharts line (the UI already uses Recharts for parameter-importance and trial-scatter). + +### A plain-language convergence verdict + +A small badge / one-liner derived from the curve, e.g.: + +- **"Converged"** — best metric flat for the last N trials (no improvement beyond epsilon). +- **"Still improving when it stopped"** — best metric improved within the last N trials → suggest re-running with a larger budget. +- **"Too few trials to tell"** — ran below the TPE warmup floor → the result is effectively random search; re-run with ≥ Standard budget. + +The verdict is the operator-facing payoff — it answers "was this enough?" without making them read a chart. Pure-domain logic over the trial series (testable without fixtures). + +### Wire the verdict into the digest / proposal framing + +When the verdict is "still improving" or "too few trials," the proposal surfaces "re-run with a larger budget" as the recommended next step *ahead of* the narrow/widen follow-ups — correcting the misattribution where an under-budgeted study's follow-ups look like the intended workflow. + +## Scope signals + +- **Backend:** small. A pure-domain convergence classifier over the ordered trial-metric series + a read-side endpoint/field to expose best-so-far series + verdict. No migration (reads existing `trials`). +- **Frontend:** moderate. One Recharts line on the study detail page + a verdict badge. Reuses existing chart infrastructure. +- **Migration:** none. +- **Config:** none (epsilon + "last N trials" window are constants, possibly shared with the auto-followup lift epsilon for consistency). +- **Audit events:** N/A (pre-`audit_log`). + +## Why deferred / not inline + +It's a genuine new analysis surface (a classifier + a chart + digest wiring), not a one-liner, and it's most useful *with* the budget presets — alone it diagnoses a problem the presets are meant to prevent. Sequencing it next to presets means the operator both avoids under-budgeting and can verify they did. + +## Relationship to other work + +- **Sibling theme:** [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md) (prevents under-budgeting; this confirms it worked) and [`feat_overnight_autopilot`](../feat_overnight_autopilot/idea.md) (each chain link gets a convergence verdict, so the morning summary can flag "link 2 was still improving — the chain may have stopped one budget short"). +- **Composes with the shipped** [`feat_pr_metric_confidence`](../../../implemented_features/2026_05_21_feat_pr_metric_confidence/) — convergence is a natural input to the PR-body confidence framing ("converged after 340 trials" is a stronger claim than "best of 12"). +- **Reuses** the parameter-importance + trial-scatter Recharts surfaces from [`feat_digest_proposal`](../../../implemented_features/2026_05_11_feat_digest_proposal/). + +## Open questions for /spec-gen + +1. Convergence definition: trailing-window-flat vs slope-of-best-so-far vs Optuna's own improvement signal — pick one defensible classifier. +2. Whether the "re-run with larger budget" recommendation is owned here or in [`feat_study_budget_presets`](../feat_study_budget_presets/idea.md)'s digest note (avoid double-ownership). +3. Whether the convergence curve is always shown or only when the verdict is non-trivial. diff --git a/docs/01_architecture/mvp2-overview.md b/docs/01_architecture/mvp2-overview.md new file mode 100644 index 00000000..5a4d843f --- /dev/null +++ b/docs/01_architecture/mvp2-overview.md @@ -0,0 +1,285 @@ +# MVP2 Overview — "Three-Engine + Real Signals" + +**Status:** Planning. MVP1 (v0.1, "The Loop") shipped; the `01_mvp1/` backlog is fully drained. This page is the MVP2 release plan: goal, scope, an organized feature list with story stubs, sequencing, and exit criteria. It is a **navigation + planning** doc — the per-feature contracts live in each feature's `feature_spec.md` (produced by `/spec-gen` from the cited `idea.md`). + +**Canonical sources this page derives from:** +- Product framing: [`docs/00_overview/relyloop-spec.md` §27 "MVP2 / v0.2"](../00_overview/relyloop-spec.md) (lines 2275–2326) + §8 SolrAdapter + §14 UBI judgments. +- Release matrix (authoritative; wins on any conflict): [`tech-stack.md` §"Canonical release matrix"](tech-stack.md). +- Live status board: [`MVP2_DASHBOARD.md`](../00_overview/MVP2_DASHBOARD.md) (regenerated from folder state). +- Sibling reading guide: [`mvp1-overview.md`](mvp1-overview.md). + +> If a statement here conflicts with the canonical release matrix in `tech-stack.md`, the matrix wins — flag the drift in your PR. + +--- + +## 1. Goal + +**Make the engine-neutral claim verifiable, and let judgments come from real users instead of only an LLM.** + +MVP1 proved the loop on Elasticsearch + OpenSearch with LLM-as-judge as the only authoritative judgment source. Two gaps remain before the positioning in [the umbrella spec §1](../00_overview/relyloop-spec.md) and [`docs/07_research/comparison.md`](../07_research/comparison.md) is *factual* rather than *rhetorical*: + +1. **"Engine-neutral" is aspirational with only two engines.** Elasticsearch, OpenSearch, and Apache Solr are the three engines the OSC / Sease / Querqy / Haystack community treats as the canonical OSS search stack. Supporting all three makes "works wherever you are" demonstrable. +2. **LLM-as-judge is a weaker trust anchor than real behavior.** For operators with production traffic, ratings derived from clicks + dwell + conversions reflect what users *find* relevant, not what an LLM *guesses* should be relevant. The optimization loop's quality ceiling is the judgment list's quality — replacing that ceiling is the single biggest believability upgrade RelyLoop can ship. + +MVP2 closes both in one release because they tell one coherent story and because **UBI on Solr is free once the adapter lands**: Solr's first-party `solr.UBIComponent` writes the same `ubi_queries` + `ubi_events` schema as the OpenSearch UBI plugin, so the engine-agnostic `UbiReader` works on Solr unchanged from day one. + +## 2. Headline + +> **RelyLoop runs on all three OSS engines (Elasticsearch, OpenSearch, Apache Solr) with UBI-derived judgments on every one of them — plus a hybrid UBI+LLM converter no competitor ships.** + +This bundle delivers **four of RelyLoop's six differentiators**: all three OSS engines + the hybrid UBI+LLM judgment source. (The other two — the Bayesian full-search-space loop and the Git-PR apply path — shipped in MVP1.) + +## 3. Audience expansion + +- Apache Solr operators (the OSC + Sease + Querqy + Quepid/Chorus community, predominantly Solr-native). +- Operators with production search traffic and UBI logging enabled on any of the three engines. +- Operators who distrust LLM-as-judge as the only trust anchor. +- **Operators who do NOT yet collect user signals** — the large majority of OSS-search deployments. MVP2 must make them *more* capable, never less, and give them a clear path to UBI when they're ready. + +## 4. Design principle — no-UBI operators stay first-class (the UBI on-ramp) + +UBI is **progressive enhancement, not a gate.** The majority of operators have no UBI plugin installed and no clickstream when they first run RelyLoop. MVP2 must leave that experience strictly better than MVP1, and turn every UBI touchpoint into an on-ramp rather than a wall. Four tenets: + +1. **LLM-as-judge remains the zero-config default.** Nothing in MVP2 degrades the no-UBI path. An operator who never enables UBI sees exactly the MVP1 experience, plus better guidance. Every UBI surface degrades gracefully to the LLM path — never to an error the operator can't act on. +2. **The tool detects readiness and nudges — it never acts on the cluster.** RelyLoop **never installs the UBI plugin, never writes to the cluster, never modifies schema** (consistent with the umbrella spec §4 non-goals). Nudges are *guidance*: detect that `ubi_queries` is absent or sparse, then point the operator at the engine-specific enablement runbook. The three engines have three different enablement paths (OpenSearch UBI plugin, o19s ES fork, Solr first-party `solr.UBIComponent`) — the nudge is engine-aware. +3. **Sparse UBI is a recommendation, not a failure.** An operator with *some* traffic is steered into hybrid mode ("UBI rates your dense head; LLM fills the tail") and told concretely what they'd gain by collecting more — not bounced with a 422. +4. **Show the value, don't just assert it.** The strongest nudge is the delta: "this UBI list covered 90% of last week's real traffic" beside "the previous LLM list rated 500 pairs on a snapshot." Surface coverage and, where a prior LLM list exists on the same query set, the metric delta — that's the moment a no-UBI operator decides UBI is worth enabling. + +### UBI readiness ladder + +Each cluster sits on a rung; the tool recommends the right judgment mode for the rung and nudges toward the next. + +| Rung | State (detected via `get_schema` probe of `ubi_queries`) | Recommended mode | Nudge | +|---|---|---|---| +| 0 — No UBI | `ubi_queries` absent | **LLM-as-judge** (unchanged from MVP1) | "Enable real user signals" card → engine-specific runbook. Non-blocking, dismissible. | +| 1 — Installed, sparse | `ubi_queries` present, below `min_impressions_threshold` for most pairs | **Hybrid UBI+LLM** (UBI head + LLM tail) | "You have early signal — here's how much more traffic strengthens it." Show current coverage %. | +| 2 — Dense head | enough impressions on the head; long tail still thin | **Hybrid UBI+LLM** (default) | "Most adopters ship from here." Surface head/tail split. | +| 3 — Full coverage | dense across the query set | **UBI threshold converter** (CTR or dwell) | "Counterfactual click models (CCM/DBN) become viable — post-MVP2." | + +This ladder is the spine of Workstream B's UX stories (B10–B13 below). It is also why the `HybridUbiLlmConverter` is the *default* recommended converter, not the conservative CTR one — hybrid is the rung most real operators occupy. + +## 5. Definition of done (release exit criteria) + +MVP2 ships when **all** of the following hold: + +- [ ] A relevance engineer can register an Apache Solr cluster (9.x or 10.x, SolrCloud or standalone) and run the full loop — register → query set → judgments → study → digest → PR — end-to-end, proven by an automated E2E test against a live Compose `solr` service. +- [ ] The `SearchAdapter` conformance/contract suite passes for Solr on every method ES + OpenSearch already pass. +- [ ] `POST /api/v1/judgment-lists/generate-from-ubi` and the `generate_judgments_from_ubi` agent tool produce mixed-source judgment lists on **all three** engines. +- [ ] The hybrid UBI+LLM converter produces a mixed `source='click'` + `source='llm'` list, and calibration stats roll up across the source mix. +- [ ] One Alembic migration extends the `clusters.engine_type` + `auth_kind` CHECK constraints to accept Solr values, round-trips cleanly (`upgrade → downgrade -1 → upgrade`), and **no other schema migration is required** (UBI rides the existing `judgments.source = 'click'` enum). +- [ ] Coverage gate holds at the MVP1 bar (80% backend) across all new code; every new endpoint has a contract test, every new service an integration test, every new domain function a unit test, the Solr loop an E2E test. +- [ ] Two new runbooks (`solr-cluster-registration.md`, `ubi-judgment-generation.md`) and the tutorial extensions (Path C "run against Solr", Step 7 "swap LLM judgments for UBI") are published. +- [ ] **The no-UBI path is provably unchanged:** an operator with no `ubi_queries` index runs the full loop on LLM-as-judge with zero new friction (regression-tested), and is shown an engine-aware "enable real user signals" nudge rather than a dead-end error. +- [ ] **Sparse UBI degrades to a recommendation:** a cluster below the impression threshold is steered into hybrid mode (or LLM fallback) with a concrete "collect more" message — never a hard 422 with no next step. +- [ ] **Run-depth is a one-action choice:** the create-study wizard offers a clearly-labeled fast path (Quick, ~minutes, for demos/testing) and a deep path (Overnight, ~1000 trials + autonomous compounding) from a single screen; a Custom budget below the TPE warmup floor warns the operator it won't converge. +- [ ] **The overnight path produces a reviewable result from one action:** selecting Overnight runs a deep study that auto-compounds while unattended and, on return, presents a single summary of what ran and the best config found — one click from a PR, with no production change made without the operator. + +## 6. Sequencing — do we gate MVP2 on the in-bucket bugs/chores? + +**No.** Unlike the MVP1 drain (which reconciled a *shipped* release's leftovers), the 6 non-feature items in `02_mvp2/` were deliberately scoped *into* this release. None of them block the two anchors, and several are best done *during/after* the anchors because they harden or test exactly what the anchors add. They are organized below as **Workstream F (Hardening & test debt)** and fold in as the relevant surfaces land — not as a pre-flight gate. + +**One standalone exception:** `bug_webhook_concurrent_merge_race_timing_sensitive` is a *real* correctness bug (the row-lock does not actually guarantee the newer-timestamp winner), not a latent-cosmetic one, and MVP2 may add lifespan startup tasks that trip it. It's a good "fix early as independent hygiene" candidate — do it first within Workstream F, but it still doesn't gate the anchors. + +**Recommended build order:** A (Solr) ∥ B (UBI) in parallel → C (three-engine enablement, depends on A) → **G (run-depth ergonomics — quick-vs-overnight; independent, high operator value, can start immediately)** → D/E (chat + search UX, independent, fill gaps) → F (hardening, folds in continuously; webhook-race first). The on-ramp UX stories (B10–B13) ship *with* B, not after — they are what protect the no-signals majority. + +--- + +## 7. Workstreams & feature list + +Six workstreams. Each lists its goal, the source `idea.md`, story stubs (outline-level — full acceptance criteria come from `/spec-gen`), and the scope signals that matter for planning (migration? new deps? priority). + +> **Anchors** = A + B. They are the release. Everything else is supporting, polish, or hardening. + +### Workstream A — Apache Solr adapter (ANCHOR) · P1 + +**Goal:** a complete `SearchAdapter` implementation for Apache Solr 9.x + 10.x (SolrCloud + standalone) so the loop runs unchanged on a third engine. +**Source:** [`infra_adapter_solr/idea.md`](../00_overview/planned_features/02_mvp2/infra_adapter_solr/idea.md) · **Spec refs:** [spec §8 SolrAdapter](../00_overview/relyloop-spec.md), [adapters.md §Cross-engine parameter naming](adapters.md). +**Migration:** one (extends `engine_type` + `auth_kind` CHECK constraints; no new tables). **New Compose service:** `solr` (`solr:10`, Apache 2.0 image, `127.0.0.1:8983`). **Est:** ~2–3 engineer-weeks (~1,200 LOC backend, ~100 LOC frontend). + +Story stubs: +- **A1 — Adapter skeleton + capability probe.** New `backend/app/adapters/solr.py`; on construction, probe Solr version, SolrCloud-vs-standalone, presence of `solr.UBIComponent`, presence of the `ltr` module; persist to `clusters.engine_config` JSONB for the search-space validator to consult. +- **A2 — `render` for `edismax`/`dismax`/`lucene`.** Emit a Solr request-parameter dict from the unified vocabulary; add `templates/solr/` Jinja templates mirroring `templates/elasticsearch/` shape; make the parameter-map's third column (the documented Solr mappings) real implementation — including richer `mm` arithmetic syntax and the `bf`-vs-`boost` additive/multiplicative split driven by `boost_fn.combine`. +- **A3 — `search_batch`.** Parallel `/select` requests over a connection pool sized by the existing `HTTPX_POOL_LIMITS` (Solr has no `_msearch` equivalent). +- **A4 — `get_schema` + `list_targets`.** Schema API (`/schema/fields|dynamicfields|fieldtypes`) → `Schema` type unchanged; CoresAdmin (standalone) / CollectionsAdmin (SolrCloud) for target listing, selected by the A1 probe. +- **A5 — `explain`.** `debugQuery=true&debug=results`, parse the `debug.explain` block. +- **A6 — Auth + migration.** Implement `solr_basic` (HTTP Basic) and `solr_apikey` (Solr 9+ JWT via `JWTAuthPlugin`); the one Alembic migration extending the `engine_type` + `auth_kind` CHECK constraints (with downgrade + round-trip per Rule #5). +- **A7 — LTR rescore (consume-only).** Render unified `rerank_model:{id,top_k}` to `rq={!ltr model=… reRankDocs=…}` applying a pre-existing `MultipleAdditiveTreesModel` from Solr's `/schema/model-store`. **Training is out of scope** (backlog). +- **A8 — Compose service + sample data.** Add the `solr` service + new optional env vars (`SOLR_HOST`/`SOLR_PORT`/`SOLR_ADMIN_USERNAME_FILE`/`SOLR_ADMIN_PASSWORD_FILE`, `*_FILE` secrets per Rule #2); seed the `products` collection from the existing `samples/products.json`. +- **A9 — Frontend.** Add `solr` to the cluster-registration `engine_type` allowlist (per the Enumerated Value Contract Discipline — ground the option in the backend Literal); Solr auth help text; a Solr engine badge on cluster cards / study headers. +- **A10 — Tests + runbook.** Unit (param rendering, LTR injection, `mm` syntax, probe parsing, error mapping), integration (live Compose Solr; LTR round-trip; UBI reader against seeded indices), contract (Protocol conformance — Solr passes every method ES/OpenSearch pass), E2E (`ui/tests/e2e/solr-study-end-to-end.spec.ts`); new `docs/03_runbooks/solr-cluster-registration.md`; tutorial Step 0 Path C. + +### Workstream B — UBI judgments (ANCHOR) · P1 + +**Goal:** click-derived, engine-agnostic judgments as a first-class source, with a differentiated hybrid UBI+LLM converter. +**Source:** [`feat_ubi_judgments/idea.md`](../00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md) · **Spec refs:** [spec §14 Click-derived judgments](../00_overview/relyloop-spec.md), §19 agent tools, §20 API surface. +**Migration:** **none** (rides the existing `judgments.source IN ('llm','human','click')` CHECK). **Est:** ~2 engineer-weeks (~600 LOC backend, ~150 LOC frontend). + +Story stubs: +- **B1 — `UbiReader` (engine-agnostic read layer).** New `backend/app/services/ubi_reader.py`; read standardized `ubi_queries` + `ubi_events` via any `SearchAdapter.search_batch` (two scrolling searches + client-side join on `query_id`); inputs `cluster_id`, `target`, `since`/`until`, optional `query_filter`, `max_queries` (default 5000). No new adapter method, no engine-specific UBI code. +- **B2 — Feature aggregation.** New `backend/app/domain/ubi/features.py`; per-(query, doc) feature vector: click count, impression count, position-bias-corrected CTR (Wang–Bendersky correction with a configurable prior), post-click dwell-time mean, conversion rate (NULL where not emitted), refinement rate. +- **B3 — `SignalsConverter` Protocol + two threshold converters.** New `backend/app/domain/ubi/converter.py`: the pure-domain Protocol `convert(features) -> ratings(0–3)`; `CtrThresholdConverter` (default, conservative; defaults 0.05/0.15/0.30) and `DwellTimeThresholdConverter` (content-discovery surfaces). +- **B4 — `HybridUbiLlmConverter` (the differentiator).** UBI rates the dense head (`impressions ≥ llm_fill_threshold`, default 20); LLM-as-judge fills the long tail below the threshold; interleave `source='click'` and `source='llm'` rows in one list. This is the operating mode most adopters ship to production (SRW's UBI path uses COEC alone — no hybrid). +- **B5 — API + worker.** `POST /api/v1/judgment-lists/generate-from-ubi` → 202 `{judgment_list_id, status:"generating"}`; new `backend/workers/judgments.py:generate_judgments_from_ubi` Arq job (pull features → run converter → optional LLM fill → INSERT `judgments` with per-row `source` → write `judgment_lists.calibration`); error envelopes `UBI_NOT_ENABLED` (412), `UBI_INSUFFICIENT_DATA` (422), `UBI_QUERY_MAPPING_AMBIGUOUS` (422). +- **B6 — Agent tool + orchestrator prompt.** `generate_judgments_from_ubi(query_set_id, cluster_id, target, since, until?, converter, llm_fill_threshold?)` mirroring `generate_judgments_llm`; orchestrator prefers UBI when the cluster has `ubi_queries` (one-shot `get_schema` probe), falls back to LLM otherwise — the chat ergonomic that earns the release name (agent-first symmetry per spec §21). +- **B7 — Calibration spot-check.** Reuse MVP1's Cohen's-kappa / agreement surface between UBI-derived ratings and a 30–50-row hand-labeled sample; account for source mix. +- **B8 — Frontend.** Source picker (LLM | UBI | Hybrid) + UBI window controls on the judgment-generation modal; insufficient-data empty state on the judgment-list detail page when the converter drops pairs. +- **B9 — Docs.** New `docs/03_runbooks/ubi-judgment-generation.md` (install the plugin, configure capture, choose a converter, calibrate thresholds); tutorial Step 7 ("swap the LLM list for a UBI-derived one" + surface the metric delta). + +**On-ramp UX for the no-signals majority (ship *with* B, per §4):** +- **B10 — UBI readiness probe + surfacing.** Reuse the `get_schema` probe for `ubi_queries` to classify each cluster on the readiness ladder (rung 0–3); expose the rung on cluster detail and as a small badge on cluster cards. Turn the `UBI_NOT_ENABLED` (412) condition from a bare error into a structured, actionable state the UI can render. No cluster writes — read-only detection. +- **B11 — Engine-aware "enable real user signals" nudge.** A dismissible card on the judgment-generation modal and cluster-detail page when UBI is absent (rung 0), with steps specific to the cluster's `engine_type` (OpenSearch UBI plugin / o19s ES fork / Solr `solr.UBIComponent`) and a deep-link to `ubi-judgment-generation.md`. Reuses the shipped `feat_contextual_help` idiom; never blocks the LLM path. Re-surfaces on next visit if dismissed but still unaddressed. +- **B12 — Sparse-data guidance, not a wall.** When UBI is present but below `min_impressions_threshold` (rung 1), the would-be `UBI_INSUFFICIENT_DATA` path instead recommends hybrid mode and shows current coverage ("~12% of your query set has enough signal — hybrid rates that head, LLM fills the rest"). The empty/partial state on the judgment-list detail page explains *why* pairs were dropped and what closes the gap. +- **B13 — Value-delta framing.** On UBI/hybrid list completion, surface coverage stats ("covered N queries / X% of traffic in the window") and, where a prior LLM list exists on the same query set, the metric/coverage delta — the concrete "here's what real signals bought you" moment that converts a no-UBI operator. Feeds naturally into the PR-body confidence framing (composes with the shipped `feat_pr_metric_confidence`). + +> **Deferred (documented, not built):** counterfactual click models (`CcmConverter`, `DbnConverter`) — same Protocol, need more impressions to be statistically valid (rung 3 unlocks them); engine-native readers (e.g. Elastic Behavioral Analytics) feeding the same Protocol. Both are post-MVP2 / backlog. + +### Workstream C — Three-engine enablement (template library + cheatsheets) · P2 + +**Goal:** make "tune any parameter on any engine" practical, not just possible, with a curated template library and per-engine tunable-params cheatsheets — including the new `templates/solr/` shapes. +**Source:** [`chore_template_library_expansion/idea.md`](../00_overview/planned_features/02_mvp2/chore_template_library_expansion/idea.md). **Depends on:** A (so Solr templates can be validated against a live engine). **Migration:** none. (Surfaced as a `chore_` but it ships user-visible content — `/spec-gen` should confirm whether it warrants a `feat_` rename per the naming convention.) + +Story stubs: +- **C1 — Curated template library.** Ship 5–6 templates under `samples/templates/` (basic multi_match, function-score decay, bool-boosted, knn-only, hybrid RRF, phrase rescore), each with `declared_params` descriptions, a `README.md`, and a hand-tuned `default_search_space.json`; install all by default on `make up`. +- **C2 — Per-engine cheatsheets.** `docs/06_vendor_docs/{elasticsearch,opensearch}-tunable-params.md` (≈15–20 knobs each: native + unified name, ranges, "when to tune", caveats, back-links to templates). **Extend to Solr** as part of this release's three-engine framing. +- **C3 — Wizard + tutorial linkage.** Engine-aware cheatsheet deep-links from the search-space glossary; inline template `README` summary in the Step-3 picker; "where to go next" tutorial section. +- **C4 — Cross-engine render smoke tests.** Each template renders against the demo cluster on its target engine(s) and returns non-empty results — reuse the `infra_adapter_elastic` integration harness, extended to Solr. + +### Workstream D — Chat polish · P2 + +**Goal:** two deferred chat-UX items that pair naturally and were explicitly held for MVP2. +**Migration:** one (D2 only). **Independent of the anchors.** + +Story stubs: +- **D1 — Last-message preview.** [`feat_chat_last_message_preview/idea.md`](../00_overview/planned_features/02_mvp2/feat_chat_last_message_preview/idea.md): add `last_message_preview` (120-char, markdown-stripped) + `last_message_at` to `ConversationSummary` via a correlated subquery (no migration, no denormalize); render a muted preview line in `ConversationList`. Open forks (which role, markdown strip, content extraction) have recommended defaults locked in the idea. +- **D2 — Long-conversation summarization.** [`bug_chat_long_conversation_truncation/bug_fix.md`](../00_overview/planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md): wrap the existing position-based truncation with a summarization pre-step that condenses the dropped portion into a system-prefix message (additive — preserves the tool-call-group boundary invariant). Latent bug (fires only >100 messages). Adds a `conversations.summary` JSONB column (migration, reversible) + a summarization prompt template. Three forks (sync vs async timing; budget line; trigger threshold) have recommended defaults; lock them at `/bug-fix` Default-mode entry. + +### Workstream E — Search UX · Backlog→P2 + +**Goal:** rank-ordered full-text search when `?q=` is present. +**Source:** [`feat_fts_rank_ordering/idea.md`](../00_overview/planned_features/02_mvp2/feat_fts_rank_ordering/idea.md). **Independent.** The `tsvector` columns + GIN indexes (`0008`–`0013`) already exist; the work is ordering + cursor encoding. + +Story stubs: +- **E1 — Rank-ordered ordering.** `ORDER BY ts_rank DESC, created_at DESC, id DESC` on the 6 search-enabled list endpoints when `?q=` is set; unchanged otherwise. +- **E2 — Float-safe cursor.** Pick one approach at spec time (rank-bucketed integer cursor vs transient materialized rank column) so keyset pagination survives `ts_rank` boundaries without violating the no-offset/limit rule. +- **E3 — Cursor invalidation + UI.** Invalidate in-flight cursors on `?q=` change (mirror the `?sort=` rule); add a "Sort by relevance" pill to the `` toolbar when `q` is active. + +### Workstream F — Hardening & test debt · mixed (do not gate the anchors) + +**Goal:** close correctness and coverage gaps that MVP2's new surfaces make reachable. Fold in continuously; the webhook race first. + +Story stubs: +- **F1 — Webhook concurrent-merge row-lock (REAL bug; do first).** [`bug_webhook_concurrent_merge_race_timing_sensitive/idea.md`](../00_overview/planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md): the `config_repos.last_merged_proposal_id` update doesn't guarantee the newer-timestamp winner under concurrency; make the compare-and-update race-free in SQL (`WHERE last_updated_at < :new_timestamp` under `SELECT … FOR UPDATE`) + add a regression test that spawns a lifespan task before the concurrent webhooks. Currently masked by an env-var gate; the first MVP2 lifespan task trips it. +- **F2 — Auto-followup parent advisory lock.** [`chore_auto_followup_parent_advisory_lock/idea.md`](../00_overview/planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md): `pg_advisory_xact_lock(hashtext(parent_id))` to serialize concurrent followup-enqueue workers (layer-3 idempotency). Sequence **after** MVP2's autonomous re-trigger paths land so the lock granularity is informed by a real failure shape (its idea says the race is "reachable in MVP2"). Watch for supersession by the unified-advisory-lock Option C in the cascade-race bug. +- **F3 — Demo-seeding async-flow integration tests.** [`chore_demo_seeding_integration_tests_rewrite/idea.md`](../00_overview/planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md): rewrite the 10 skipped sync-contract tests for the async enqueue+poll flow (in-process Arq worker fixture + "POST then poll to terminal" helpers). Closes the coverage gap that let `bug_demo_reseed_button_silent_enqueue_failure` ship. +- **F4 — Studies-POST Arq spy fixture.** [`chore_studies_post_arq_spy_fixture/idea.md`](../00_overview/planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md): a `SpyArqPool` fixture so rejection-path tests can positively assert "no job enqueued." Natural to extend to the new UBI generate endpoint (B5) and the other enqueueing POSTs. +- **F5 — Arq subprocess resume test.** [`infra_arq_subprocess_test/idea.md`](../00_overview/planned_features/02_mvp2/infra_arq_subprocess_test/idea.md): spawn a real `arq … WorkerSettings` subprocess, SIGTERM mid-loop, restart, assert trials resume — a narrow Arq-version + cron-registry regression guard. **Trigger-locked:** ship when the `arq` pin bumps, a third cron lands, or MVP3 hardening opts in. May not fire within MVP2; keep as standby. + +### Workstream G — Run-depth ergonomics: quick-for-demos vs one-click-overnight · G1 = P1 + +**Goal:** make it *obvious and one-action* to choose between a **fast shallow run** (demos, testing, smoke-checking a search space) and a **deep overnight run that compounds automatically and produces a far better result from a single user action**. This is the "set it before I log off, wake up to results worth a PR" experience — and equally the "give me something in 60 seconds for a demo" experience — from the same screen. + +**Why this is in MVP2, not deferred:** an operator dogfooding trace (2026-05-29) found 6 of 7 real studies ran `max_trials` of 12–15 — *below* Optuna TPE's ~10-trial random-search warmup ([`optimization.md`](optimization.md)). Those studies never actually engaged the Bayesian optimizer; they were effectively random search, and the digest's narrow/widen follow-ups were compensating for under-budgeting, which *felt* like mandatory manual iteration. The capability to run deep and compound overnight (`auto_followup_depth`, [shipped](../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/)) had **zero usage** — it was a hidden config key. G fixes the defaults and surfaces the autopilot so the loop delivers a great result from one execution. + +**Two paths, one screen** — the create-study wizard presents a clear choice: + +``` +How deep should this run go? + ( ) Quick look ~30 trials minutes — demos, testing, smoke-check a search space + (•) Standard ~200 trials ~tens of min — the everyday "give me a real answer" + ( ) 🌙 Overnight ~1000 trials + auto-compound — deepest single-action result; review in the morning + ( ) Custom… power users +``` + +Story stubs: +- **G1 — Budget presets + sub-warmup guard · P1.** [`feat_study_budget_presets/idea.md`](../00_overview/planned_features/02_mvp2/feat_study_budget_presets/idea.md): replace the bare `max_trials` input with the Quick / Standard / Overnight / Custom selector (grounded in a backend constant per the Enumerated Value Contract Discipline); warn non-blockingly when a Custom value falls below the TPE warmup floor ("at 12 trials this is essentially random search — use ≥ Standard for a result worth a PR"). No migration. **This is the single change that most directly fixes the dogfooding friction** — and the "Quick look" preset is what makes fast demo/test runs trivial. +- **G2 — One-click overnight autopilot · P2.** [`feat_overnight_autopilot/idea.md`](../00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md): promote the shipped `auto_followup_depth` chaining to a first-class "🌙 Overnight" path — selecting it sets a deep trial budget *and* enables autonomous compounding (narrow around the winner, re-run, repeat; self-terminates on no-lift / depth / budget). Plain-language copy makes the human-approval boundary explicit ("no production change happens without your review — you still open every PR"). Plus a **morning results summary**: a chain-view panel showing each link's best metric, cumulative lift, the best config across the whole chain, and one click to the proposal that carries it. No migration (reads existing `parent_study_id` links + existing config field). +- **G3 — Convergence indicator · P2.** [`feat_study_convergence_indicator/idea.md`](../00_overview/planned_features/02_mvp2/feat_study_convergence_indicator/idea.md): a best-so-far curve + plain verdict ("Converged" / "Still improving when it stopped" / "Too few trials to tell") on the study detail page, so the operator can *see* whether a run went deep enough — and so the proposal recommends "re-run deeper" ahead of narrow/widen when a study stopped early. No migration (reads existing `trials`). + +> **The three compose:** G1 prevents under-budgeting and gives the fast demo path; G2 makes the deep path one action and unattended; G3 confirms depth was sufficient. Together they are the operator-facing answer to "quick when I'm testing, devastatingly deep when I run it overnight." All compose with the MVP2 UBI anchor — an overnight compounding chain against a fresh UBI judgment list is the strongest result RelyLoop can produce. + +--- + +## 8. Story summary (one line each) + +| WS | ID | Story | Type | Migration | Priority | +|---|---|---|---|---|---| +| A | A1 | Solr adapter skeleton + capability probe | infra | — | P1 | +| A | A2 | `render` edismax/dismax/lucene + `templates/solr/` | infra | — | P1 | +| A | A3 | `search_batch` parallel `/select` + pool | infra | — | P1 | +| A | A4 | `get_schema` + `list_targets` (Cores/Collections) | infra | — | P1 | +| A | A5 | `explain` via `debugQuery` | infra | — | P1 | +| A | A6 | Auth (`solr_basic`/`solr_apikey`) + CHECK migration | infra | **1** | P1 | +| A | A7 | LTR rescore injection (consume-only) | infra | — | P1 | +| A | A8 | Compose `solr` service + sample data + env vars | infra | — | P1 | +| A | A9 | Frontend: engine_type option + auth help + badge | feat | — | P1 | +| A | A10 | Tests (unit/integration/contract/E2E) + runbook + Path C | infra | — | P1 | +| B | B1 | `UbiReader` engine-agnostic read layer | feat | — | P1 | +| B | B2 | Feature aggregation (CTR/dwell/conversion/refinement) | feat | — | P1 | +| B | B3 | `SignalsConverter` Protocol + CTR + dwell converters | feat | — | P1 | +| B | B4 | `HybridUbiLlmConverter` (the differentiator) | feat | — | P1 | +| B | B5 | `POST …/generate-from-ubi` + worker + error envelopes | feat | — | P1 | +| B | B6 | `generate_judgments_from_ubi` agent tool + prompt | feat | — | P1 | +| B | B7 | Calibration spot-check across source mix | feat | — | P1 | +| B | B8 | Frontend: source picker + window controls + empty state | feat | — | P1 | +| B | B9 | UBI runbook + tutorial Step 7 | docs | — | P1 | +| B | B10 | UBI readiness probe + ladder surfacing (read-only) | feat | — | P1 | +| B | B11 | Engine-aware "enable signals" nudge card | feat | — | P1 | +| B | B12 | Sparse-data → hybrid recommendation (not a wall) | feat | — | P1 | +| B | B13 | Value-delta framing (coverage + LLM→UBI delta) | feat | — | P1 | +| C | C1 | Curated 5–6 template library | feat | — | P2 | +| C | C2 | Per-engine tunable-params cheatsheets (ES/OS/Solr) | docs | — | P2 | +| C | C3 | Wizard + tutorial linkage | feat | — | P2 | +| C | C4 | Cross-engine render smoke tests | infra | — | P2 | +| D | D1 | Chat last-message preview | feat | — | P2 | +| D | D2 | Long-conversation summarization | bug | **1** | P2 | +| E | E1 | Rank-ordered FTS ordering | feat | — | P2 | +| E | E2 | Float-safe cursor encoding | feat | 0–1 | P2 | +| E | E3 | Cursor invalidation + relevance pill | feat | — | P2 | +| F | F1 | Webhook merge row-lock correctness (real bug) | bug | — | P1 | +| F | F2 | Auto-followup parent advisory lock | chore | — | P2 | +| F | F3 | Demo-seeding async-flow integration tests | chore | — | P2 | +| F | F4 | Studies-POST Arq spy fixture | chore | — | P2 | +| F | F5 | Arq subprocess resume test | infra | — | Backlog/trigger | +| G | G1 | Budget presets (Quick/Standard/Overnight) + sub-warmup guard | feat | — | **P1** | +| G | G2 | One-click overnight autopilot + morning chain summary | feat | — | P2 | +| G | G3 | Study convergence indicator + "re-run deeper" nudge | feat | — | P2 | + +**Migration budget:** **two** new Alembic migrations total (A6 Solr CHECK constraints; D2 `conversations.summary`) — plus possibly one for E2 depending on the cursor approach chosen at spec time. UBI (the headline judgment work) and all of Workstream G require **zero** schema change. + +## 9. Non-goals (explicitly NOT in MVP2) + +Per [spec §27](../00_overview/relyloop-spec.md): + +- **No second observability stack.** Langfuse + ClickHouse + SigNoz + the `audit_log` table land at **MVP3** ("Observable"). +- **No LangGraph / `PostgresSaver` / RFC 7807 / `Idempotency-Key` everywhere** — those are **GA v1**. +- **No multi-Git provider** (GitLab, Bitbucket) — backlog. GitHub remains the only provider. +- **No multi-tenancy** — backlog. Single-tenant through GA v1. +- **No native multi-LLM provider SDKs** (Anthropic, Bedrock, Vertex, Azure) — backlog. OpenAI-compatible endpoints keep working via `OPENAI_BASE_URL`. +- **No LTR training** — MVP2's Solr LTR is consume-only; cross-engine training is backlog. +- **No real-time signal streaming / Path B** — UBI ratings are computed batch-wise at judgment-list creation, strictly offline. +- **No cluster-side actions to enable UBI.** RelyLoop never installs the UBI plugin, never writes the `ubi_queries`/`ubi_events` indices, never modifies the operator's cluster. The on-ramp (§4) is *guidance only* — detection + engine-specific runbook links. Enabling UBI is always the operator's action on their own infrastructure. +- **No Lucidworks Fusion** — dropped outright ([`chore_drop_fusion_scope`](../00_overview/planned_features/chore_drop_fusion_scope/idea.md)). + +## 10. Open questions for `/spec-gen` + +Resolve these when each feature's spec is authored: + +1. **(A) Solr LTR test fixture** — does the E2E load a real `MultipleAdditiveTreesModel` into Compose Solr, or assert the `rq={!ltr …}` render shape only? (A real model makes the E2E heavier but proves the rescore round-trip.) +2. **(B) `UBI_QUERY_MAPPING_AMBIGUOUS` tiebreaker** — when one UBI `user_query` string maps to multiple `query_set` entries, what's the operator-facing disambiguation contract? +3. **(B) Hybrid converter cost accounting** — does the LLM-fill tail draw from the same `openai_daily_budget_usd` line as `generate_judgments_llm`, or its own? +4. **(C) `feat_` vs `chore_` for the template library** — it ships user-visible content; confirm the rename per the naming convention. +5. **(D2) Summarization timing/budget/trigger** — accept the three recommended defaults (sync; same budget line; message-count primary + token-count safety), or revisit? +6. **(E2) Cursor strategy** — rank-bucketed integer cursor vs transient materialized rank column (drives whether E needs a migration). +7. **(B10–B11) Nudge persistence + cadence** — where is "dismissed" state stored (per-cluster row? client localStorage like other contextual-help dismissals?), and does the nudge re-surface on a schedule or only while the underlying readiness rung is unchanged? +8. **(B10) Readiness thresholds** — what impression counts define the rung 1→2→3 boundaries, and are they operator-configurable or fixed defaults for MVP2? +9. **(B13) Delta baseline** — when no prior LLM list exists on the query set, what does the value-delta surface show instead (coverage-only? a one-off LLM spot-rating for comparison? nothing)? +10. **(G1) Exact preset trial counts** — confirm Quick/Standard/Overnight numbers (~30/200/1000?) and whether the Overnight preset also bumps `parallelism` for faster wall-clock; set the sub-warmup floor (fixed constant vs derived from `n_startup_trials`). +11. **(G2) Overnight = preset + autopilot, coupled or independent?** — does selecting the "🌙 Overnight" preset auto-enable `auto_followup_depth` (and at what default depth), or are deep-budget and auto-compound separate toggles? Where does the morning summary live (study-detail chain panel, a `/studies` "ran while away" card, or both)? + +## 11. Where to look next + +- Live status: [`MVP2_DASHBOARD.md`](../00_overview/MVP2_DASHBOARD.md) · roadmap: [`DASHBOARD.md`](../00_overview/DASHBOARD.md) +- Product framing: [`relyloop-spec.md` §27](../00_overview/relyloop-spec.md) +- Release matrix (authoritative): [`tech-stack.md`](tech-stack.md) +- Adapter contract: [`adapters.md`](adapters.md) · data model: [`data-model.md`](data-model.md) · API conventions: [`api-conventions.md`](api-conventions.md) +- MVP1 reading guide (the sibling of this doc): [`mvp1-overview.md`](mvp1-overview.md) From ce0e9381c863b03d02dc318510a93d4bf46bcfc4 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 29 May 2026 08:08:45 -0400 Subject: [PATCH 2/5] docs(mvp2): correct auto_followup_depth accepted range (0-5, not 1-5) Accuracy fix during pre-finalize verification: the StudyConfigSpec validator at schemas.py:645 accepts 0<=depth<=5 (0 = no chaining), so the prose 'when set (1-5)' understated the field's accepted range. Clarified that the validator accepts 0-5 where 0 disables chaining and 1-5 enables it. The wizard-copy mockup '(1-5)' is correct as-is (0 is the off state, not a value the operator picks in a 'compound up to N times' control). Signed-off-by: SoundMindsAI Co-Authored-By: Claude Opus 4.8 (1M context) --- .../planned_features/02_mvp2/feat_overnight_autopilot/idea.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md index 157a632c..199a8c32 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md @@ -10,7 +10,7 @@ The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: -1. **`auto_followup_depth` is a hidden config key.** When set (1–5, [`schemas.py:645`](../../../../../backend/app/api/v1/schemas.py#L645)), a completed study automatically narrows the search space around its winner, decrements the depth, and spawns a child study — **zero human intervention between iterations** ([`backend/workers/auto_followup.py`](../../../../../backend/workers/auto_followup.py)). The chain self-terminates on depth exhaustion, sub-epsilon lift (<0.5%), budget at 80%, or parent failure. This is exactly the operator's "wake up to a few results" ask. But it is not exposed as a first-class control in the create-study wizard — the operator never knew it existed, so all 7 studies ran one-shot. +1. **`auto_followup_depth` is a hidden config key.** When set (the validator accepts `0–5`, where `0` = no chaining, so `1–5` enables it — [`schemas.py:645`](../../../../../backend/app/api/v1/schemas.py#L645)), a completed study automatically narrows the search space around its winner, decrements the depth, and spawns a child study — **zero human intervention between iterations** ([`backend/workers/auto_followup.py`](../../../../../backend/workers/auto_followup.py)). The chain self-terminates on depth exhaustion, sub-epsilon lift (<0.5%), budget at 80%, or parent failure. This is exactly the operator's "wake up to a few results" ask. But it is not exposed as a first-class control in the create-study wizard — the operator never knew it existed, so all 7 studies ran one-shot. 2. **No "what happened overnight" surface.** Even with chaining on, there is no single place that says "here are the 3 studies that ran while you slept, here's the best config each found, here's the cumulative lift, here's the one that's ready to become a PR." The operator would have to piece it together from the studies list + individual proposals. From 8665bc0857e7cf6169bcb0ae93d42cba4d11edb7 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 29 May 2026 08:24:35 -0400 Subject: [PATCH 3/5] docs(mvp2): promote UBI on-ramp (B10-B13) to a first-class idea file + dashboards The no-signals-majority UX (readiness ladder + engine-aware nudge + sparse-data recommendation + value-delta framing) was described in mvp2-overview.md Workstream B but lived only as story stubs inside feat_ubi_judgments. Splits B10-B13 into one coherent feat_ubi_onramp/idea.md (P1, ships with the UBI anchor). Includes lockstep dashboard regen for the new 02_mvp2 folder. Zero migration. Signed-off-by: SoundMindsAI Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/00_overview/MVP2_DASHBOARD.md | 33 +++---- docs/00_overview/mvp2_dashboard.html | 21 ++++- .../02_mvp2/feat_ubi_onramp/idea.md | 89 +++++++++++++++++++ docs/01_architecture/mvp2-overview.md | 2 +- 4 files changed, 124 insertions(+), 21 deletions(-) create mode 100644 docs/00_overview/planned_features/02_mvp2/feat_ubi_onramp/idea.md diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 978ed6ac..0d96aba1 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -15,14 +15,14 @@ Pull from the Idea backlog or capture a new feature spec. | Metric | Value | |---|---| | Scoped items done | **1 / 1** (100%) — feat_/infra_/chore_/epic_ past idea stage | -| Pending work | **14** items (every not-done feat/infra/chore/bug across all priorities) | +| Pending work | **15** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | -| → P1 | **3** high-value, ready when P0 clears | +| → P1 | **4** high-value, ready when P0 clears | | → P2 (default) | 7 important to file, not blocking | | → Backlog | 4 captured for record, not planned | | Open bugs | 2 | | Legacy "Path to MVP2" | 6 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 8 idea-only feat/infra (not yet scoped into MVP2) | +| Backlog ideas | 9 idea-only feat/infra (not yet scoped into MVP2) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -45,24 +45,25 @@ _None._ _None._ -### Idea (14) +### Idea (15) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| | 1 | P1 | [feat_study_budget_presets](planned_features/02_mvp2/feat_study_budget_presets/idea.md) | Feature | Three connected gaps make under-budgeting the default outcome: | — | Idea — surfaced from an operator dogfooding review of MVP1 studies (2026-05-29). Highest-leverage item in the "overnight autopilot ergonomics" theme. | | 2 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/idea.md) | Feature | MVP1 ships with **LLM-as-judge** as the only authoritative judgment source. The architecture anticipated this would change — the `judgments.source` CHECK already accepts `click`… | — | Idea — bundled with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) into MVP2 / v0.2 "Three-Engine + Real Signals" | -| 3 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | -| 4 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../feature_templates/README.md). | -| 5 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | -| 6 | P2 | [feat_study_convergence_indicator](planned_features/02_mvp2/feat_study_convergence_indicator/idea.md) | Feature | After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. | -| 7 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | -| 8 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | -| 9 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| 10 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | -| 11 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | -| 12 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | -| 13 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | -| 14 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | +| 3 | P1 | [feat_ubi_onramp](planned_features/02_mvp2/feat_ubi_onramp/idea.md) | Feature | `feat_ubi_judgments` adds click-derived judgments, but a UBI-centric release has a well-known failure mode: it makes the operators who *don't* have UBI feel like second-class citizens, when in reality | — | Idea — split out from [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) as a first-class feature (2026-05-29) so the no-signals-majority UX is dashboard-visible and not treated as optional UBI polish. | +| 4 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | +| 5 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../feature_templates/README.md). | +| 6 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | +| 7 | P2 | [feat_study_convergence_indicator](planned_features/02_mvp2/feat_study_convergence_indicator/idea.md) | Feature | After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. | +| 8 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | +| 9 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | +| 10 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | +| 11 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | +| 12 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | +| 13 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | +| 14 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | +| 15 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | ## Dependency graph diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index e7b96810..483584f2 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -403,7 +403,7 @@

MVP2 Progress

Pending work
-
14
+
15
every not-done feat/infra/chore/bug across all priorities
@@ -420,7 +420,7 @@

MVP2 Progress

P1
-
3
+
4
high-value, ready when P0 clears
@@ -442,7 +442,7 @@

MVP2 Progress

Backlog ideas: - 8 idea-only feat/infra folders (not yet scoped into MVP2) + 9 idea-only feat/infra folders (not yet scoped into MVP2) In flight: @@ -463,7 +463,7 @@

Pipeline

-

Idea 14

+

Idea 15

Study Budget Presets
@@ -491,6 +491,19 @@

Idea 14

+
+
Ubi Onramp
+
+ Feature + P1 + +
+
`feat_ubi_judgments` adds click-derived judgments, but a UBI-centric release has a well-known failure mode: it makes the operators who *don't* have UBI feel like second-class citizens, when in reality
+ + +
+ +
Adapter Solr
diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_onramp/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_ubi_onramp/idea.md new file mode 100644 index 00000000..d95a112a --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/feat_ubi_onramp/idea.md @@ -0,0 +1,89 @@ +# UBI on-ramp — keep no-signals operators first-class and nudge them toward real user signals + +**Date:** 2026-05-29 +**Status:** Idea — split out from [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) as a first-class feature (2026-05-29) so the no-signals-majority UX is dashboard-visible and not treated as optional UBI polish. +**Priority:** P1 — ships *with* the UBI anchor, not after. The majority of OSS-search deployments have no UBI plugin installed; this feature is what keeps MVP2 a strict upgrade for them rather than a release aimed only at the traffic-rich minority. See [`mvp2-overview.md` §4 "Design principle — no-UBI operators stay first-class"](../../../../01_architecture/mvp2-overview.md). +**Origin:** Operator review of the MVP2 plan (2026-05-29) — explicit requirement that UBI work "still has great support for apps that do not yet collect user signals" and "nudges the user to help them know how to improve." These were the B10–B13 story stubs inside `feat_ubi_judgments`; promoted to their own folder per an explicit granularity decision. +**Depends on:** [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) (the `UbiReader` / `SignalsConverter` / `generate-from-ubi` substrate this feature surfaces and degrades gracefully from). The engine-aware nudge (capability 2) covers all three engines only **after** [`infra_adapter_solr`](../infra_adapter_solr/idea.md) extends `engine_type` to accept `solr` — until then it is ES + OpenSearch, exactly like the rest of MVP1. + +## Problem + +`feat_ubi_judgments` adds click-derived judgments, but a UBI-centric release has a well-known failure mode: it makes the operators who *don't* have UBI feel like second-class citizens, when in reality they are the majority. Three concrete gaps if the on-ramp isn't built: + +1. **The `UBI_NOT_ENABLED` (412) error is a dead-end, not an on-ramp.** A no-UBI operator who tries the UBI path gets a bare error code at the exact moment of maximum intent. There is no detection-and-guidance surface that says "here's how to turn on real user signals on *your* engine." + +2. **Sparse UBI degrades into a hard failure (`UBI_INSUFFICIENT_DATA`, 422) rather than a recommendation.** An operator with *some* traffic should be steered into hybrid mode ("UBI rates the dense head, LLM fills the tail") and told concretely what they'd gain by collecting more — not bounced. + +3. **The value of real signals is asserted, never shown.** The strongest nudge is the delta — "this UBI list covered 90% of last week's real traffic" beside "the previous LLM list rated 500 pairs on a snapshot." Without it, a no-UBI operator has no concrete reason to invest in enabling UBI. + +A hard constraint frames all three: **RelyLoop never installs the plugin, never writes to the cluster, never modifies schema** (umbrella spec §4 non-goals). Every nudge is *guidance only* — detection plus engine-specific runbook links. Enabling UBI is always the operator's action on their own infrastructure. + +## UBI readiness ladder + +Each cluster sits on a rung (detected read-only via a `get_schema` probe for the `ubi_queries` index); the tool recommends the right judgment mode and nudges toward the next rung. + +| Rung | State | Recommended mode | Nudge | +|---|---|---|---| +| 0 — No UBI | `ubi_queries` absent | **LLM-as-judge** (unchanged from MVP1) | "Enable real user signals" card → engine-specific runbook. Non-blocking, dismissible. | +| 1 — Installed, sparse | `ubi_queries` present, below `min_impressions_threshold` for most pairs | **Hybrid UBI+LLM** (UBI head + LLM tail) | "You have early signal — here's how much more traffic strengthens it." Show coverage %. | +| 2 — Dense head | enough impressions on the head; long tail thin | **Hybrid UBI+LLM** (default) | "Most adopters ship from here." Show head/tail split. | +| 3 — Full coverage | dense across the query set | **UBI threshold converter** (CTR or dwell) | "Counterfactual click models (CCM/DBN) become viable — post-MVP2." | + +This ladder is *why* `HybridUbiLlmConverter` is the default recommended converter (the rung most real operators occupy), not the conservative CTR one. + +## Proposed capabilities + +### 1. Readiness probe + ladder surfacing (read-only) — was B10 + +- Reuse the `get_schema` probe for `ubi_queries` to classify each cluster on the rung 0–3 ladder; expose the rung on cluster detail and as a small badge on cluster cards. +- Turn the `UBI_NOT_ENABLED` (412) condition (defined in [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) §"API surface") from a bare error into a structured, renderable state the UI consumes. +- **No cluster writes** — detection only. + +### 2. Engine-aware "enable real user signals" nudge — was B11 + +- A dismissible card on the judgment-generation modal and cluster-detail page when UBI is absent (rung 0), with steps specific to the cluster's `engine_type`: + - **OpenSearch** → OpenSearch UBI plugin + - **Elasticsearch** → o19s ES UBI fork + - **Solr** → first-party `solr.UBIComponent` (available once [`infra_adapter_solr`](../infra_adapter_solr/idea.md) lands; `engine_type` accepts only `elasticsearch | opensearch` today per [`backend/app/db/models/cluster.py:30`](../../../../../backend/app/db/models/cluster.py#L30)) +- Deep-links to the new `docs/03_runbooks/ubi-judgment-generation.md` (authored in `feat_ubi_judgments` B9). +- Reuses the shipped contextual-help idiom — [`feat_contextual_help`](../../../implemented_features/2026_05_15_feat_contextual_help/) ([`ui/src/components/common/help-popover.tsx`](../../../../../ui/src/components/common/help-popover.tsx)) — and the existing client-side dismissal pattern ([`ui/src/hooks/use-local-storage-set.ts`](../../../../../ui/src/hooks/use-local-storage-set.ts), as used by [`demo-data-banner.tsx`](../../../../../ui/src/components/dashboard/demo-data-banner.tsx)). +- **Never blocks the LLM path.** Re-surfaces on next visit if dismissed but the underlying rung is still 0. + +### 3. Sparse-data guidance, not a wall — was B12 + +- When UBI is present but below `min_impressions_threshold` (rung 1), the would-be `UBI_INSUFFICIENT_DATA` (422) path instead **recommends hybrid mode** and shows current coverage ("~12% of your query set has enough signal — hybrid rates that head, LLM fills the rest"). +- The empty/partial state on the judgment-list detail page explains *why* pairs were dropped and what closes the gap. + +### 4. Value-delta framing — was B13 + +- On UBI/hybrid list completion, surface coverage stats ("covered N queries / X% of traffic in the window"). +- Where a prior LLM list exists on the same query set, show the metric/coverage delta — the concrete "here's what real signals bought you" moment. +- Feeds the PR-body confidence framing (composes with shipped [`feat_pr_metric_confidence`](../../../implemented_features/2026_05_21_feat_pr_metric_confidence/)): "scored against 50,000 UBI-derived ratings covering 90% of last week's traffic" is a far stronger claim than "500 LLM ratings on a snapshot." + +## Scope signals + +- **Backend:** small. The readiness classifier is a read-side wrapper over the same `get_schema` probe `feat_ubi_judgments` already uses; turning `UBI_NOT_ENABLED` / `UBI_INSUFFICIENT_DATA` into structured states (vs. bare errors) is a response-shape change, not new core logic. No new cluster-write path. +- **Frontend:** moderate — the bulk of this feature. Rung badge + readiness surfacing; the engine-aware nudge card; sparse-data recommendation copy; the value-delta surface on list completion. +- **Migration:** **none.** Pure read-side detection + UX over the existing `judgments` substrate (rides `feat_ubi_judgments`, which is itself zero-migration). +- **Config:** the `min_impressions_threshold` / rung boundaries are constants (possibly an optional setting); no required new config. +- **Audit events:** N/A (MVP2 is pre-`audit_log`). + +## Why split from `feat_ubi_judgments` (not left as story stubs) + +- **Dashboard visibility.** As stubs inside the anchor's idea, the no-signals-majority UX is invisible to `/pipeline status` and the MVP2 dashboard, and risks being treated as optional polish at implementation time. As a P1 folder it is tracked and sequenced explicitly. +- **Clean ownership boundary.** `feat_ubi_judgments` owns the *machinery* (reader, converters, endpoint, agent tool); this feature owns the *no-UBI / partial-UBI operator experience* on top of it. Two reviewable units, one coherent each. +- **Single coherent system, one file.** Capabilities 1–4 share one detection mechanism (the `ubi_queries` probe), one dismissal-state question, and one engine-aware-copy concern — so they belong in one idea, not four. + +## Relationship to other work + +- **Depends on / surfaces** [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) (the anchor) — ships in the same release, sequenced together. +- **Engine-aware nudge spans all three engines only after** [`infra_adapter_solr`](../infra_adapter_solr/idea.md). +- **Reuses** [`feat_contextual_help`](../../../implemented_features/2026_05_15_feat_contextual_help/) (the nudge idiom + dismissal pattern). +- **Feeds** [`feat_pr_metric_confidence`](../../../implemented_features/2026_05_21_feat_pr_metric_confidence/) (value-delta → PR confidence framing). +- **Tracked in** [`mvp2-overview.md` Workstream B](../../../../01_architecture/mvp2-overview.md) (capabilities 1–4 = the former B10–B13). + +## Open questions for /spec-gen + +1. **Nudge persistence + cadence** — store "dismissed" client-side (localStorage, like `demo-data-banner`) or per-cluster server-side? Re-surface on a schedule or only while the readiness rung is unchanged? +2. **Readiness thresholds** — what impression counts define the rung 1→2→3 boundaries; operator-configurable or fixed defaults for MVP2? +3. **Value-delta baseline** — when no prior LLM list exists on the query set, what does capability 4 show (coverage-only? a one-off LLM spot-rating for comparison? nothing)? diff --git a/docs/01_architecture/mvp2-overview.md b/docs/01_architecture/mvp2-overview.md index 5a4d843f..d8419b86 100644 --- a/docs/01_architecture/mvp2-overview.md +++ b/docs/01_architecture/mvp2-overview.md @@ -125,7 +125,7 @@ Story stubs: - **B8 — Frontend.** Source picker (LLM | UBI | Hybrid) + UBI window controls on the judgment-generation modal; insufficient-data empty state on the judgment-list detail page when the converter drops pairs. - **B9 — Docs.** New `docs/03_runbooks/ubi-judgment-generation.md` (install the plugin, configure capture, choose a converter, calibrate thresholds); tutorial Step 7 ("swap the LLM list for a UBI-derived one" + surface the metric delta). -**On-ramp UX for the no-signals majority (ship *with* B, per §4):** +**On-ramp UX for the no-signals majority (ship *with* B, per §4)** — owned by the dedicated [`feat_ubi_onramp/idea.md`](../00_overview/planned_features/02_mvp2/feat_ubi_onramp/idea.md) (split from `feat_ubi_judgments` 2026-05-29 so the no-signals-majority UX is first-class and dashboard-visible). The four stories below map 1:1 to its four capabilities: - **B10 — UBI readiness probe + surfacing.** Reuse the `get_schema` probe for `ubi_queries` to classify each cluster on the readiness ladder (rung 0–3); expose the rung on cluster detail and as a small badge on cluster cards. Turn the `UBI_NOT_ENABLED` (412) condition from a bare error into a structured, actionable state the UI can render. No cluster writes — read-only detection. - **B11 — Engine-aware "enable real user signals" nudge.** A dismissible card on the judgment-generation modal and cluster-detail page when UBI is absent (rung 0), with steps specific to the cluster's `engine_type` (OpenSearch UBI plugin / o19s ES fork / Solr `solr.UBIComponent`) and a deep-link to `ubi-judgment-generation.md`. Reuses the shipped `feat_contextual_help` idiom; never blocks the LLM path. Re-surfaces on next visit if dismissed but still unaddressed. - **B12 — Sparse-data guidance, not a wall.** When UBI is present but below `min_impressions_threshold` (rung 1), the would-be `UBI_INSUFFICIENT_DATA` path instead recommends hybrid mode and shows current coverage ("~12% of your query set has enough signal — hybrid rates that head, LLM fills the rest"). The empty/partial state on the judgment-list detail page explains *why* pairs were dropped and what closes the gap. From 3c62376fbe20d64321f6893f53d4a85a83ab844b Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 29 May 2026 08:45:13 -0400 Subject: [PATCH 4/5] docs: position normalization stage + opt-in query-normalization-tuning idea Adds operator guidance on where RelyLoop fits in the relevance pipeline (optimization.md new section) and an exploratory 00_unsure idea to make pre-query normalization an opt-in categorical search-space parameter (off by default; verbatim pass-through stays default). Analyzer-level changes remain a permanent non-goal. Includes lockstep dashboard regen. Zero migration; docs + idea only. Signed-off-by: SoundMindsAI Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/00_overview/UNSURE_DASHBOARD.md | 11 ++-- .../feat_query_normalization_tuning/idea.md | 60 +++++++++++++++++++ docs/00_overview/unsure_dashboard.html | 21 +++++-- docs/01_architecture/optimization.md | 20 +++++++ 4 files changed, 103 insertions(+), 9 deletions(-) create mode 100644 docs/00_overview/planned_features/00_unsure/feat_query_normalization_tuning/idea.md diff --git a/docs/00_overview/UNSURE_DASHBOARD.md b/docs/00_overview/UNSURE_DASHBOARD.md index f4f330d5..0cd9f274 100644 --- a/docs/00_overview/UNSURE_DASHBOARD.md +++ b/docs/00_overview/UNSURE_DASHBOARD.md @@ -15,14 +15,14 @@ Pull from the Idea backlog or capture a new feature spec. | Metric | Value | |---|---| | Scoped items done | **0 / 0** (0%) — feat_/infra_/chore_/epic_ past idea stage | -| Pending work | **1** items (every not-done feat/infra/chore/bug across all priorities) | +| Pending work | **2** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | | → P1 | **0** high-value, ready when P0 clears | -| → P2 (default) | 1 important to file, not blocking | +| → P2 (default) | 2 important to file, not blocking | | → Backlog | 0 captured for record, not planned | | Open bugs | 1 | | Legacy "Path to UNSURE" | 1 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 0 idea-only feat/infra (not yet scoped into UNSURE) | +| Backlog ideas | 1 idea-only feat/infra (not yet scoped into UNSURE) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -43,11 +43,12 @@ _None._ _None._ -### Idea (1) +### Idea (2) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P2 | [bug_seed_meaningful_demos_silent_bulk_errors](planned_features/00_unsure/bug_seed_meaningful_demos_silent_bulk_errors/idea.md) | Bug | [`scripts/seed_meaningful_demos.py:917-935`](../../scripts/seed_meaningful_demos.py#L917-L935) bulk-indexes 1000 Amazon ESCI products into a dedicated index per demo scenario: | — | Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep | +| 1 | P2 | [feat_query_normalization_tuning](planned_features/00_unsure/feat_query_normalization_tuning/idea.md) | Feature | A relevance pipeline runs in stages: (1) query understanding / normalization → (2) retrieval → (3) ranking / boosting → (4) re-ranking. RelyLoop tunes **stage 3 only**. But stage 1 is often where the | — | Idea — exploratory. Placed in `00_unsure/` because the release target is genuinely unresolved: the core capability is small and fits the existing parameter model, but a prod-reproducibility question (below) must be answered before it can be committed to a release. | +| 2 | P2 | [bug_seed_meaningful_demos_silent_bulk_errors](planned_features/00_unsure/bug_seed_meaningful_demos_silent_bulk_errors/idea.md) | Bug | [`scripts/seed_meaningful_demos.py:917-935`](../../scripts/seed_meaningful_demos.py#L917-L935) bulk-indexes 1000 Amazon ESCI products into a dedicated index per demo scenario: | — | Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep | ## Dependency graph diff --git a/docs/00_overview/planned_features/00_unsure/feat_query_normalization_tuning/idea.md b/docs/00_overview/planned_features/00_unsure/feat_query_normalization_tuning/idea.md new file mode 100644 index 00000000..71ce98a0 --- /dev/null +++ b/docs/00_overview/planned_features/00_unsure/feat_query_normalization_tuning/idea.md @@ -0,0 +1,60 @@ +# Query normalization as a tunable, opt-in query-time parameter + +**Date:** 2026-05-29 +**Status:** Idea — exploratory. Placed in `00_unsure/` because the release target is genuinely unresolved: the core capability is small and fits the existing parameter model, but a prod-reproducibility question (below) must be answered before it can be committed to a release. +**Priority:** P2 (exploratory) — a scope-respecting differentiator in the "query understanding" space no competitor's *optimizer* touches, but gated on a design decision, not ready to schedule. +**Origin:** Operator observation (2026-05-29) from a real-world search-relevance exercise: "before we ever discussed boosting or parameters, we normalized the incoming query — lowercasing, whitespace removal, expanding `what's` → `what is`." This is the **query-understanding / query-rewriting** stage, which sits *upstream* of the ranking stage RelyLoop tunes. RelyLoop currently passes `query_text` through verbatim ([`ElasticAdapter.render`](../../../../../backend/app/adapters/elastic.py)) and has no parameter representing normalization. No prior backlog idea existed for this — confirmed by search. +**Depends on:** MVP1 study lifecycle + search space (shipped). Composes with — does not block — the MVP2 anchors. + +## Problem + +A relevance pipeline runs in stages: (1) query understanding / normalization → (2) retrieval → (3) ranking / boosting → (4) re-ranking. RelyLoop tunes **stage 3 only**. But stage 1 is often where the largest relevance wins hide (vocabulary mismatch, not bad boosts, is the common cause of zero-results), and operators routinely tune it by hand — exactly what the originating exercise did. + +"Normalization" is two different mechanisms, and only one is a candidate here: + +- **Analyzer-level** (lowercase, stemming, stopwords, synonyms as token filters): governed by index analyzers with index-time/query-time symmetry; changing it requires reindexing. **Permanently out of scope** per umbrella spec §4 ("Make schema/mapping/analyzer changes" is a non-goal). RelyLoop reads analyzer names, never writes them. This idea does **not** touch this. +- **Pre-query rewriting** (contraction expansion, whitespace/case normalization applied to the *query string* before it reaches the engine, light spell-normalization): an application-layer transform. It does **not** touch the cluster, so it is query-time — RelyLoop's domain — and RelyLoop simply has no parameter for it today. **This is the candidate.** + +So the gap: RelyLoop can tell an operator the best `title_boost`, but it cannot empirically answer "does expanding contractions on the incoming query improve nDCG on my judgment set?" — even though that's a query-time question its loop is built to answer. + +## Proposed capability (sketch) + +Make the normalization choice a **categorical search-space parameter** — RelyLoop already supports categoricals ([`CategoricalParam`](../../../../../backend/app/domain/study/search_space.py)). A template could declare: + +``` +query_normalizer: { "type": "categorical", + "choices": ["none", "lowercase", "lowercase+trim", "lowercase+trim+expand_contractions"] } +``` + +A small **pre-render hook** applies the selected normalizer to `query_text` *before* the template interpolates it (`query_text` is the implicit render param — [`template_validator.py:53`](../../../../../backend/app/domain/study/template_validator.py#L53)). The Optuna loop then *discovers* whether a normalization rule helps the judgment set, the same way it discovers the best boost — and the winning normalizer travels in the proposal/PR like any other parameter. + +Normalizers are a small, **pure-domain, deterministic** library (`none` / `lowercase` / `trim` / `expand_contractions` from a small dictionary). No LLM, no cluster write, no new external dependency. + +## The opt-in requirement (operator's explicit ask) + +**This must be optional and off by default.** Two reasons, the second decisive: + +1. **Not every operator can act on normalization.** Some have no control over their query-rewriting layer (it lives in a separate team's service, or a vendor front-end). For them, a normalization parameter would tune something they can't deploy. Default behavior stays exactly as today: `query_text` passes through verbatim; the parameter only exists when a template opts in by declaring it. +2. **Prod-reproducibility is the gating design question.** If RelyLoop applies a rewrite at *its* query-time but the operator's *production* query pipeline does not apply the same rewrite, the winning config will not reproduce in production — RelyLoop would be optimizing against a query the live system never issues. This violates the core invariant that a merged proposal reproduces the measured gain. **Therefore the rewrite must be one the operator can also deploy** (e.g. carried in the config repo via the Git-PR apply path, or explicitly acknowledged as "you must replicate this normalizer in your query service"). Until this is resolved, the feature can't be committed to a release — hence `00_unsure/`. + +## Open questions (must resolve before promoting out of `00_unsure/`) + +1. **Prod reproducibility** — how does the chosen normalizer get deployed in the operator's actual query pipeline? Options: (a) the apply-path PR carries a normalizer declaration the operator's search service reads; (b) the proposal body documents the required normalizer and the operator replicates it manually; (c) scope to only normalizers the engine itself can apply at query time (narrow). This decision determines whether the feature is viable at all. +2. **Normalizer library scope** — which rules ship? Lowercasing and trim are safe and engine-symmetric-ish; contraction expansion needs a dictionary (English-only? operator-supplied?); spell-correction is probably too far (drifts toward the analyzer/index-time boundary and needs a corpus). +3. **Overlap with analyzers** — lowercasing at query-time may *duplicate or conflict* with a lowercase token filter already in the index analyzer (double-normalization is usually harmless but worth validating). The capability probe / schema read could warn when a chosen normalizer is redundant with the field's analyzer. +4. **Is categorical the right shape, or should it be a typed sub-object** (an ordered list of normalization steps)? Categorical keeps it inside the existing search-space model with zero schema change; a sub-object is more expressive but more work. + +## Scope signals (rough, pending the open questions) + +- **Backend:** small if scoped to a pure-domain normalizer library + a pre-render hook on `query_text` + categorical wiring. Larger if prod-reproducibility requires apply-path changes (question 1). +- **Frontend:** small — the normalizer is just another categorical in the search-space builder. +- **Migration:** none for the tuning mechanism (rides existing `CategoricalParam`); possibly one if the apply-path carries a normalizer declaration. +- **Config:** none required. +- **Audit events:** N/A (pre-`audit_log`). + +## Relationship to other work + +- **Documented in** [`docs/01_architecture/optimization.md` §"Where RelyLoop fits in your relevance pipeline"](../../../../01_architecture/optimization.md) (the Tier-1 boundary doc that landed with this idea) — that section is the operator-facing "normalize first; RelyLoop tunes ranking" guidance and points here for the tunable extension. +- **Distinct from analyzer changes** — those stay a permanent non-goal (umbrella spec §4); this is strictly query-time string rewriting that never touches the cluster. +- **Composes with the apply path** ([`apply-path.md`](../../../../01_architecture/apply-path.md)) — if resolved via option (a), the winning normalizer ships in the config-repo PR like any other tuned parameter. +- **Mirrors the UBI on-ramp opt-in philosophy** ([`feat_ubi_onramp`](../../02_mvp2/feat_ubi_onramp/idea.md)) — a capability for operators who *can* participate must not degrade the experience for those who can't; default behavior is unchanged. diff --git a/docs/00_overview/unsure_dashboard.html b/docs/00_overview/unsure_dashboard.html index ea69117e..0d180ced 100644 --- a/docs/00_overview/unsure_dashboard.html +++ b/docs/00_overview/unsure_dashboard.html @@ -403,7 +403,7 @@

UNSURE Progress

Pending work
-
1
+
2
every not-done feat/infra/chore/bug across all priorities
@@ -425,7 +425,7 @@

UNSURE Progress

P2 (default)
-
1
+
2
important to file, not blocking
@@ -442,7 +442,7 @@

UNSURE Progress

Backlog ideas: - 0 idea-only feat/infra folders (not yet scoped into UNSURE) + 1 idea-only feat/infra folders (not yet scoped into UNSURE) In flight: @@ -463,7 +463,20 @@

Pipeline

-

Idea 1

+

Idea 2

+ +
+
Query Normalization Tuning
+
+ Feature + P2 + +
+
A relevance pipeline runs in stages: (1) query understanding / normalization → (2) retrieval → (3) ranking / boosting → (4) re-ranking. RelyLoop tunes **stage 3 only**. But stage 1 is often where the
+ + +
+
Seed Meaningful Demos Silent Bulk Errors
diff --git a/docs/01_architecture/optimization.md b/docs/01_architecture/optimization.md index e689cb3f..faca82fd 100644 --- a/docs/01_architecture/optimization.md +++ b/docs/01_architecture/optimization.md @@ -176,6 +176,26 @@ live as module constants in contract is reviewed in [`feat_pr_metric_confidence/feature_spec.md`](../00_overview/planned_features/feat_pr_metric_confidence/feature_spec.md) §7 (FR-4 / FR-4a) and §12 (AC-3 through AC-17). +## Where RelyLoop fits in your relevance pipeline (and what comes *before* it) + +A search-relevance pipeline runs in stages, and RelyLoop deliberately operates at **one** of them. Knowing which stage matters, because the stage *before* RelyLoop is often where the biggest wins hide — and it is not something RelyLoop tunes. + +| Stage | What it does | Owned by | RelyLoop? | +|---|---|---|---| +| 1. **Query understanding / normalization** | Transform the *incoming query string* before it hits the engine: lowercasing, whitespace trimming, contraction expansion (`what's` → `what is`), spelling correction, synonym/abbreviation expansion, intent/entity detection. Fixes *vocabulary mismatch*. | The operator's query-rewriting layer (e.g. Querqy, a preprocessing service) **or** the engine's analyzers | **No** (see below) | +| 2. **Retrieval** | Match the query against the index; pull candidate documents. | The engine, driven by the query template | RelyLoop *renders* the query here | +| 3. **Ranking / boosting** | Field boosts, function scores, tie-breakers, fuzziness, slop, `min_should_match`, rerankers. | Query-time parameters | **Yes — this is RelyLoop's tuning surface** | +| 4. **Re-ranking / business rules / personalization** | LTR rerank, pinned results, merchandising rules. | Operator (LTR consume-only in MVP2; rules are out of scope) | Partial (LTR consume-only) | + +**RelyLoop tunes stage 3.** It passes `query_text` through to the engine **verbatim** — no lowercasing, no trimming, no rewriting ([`ElasticAdapter.render`](../../backend/app/adapters/elastic.py); the template interpolates `{{ query_text }}` raw). + +Normalization splits into two mechanisms, both currently outside RelyLoop's tuning boundary, for *different* reasons: + +- **Analyzer-level normalization** (lowercase, stemming, stopwords, synonyms as token filters) is governed by the index's **analyzers**, with index-time/query-time *symmetry* — the same analysis runs on documents at index time and on the query at search time. Changing it usually requires **reindexing**, which is why it sits behind the umbrella spec §4 non-goal: *"Make schema/mapping/analyzer changes. Tuning is restricted to query-time parameters."* RelyLoop *reads* analyzer names (the schema browser shows them) but never writes them. **This boundary is permanent.** +- **Pre-query rewriting** (contraction expansion, spell-correction, query expansion) is an *application-layer* transform of the query string before it reaches the engine. It does **not** require touching the cluster, so it is technically query-time — RelyLoop's domain — but RelyLoop has no parameter for it today. Tuning it is captured as an exploratory idea: [`feat_query_normalization_tuning`](../00_overview/planned_features/00_unsure/feat_query_normalization_tuning/idea.md) (release target unresolved pending the prod-reproducibility question — a rewrite RelyLoop applies only reproduces in production if the operator's query pipeline applies the same rewrite). + +**Operator guidance:** do your query-understanding work (stage 1) *before* leaning on RelyLoop for stage 3. Ranking tuning cannot recover a query whose terms never matched the index in the first place — the canonical failure is an analyzer that strips `not` as a stopword, turning "not waterproof" into "waterproof". RelyLoop sharpens *how matched candidates are scored*; it does not fix *whether the query matched*. + ## Cross-references - Stack choices (Optuna + ir_measures pinned in `pyproject.toml`): [`tech-stack.md`](tech-stack.md) From dceabf794f3f04f6a2e0d147e4419494479a9d41 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Fri, 29 May 2026 09:13:54 -0400 Subject: [PATCH 5/5] docs(mvp2): idea-preflight accuracy pass + Gemini adjudication fixes + dashboards idea-preflight across the MVP2 bucket + adjudicated 6 Gemini comments on PR #312. Stale-link root cause: ideas moved into planned_features/02_mvp2// during the MVP-bucket reorg without updating relative links; several siblings since shipped to implemented_features/. Link fixes (8 files): shipped siblings -> implemented_features/ homes; comparison.md + feature_templates/README.md depth; truncation_mvp2 -> _truncation; 00_unsure bug_seed link depth. infra_adapter_solr deep audit: removed false HTTPX_POOL_LIMITS claim; removed nonexistent templates/elasticsearch/ reference; added open questions + feat_ubi_onramp coordination. Gemini adjudication (four-quadrant): #1-3 REJECTED as false positives (Gemini miscounted directory depth; flagged links resolve, its suggestions resolve above repo root). #4 ACCEPTED (link CLAUDE.md for Rule #5/#2). #6 ACCEPTED (drop markdown bold from overnight-autopilot first sentence). #5 auto-resolved by upstream _mvp2 fix + regen. Lockstep dashboard regen. Docs only. Zero code/migration. Signed-off-by: SoundMindsAI Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/00_overview/MVP2_DASHBOARD.md | 4 ++-- docs/00_overview/mvp2_dashboard.html | 2 +- .../idea.md | 6 +++--- .../idea.md | 2 +- .../idea.md | 2 +- .../idea.md | 2 +- .../feat_chat_last_message_preview/idea.md | 6 +++--- .../02_mvp2/feat_fts_rank_ordering/idea.md | 2 +- .../02_mvp2/feat_overnight_autopilot/idea.md | 2 +- .../02_mvp2/feat_ubi_judgments/idea.md | 6 +++--- .../02_mvp2/infra_adapter_solr/idea.md | 20 +++++++++++++------ docs/01_architecture/mvp2-overview.md | 4 ++-- 12 files changed, 33 insertions(+), 25 deletions(-) diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 0d96aba1..da20db32 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -53,8 +53,8 @@ _None._ | 2 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/idea.md) | Feature | MVP1 ships with **LLM-as-judge** as the only authoritative judgment source. The architecture anticipated this would change — the `judgments.source` CHECK already accepts `click`… | — | Idea — bundled with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) into MVP2 / v0.2 "Three-Engine + Real Signals" | | 3 | P1 | [feat_ubi_onramp](planned_features/02_mvp2/feat_ubi_onramp/idea.md) | Feature | `feat_ubi_judgments` adds click-derived judgments, but a UBI-centric release has a well-known failure mode: it makes the operators who *don't* have UBI feel like second-class citizens, when in reality | — | Idea — split out from [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md) as a first-class feature (2026-05-29) so the no-signals-majority UX is dashboard-visible and not treated as optional UBI polish. | | 4 | P1 | [infra_adapter_solr](planned_features/02_mvp2/infra_adapter_solr/idea.md) | Infra | After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "engine-neutral" positioning is aspirational until a third engine ships. Apache Solr is the right third engine because: | — | Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) | -| 5 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../feature_templates/README.md). | -| 6 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | +| 5 | P2 | [feat_chat_last_message_preview](planned_features/02_mvp2/feat_chat_last_message_preview/idea.md) | Feature | The `/chat` list page ([ui/src/app/chat/page.tsx](../../ui/src/app/chat/page.tsx)) renders each conversation row as `title + relative timestamp + "{N} messages"` via… | — | Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../../feature_templates/README.md). | +| 6 | P2 | [feat_overnight_autopilot](planned_features/02_mvp2/feat_overnight_autopilot/idea.md) | Feature | The "Karpathy overnight loop" is already implemented and already autonomous, but an operator has no way to discover or trust it: | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The autonomous-chaining *engine* already shipped; this is the ergonomics layer that makes it discoverable. | | 7 | P2 | [feat_study_convergence_indicator](planned_features/02_mvp2/feat_study_convergence_indicator/idea.md) | Feature | After a study completes, the UI shows the best metric and a trials table, but **nothing tells the operator whether the metric had plateaued or was still climbing when the study stopped.** This is the | — | Idea — surfaced from an operator dogfooding review (2026-05-29). The feedback half of the "overnight autopilot ergonomics" theme. | | 8 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md) | Chore | The async flow's contract: | — | Idea — chore captured during PR #286 | | 9 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index 483584f2..b0ebe6ca 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -537,7 +537,7 @@

Idea 15

P2
-
The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it:
+
The "Karpathy overnight loop" is already implemented and already autonomous, but an operator has no way to discover or trust it:
diff --git a/docs/00_overview/planned_features/00_unsure/bug_seed_meaningful_demos_silent_bulk_errors/idea.md b/docs/00_overview/planned_features/00_unsure/bug_seed_meaningful_demos_silent_bulk_errors/idea.md index be74c8fb..de03467c 100644 --- a/docs/00_overview/planned_features/00_unsure/bug_seed_meaningful_demos_silent_bulk_errors/idea.md +++ b/docs/00_overview/planned_features/00_unsure/bug_seed_meaningful_demos_silent_bulk_errors/idea.md @@ -3,7 +3,7 @@ **Date:** 2026-05-28 **Status:** Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep **Priority:** P2 — latent. Affects operators running `make seed-demo` locally; same `unavailable_shards_exception` race that broke CI's `seed-es` step will silently corrupt the seeded demo data here, with no logs and no exit-code signal. -**Origin:** Surfaced while tracing the bulk-error handling in `seed_es.py`. The sibling script `scripts/seed_meaningful_demos.py` has its OWN bulk-index loop using `urllib.request` (not httpx) that never inspects the response body — fundamentally the same write-allocation race exposed in [bug_smoke_seed_es_unavailable_shards_race](../bug_smoke_seed_es_unavailable_shards_race/idea.md), but unguarded. +**Origin:** Surfaced while tracing the bulk-error handling in `seed_es.py`. The sibling script `scripts/seed_meaningful_demos.py` has its OWN bulk-index loop using `urllib.request` (not httpx) that never inspects the response body — fundamentally the same write-allocation race exposed in [bug_smoke_seed_es_unavailable_shards_race](../../../implemented_features/2026_05_29_bug_smoke_seed_es_unavailable_shards_race/idea.md), but unguarded. **Depends on:** None. ## Problem @@ -23,7 +23,7 @@ ES bulk semantics return HTTP 200 even when the primary shard is INITIALIZING Net: operators running `make seed-demo` get incomplete demo data with no signal. Studies created against the demo cluster will return zero results or fewer-than-expected hits, looking like a configuration problem. -**Why latent now:** [`chore_drop_demo_seed_from_ci`](../../implemented_features/2026_05_28_chore_drop_demo_seed_from_ci/idea.md) removed `make seed-demo FORCE=1` from `pr.yml`, so CI doesn't exercise this path. Local-dev operators still hit it via the home page's "Reset to demo state" button + the `chore_tutorial_polish` walkthrough. +**Why latent now:** [`chore_drop_demo_seed_from_ci`](../../../implemented_features/2026_05_28_chore_drop_demo_seed_from_ci/idea.md) removed `make seed-demo FORCE=1` from `pr.yml`, so CI doesn't exercise this path. Local-dev operators still hit it via the home page's "Reset to demo state" button + the `chore_tutorial_polish` walkthrough. ## Proposed capabilities @@ -53,6 +53,6 @@ Just check `payload.get("errors")` and exit non-zero with a clear message. Doesn ## Relationship to other work -- **Mirror of [`bug_smoke_seed_es_unavailable_shards_race`](../bug_smoke_seed_es_unavailable_shards_race/idea.md)** — same root cause, different script, different HTTP library. Pick this up after the sibling bug ships so the retry pattern is established. +- **Mirror of [`bug_smoke_seed_es_unavailable_shards_race`](../../../implemented_features/2026_05_29_bug_smoke_seed_es_unavailable_shards_race/idea.md)** — same root cause, different script, different HTTP library. Pick this up after the sibling bug ships so the retry pattern is established. - **Composes with the "extract `_bulk_with_retry` to shared module" refactor** the bug_smoke fix noted but deferred — best to land both together. - **Surfaces on `make seed-demo` only.** Not a CI blocker; affects local operator first-impression quality. diff --git a/docs/00_overview/planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md b/docs/00_overview/planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md index c1c03d25..d96a3cb0 100644 --- a/docs/00_overview/planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md @@ -36,7 +36,7 @@ Lock scope is the transaction — `pg_advisory_xact_lock` is automatically relea ## Co-evolution with `bug_auto_followup_completed_parent_stop_chain_race` -This chore composes with the bug fix at [`bug_auto_followup_completed_parent_stop_chain_race`](../bug_auto_followup_completed_parent_stop_chain_race/idea.md) — both races have the same shape (cascade-vs-worker timing) but at different layers: +This chore composes with the bug fix at [`bug_auto_followup_completed_parent_stop_chain_race`](../../../implemented_features/2026_05_26_bug_auto_followup_completed_parent_stop_chain_race/idea.md) — both races have the same shape (cascade-vs-worker timing) but at different layers: - The bug fix (Option A) zeroes `parent.config["auto_followup_depth"]` in the cascade to short-circuit the **pending worker's gate**. - This chore (the advisory lock) coordinates **concurrent worker invocations** so the second one observes the first one's child. diff --git a/docs/00_overview/planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md b/docs/00_overview/planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md index 6fb848d5..7de3c64a 100644 --- a/docs/00_overview/planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/idea.md @@ -31,7 +31,7 @@ None of this is integration-tested. A future regression that: - Forgets to register `run_demo_reseed` in `WorkerSettings.functions` → no test fails. - Breaks the Redis status key shape → no test fails (the unit test covers the *Pydantic* shape but not the persistence path through the real Redis). -[`bug_demo_reseed_button_silent_enqueue_failure`](../bug_demo_reseed_button_silent_enqueue_failure/idea.md) is a real bug that would have been caught by integration tests on the async flow. +[`bug_demo_reseed_button_silent_enqueue_failure`](../../../implemented_features/2026_05_29_bug_demo_reseed_button_silent_enqueue_failure/idea.md) is a real bug that would have been caught by integration tests on the async flow. ## Why deferred diff --git a/docs/00_overview/planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md b/docs/00_overview/planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md index 712e3a65..c79d84a9 100644 --- a/docs/00_overview/planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md @@ -30,4 +30,4 @@ This means every studies-POST integration test today runs without a real Arq poo ## Coordinates with -- [`infra_study_preflight_real_engine_integration`](../infra_study_preflight_real_engine_integration/idea.md) — also deferred from the same phase-gate review; both could be picked up together in an infra-sweep PR. +- [`infra_study_preflight_real_engine_integration`](../../../implemented_features/2026_05_25_infra_study_preflight_real_engine_integration/idea.md) — also deferred from the same phase-gate review; both could be picked up together in an infra-sweep PR. diff --git a/docs/00_overview/planned_features/02_mvp2/feat_chat_last_message_preview/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_chat_last_message_preview/idea.md index 0c8f36f6..1ce02017 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_chat_last_message_preview/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/feat_chat_last_message_preview/idea.md @@ -1,7 +1,7 @@ # feat_chat_last_message_preview **Type:** feat (chat UX polish — deferred) -**Status:** Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../feature_templates/README.md). +**Status:** Held for MVP2 (decided 2026-05-13). No technical dependency on MVP2 infra; bundling with [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation/idea.md) as chat polish. `feat_chat_agent` has been live since 2026-05-12 (PR #60) and no operator has asked for the preview yet. Folder renamed from `chore_chat_last_message_preview` 2026-05-14 per `/idea-preflight` audit — `chore_` is reserved for changes with no user-visible behavior per [feature_templates/README.md](../../feature_templates/README.md). **Date:** 2026-05-12 **Origin:** GPT-5.5 cycle-2 finding F15 against `feat_chat_agent` implementation plan ([deferred-work entries](../../../implemented_features/2026_05_12_feat_chat_agent/implementation_plan.md), F15 row in the rating table); Story 5.1 capture. @@ -118,7 +118,7 @@ These need spec-time decisions the idea cannot lock from the codebase alone: Originally captured as `chore_chat_last_message_preview` (Story 5.1, 2026-05-12). Renamed to `feat_chat_last_message_preview` on 2026-05-14 per `/idea-preflight` -audit: [feature_templates/README.md](../feature_templates/README.md) reserves +audit: [feature_templates/README.md](../../feature_templates/README.md) reserves `chore_` for "Refactor, rename, dead-code removal, tech debt — no user-visible behavior change," and a preview line in the conversation list is a user-visible addition. Mirrors the 2026-05-13 `bug_chat_long_conversation_truncation` → @@ -134,5 +134,5 @@ F15 row, deferred-work table) left untouched as frozen-at-ship-time. [`implementation_plan.md`](../../../implemented_features/2026_05_12_feat_chat_agent/implementation_plan.md) (cycle-2 F15 row + Story 5.1 deferred-work table). - Companion deferred work: - [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation_mvp2/idea.md) + [`bug_chat_long_conversation_truncation_mvp2`](../bug_chat_long_conversation_truncation/idea.md) — both are MVP2 chat polish items and could ship together. diff --git a/docs/00_overview/planned_features/02_mvp2/feat_fts_rank_ordering/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_fts_rank_ordering/idea.md index a91ef65c..61585903 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_fts_rank_ordering/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/feat_fts_rank_ordering/idea.md @@ -3,7 +3,7 @@ **Date:** 2026-05-16 **Status:** Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. **Priority:** Backlog — explicitly held for MVP2 (folder name suffix). Re-evaluate when MVP2 work begins; the tsvector + GIN indexes are already on disk so the actual implementation is small. -**Origin:** [`feat_data_table_primitive/feature_spec.md` §16 lines 896–900](../feat_data_table_primitive/feature_spec.md). The 6 Postgres `tsvector` columns + GIN indexes already exist (migrations `0008`–`0013`); the `plainto_tsquery('english', :q)` predicate is wired into 6 list endpoints. Only the ORDER BY and the cursor encoding need to change. +**Origin:** [`feat_data_table_primitive/feature_spec.md` §16 lines 896–900](../../../implemented_features/2026_05_16_feat_data_table_primitive/feature_spec.md). The 6 Postgres `tsvector` columns + GIN indexes already exist (migrations `0008`–`0013`); the `plainto_tsquery('english', :q)` predicate is wired into 6 list endpoints. Only the ORDER BY and the cursor encoding need to change. **Depends on:** `feat_data_table_primitive` (PR open on `feat/data-table-primitive`). Cursor pagination is already keyset-based on `(created_at, id)`. ## Problem diff --git a/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md index 199a8c32..f12bdd0b 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/feat_overnight_autopilot/idea.md @@ -8,7 +8,7 @@ ## Problem -The "Karpathy overnight loop" is **already implemented** and **already autonomous**, but an operator has no way to discover or trust it: +The "Karpathy overnight loop" is already implemented and already autonomous, but an operator has no way to discover or trust it: 1. **`auto_followup_depth` is a hidden config key.** When set (the validator accepts `0–5`, where `0` = no chaining, so `1–5` enables it — [`schemas.py:645`](../../../../../backend/app/api/v1/schemas.py#L645)), a completed study automatically narrows the search space around its winner, decrements the depth, and spawns a child study — **zero human intervention between iterations** ([`backend/workers/auto_followup.py`](../../../../../backend/workers/auto_followup.py)). The chain self-terminates on depth exhaustion, sub-epsilon lift (<0.5%), budget at 80%, or parent failure. This is exactly the operator's "wake up to a few results" ask. But it is not exposed as a first-class control in the create-study wizard — the operator never knew it existed, so all 7 studies ran one-shot. diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md index 3ceacdc4..accc4513 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md @@ -2,7 +2,7 @@ **Date:** 2026-05-22 (refreshed 2026-05-27 for the positioning reframe) **Status:** Idea — bundled with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) into MVP2 / v0.2 "Three-Engine + Real Signals" -**Priority:** P1 — MVP2 ships UBI + Solr together; the hybrid UBI+LLM converter is the differentiated capability vs OpenSearch SRW (which has UBI-via-COEC GA but no hybrid mode, no full-search-space Bayesian optimizer to feed). See [`docs/07_research/comparison.md`](../../../07_research/comparison.md) for the citation-backed competitive position. +**Priority:** P1 — MVP2 ships UBI + Solr together; the hybrid UBI+LLM converter is the differentiated capability vs OpenSearch SRW (which has UBI-via-COEC GA but no hybrid mode, no full-search-space Bayesian optimizer to feed). See [`docs/07_research/comparison.md`](../../../../07_research/comparison.md) for the citation-backed competitive position. **Origin:** Originally prompted by an external review on 2026-05-22 (LinkedIn outreach to a senior search engineer at a relevance-tooling company who pushed back on LLM-as-judge as the only authoritative judgment source). The 2026-05-27 reframe bundled this work with the Solr adapter into one MVP2 release because Solr's first-party `solr.UBIComponent` writes the same UBI schema as the OpenSearch UBI plugin — UBI on Solr is free once the adapter ships, and the combined release tells the engine-neutral story coherently. **Depends on:** MVP1 shipped (`judgments` + `judgment_lists` tables, `ElasticAdapter` with `SearchAdapter.search_batch`, `generate_judgments_llm` agent tool pattern). Co-ships with [`infra_adapter_solr`](../infra_adapter_solr/idea.md) in MVP2. @@ -78,8 +78,8 @@ Single-tier — small, additive, no schema migration. Five capability blocks bel ## Relationship to other work - **Cleans up [`docs/00_overview/relyloop-spec.md`](../../../relyloop-spec.md) §14 + §19 + §27** — the spec previously framed click data as a per-engine adapter concern with engine-specific timelines. The §14 patch (landing with this idea) re-anchors the architecture around the engine-neutral UBI schema (which works across all three OSS engines via their respective UBI implementations), with engine-native readers (Elastic Behavioral Analytics, etc.) as thin extensions feeding the same `SignalsConverter` Protocol when an adopter needs them. -- **Composes with [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md)** — auto-chained follow-up studies become dramatically more useful with a continuously-refreshed UBI judgment list than with a snapshot LLM-as-judge list. The two features are complementary; UBI ships first. +- **Composes with [`feat_auto_followup_studies`](../../../implemented_features/2026_05_24_feat_auto_followup_studies/idea.md)** — auto-chained follow-up studies become dramatically more useful with a continuously-refreshed UBI judgment list than with a snapshot LLM-as-judge list. The two features are complementary; UBI ships first. - **Composes with [`feat_pr_metric_confidence`](../../../implemented_features/2026_05_21_feat_pr_metric_confidence/)** (shipped 2026-05-21) — the confidence framing in the PR body becomes meaningfully stronger when "the metric was scored against 50,000 UBI-derived ratings covering 90% of last week's traffic" replaces "the metric was scored against 500 LLM ratings against a snapshot query set." -- **Composes with [`feat_study_baseline_trial`](../feat_study_baseline_trial/idea.md) + [`feat_config_repo_baseline_tracking`](../feat_config_repo_baseline_tracking/idea.md)** — once UBI is the judgment source, "the baseline metric on the live config" becomes a meaningful absolute number rather than a synthetic LLM-rated approximation. Materially raises the credibility of every winning trial. +- **Composes with [`feat_study_baseline_trial`](../../../implemented_features/2026_05_25_feat_study_baseline_trial/idea.md) + [`feat_config_repo_baseline_tracking`](../../../implemented_features/2026_05_23_feat_config_repo_baseline_tracking/idea.md)** — once UBI is the judgment source, "the baseline metric on the live config" becomes a meaningful absolute number rather than a synthetic LLM-rated approximation. Materially raises the credibility of every winning trial. - **Does NOT block MVP2 "Observable"** — Langfuse and SigNoz instrumentation can layer on top of `generate_judgments_from_ubi` exactly as it would on top of `generate_judgments_llm`. The `langfuse_trace_id` lineage column landing at MVP2 will be NULL for `source='click'` rows (which never invoke an LLM) and populated for `source='llm'` rows in the hybrid case — same column, source-dependent fill. - **Does NOT block later engine work** — the MVP2 `SignalsConverter` Protocol is engine-agnostic. New adapters added in later releases contribute their own engine-native reader (where they have one) feeding the same Protocol; the converter library and the API surface are unchanged regardless of which engines ship. diff --git a/docs/00_overview/planned_features/02_mvp2/infra_adapter_solr/idea.md b/docs/00_overview/planned_features/02_mvp2/infra_adapter_solr/idea.md index b3febd97..273dde42 100644 --- a/docs/00_overview/planned_features/02_mvp2/infra_adapter_solr/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/infra_adapter_solr/idea.md @@ -3,7 +3,7 @@ **Date:** 2026-05-27 **Status:** Idea — anchor feature for MVP2 / v0.2 "Three-Engine + Real Signals" (bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)) **Priority:** P1 — MVP2 is named for the bundle of this adapter + UBI judgments; together they ship four of RelyLoop's six differentiators (all three OSS engines + hybrid UBI+LLM) -**Origin:** Positioning reframe on 2026-05-27 (see [`chore_drop_fusion_scope/idea.md`](../chore_drop_fusion_scope/idea.md) for the paired Fusion-drop rationale and [`docs/07_research/comparison.md`](../../../07_research/comparison.md) for the moat analysis). Replaces the previously-planned Lucidworks Fusion adapter as the next engine target. +**Origin:** Positioning reframe on 2026-05-27 (see [`chore_drop_fusion_scope/idea.md`](../../../implemented_features/2026_05_28_chore_drop_fusion_scope/idea.md) for the paired Fusion-drop rationale — shipped 2026-05-28 — and [`docs/07_research/comparison.md`](../../../../07_research/comparison.md) for the moat analysis). Replaces the previously-planned Lucidworks Fusion adapter as the next engine target. **Depends on:** MVP1 shipped (`ElasticAdapter`, `SearchAdapter` Protocol, study lifecycle, judgment lists, PR worker). Co-released with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md): Solr's `solr.UBIComponent` writes the same `ubi_queries` + `ubi_events` schema, so the MVP2 `UbiReader` works unchanged against a Solr cluster from day one. ## Problem @@ -15,7 +15,7 @@ After MVP1.5, RelyLoop runs against Elasticsearch and OpenSearch — but the "en 3. **Quepid + Chorus user base is Solr-native.** OSC's primary reference stack is Solr-based. Operators who already run Quepid for manual relevance evaluation are the natural adopters for RelyLoop's Bayesian-loop upgrade on the same engine they already manage. 4. **LTR is stable.** Solr 10 (March 2026) ships `modules/ltr` with `LinearModel`, `MultipleAdditiveTreesModel` (XGBoost-compatible), and `NeuralNetworkModel`. Stable since Solr 6. The de facto OSS LTR baseline outside ES native LTR ([Sease: Solr 10 LTR overview](https://sease.io/2026/03/apache-solr-10-what-is-new-for-vector-search-and-ltr.html)). -The Lucidworks Fusion adapter that previously occupied this slot is dropped — see [`chore_drop_fusion_scope`](../chore_drop_fusion_scope/idea.md) for the rationale (vendor entanglement, narrower audience overlap with the Quepid/Chorus community, materially higher build cost). +The Lucidworks Fusion adapter that previously occupied this slot is dropped — see [`chore_drop_fusion_scope`](../../../implemented_features/2026_05_28_chore_drop_fusion_scope/idea.md) for the rationale (vendor entanglement, narrower audience overlap with the Quepid/Chorus community, materially higher build cost). ## Proposed capabilities @@ -23,8 +23,8 @@ The Lucidworks Fusion adapter that previously occupied this slot is dropped — - **Location:** new module `backend/app/adapters/solr.py` implementing the `SearchAdapter` Protocol from [`backend/app/adapters/protocol.py`](../../../../../backend/app/adapters/protocol.py). - **Engine support:** Solr 9.x (current widely-deployed) + Solr 10.x (released 2026-03). SolrCloud and standalone modes both supported. Solr 8.x and earlier explicitly out of scope. -- **`search_batch`:** parallel `/select` requests with a connection pool. Solr has no `_msearch` equivalent; the JSON Request API allows multi-query but is awkward and undertested across versions. Connection pool sized via existing settings (`HTTPX_POOL_LIMITS`). -- **`render`:** produces a Solr request parameter dict (later URL-encoded). Supports `edismax` (primary), `dismax`, and `lucene` parsers. Templates live under `templates/solr/` as Jinja templates that emit parameter maps, mirroring `templates/elasticsearch/` shape. +- **`search_batch`:** parallel `/select` requests with a connection pool. Solr has no `_msearch` equivalent; the JSON Request API allows multi-query but is awkward and undertested across versions. Connection pool sized via the same inline `httpx.AsyncClient` pattern the ElasticAdapter uses today (`timeout=Timeout(10.0, connect=2.0)`, see [`backend/app/adapters/elastic.py:120`](../../../../../backend/app/adapters/elastic.py#L120)); a settings-level pool tunable can be introduced if Solr's per-query parallelism warrants it (open at spec time — there is no `HTTPX_POOL_LIMITS` setting today, verified 2026-05-29). +- **`render`:** produces a Solr request parameter dict (later URL-encoded). Supports `edismax` (primary), `dismax`, and `lucene` parsers. Templates live under `templates/solr/` as Jinja templates that emit parameter maps. **Template-path convention is unresolved as of 2026-05-29:** the repo's only existing template is at [`samples/templates/product_search.j2`](../../../../../samples/templates/product_search.j2); the repo-root `templates/` directory exists but is empty (`.keep` only). The sibling [`chore_template_library_expansion`](../chore_template_library_expansion/idea.md) proposes expanding `samples/templates/`, not introducing `templates//`. Pick one convention in `/spec-gen` and apply it uniformly to both adapters (open question listed below). - **`get_schema`:** uses Solr's Schema API (`/schema/fields`, `/schema/dynamicfields`, `/schema/fieldtypes`). Result shape matches `Schema` type unchanged. - **`list_targets`:** uses CoresAdmin API (`/admin/cores?action=STATUS`) for standalone; CollectionsAdmin (`/admin/collections?action=LIST`) for SolrCloud. Selects automatically based on a startup capability probe. - **`explain`:** uses `debugQuery=true&debug=results` and parses `debug.explain` from the response. @@ -90,8 +90,16 @@ Solr-specific notes: ## Relationship to other work -- **Replaces the previously-planned Lucidworks Fusion adapter** as the next engine target. See [`chore_drop_fusion_scope/idea.md`](../chore_drop_fusion_scope/idea.md) for why Fusion was dropped. +- **Replaces the previously-planned Lucidworks Fusion adapter** as the next engine target. See [`chore_drop_fusion_scope`](../../../implemented_features/2026_05_28_chore_drop_fusion_scope/idea.md) (shipped 2026-05-28) for why Fusion was dropped. - **Bundled with [`feat_ubi_judgments`](../feat_ubi_judgments/idea.md)** in MVP2 — Solr's `solr.UBIComponent` writes the same UBI schema; the UBI reader and hybrid UBI+LLM converter work on Solr unchanged from day one. +- **Required by [`feat_ubi_onramp`](../feat_ubi_onramp/idea.md)** — the engine-aware "enable real user signals" nudge spans all three engines only after the `engine_type` CHECK constraint extension here lands. Until then the on-ramp covers `elasticsearch | opensearch` (current values at [`cluster.py:30`](../../../../../backend/app/db/models/cluster.py#L30)). +- **Pairs with [`chore_template_library_expansion`](../chore_template_library_expansion/idea.md)** (Workstream C in [`mvp2-overview.md`](../../../../01_architecture/mvp2-overview.md)) — that idea ships the curated multi-engine template library (including Solr templates) and the per-engine tunable-params cheatsheets. Coordinate the template-path convention with it (see open questions). - **Multi-Git provider abstraction (GitLab, Bitbucket) is in the backlog** — was previously bundled with the Fusion-era MVP3; reframed as backlog because it serves a smaller adopter axis than the engine sweep + observability path. GitHub remains the only Git provider through GA v1. -- **Unlocks the verifiable "engine-neutral" claim** in [`docs/07_research/comparison.md`](../../../07_research/comparison.md) and the umbrella spec §1. The claim is rhetorical at MVP1; it becomes factual at MVP2. +- **Unlocks the verifiable "engine-neutral" claim** in [`docs/07_research/comparison.md`](../../../../07_research/comparison.md) and the umbrella spec §1. The claim is rhetorical at MVP1; it becomes factual at MVP2. - **MVP3 "Observable" follows** — Langfuse + SigNoz + audit-log immutability + lineage layers on top of all three engines and both judgment sources in one go. + +## Open questions for /spec-gen + +1. **Template-path convention.** Adopt `templates//` (this idea's current text) or extend `samples/templates/` (the convention `chore_template_library_expansion` proposes)? Whichever wins must apply uniformly — splitting "Solr at one path, ES/OS at another" is the worst outcome. Recommended: `samples/templates//` (extends today's location without introducing a second top-level templates dir, and matches the sibling chore's path baseline). +2. **httpx pool tunable.** Introduce a settings-level pool limit (`HTTPX_POOL_LIMITS` or similar) as part of this work, or rely on the inline `AsyncClient` defaults the ElasticAdapter uses today? The latter ships sooner; the former is the right shape if Solr's parallel `/select` warrants per-engine tuning. +3. **LTR test fixture** (also tracked in [`mvp2-overview.md` §10](../../../../01_architecture/mvp2-overview.md)) — load a real `MultipleAdditiveTreesModel` into Compose Solr for the E2E, or assert the `rq={!ltr …}` render shape only? diff --git a/docs/01_architecture/mvp2-overview.md b/docs/01_architecture/mvp2-overview.md index d8419b86..772d9b68 100644 --- a/docs/01_architecture/mvp2-overview.md +++ b/docs/01_architecture/mvp2-overview.md @@ -102,9 +102,9 @@ Story stubs: - **A3 — `search_batch`.** Parallel `/select` requests over a connection pool sized by the existing `HTTPX_POOL_LIMITS` (Solr has no `_msearch` equivalent). - **A4 — `get_schema` + `list_targets`.** Schema API (`/schema/fields|dynamicfields|fieldtypes`) → `Schema` type unchanged; CoresAdmin (standalone) / CollectionsAdmin (SolrCloud) for target listing, selected by the A1 probe. - **A5 — `explain`.** `debugQuery=true&debug=results`, parse the `debug.explain` block. -- **A6 — Auth + migration.** Implement `solr_basic` (HTTP Basic) and `solr_apikey` (Solr 9+ JWT via `JWTAuthPlugin`); the one Alembic migration extending the `engine_type` + `auth_kind` CHECK constraints (with downgrade + round-trip per Rule #5). +- **A6 — Auth + migration.** Implement `solr_basic` (HTTP Basic) and `solr_apikey` (Solr 9+ JWT via `JWTAuthPlugin`); the one Alembic migration extending the `engine_type` + `auth_kind` CHECK constraints (with downgrade + round-trip per [CLAUDE.md](../../CLAUDE.md) Absolute Rule #5 — every migration has a reversible `downgrade()`). - **A7 — LTR rescore (consume-only).** Render unified `rerank_model:{id,top_k}` to `rq={!ltr model=… reRankDocs=…}` applying a pre-existing `MultipleAdditiveTreesModel` from Solr's `/schema/model-store`. **Training is out of scope** (backlog). -- **A8 — Compose service + sample data.** Add the `solr` service + new optional env vars (`SOLR_HOST`/`SOLR_PORT`/`SOLR_ADMIN_USERNAME_FILE`/`SOLR_ADMIN_PASSWORD_FILE`, `*_FILE` secrets per Rule #2); seed the `products` collection from the existing `samples/products.json`. +- **A8 — Compose service + sample data.** Add the `solr` service + new optional env vars (`SOLR_HOST`/`SOLR_PORT`/`SOLR_ADMIN_USERNAME_FILE`/`SOLR_ADMIN_PASSWORD_FILE`, `*_FILE` mounted secrets per [CLAUDE.md](../../CLAUDE.md) Absolute Rule #2 — secrets via mounted files, never bare env vars); seed the `products` collection from the existing `samples/products.json`. - **A9 — Frontend.** Add `solr` to the cluster-registration `engine_type` allowlist (per the Enumerated Value Contract Discipline — ground the option in the backend Literal); Solr auth help text; a Solr engine badge on cluster cards / study headers. - **A10 — Tests + runbook.** Unit (param rendering, LTR injection, `mm` syntax, probe parsing, error mapping), integration (live Compose Solr; LTR round-trip; UBI reader against seeded indices), contract (Protocol conformance — Solr passes every method ES/OpenSearch pass), E2E (`ui/tests/e2e/solr-study-end-to-end.spec.ts`); new `docs/03_runbooks/solr-cluster-registration.md`; tutorial Step 0 Path C.