diff --git a/docs/00_overview/DASHBOARD.md b/docs/00_overview/DASHBOARD.md index ca08ff3d..4d45ef28 100644 --- a/docs/00_overview/DASHBOARD.md +++ b/docs/00_overview/DASHBOARD.md @@ -6,8 +6,8 @@ _Top-level index across MVP1 → GA v1+ as of **2026-05-29**. Click a release na | Release | Theme | Progress | Status | |---|---|---|---| -| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 95 / 95 scoped done | **Complete** | -| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 1 / 2 scoped done · 10 remaining | **In progress** | +| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 96 / 96 scoped done | **Complete** | +| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 1 / 1 scoped done · 9 remaining | **In progress** | | MVP3 / v0.3 | Observable | — | **Not yet scoped** | | [GA v1 / v1.0](GA_DASHBOARD.md) | Production-ready | 1 item(s) queued | **Held / queued** | diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index 4d90eca2..0703ac66 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -14,7 +14,7 @@ Pull from the Idea backlog or capture a new feature spec. | Metric | Value | |---|---| -| Scoped items done | **95 / 95** (100%) — feat_/infra_/chore_/epic_ past idea stage | +| Scoped items done | **96 / 96** (100%) — feat_/infra_/chore_/epic_ past idea stage | | Pending work | **0** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | | → P1 | **0** high-value, ready when P0 clears | @@ -27,7 +27,7 @@ Pull from the Idea backlog or capture a new feature spec. ## Pipeline -### Done (130) +### Done (131) | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---| @@ -62,6 +62,7 @@ Pull from the Idea backlog or capture a new feature spec. | [feat_study_preflight_overlap_probe](implemented_features/2026_05_22_feat_study_preflight_overlap_probe/feature_spec.md) | Feature | `POST /api/v1/studies` issues a single bounded `ids`-existence query against the study's target asking "for the *first* query in the query set that has any judgments (chosen deterministically by `id A | — | [PR #193](https://github.com/SoundMindsAI/relyloop/pull/193) merged 2026-05-21 | | [feat_study_sub_warmup_guard](implemented_features/2026_05_29_feat_study_sub_warmup_guard/feature_spec.md) | Feature | A non-blocking inline warning appears under the `max_trials` input whenever the derived preset is `custom` AND `max_trials < STUDIES_TPE_WARMUP_FLOOR (= 50)`, naming Focused/Standard as one-click reme | — | [PR #316](https://github.com/SoundMindsAI/relyloop/pull/316) merged 2026-05-29 | | [feat_study_target_judgment_mismatch_guard](implemented_features/2026_05_21_feat_study_target_judgment_mismatch_guard/feature_spec.md) | Feature | `POST /api/v1/studies` rejects the mismatch at create time with a specific machine-readable error code (`JUDGMENT_TARGET_MISMATCH`, 422). | — | [PR #184](https://github.com/SoundMindsAI/relyloop/pull/184) merged 2026-05-21 | +| [feat_ubi_judgments](implemented_features/2026_05_29_feat_ubi_judgments/feature_spec.md) | Feature | Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic | — | [PR #317](https://github.com/SoundMindsAI/relyloop/pull/317) merged 2026-05-29 | | [infra_adapter_elastic](implemented_features/2026_05_10_infra_adapter_elastic/feature_spec.md) | Infra | A single `ElasticAdapter` implements the `SearchAdapter` Protocol and serves both Elasticsearch (8.11+ / 9.x) and OpenSearch (2.x / 3.x), distinguished by a `engine_type` column. | — | [PR #16](https://github.com/SoundMindsAI/relyloop/pull/16) merged 2026-05-10 | | [infra_ci_smoke_makeup](implemented_features/2026_05_13_infra_ci_smoke_makeup/idea.md) | Infra | CI runs `make test-unit && make test-integration && make test-contract` against a service-container Postgres on `localhost:5432` — a synthetic environment that masks every real-world `make up` failure | — | Complete | | [infra_dashboard_regen_pre_commit_conflict](implemented_features/2026_05_14_infra_dashboard_regen_pre_commit_conflict/idea.md) | Infra | The pre-commit pipeline does: | — | Complete | @@ -379,6 +380,8 @@ graph LR class chore_state_md_size_compression done; feat_study_sub_warmup_guard["study sub warmup guard"] class feat_study_sub_warmup_guard done; + feat_ubi_judgments["ubi judgments"] + class feat_ubi_judgments done; feat_study_lifecycle --> feat_digest_proposal feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_llm_judgments diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 3e7f252a..bffee242 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -6,30 +6,24 @@ _Reflects feature-folder state as of **2026-05-29** (latest mtime of any planned ## Next up -**[feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md)** — Feature, currently in **Implementing** +All scoped MVP2 features shipped 🎉 -> Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters - -Implementation in progress — resume to finish - -```bash -/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md --all # resume in-progress -``` +Pull from the Idea backlog or capture a new feature spec. ## MVP2 Progress | Metric | Value | |---|---| -| Scoped items done | **1 / 2** (50%) — feat_/infra_/chore_/epic_ past idea stage | -| Pending work | **18** items (every not-done feat/infra/chore/bug across all priorities) | +| Scoped items done | **1 / 1** (100%) — feat_/infra_/chore_/epic_ past idea stage | +| Pending work | **17** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | -| → P1 | **3** high-value, ready when P0 clears | +| → P1 | **2** high-value, ready when P0 clears | | → P2 (default) | 11 important to file, not blocking | | → Backlog | 4 captured for record, not planned | | Open bugs | 3 | -| Legacy "Path to MVP2" | 10 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | +| Legacy "Path to MVP2" | 9 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | | Backlog ideas | 8 idea-only feat/infra (not yet scoped into MVP2) | -| In flight | 1 feature(s) actively shipping | +| In flight | 0 feature(s) actively shipping | ## Pipeline @@ -39,11 +33,9 @@ Implementation in progress — resume to finish |---|---|---|---|---| | [feat_contextual_help_mvp2](implemented_features/2026_05_15_feat_contextual_help_mvp2/idea.md) | Feature | Phase 1 covered the create-study modal + study-detail surface — the steepest onboarding cliff. Two clusters of surfaces remain that a relevance engineer encounters after running their first study: | — | [PR #124](https://github.com/SoundMindsAI/relyloop/pull/124) merged 2026-05-15 | -### Implementing (1) +### Implementing (0) -| # | Priority | Feature | Type | One-liner | Depends on | Status | -|---|---|---|---|---|---|---| -| 1 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md) | Feature | Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic | — | — | +_None._ ### Plan (0) @@ -86,8 +78,6 @@ graph LR classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; - feat_ubi_judgments["ubi judgments"] - class feat_ubi_judgments implement; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done; ``` diff --git a/docs/00_overview/dashboard.html b/docs/00_overview/dashboard.html index 3e3f346b..6ca32bca 100644 --- a/docs/00_overview/dashboard.html +++ b/docs/00_overview/dashboard.html @@ -384,7 +384,7 @@

Releases

MVP1 / v0.1
The Loop
-
95 / 95 scoped done
+
96 / 96 scoped done
Complete
@@ -392,7 +392,7 @@

Releases

MVP2 / v0.2
Three-Engine + Real Signals
-
1 / 2 scoped done · 10 remaining
+
1 / 1 scoped done · 9 remaining
In progress
diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md b/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/feature_spec.md similarity index 100% rename from docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md rename to docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/feature_spec.md diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md b/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/idea.md similarity index 100% rename from docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/idea.md rename to docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/idea.md diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md b/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/implementation_plan.md similarity index 99% rename from docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md rename to docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/implementation_plan.md index cd3c5a2d..939cd102 100644 --- a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md +++ b/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/implementation_plan.md @@ -1,7 +1,7 @@ # Implementation Plan — UBI Judgments (engine-neutral User Behavior Insights) **Date:** 2026-05-29 -**Status:** Approved (cross-model converged at 3-cycle cap; all 3 GPT-5.5 findings accepted — see footer) +**Status:** Complete (PR #317, squash-merged 2026-05-29). 13/13 stories shipped; Gemini (6) + GPT-5.5 final review (6) adjudicated; deferrals → `chore_ubi_reader_search_after_pagination`, `chore_ubi_hybrid_template_render`, `feat_demo_ubi_study_comparison`. **Primary spec:** [feature_spec.md](feature_spec.md) (Approved 2026-05-29; cross-model converged at cycle-3 cap with all 10 findings accepted — see spec §19 D-10) **Policy source(s):** - [docs/01_architecture/api-conventions.md](../../../../01_architecture/api-conventions.md) — `/api/v1/` prefix, error envelope `{"detail":{"error_code","message","retryable"}}`, cursor pagination, `X-Total-Count` diff --git a/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/pipeline_status.md b/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/pipeline_status.md new file mode 100644 index 00000000..942d3e09 --- /dev/null +++ b/docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/pipeline_status.md @@ -0,0 +1,53 @@ +# Pipeline Status — UBI Judgments (engine-neutral User Behavior Insights) + +## Idea +- Status: Complete +- File: [`idea.md`](idea.md) +- Origin: external review 2026-05-22; 2026-05-27 reframe (Solr bundle); 2026-05-29 `feat_ubi_onramp` merged back in; 2026-05-29 preflight refresh + +## Spec +- Status: Approved +- Date: 2026-05-29 +- File: [`feature_spec.md`](feature_spec.md) +- Cross-model review: GPT-5.5 passed (3 cycles; cap hit; 10 findings — 1 H + 1 M cycle 1/2 + 2 H + 4 M + 2 L cycle 3 — all accepted and applied in place; see spec D-10) +- Phases: 1 default (single-phase delivery); contingency Phase 2 split decided at impl-plan-gen time if bundled diff exceeds ~1500 LOC +- Scope: ~700 LOC backend + ~350 LOC frontend + ~300 LOC tests + 1 additive Alembic migration (`0021_judgment_lists_generation_params.py`) + +## Plan +- Status: Approved +- Date: 2026-05-29 +- File: [`implementation_plan.md`](implementation_plan.md) +- Cross-model review: GPT-5.5 passed (3 cycles; cap hit; all 3 findings accepted — see plan footer) +- Stories: 14 across 5 epics +- Phases covered: Phase 1 (all 11 FRs, single-phase delivery per spec D-6) + +## Implementation +- Status: **Complete (PR #317, squash-merged 2026-05-29)** +- CI: `pr.yml` green on the merge SHA; DCO + secrets-defense green. +- Stories: 13 of 13 shipped (all 5 epics). Story 5.2's E2E half + the + hybrid-template-render contract cleanup were the only deferrals — both + resolved below. +- Reviews: Gemini Code Assist (6 findings, all accepted + fixed) + GPT-5.5 + final cross-model review (6 findings: 4 fixed, 1 documented, 1 + analyzed as working-as-designed) — adjudication tables posted on the PR. +- Tests at merge: 1,719 backend unit + 931 UI vitest + 4 UBI E2E (live + ES + worker, no mocking) all green; mypy --strict clean (507 files). + +### Notable: E2E surfaced a real backend bug +The rung-3 E2E (real engine) caught what stubbed unit tests structurally +could not: `UbiReader` requested `size=50000` > the engine's default +`index.max_result_window` (10000) → "all shards failed" → swallowed → +spurious `UBI_INSUFFICIENT_DATA` on dense clusters. Fixed (cap at 10k + +clamp + regression guard); full-traffic aggregation deferred to +`chore_ubi_reader_search_after_pagination`. + +### Deferred follow-ups (all captured as idea files) +- `chore_ubi_reader_search_after_pagination` (P2) — exact full-traffic + UBI aggregation via `search_after` (current: 10k-event sample). +- `chore_ubi_hybrid_template_render` (P3) — drop the now-vestigial + `current_template_id` requirement for hybrid (a product/contract + decision; the worker's per-pair `get_document` scoring is correct per + FR-2). +- `feat_demo_ubi_study_comparison` (P1) — synthetic UBI in the demo + reseed + UBI-vs-LLM study comparison (operator-requested). +- **Resume command:** `/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md 2.1` (the second argument is the next story ID; `--all` would batch-run Epics 2-5, which is the right call if the next session has token budget for it). diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index de44ae8b..e0628312 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -397,7 +397,7 @@

MVP1 Progress

Scoped items done
-
95 / 95
+
96 / 96
100% of feat_/infra_/chore_/epic_ items past idea stage
@@ -483,7 +483,7 @@

Implementing 0

-

Done 130

+

Done 131

@@ -888,6 +888,19 @@

Done 130

+
+ +
+ Feature + + PR #317 merged 2026-05-29 +
+
Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic
+ + +
+ +
@@ -2377,6 +2390,8 @@

Dependency graph (feat_ + infra_)

class chore_state_md_size_compression done; feat_study_sub_warmup_guard["study sub warmup guard"] class feat_study_sub_warmup_guard done; + feat_ubi_judgments["ubi judgments"] + class feat_ubi_judgments done; feat_study_lifecycle --> feat_digest_proposal feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_llm_judgments @@ -2620,6 +2635,8 @@

Dependency graph (feat_ + infra_)

class chore_state_md_size_compression done; feat_study_sub_warmup_guard["study sub warmup guard"] class feat_study_sub_warmup_guard done; + feat_ubi_judgments["ubi judgments"] + class feat_ubi_judgments done; feat_study_lifecycle --> feat_digest_proposal feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_llm_judgments diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index 441c336d..3819eb67 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -382,12 +382,12 @@

RelyLoop MVP2 Dashboard

-
-
Next up — Feature, currently in Implementing
- -
Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters
-
Implementation in progress — resume to finish
- /impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md --all # resume in-progress +
+
Next up
+
All scoped MVP2 features shipped 🎉
+
+ Pull from the Idea backlog or capture a new feature spec. +
@@ -395,15 +395,15 @@

RelyLoop MVP2 Dashboard

MVP2 Progress

-
+
Scoped items done
-
1 / 2
-
50% of feat_/infra_/chore_/epic_ items past idea stage
-
+
1 / 1
+
100% of feat_/infra_/chore_/epic_ items past idea stage
+
Pending work
-
18
+
17
every not-done feat/infra/chore/bug across all priorities
@@ -420,7 +420,7 @@

MVP2 Progress

P1
-
3
+
2
high-value, ready when P0 clears
@@ -435,7 +435,7 @@

MVP2 Progress

Legacy "Path to MVP2"
-
10
+
9
scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)
@@ -446,7 +446,7 @@

MVP2 Progress

In flight: - 1 feature(s) actively shipping + 0 feature(s) actively shipping
@@ -698,19 +698,7 @@

Plan 0

-

Implementing 1

- -
- -
- Feature - P1 - -
-
Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic
- - -
+

Implementing 0

@@ -742,8 +730,6 @@

Dependency graph (feat_ + infra_)

classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; - feat_ubi_judgments["ubi judgments"] - class feat_ubi_judgments implement; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done;
diff --git a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/pipeline_status.md b/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/pipeline_status.md deleted file mode 100644 index 570ce1d7..00000000 --- a/docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/pipeline_status.md +++ /dev/null @@ -1,64 +0,0 @@ -# Pipeline Status — UBI Judgments (engine-neutral User Behavior Insights) - -## Idea -- Status: Complete -- File: [`idea.md`](idea.md) -- Origin: external review 2026-05-22; 2026-05-27 reframe (Solr bundle); 2026-05-29 `feat_ubi_onramp` merged back in; 2026-05-29 preflight refresh - -## Spec -- Status: Approved -- Date: 2026-05-29 -- File: [`feature_spec.md`](feature_spec.md) -- Cross-model review: GPT-5.5 passed (3 cycles; cap hit; 10 findings — 1 H + 1 M cycle 1/2 + 2 H + 4 M + 2 L cycle 3 — all accepted and applied in place; see spec D-10) -- Phases: 1 default (single-phase delivery); contingency Phase 2 split decided at impl-plan-gen time if bundled diff exceeds ~1500 LOC -- Scope: ~700 LOC backend + ~350 LOC frontend + ~300 LOC tests + 1 additive Alembic migration (`0021_judgment_lists_generation_params.py`) - -## Plan -- Status: Approved -- Date: 2026-05-29 -- File: [`implementation_plan.md`](implementation_plan.md) -- Cross-model review: GPT-5.5 passed (3 cycles; cap hit; all 3 findings accepted — see plan footer) -- Stories: 14 across 5 epics -- Phases covered: Phase 1 (all 11 FRs, single-phase delivery per spec D-6) - -## Implementation -- Status: **In progress — Epic 1 of 5 complete** (branch `feat/ubi-judgments`, parent `main` at `68fa357c`) -- Pacing: Epic by epic, pause between each (operator confirmation 2026-05-29; multi-session execution to respect single-conversation context limits) - -### Done - -| Epic | Story | Description | Commit | -|---|---|---|---| -| 1 | 1.1 | Migration `0021_judgment_lists_generation_params` (JSONB column for UBI worker resume) | `5acdee15` | -| 1 | 1.2 | `domain/ubi/` pure-domain library (features, async converter Protocol + 3 impls, position-bias prior, 58 unit tests) | `6036586a` | -| — | — | Planning bundle (idea refresh + spec + plan + dashboards) | `84c810aa` | - -### Next session — resume at Epic 2 - -Run: `/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md 2.1` - -Epic 2 stories (the next 3 to ship): - -| Story | Description | Estimated scope | -|---|---|---| -| 2.1 | `UbiReader` service — engine-agnostic two-index scan + client-side join | ~200 LOC backend + 2 integration tests (the canned-features + the no-cluster-writes guard) | -| 2.2 | Readiness service + `start_ubi_judgment_generation` dispatcher refactor — shared helpers extracted from `start_judgment_generation` (parity-preserving) | ~250 LOC backend + 1 integration test for the dispatcher matrix | -| 2.3 | `_SourceBreakdown` evolution + `JudgmentSourceFilterWire` widening + 4 new wire Literals (`UbiConverterKind`, `JudgmentGenerationMethodWire`, `UbiReadinessRungWire`, `UbiMappingStrategyWire`) | ~100 LOC backend + 2 contract tests + 1 unit test | - -Sequencing: 2.1 first (used by 2.2), then 2.2 + 2.3 in parallel (different files, no shared state). Branch tip remains `feat/ubi-judgments`; no rebase needed. - -### Remaining work (Epics 3–5) - -| Epic | Stories | Theme | -|---|---|---| -| 3 | 3.1, 3.2, 3.3, 3.4 | API endpoints + worker + agent tool | -| 4 | 4.1, 4.2, 4.3 | Frontend (enums + hook + badge, dialog method picker + nudge + sparse card, value-delta + recovery cards) | -| 5 | 5.1, 5.2 | Docs (runbook + glossary + FAQ + tutorial + umbrella spec patches) + E2E suite (4 specs + `seed_ubi.ts` helper) | - -After Epic 5 completes: post-implementation ceremony (test coverage audit, deferred work extraction, tangential sweep, guide impact, push, CI watch, Gemini adjudication, final GPT-5.5 review, finalization to `implemented_features/2026_MM_DD_feat_ubi_judgments/`). - -## Notes for `/impl-execute` resume -- **Epic 1 is durable** — the migration is round-trip-clean against the local Postgres, the domain library has 58 passing unit tests, and the branch is push-ready (will push at end of this session per the operator's "push after Epic 1" instruction). -- **No code in subsequent stories depends on un-pushed Epic 1 state** — Story 2.1 imports `backend.app.domain.ubi.features.UbiEvent` and `aggregate_features`; Story 2.2 imports `backend.app.services.ubi_reader` (Story 2.1's output) + the existing dispatcher helpers; etc. -- **The plan's §11 consistency review remains green** — none of the cycle-3 fixes contradicted Epic 1 stories (the `generation_kind: 'ubi'` discriminator goes in the JSONB at INSERT time in Story 2.2's dispatcher; the column itself was added in Story 1.1). -- **Resume command:** `/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md 2.1` (the second argument is the next story ID; `--all` would batch-run Epics 2-5, which is the right call if the next session has token budget for it). diff --git a/docs/08_guides/README.md b/docs/08_guides/README.md index 059c71ca..0bf62d21 100644 --- a/docs/08_guides/README.md +++ b/docs/08_guides/README.md @@ -60,6 +60,7 @@ Shipped guides: **Workflows still NOT covered by a guide:** - **D4 (full) — Open PR end-to-end**: the trigger is captured in guide 02; the worker's GitHub interaction requires a registered config-repo + real PAT, which is operator-credential territory. +- **UBI judgment generation**: guide 09 covers the LLM-as-judge path; the UBI (click/dwell) alternative needs a cluster with captured behavior traffic, which is operator-data territory. See the [UBI judgment-generation runbook](../03_runbooks/ubi-judgment-generation.md) and the optional UBI upgrade step in [`tutorial-first-study.md`](tutorial-first-study.md). ### Adding or regenerating a guide diff --git a/docs/08_guides/llm-endpoint-setup.md b/docs/08_guides/llm-endpoint-setup.md index a15f1caa..28a94b0c 100644 --- a/docs/08_guides/llm-endpoint-setup.md +++ b/docs/08_guides/llm-endpoint-setup.md @@ -7,6 +7,8 @@ RelyLoop talks to **any OpenAI-compatible endpoint** via a single environment va This guide walks through the most common configurations side by side. +> **Not the only judgment source.** This guide configures the LLM endpoint that powers LLM-as-judge generation, the digest narrative, and the chat agent. If your cluster has captured click/dwell traffic, User Behavior Insights (UBI) generates judgments from real user behavior with no external LLM — see the [UBI judgment-generation runbook](../03_runbooks/ubi-judgment-generation.md). + ## The pattern in one paragraph RelyLoop uses the `openai` Python SDK pointed at whatever URL you set in `OPENAI_BASE_URL`. The SDK speaks the OpenAI Chat Completions wire protocol (`POST /v1/chat/completions`). Anything that serves that protocol — OpenAI itself, Azure OpenAI's OpenAI-compatible mode, Ollama, LM Studio, vLLM, HuggingFace TGI, OpenRouter, LiteLLM proxy, or any other compatible server — works without changing a line of RelyLoop code. The API key is mounted at `./secrets/openai_key` (or whatever path you point `OPENAI_API_KEY_FILE` at) per [`CLAUDE.md`](../../CLAUDE.md) Absolute Rule #2 on secrets handling. diff --git a/docs/08_guides/quick-tour.md b/docs/08_guides/quick-tour.md index f5c46e51..138d8459 100644 --- a/docs/08_guides/quick-tour.md +++ b/docs/08_guides/quick-tour.md @@ -359,10 +359,12 @@ Three points to land, in order: optimization engine doesn't care which backend implements the search; it tunes whatever knobs the adapter exposes. Your existing Fusion query pipelines map directly. -2. **Real signals next release.** MVP1.5 ("Real Signals") replaces - LLM-as-judge with UBI click/dwell data as a first-class judgment - source. Your existing Fusion Signals capture maps directly — you - grade studies against your users' real behavior, not an LLM's guess. +2. **Real signals, available now.** User Behavior Insights (UBI) is a + first-class judgment source alongside LLM-as-judge: when your cluster + has captured click/dwell traffic, you grade studies against your + users' real behavior instead of an LLM's guess. UBI is engine-neutral + (Elasticsearch, OpenSearch, Solr) and complements the LLM path — the + demo above uses LLM because it needs no traffic to run. 3. **Open source, self-hosted.** Apache 2.0. Run it on a laptop, run it on your own infra. The PR-based ship workflow keeps your existing CI and existing reviewers in charge of production. diff --git a/docs/08_guides/workflows-overview.md b/docs/08_guides/workflows-overview.md index b4c29dbf..0e3fa9b9 100644 --- a/docs/08_guides/workflows-overview.md +++ b/docs/08_guides/workflows-overview.md @@ -101,6 +101,14 @@ These define *what you're tuning* (the query template knobs) and *what good look - **Side effects:** Creates a `judgment_lists` row (status=`generating`), enqueues the worker. Cost-gated by daily OpenAI budget. ~$0.01–$0.05 with `gpt-4o-mini` on the 48-query tutorial set. - **Auto-recovery:** If the worker crashes mid-list, the boot-time sweep + the every-15-minute `resume_stuck_judgment_lists` cron re-enqueue stuck lists (capped at 24 attempts/day to prevent infinite loops). See [`feat_judgments_periodic_resume_sweep`](../00_overview/implemented_features/2026_05_14_feat_judgments_periodic_resume_sweep/). +#### B4a. Generate judgments via UBI (User Behavior Insights) +- **Solves:** Grading study trials against *real user behavior* (clicks, dwell time) instead of an LLM's prediction. Complementary to B4 — UBI needs a cluster that has captured click/dwell traffic; LLM works with zero traffic. Engine-neutral: works on Elasticsearch, OpenSearch, and Solr from the same `ubi_queries` + `ubi_events` schema. +- **Readiness probe:** `GET /api/v1/clusters/{id}/ubi-readiness` reports a rung (`rung_0`..`rung_3`) from the captured event volume. The "Generate judgments" modal surfaces an on-ramp nudge below `rung_3` and a sparse-data card when coverage is thin. +- **How (UI):** [`/query-sets/[id]`](../../ui/src/app/query-sets/[id]/page.tsx) → "Generate judgments" modal → **Method** picker: `UBI (click-through)`, `UBI (dwell-time)`, or `Hybrid UBI + LLM` (UBI for high-signal pairs, LLM fills the long tail). The detail page then shows a value-delta card comparing the UBI ratings against any prior LLM list. +- **How (API):** `POST /api/v1/judgments/generate-from-ubi` returns `202 ACCEPTED` with a `judgment_list_id`. The `generate_judgments_from_ubi` worker runs the converter; the chat agent exposes the same path via the `generate_judgments_from_ubi` tool. +- **Side effects:** Creates a `judgment_lists` row (status=`generating`, `source='click'`). Read-only against the cluster — RelyLoop never writes to UBI indices. Pure-UBI converters cost no LLM tokens; hybrid mode costs match B4 for the filled pairs only. +- **Runbook:** [`../03_runbooks/ubi-judgment-generation.md`](../03_runbooks/ubi-judgment-generation.md) for per-engine UBI capture setup + converter selection. + #### B5. Import pre-curated judgments (tutorial / sideload path) - **Solves:** Bypassing LLM generation when you already have human-labeled judgments (e.g., from Amazon ESCI, a previous tool, or hand-curation). - **How (API):** `POST /api/v1/judgment-lists/import` — bulk-insert with strict validation (every `query_id` must exist in the query set, duplicate `(query_id, doc_id)` rejected). Sets status=`complete` immediately, no LLM call. @@ -108,7 +116,7 @@ These define *what you're tuning* (the query template knobs) and *what good look #### B6. Review and override individual judgments - **Solves:** The LLM gets things wrong — sometimes spectacularly. Engineers need to inspect the (query, doc, rating, LLM-reasoning) tuples and override the bad ones, *without* re-running generation. -- **How (UI):** [`/judgments/[id]`](../../ui/src/app/judgments/[id]/page.tsx) — table with a source filter (`llm` / `human`), inline override. +- **How (UI):** [`/judgments/[id]`](../../ui/src/app/judgments/[id]/page.tsx) — table with a source filter (`llm` / `human` / `click`), inline override. - **How (API):** `PATCH /api/v1/judgment-lists/{id}/judgments/{judgment_id}` — UPSERT semantics; the human override coexists with the LLM judgment and supersedes it for scoring. #### B7. Calibrate LLM judgments against human ground truth diff --git a/state.md b/state.md index 864949f1..b123ce3f 100644 --- a/state.md +++ b/state.md @@ -14,8 +14,8 @@ MVP1 (v0.1) **shipped** — all six differentiators live (Bayesian/TPE optimizer ## Current branch / execution context -- **Branch:** `feat/ubi-judgments`. `feat_ubi_judgments` shipped end-to-end (11 of 13 stories: migration + domain + reader + readiness service + dispatcher + 2 endpoints + worker + agent tool + frontend dialog + value-delta cards + runbook/FAQ/data-model docs). Stories 5.2 (E2E suite) + the 7-of-10 Story 5.1 doc artifacts deferred to `chore_ubi_e2e_suite` + `chore_ubi_docs_followup`; DB-backed integration tests deferred to `chore_ubi_integration_tests`. All three follow-up idea files committed. -- **Active feature:** `feat_ubi_judgments` PR pending push. +- **Branch:** none active — `feat_ubi_judgments` (PR #317) squash-merged to `main` 2026-05-29; folder finalized to `implemented_features/2026_05_29_feat_ubi_judgments/` via the `docs/finalize-ubi-judgments` docs PR. See `/pipeline status` for the next MVP2 item. +- **Active feature:** none in flight. - **Alembic head:** `0021_judgment_lists_generation_params` (added by `feat_ubi_judgments` Story 1.1 — JSONB column for UBI worker resume payload). - **Python:** 3.13. **Frontend stack:** Next 16 (App Router + Turbopack), React 19, Tailwind 4 (CSS-first), Vitest 4, ESLint 9 (flat), TypeScript 6, Playwright (chromium, single worker) for E2E. - **Coverage gates:** backend 80% (`fail_under` in pyproject), UI vitest + tsc + ESLint + Next build, plus a full-stack smoke E2E job. Live pass counts: see the latest `pr.yml` run (the historical per-feature counts moved to `state_history.md`). @@ -24,24 +24,24 @@ MVP1 (v0.1) **shipped** — all six differentiators live (Bayesian/TPE optimizer Detail + reasoning for each is in [`state_history.md`](state_history.md). -- **2026-05-29** — `feat_ubi_judgments` (PR pending push). MVP2 second feature: engine-neutral User Behavior Insights judgment generation. Ships 11 of 13 planned stories: migration 0021 (judgment_lists.generation_params JSONB) → domain/ubi/ pure-domain library (FeatureVec, async SignalsConverter Protocol + 3 impls, position-bias prior) → UbiReader (engine-neutral two-index scan + client-side join, no new adapter method) → ubi_readiness classifier (rung_0..rung_3, 60s Redis cache) → start_ubi_judgment_generation dispatcher (refactor extracts 5 shared helpers; LLM dispatcher parity preserved by all 12 existing tests) → 5 new wire Literals + _SourceBreakdown three-term evolution (FR-10) → GET /clusters/{id}/ubi-readiness + POST /judgments/generate-from-ubi endpoints → generate_judgments_from_ubi Arq worker with mapping_strategy + hybrid LLM-fill callback → 21st agent tool + orchestrator prompt update → frontend method picker dialog + on-ramp nudge + sparse-data card + value-delta + ambiguous-skip recovery cards → operator runbook + 3 FAQ entries + data-model patches. Test count: +110 new tests across unit (~95) + contract (~17) layers; all 1715 backend unit + 931 UI vitest green; mypy --strict clean across 501 files. Stories 5.2 (E2E) + remaining Story 5.1 docs + DB-backed integration tests deferred via 3 follow-up idea files (chore_ubi_e2e_suite, chore_ubi_docs_followup, chore_ubi_integration_tests). +- **2026-05-29** — `feat_ubi_judgments` (PR #317, squash-merged). MVP2 second feature: engine-neutral User Behavior Insights judgment generation, shipped end-to-end (all 13 stories incl. E2E + DB-backed integration tests + operator docs — none deferred). Migration 0021 (judgment_lists.generation_params JSONB) → domain/ubi/ pure-domain library (FeatureVec, async SignalsConverter Protocol + 3 impls, position-bias prior) → UbiReader (engine-neutral two-index scan + client-side join, no new adapter method) → ubi_readiness classifier (rung_0..rung_3, 60s Redis cache) → start_ubi_judgment_generation dispatcher (refactor extracts 5 shared helpers; LLM dispatcher parity preserved by all 12 existing tests) → 5 new wire Literals + _SourceBreakdown three-term evolution (FR-10) → GET /clusters/{id}/ubi-readiness + POST /judgments/generate-from-ubi endpoints → generate_judgments_from_ubi Arq worker with mapping_strategy + hybrid LLM-fill callback → 21st agent tool + orchestrator prompt update → frontend method picker dialog + on-ramp nudge + sparse-data card + value-delta + ambiguous-skip recovery cards → operator runbook + 3 FAQ entries + data-model patches → 4 Playwright E2E specs (rung_0/rung_3/hybrid/source-filter) green against the live ES-backed stack. **Real-engine E2E caught a production bug**: UbiReader requested `size=50000 > ES index.max_result_window (10000)` → "all shards failed" swallowed by the adapter → spurious `UBI_INSUFFICIENT_DATA` on dense clusters; fixed by clamping both index scans to `ES_MAX_RESULT_WINDOW=10000` + regression guard. Cross-model review: 6 Gemini findings + 6 GPT-5.5 findings all adjudicated (fixed or documented as working-as-designed). Remaining follow-ups are pure deferrals, not gaps: `chore_ubi_reader_search_after_pagination` (P2, >10k-event clusters), `chore_ubi_hybrid_template_render` (P3, vestigial-template contract cleanup — current behavior correct per FR-2), `feat_demo_ubi_study_comparison` (P1, side-by-side UBI-vs-LLM demo study). - **2026-05-29** — `feat_study_sub_warmup_guard` (PR #316). First MVP2 feature ships. Closes the Custom-mode sub-warmup gap left open by `chore_study_default_stop_conditions` (2026-05-23): adds a non-blocking inline amber warning to the create-study modal's Step 5 when operators enter `max_trials < 50` in Custom mode. Hoists the inline `50` at `optuna_runtime.py:154` to a module-level `STUDIES_TPE_WARMUP_FLOOR` constant; frontend `SUB_WARMUP_FLOOR` mirrors with the `// Values must match` discipline comment + value-lock unit test. 3 new backend pytest assertions (value lock + `floor-1=49` boundary + `floor=50` boundary using constant); 5 new vitest cases (AC-1..AC-4 + AC-6 submit-non-blocking). Single-phase per spec D-6; digest narrative routed to `feat_study_convergence_indicator`. Cross-model review: spec converged at cycle 3 (13/13 accepted), plan converged at cycle 3 (8 accepted + 1 rejected with counter-evidence), phase-gate converged at cycle 2 (clean), final GPT-5.5 + 1 Gemini finding accepted. - **2026-05-29** — `docs: reclassify 2 deferred MVP1 items → 99_backlog/03_mvp3` (PR #310, docs-only). Empties `01_mvp1/` — MVP1 actionable backlog fully drained. `chore_demo_reseed_stale_recovery_atomic_cas` → `99_backlog/` (already Priority: Backlog); `infra_agent_sibling_worktree_isolation` → `99_backlog/` (phases 1+2 shipped, only phase3 remains, defer-until-incident). Dashboards regenerated. - **2026-05-29** — `bug_smoke_studies_data_table_search_flake` (PR #308 + finalization #309). Hardened the flaky `studies-data-table.spec.ts:20` search-visibility assertion: scoped it to the `studies-table` element + 15s web-first timeout to ride out the debounce→refetch→render race on slow CI runners. e2e-only; no product change. - **2026-05-29** — `ci(pr): SKIP_HEAVY_CI kill-switch` (PR #307, infra). Added an `if:` guard on the 5 `pr.yml` jobs over 1 min so a repo variable can skip them (temporary GitHub Actions budget measure). See the Active CI note above — variable currently set, auto-restores ~2026-06-01. ## In flight -- `feat_ubi_judgments` (PR pending push). Ships the UBI substrate + API + worker + frontend + operator docs. Three follow-up idea files committed for the deferred sub-scope (E2E, doc completeness, DB-backed integration tests). +- _None._ `feat_ubi_judgments` (PR #317) merged 2026-05-29; finalization docs PR (`docs/finalize-ubi-judgments`) in review. ## Queued (priority-ordered by dashboard / dep graph) **Source of truth:** [`docs/00_overview/DASHBOARD.md`](docs/00_overview/DASHBOARD.md) + [`docs/00_overview/MVP1_DASHBOARD.md`](docs/00_overview/MVP1_DASHBOARD.md) (regenerated by the `mvp1-dashboard-regen` pre-commit hook). Run `/pipeline status` for the live view. -**MVP1 backlog is fully drained** (`01_mvp1/` empty as of PR #310). The next stop is **MVP2 / v0.2 — "Three-Engine + Real Signals"**. The `02_mvp2/` bucket currently holds 11 folders (run `ls docs/00_overview/planned_features/02_mvp2/` for the live list): +**MVP1 backlog is fully drained** (`01_mvp1/` empty as of PR #310). The next stop is **MVP2 / v0.2 — "Three-Engine + Real Signals"**. With `feat_ubi_judgments` merged, the `02_mvp2/` bucket holds 18 folders (run `ls docs/00_overview/planned_features/02_mvp2/` for the live list): -- **Headliners:** `infra_adapter_solr` (Apache Solr adapter), `feat_ubi_judgments` (UBI judgment source), `feat_chat_last_message_preview`, `feat_fts_rank_ordering`. -- **Bugs held for MVP2:** `bug_chat_long_conversation_truncation` (investigation `bug_fix.md` exists; pullable forward but deferred for scope discipline — latency-of-impact is zero today), `bug_webhook_concurrent_merge_race_timing_sensitive`. -- **Chores:** `chore_auto_followup_parent_advisory_lock`, `chore_demo_seeding_integration_tests_rewrite`, `chore_studies_post_arq_spy_fixture`, `chore_template_library_expansion`, `infra_arq_subprocess_test`. +- **Headliners:** `infra_adapter_solr` (Apache Solr adapter), `feat_fts_rank_ordering`, `feat_chat_last_message_preview`, `feat_overnight_autopilot`, `feat_query_normalization_tuning`, `feat_study_convergence_indicator`, `feat_demo_ubi_study_comparison` (side-by-side UBI-vs-LLM demo study, P1 — spun out of `feat_ubi_judgments`). +- **Bugs held for MVP2:** `bug_chat_long_conversation_truncation` (investigation `bug_fix.md` exists; pullable forward but deferred for scope discipline — latency-of-impact is zero today), `bug_webhook_concurrent_merge_race_timing_sensitive`, `bug_seed_meaningful_demos_silent_bulk_errors`. +- **Chores/infra:** `chore_auto_followup_parent_advisory_lock`, `chore_demo_seeding_integration_tests_rewrite`, `chore_studies_post_arq_spy_fixture`, `chore_template_library_expansion`, `chore_ubi_reader_search_after_pagination` (P2, search_after for >10k-event clusters — spun out of `feat_ubi_judgments`), `chore_ubi_hybrid_template_render` (P3, vestigial-template contract cleanup — spun out of `feat_ubi_judgments`), `infra_arq_subprocess_test`. **Other buckets:** `03_mvp3/` (Observable — includes `infra_optuna_orphan_reaper`, deferred from MVP1 per spec §11 operational tolerance), `04_ga/`, `99_backlog/` (4 defer-until-incident items), `00_unsure/` (`bug_seed_meaningful_demos_silent_bulk_errors`).