Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/00_overview/DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ _Top-level index across MVP1 → GA v1+ as of **2026-05-29**. Click a release na

| Release | Theme | Progress | Status |
|---|---|---|---|
| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 95 / 95 scoped done | **Complete** |
| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 1 / 2 scoped done · 10 remaining | **In progress** |
| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 96 / 96 scoped done | **Complete** |
| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 1 / 1 scoped done · 9 remaining | **In progress** |
| MVP3 / v0.3 | Observable | — | **Not yet scoped** |
| [GA v1 / v1.0](GA_DASHBOARD.md) | Production-ready | 1 item(s) queued | **Held / queued** |

Expand Down
7 changes: 5 additions & 2 deletions docs/00_overview/MVP1_DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Pull from the Idea backlog or capture a new feature spec.

| Metric | Value |
|---|---|
| Scoped items done | **95 / 95** (100%) — feat_/infra_/chore_/epic_ past idea stage |
| Scoped items done | **96 / 96** (100%) — feat_/infra_/chore_/epic_ past idea stage |
| Pending work | **0** items (every not-done feat/infra/chore/bug across all priorities) |
| → P0 — do next | **0** unblocking / paying daily cost |
| → P1 | **0** high-value, ready when P0 clears |
Expand All @@ -27,7 +27,7 @@ Pull from the Idea backlog or capture a new feature spec.

## Pipeline

### Done (130)
### Done (131)

| Feature | Type | One-liner | Depends on | Status |
|---|---|---|---|---|
Expand Down Expand Up @@ -62,6 +62,7 @@ Pull from the Idea backlog or capture a new feature spec.
| [feat_study_preflight_overlap_probe](implemented_features/2026_05_22_feat_study_preflight_overlap_probe/feature_spec.md) | Feature | `POST /api/v1/studies` issues a single bounded `ids`-existence query against the study's target asking "for the *first* query in the query set that has any judgments (chosen deterministically by `id A | — | [PR #193](https://github.com/SoundMindsAI/relyloop/pull/193) merged 2026-05-21 |
| [feat_study_sub_warmup_guard](implemented_features/2026_05_29_feat_study_sub_warmup_guard/feature_spec.md) | Feature | A non-blocking inline warning appears under the `max_trials` input whenever the derived preset is `custom` AND `max_trials < STUDIES_TPE_WARMUP_FLOOR (= 50)`, naming Focused/Standard as one-click reme | — | [PR #316](https://github.com/SoundMindsAI/relyloop/pull/316) merged 2026-05-29 |
| [feat_study_target_judgment_mismatch_guard](implemented_features/2026_05_21_feat_study_target_judgment_mismatch_guard/feature_spec.md) | Feature | `POST /api/v1/studies` rejects the mismatch at create time with a specific machine-readable error code (`JUDGMENT_TARGET_MISMATCH`, 422). | — | [PR #184](https://github.com/SoundMindsAI/relyloop/pull/184) merged 2026-05-21 |
| [feat_ubi_judgments](implemented_features/2026_05_29_feat_ubi_judgments/feature_spec.md) | Feature | Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic | — | [PR #317](https://github.com/SoundMindsAI/relyloop/pull/317) merged 2026-05-29 |
| [infra_adapter_elastic](implemented_features/2026_05_10_infra_adapter_elastic/feature_spec.md) | Infra | A single `ElasticAdapter` implements the `SearchAdapter` Protocol and serves both Elasticsearch (8.11+ / 9.x) and OpenSearch (2.x / 3.x), distinguished by a `engine_type` column. | — | [PR #16](https://github.com/SoundMindsAI/relyloop/pull/16) merged 2026-05-10 |
| [infra_ci_smoke_makeup](implemented_features/2026_05_13_infra_ci_smoke_makeup/idea.md) | Infra | CI runs `make test-unit && make test-integration && make test-contract` against a service-container Postgres on `localhost:5432` — a synthetic environment that masks every real-world `make up` failure | — | Complete |
| [infra_dashboard_regen_pre_commit_conflict](implemented_features/2026_05_14_infra_dashboard_regen_pre_commit_conflict/idea.md) | Infra | The pre-commit pipeline does: | — | Complete |
Expand Down Expand Up @@ -379,6 +380,8 @@ graph LR
class chore_state_md_size_compression done;
feat_study_sub_warmup_guard["study sub warmup guard"]
class feat_study_sub_warmup_guard done;
feat_ubi_judgments["ubi judgments"]
class feat_ubi_judgments done;
feat_study_lifecycle --> feat_digest_proposal
feat_llm_judgments --> feat_digest_proposal
infra_foundation --> feat_llm_judgments
Expand Down
28 changes: 9 additions & 19 deletions docs/00_overview/MVP2_DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,24 @@ _Reflects feature-folder state as of **2026-05-29** (latest mtime of any planned

## Next up

**[feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md)** — Feature, currently in **Implementing**
All scoped MVP2 features shipped 🎉

> Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters

Implementation in progress — resume to finish

```bash
/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md --all # resume in-progress
```
Pull from the Idea backlog or capture a new feature spec.

## MVP2 Progress

| Metric | Value |
|---|---|
| Scoped items done | **1 / 2** (50%) — feat_/infra_/chore_/epic_ past idea stage |
| Pending work | **18** items (every not-done feat/infra/chore/bug across all priorities) |
| Scoped items done | **1 / 1** (100%) — feat_/infra_/chore_/epic_ past idea stage |
| Pending work | **17** items (every not-done feat/infra/chore/bug across all priorities) |
| → P0 — do next | **0** unblocking / paying daily cost |
| → P1 | **3** high-value, ready when P0 clears |
| → P1 | **2** high-value, ready when P0 clears |
| → P2 (default) | 11 important to file, not blocking |
| → Backlog | 4 captured for record, not planned |
| Open bugs | 3 |
| Legacy "Path to MVP2" | 10 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) |
| Legacy "Path to MVP2" | 9 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) |
| Backlog ideas | 8 idea-only feat/infra (not yet scoped into MVP2) |
| In flight | 1 feature(s) actively shipping |
| In flight | 0 feature(s) actively shipping |

## Pipeline

Expand All @@ -39,11 +33,9 @@ Implementation in progress — resume to finish
|---|---|---|---|---|
| [feat_contextual_help_mvp2](implemented_features/2026_05_15_feat_contextual_help_mvp2/idea.md) | Feature | Phase 1 covered the create-study modal + study-detail surface — the steepest onboarding cliff. Two clusters of surfaces remain that a relevance engineer encounters after running their first study: | — | [PR #124](https://github.com/SoundMindsAI/relyloop/pull/124) merged 2026-05-15 |

### Implementing (1)
### Implementing (0)

| # | Priority | Feature | Type | One-liner | Depends on | Status |
|---|---|---|---|---|---|---|
| 1 | P1 | [feat_ubi_judgments](planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md) | Feature | Operators with the OpenSearch / ES UBI plugin installed (today; Solr's first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic | — | — |
_None._

### Plan (0)

Expand Down Expand Up @@ -86,8 +78,6 @@ graph LR
classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e;
classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af;
classDef idea fill:#f1f5f9,stroke:#334155,color:#334155;
feat_ubi_judgments["ubi judgments"]
class feat_ubi_judgments implement;
feat_contextual_help_mvp2["contextual help mvp2"]
class feat_contextual_help_mvp2 done;
```
Expand Down
4 changes: 2 additions & 2 deletions docs/00_overview/dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -384,15 +384,15 @@ <h2>Releases</h2>
<div class="roadmap-row">
<div class="release-name"><a href="mvp1_dashboard.html">MVP1 / v0.1</a></div>
<div class="theme">The Loop</div>
<div class="progress">95 / 95 scoped done</div>
<div class="progress">96 / 96 scoped done</div>
<span class="state-pill complete">Complete</span>
</div>


<div class="roadmap-row">
<div class="release-name"><a href="mvp2_dashboard.html">MVP2 / v0.2</a></div>
<div class="theme">Three-Engine + Real Signals</div>
<div class="progress">1 / 2 scoped done · 10 remaining</div>
<div class="progress">1 / 1 scoped done · 9 remaining</div>
<span class="state-pill in_progress">In progress</span>
</div>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Implementation Plan — UBI Judgments (engine-neutral User Behavior Insights)

**Date:** 2026-05-29
**Status:** Approved (cross-model converged at 3-cycle cap; all 3 GPT-5.5 findings accepted — see footer)
**Status:** Complete (PR #317, squash-merged 2026-05-29). 13/13 stories shipped; Gemini (6) + GPT-5.5 final review (6) adjudicated; deferrals → `chore_ubi_reader_search_after_pagination`, `chore_ubi_hybrid_template_render`, `feat_demo_ubi_study_comparison`.
**Primary spec:** [feature_spec.md](feature_spec.md) (Approved 2026-05-29; cross-model converged at cycle-3 cap with all 10 findings accepted — see spec §19 D-10)
**Policy source(s):**
- [docs/01_architecture/api-conventions.md](../../../../01_architecture/api-conventions.md) — `/api/v1/<resource>` prefix, error envelope `{"detail":{"error_code","message","retryable"}}`, cursor pagination, `X-Total-Count`
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Pipeline Status — UBI Judgments (engine-neutral User Behavior Insights)

## Idea
- Status: Complete
- File: [`idea.md`](idea.md)
- Origin: external review 2026-05-22; 2026-05-27 reframe (Solr bundle); 2026-05-29 `feat_ubi_onramp` merged back in; 2026-05-29 preflight refresh

## Spec
- Status: Approved
- Date: 2026-05-29
- File: [`feature_spec.md`](feature_spec.md)
- Cross-model review: GPT-5.5 passed (3 cycles; cap hit; 10 findings — 1 H + 1 M cycle 1/2 + 2 H + 4 M + 2 L cycle 3 — all accepted and applied in place; see spec D-10)
- Phases: 1 default (single-phase delivery); contingency Phase 2 split decided at impl-plan-gen time if bundled diff exceeds ~1500 LOC
- Scope: ~700 LOC backend + ~350 LOC frontend + ~300 LOC tests + 1 additive Alembic migration (`0021_judgment_lists_generation_params.py`)

## Plan
- Status: Approved
- Date: 2026-05-29
- File: [`implementation_plan.md`](implementation_plan.md)
- Cross-model review: GPT-5.5 passed (3 cycles; cap hit; all 3 findings accepted — see plan footer)
- Stories: 14 across 5 epics
- Phases covered: Phase 1 (all 11 FRs, single-phase delivery per spec D-6)

## Implementation
- Status: **Complete (PR #317, squash-merged 2026-05-29)**
- CI: `pr.yml` green on the merge SHA; DCO + secrets-defense green.
- Stories: 13 of 13 shipped (all 5 epics). Story 5.2's E2E half + the
hybrid-template-render contract cleanup were the only deferrals — both
resolved below.
- Reviews: Gemini Code Assist (6 findings, all accepted + fixed) + GPT-5.5
final cross-model review (6 findings: 4 fixed, 1 documented, 1
analyzed as working-as-designed) — adjudication tables posted on the PR.
- Tests at merge: 1,719 backend unit + 931 UI vitest + 4 UBI E2E (live
ES + worker, no mocking) all green; mypy --strict clean (507 files).

### Notable: E2E surfaced a real backend bug
The rung-3 E2E (real engine) caught what stubbed unit tests structurally
could not: `UbiReader` requested `size=50000` > the engine's default
`index.max_result_window` (10000) → "all shards failed" → swallowed →
spurious `UBI_INSUFFICIENT_DATA` on dense clusters. Fixed (cap at 10k +
clamp + regression guard); full-traffic aggregation deferred to
`chore_ubi_reader_search_after_pagination`.

### Deferred follow-ups (all captured as idea files)
- `chore_ubi_reader_search_after_pagination` (P2) — exact full-traffic
UBI aggregation via `search_after` (current: 10k-event sample).
- `chore_ubi_hybrid_template_render` (P3) — drop the now-vestigial
`current_template_id` requirement for hybrid (a product/contract
decision; the worker's per-pair `get_document` scoring is correct per
FR-2).
- `feat_demo_ubi_study_comparison` (P1) — synthetic UBI in the demo
reseed + UBI-vs-LLM study comparison (operator-requested).
- **Resume command:** `/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md 2.1` (the second argument is the next story ID; `--all` would batch-run Epics 2-5, which is the right call if the next session has token budget for it).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since the status of the feat_ubi_judgments feature is now marked as Complete, the Resume command section is obsolete and can be removed.

Additionally, the path in the command (planned_features/02_mvp2/feat_ubi_judgments/...) is now broken because the folder has been moved to implemented_features/2026_05_29_feat_ubi_judgments/.

21 changes: 19 additions & 2 deletions docs/00_overview/mvp1_dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -397,7 +397,7 @@ <h2>MVP1 Progress</h2>
<div class="kpi-row">
<div class="kpi complete">
<div class="label">Scoped items done</div>
<div class="value">95 / 95</div>
<div class="value">96 / 96</div>
<div class="sub">100% of feat_/infra_/chore_/epic_ items past idea stage</div>
<div class="bar"><span style="width:100%"></span></div>
</div>
Expand Down Expand Up @@ -483,7 +483,7 @@ <h3>Implementing <span class="count">0</span></h3>
</div>

<div class="col done">
<h3>Done <span class="count">130</span></h3>
<h3>Done <span class="count">131</span></h3>

<div class="card feat" data-prefix="feat" data-priority="P2">
<div class="name"><a href="../../docs/00_overview/implemented_features/2026_05_21_feat_agent_propose_search_space/feature_spec.md">Agent Propose Search Space</a></div>
Expand Down Expand Up @@ -888,6 +888,19 @@ <h3>Done <span class="count">130</span></h3>
</div>


<div class="card feat" data-prefix="feat" data-priority="P2">
<div class="name"><a href="../../docs/00_overview/implemented_features/2026_05_29_feat_ubi_judgments/feature_spec.md">Ubi Judgments</a></div>
<div class="meta">
<span class="badge feat">Feature</span>

<a class="pr" href="https://github.com/SoundMindsAI/relyloop/pull/317">PR #317</a> <span>merged 2026-05-29</span>
</div>
<div class="one-liner">Operators with the OpenSearch / ES UBI plugin installed (today; Solr&#x27;s first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic</div>


</div>


<div class="card infra" data-prefix="infra" data-priority="P2">
<div class="name"><a href="../../docs/00_overview/implemented_features/2026_05_10_infra_adapter_elastic/feature_spec.md">Adapter Elastic</a></div>
<div class="meta">
Expand Down Expand Up @@ -2377,6 +2390,8 @@ <h2>Dependency graph (feat_ + infra_)</h2>
class chore_state_md_size_compression done;
feat_study_sub_warmup_guard[&quot;study sub warmup guard&quot;]
class feat_study_sub_warmup_guard done;
feat_ubi_judgments[&quot;ubi judgments&quot;]
class feat_ubi_judgments done;
feat_study_lifecycle --&gt; feat_digest_proposal
feat_llm_judgments --&gt; feat_digest_proposal
infra_foundation --&gt; feat_llm_judgments
Expand Down Expand Up @@ -2620,6 +2635,8 @@ <h2>Dependency graph (feat_ + infra_)</h2>
class chore_state_md_size_compression done;
feat_study_sub_warmup_guard[&quot;study sub warmup guard&quot;]
class feat_study_sub_warmup_guard done;
feat_ubi_judgments[&quot;ubi judgments&quot;]
class feat_ubi_judgments done;
feat_study_lifecycle --&gt; feat_digest_proposal
feat_llm_judgments --&gt; feat_digest_proposal
infra_foundation --&gt; feat_llm_judgments
Expand Down
46 changes: 15 additions & 31 deletions docs/00_overview/mvp2_dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -382,28 +382,28 @@ <h1>RelyLoop MVP2 Dashboard</h1>
<main>

<section>
<div class="next-up">
<div class="eyebrow">Next up — Feature, currently in <strong>Implementing</strong></div>
<div class="title"><a href="../../docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md">Ubi Judgments</a></div>
<div class="one-liner">Operators with the OpenSearch / ES UBI plugin installed (today; Solr&#x27;s first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real click + dwell behavior via three converters </div>
<div class="stage-hint">Implementation in progress — resume to finish</div>
<code class="cmd">/impl-execute docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/implementation_plan.md --all # resume in-progress</code>
<div class="next-up done">
<div class="eyebrow">Next up</div>
<div class="title">All scoped MVP2 features shipped 🎉</div>
<div class="one-liner">
Pull from the Idea backlog or capture a new feature spec.
</div>
</div>
</section>


<section>
<h2>MVP2 Progress</h2>
<div class="kpi-row">
<div class="kpi ">
<div class="kpi complete">
<div class="label">Scoped items done</div>
<div class="value">1 / 2</div>
<div class="sub">50% of feat_/infra_/chore_/epic_ items past idea stage</div>
<div class="bar"><span style="width:50%"></span></div>
<div class="value">1 / 1</div>
<div class="sub">100% of feat_/infra_/chore_/epic_ items past idea stage</div>
<div class="bar"><span style="width:100%"></span></div>
</div>
<div class="kpi warn">
<div class="label">Pending work</div>
<div class="value">18</div>
<div class="value">17</div>
<div class="sub">every not-done feat/infra/chore/bug across all priorities</div>
</div>
<div class="kpi bug">
Expand All @@ -420,7 +420,7 @@ <h2>MVP2 Progress</h2>
<div class="kpi-row">
<div class="kpi">
<div class="label">P1</div>
<div class="value">3</div>
<div class="value">2</div>
<div class="sub">high-value, ready when P0 clears</div>
</div>
<div class="kpi">
Expand All @@ -435,7 +435,7 @@ <h2>MVP2 Progress</h2>
</div>
<div class="kpi">
<div class="label">Legacy "Path to MVP2"</div>
<div class="value">10</div>
<div class="value">9</div>
<div class="sub">scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)</div>
</div>
</div>
Expand All @@ -446,7 +446,7 @@ <h2>MVP2 Progress</h2>
</span>
<span>
<strong>In flight:</strong>
1 feature(s) actively shipping
0 feature(s) actively shipping
</span>
</div>
</section>
Expand Down Expand Up @@ -698,19 +698,7 @@ <h3>Plan <span class="count">0</span></h3>
</div>

<div class="col implement">
<h3>Implementing <span class="count">1</span></h3>

<div class="card feat" data-prefix="feat" data-priority="P1">
<div class="name"><a href="../../docs/00_overview/planned_features/02_mvp2/feat_ubi_judgments/feature_spec.md">Ubi Judgments</a></div>
<div class="meta">
<span class="badge feat">Feature</span>
<span class="badge priority" data-priority="P1">P1</span>

</div>
<div class="one-liner">Operators with the OpenSearch / ES UBI plugin installed (today; Solr&#x27;s first-party `solr.UBIComponent` lights up with the sibling `infra_adapter_solr` MVP2 release) can derive judgments from real clic</div>


</div>
<h3>Implementing <span class="count">0</span></h3>

</div>

Expand Down Expand Up @@ -742,8 +730,6 @@ <h2>Dependency graph (feat_ + infra_)</h2>
classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e;
classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af;
classDef idea fill:#f1f5f9,stroke:#334155,color:#334155;
feat_ubi_judgments[&quot;ubi judgments&quot;]
class feat_ubi_judgments implement;
feat_contextual_help_mvp2[&quot;contextual help mvp2&quot;]
class feat_contextual_help_mvp2 done;</div>
<noscript><pre>graph LR
Expand All @@ -752,8 +738,6 @@ <h2>Dependency graph (feat_ + infra_)</h2>
classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e;
classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af;
classDef idea fill:#f1f5f9,stroke:#334155,color:#334155;
feat_ubi_judgments[&quot;ubi judgments&quot;]
class feat_ubi_judgments implement;
feat_contextual_help_mvp2[&quot;contextual help mvp2&quot;]
class feat_contextual_help_mvp2 done;</pre></noscript>
</div>
Expand Down
Loading
Loading