From c3cfb17062f1838f61690ea4cce0d202c25f16ce Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Sat, 23 May 2026 20:51:30 -0400 Subject: [PATCH 01/15] docs(digest-executable-followups): idea-preflight patches + feature spec (3 GPT-5.5 cycles) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundles two stages: **Idea preflight (11 edits, 1 file)** — pre-spec audit grounded every concrete claim against the current codebase: - Fix 4 broken sibling links to feat_auto_followup_studies (folder moved to implemented_features/2026_05_24_* on 2026-05-24). - Reframe feat_auto_followup_studies as already-shipped substrate (PR #223 squash 20cf183a) rather than future coordination concern. - Fix line range digest.py:168-189 -> digest.py:169-182 (2 sites). - Fix column claim: digests.followups JSONB is wrong; actual is digests.suggested_followups ARRAY(Text) per backend/app/db/models/digest.py:49. This changes the migration story from "strictly additive" to "two migrations including a column-type change" with USING-clause backfill discipline. - Capture: SuggestedFollowupsPanel has a dead "Create study from this hypothesis" button (link constructed but /studies never reads the param). Subsume into the new structured flow. - Add parent_proposal_id FK alongside parent_proposal_followup_index (the index alone is unmoored without the proposal ID). - Bump scope estimates +50 LOC each layer. **Feature spec (Generate mode + 3 GPT-5.5 cycles to convergence):** - 13 FRs / 13 ACs / 3 phases (Phase 1 in scope; Phase 2 swap_template + Phase 3 edit_template deferred with idea files). - 3 new error codes: PROPOSAL_NOT_FOUND (404), DIGEST_NOT_FOUND (404 retryable), FOLLOWUP_INDEX_OUT_OF_RANGE (422). - Migration discipline: PL/pgSQL helper functions for the ARRAY(Text) -> JSONB type change (subqueries not allowed in ALTER COLUMN TYPE ... USING per empirical Postgres-16 verification); BEFORE DELETE trigger (NOT ON DELETE SET NULL) for the parent_proposal lineage pair invariant. - Cross-model review: 17 accepted + 1 rejected (D-17 — CLAUDE.md Absolute Rule #8 mandates persisted lineage capture, the response example showing openai:gpt-4o-2024-08-06 is lineage data, not hardcoded model usage). - Decision log D-13 through D-29 capture every adjudication. Co-Authored-By: Claude Opus 4.7 --- docs/00_overview/DASHBOARD.md | 2 +- docs/00_overview/MVP1_DASHBOARD.md | 51 +- docs/00_overview/dashboard.html | 2 +- docs/00_overview/mvp1_dashboard.html | 57 +- .../feature_spec.md | 793 ++++++++++++++++++ .../feat_digest_executable_followups/idea.md | 37 +- .../phase2_idea.md | 44 + .../phase3_idea.md | 43 + .../pipeline_status.md | 22 + 9 files changed, 987 insertions(+), 64 deletions(-) create mode 100644 docs/02_product/planned_features/feat_digest_executable_followups/feature_spec.md create mode 100644 docs/02_product/planned_features/feat_digest_executable_followups/phase2_idea.md create mode 100644 docs/02_product/planned_features/feat_digest_executable_followups/phase3_idea.md create mode 100644 docs/02_product/planned_features/feat_digest_executable_followups/pipeline_status.md diff --git a/docs/00_overview/DASHBOARD.md b/docs/00_overview/DASHBOARD.md index d5c9106e..11fc8cd2 100644 --- a/docs/00_overview/DASHBOARD.md +++ b/docs/00_overview/DASHBOARD.md @@ -6,7 +6,7 @@ _Top-level index across MVP1 → GA v1+ as of **2026-05-24**. Click a release na | Release | Theme | Progress | Status | |---|---|---|---| -| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 72 / 72 scoped done · 7 remaining | **In progress** | +| [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 72 / 73 scoped done · 8 remaining | **In progress** | | [MVP1.5 / v0.1.5](MVP1_5_DASHBOARD.md) | Real Signals | 1 item(s) queued | **Held / queued** | | [MVP2 / v0.2](MVP2_DASHBOARD.md) | Observable | 1 / 1 scoped done · 1 remaining | **In progress** | | MVP3 / v0.3 | Production Stacks | — | **Not yet scoped** | diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index 35c9f3f1..b02d4905 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -6,23 +6,29 @@ _Reflects feature-folder state as of **2026-05-24** (latest mtime of any planned ## Next up -All scoped MVP1 features shipped 🎉 +**[feat_digest_executable_followups](../02_product/planned_features/feat_digest_executable_followups/feature_spec.md)** — Feature, currently in **Spec** -Pull from the Idea backlog or capture a new feature spec. +> The LLM emits a discriminated union (`narrow` \| `widen` \| `text`) for each followup with a structured `search_space` when applicable. + +Spec exists; run /pipeline to generate the implementation plan + ship + +```bash +/pipeline docs/02_product/planned_features/feat_digest_executable_followups --auto +``` ## MVP1 Progress | Metric | Value | |---|---| -| Scoped items done | **72 / 72** (100%) — feat_/infra_/chore_/epic_ past idea stage | +| Scoped items done | **72 / 73** (99%) — feat_/infra_/chore_/epic_ past idea stage | | Pending work | **13** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | | → P1 | **0** high-value, ready when P0 clears | | → P2 (default) | 12 important to file, not blocking | | → Backlog | 1 captured for record, not planned | | Open bugs | 0 | -| Legacy "Path to MVP1" | 7 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 6 idea-only feat/infra (not yet scoped into MVP1) | +| Legacy "Path to MVP1" | 8 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | +| Backlog ideas | 5 idea-only feat/infra (not yet scoped into MVP1) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -129,27 +135,28 @@ _None._ _None._ -### Spec (0) +### Spec (1) -_None._ +| # | Priority | Feature | Type | One-liner | Depends on | Status | +|---|---|---|---|---|---|---| +| 1 | P2 | [feat_digest_executable_followups](../02_product/planned_features/feat_digest_executable_followups/feature_spec.md) | Feature | The LLM emits a discriminated union (`narrow` \| `widen` \| `text`) for each followup with a structured `search_space` when applicable. | — | [PR #223](https://github.com/SoundMindsAI/relyloop/pull/223) merged 2026-05-24 | -### Idea (13) +### Idea (12) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P2 | [feat_digest_executable_followups](../02_product/planned_features/feat_digest_executable_followups/idea.md) | Feature | The digest worker's LLM contract at [`backend/workers/digest.py:168-189`](../../backend/workers/digest.py) defines `suggested_followups` as a flat `array of string`: | — | Idea — surfaced during the 2026-05-21 Karpathy-loop audit. | -| 2 | P2 | [feat_home_demo_reseed_endpoint](../02_product/planned_features/feat_home_demo_reseed_endpoint/idea.md) | Feature | Phase 1 ships the banner + badges that signal "this is demo data." It does NOT close the recovery loop for operators who blew away their dev DB and want to re-seed the meaningful demos from inside the | — | Idea — deferred Phase 2 work from `feat_home_first_run_demo_nudge` (Phase 1 merged 2026-05-22 as PR #188 squash `21325432`) | -| 3 | P2 | [feat_study_baseline_trial](../02_product/planned_features/feat_study_baseline_trial/idea.md) | Feature | `studies.baseline_metric` exists as a column on the `studies` table (declared in `feat_study_lifecycle` Phase 1, [`backend/app/db/models/study.py:76`](../../backend/app/db/models/study.py#L76)) with t | — | Idea — deferred Phase 2 work from `feat_pr_metric_confidence` (Phase 1 merged 2026-05-21 as PR #180 squash `d0a8358`). | -| 4 | P2 | [feat_study_clone_from_previous](../02_product/planned_features/feat_study_clone_from_previous/idea.md) | Feature | A relevance engineer's normal workflow after the first study completes: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| 5 | P2 | [infra_agent_sibling_worktree_isolation](../02_product/planned_features/infra_agent_sibling_worktree_isolation/idea.md) | Infra | Running an autonomous agent in a sibling git worktree while the operator's main checkout has the Docker Compose stack up exposes two surfaces that aren't designed for parallel work: | — | Idea — tangential observations from the autonomous `chore_reconciler_terminal_closed_no_poll` agent run (PR #216, merged 2026-05-23) | -| 6 | P2 | [infra_study_preflight_real_engine_integration](../02_product/planned_features/infra_study_preflight_real_engine_integration/idea.md) | Infra | `feat_study_preflight_overlap_probe`'s integration tests (AC-1 through AC-4b in [`backend/tests/integration/test_studies_api.py`](../../backend/tests/integration/test_studies_api.py)) use… | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | -| 7 | P2 | [chore_auto_followup_completed_parent_stop_chain_race](../02_product/planned_features/chore_auto_followup_completed_parent_stop_chain_race/idea.md) | Chore | The cycle-3 C3-1 cascade-cancel design tolerates terminal parents (cascade traverses through `completed` intermediates to reach in-flight descendants). But the FR-1 digest trigger fires `enqueue_follo | — | Idea — surfaced during the Epic 1+2 phase-gate GPT-5.5 review of `feat_auto_followup_studies` (cumulative-diff review finding F2, accepted in part as a future-work capture) | -| 8 | P2 | [chore_auto_followup_e2e_chain_seed_helper](../02_product/planned_features/chore_auto_followup_e2e_chain_seed_helper/idea.md) | Chore | `feat_auto_followup_studies` Story 3.3 specified a Playwright E2E spec that seeds a 3-node chain (root R → middle M → leaf L) and asserts: | — | Idea | -| 9 | P2 | [chore_dashboard_regen_quoted_pr_false_positive](../02_product/planned_features/chore_dashboard_regen_quoted_pr_false_positive/idea.md) | Chore | [`_extract_pr_number`](../../scripts/build_mvp1_dashboard.py#L572)'s priority-3 fuzzy match has two regexes: | — | Idea — surfaced during `chore_dashboard_pr_extraction_from_idea` empirical verification (2026-05-23) | -| 10 | P2 | [chore_e2e_seed_acme_idea_obsolete](../02_product/planned_features/chore_e2e_seed_acme_idea_obsolete/idea.md) | Chore | [`chore_e2e_seed_acme_helper_dead/idea.md`](../02_product/planned_features/chore_e2e_seed_acme_helper_dead/idea.md) (dated 2026-05-21) proposed two paths: | — | Idea — surfaced during `chore_migration_test_head_brittleness` `/idea-preflight` pick (2026-05-23) | -| 11 | P2 | [chore_studies_post_arq_spy_fixture](../02_product/planned_features/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | -| 12 | P2 | [chore_template_library_expansion](../02_product/planned_features/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| 13 | Backlog | [chore_e2e_seed_acme_helper_dead](../02_product/planned_features/chore_e2e_seed_acme_helper_dead/idea.md) | Chore | `seedAcmeProductsChain` is a 140-line helper that constructs a cluster + query_set + template + judgment_list + study + optional proposal/digest chain "Acme Products" demo scenario. The function is co | — | Idea — surfaced during `chore_e2e_test_rows_isolation` Story 1.2 coverage audit | +| 1 | P2 | [feat_home_demo_reseed_endpoint](../02_product/planned_features/feat_home_demo_reseed_endpoint/idea.md) | Feature | Phase 1 ships the banner + badges that signal "this is demo data." It does NOT close the recovery loop for operators who blew away their dev DB and want to re-seed the meaningful demos from inside the | — | Idea — deferred Phase 2 work from `feat_home_first_run_demo_nudge` (Phase 1 merged 2026-05-22 as PR #188 squash `21325432`) | +| 2 | P2 | [feat_study_baseline_trial](../02_product/planned_features/feat_study_baseline_trial/idea.md) | Feature | `studies.baseline_metric` exists as a column on the `studies` table (declared in `feat_study_lifecycle` Phase 1, [`backend/app/db/models/study.py:76`](../../backend/app/db/models/study.py#L76)) with t | — | Idea — deferred Phase 2 work from `feat_pr_metric_confidence` (Phase 1 merged 2026-05-21 as PR #180 squash `d0a8358`). | +| 3 | P2 | [feat_study_clone_from_previous](../02_product/planned_features/feat_study_clone_from_previous/idea.md) | Feature | A relevance engineer's normal workflow after the first study completes: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | +| 4 | P2 | [infra_agent_sibling_worktree_isolation](../02_product/planned_features/infra_agent_sibling_worktree_isolation/idea.md) | Infra | Running an autonomous agent in a sibling git worktree while the operator's main checkout has the Docker Compose stack up exposes two surfaces that aren't designed for parallel work: | — | Idea — tangential observations from the autonomous `chore_reconciler_terminal_closed_no_poll` agent run (PR #216, merged 2026-05-23) | +| 5 | P2 | [infra_study_preflight_real_engine_integration](../02_product/planned_features/infra_study_preflight_real_engine_integration/idea.md) | Infra | `feat_study_preflight_overlap_probe`'s integration tests (AC-1 through AC-4b in [`backend/tests/integration/test_studies_api.py`](../../backend/tests/integration/test_studies_api.py)) use… | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | +| 6 | P2 | [chore_auto_followup_completed_parent_stop_chain_race](../02_product/planned_features/chore_auto_followup_completed_parent_stop_chain_race/idea.md) | Chore | The cycle-3 C3-1 cascade-cancel design tolerates terminal parents (cascade traverses through `completed` intermediates to reach in-flight descendants). But the FR-1 digest trigger fires `enqueue_follo | — | Idea — surfaced during the Epic 1+2 phase-gate GPT-5.5 review of `feat_auto_followup_studies` (cumulative-diff review finding F2, accepted in part as a future-work capture) | +| 7 | P2 | [chore_auto_followup_e2e_chain_seed_helper](../02_product/planned_features/chore_auto_followup_e2e_chain_seed_helper/idea.md) | Chore | `feat_auto_followup_studies` Story 3.3 specified a Playwright E2E spec that seeds a 3-node chain (root R → middle M → leaf L) and asserts: | — | Idea | +| 8 | P2 | [chore_dashboard_regen_quoted_pr_false_positive](../02_product/planned_features/chore_dashboard_regen_quoted_pr_false_positive/idea.md) | Chore | [`_extract_pr_number`](../../scripts/build_mvp1_dashboard.py#L572)'s priority-3 fuzzy match has two regexes: | — | Idea — surfaced during `chore_dashboard_pr_extraction_from_idea` empirical verification (2026-05-23) | +| 9 | P2 | [chore_e2e_seed_acme_idea_obsolete](../02_product/planned_features/chore_e2e_seed_acme_idea_obsolete/idea.md) | Chore | [`chore_e2e_seed_acme_helper_dead/idea.md`](../02_product/planned_features/chore_e2e_seed_acme_helper_dead/idea.md) (dated 2026-05-21) proposed two paths: | — | Idea — surfaced during `chore_migration_test_head_brittleness` `/idea-preflight` pick (2026-05-23) | +| 10 | P2 | [chore_studies_post_arq_spy_fixture](../02_product/planned_features/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | +| 11 | P2 | [chore_template_library_expansion](../02_product/planned_features/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | +| 12 | Backlog | [chore_e2e_seed_acme_helper_dead](../02_product/planned_features/chore_e2e_seed_acme_helper_dead/idea.md) | Chore | `seedAcmeProductsChain` is a 140-line helper that constructs a cluster + query_set + template + judgment_list + study + optional proposal/digest chain "Acme Products" demo scenario. The function is co | — | Idea — surfaced during `chore_e2e_test_rows_isolation` Story 1.2 coverage audit | ## Dependency graph @@ -162,6 +169,8 @@ graph LR classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + feat_digest_executable_followups["digest executable followups"] + class feat_digest_executable_followups spec; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] diff --git a/docs/00_overview/dashboard.html b/docs/00_overview/dashboard.html index d4f242d6..38d88ae8 100644 --- a/docs/00_overview/dashboard.html +++ b/docs/00_overview/dashboard.html @@ -384,7 +384,7 @@

Releases

MVP1 / v0.1
The Loop
-
72 / 72 scoped done · 7 remaining
+
72 / 73 scoped done · 8 remaining
In progress
diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index ad7f8599..15424fef 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -382,12 +382,12 @@

RelyLoop MVP1 Dashboard

-
-
Next up
-
All scoped MVP1 features shipped 🎉
-
- Pull from the Idea backlog or capture a new feature spec. -
+
+
Next up — Feature, currently in Spec
+
Digest Executable Followups
+
The LLM emits a discriminated union (`narrow` | `widen` | `text`) for each followup with a structured `search_space` when applicable.
+
Spec exists; run /pipeline to generate the implementation plan + ship
+ /pipeline docs/02_product/planned_features/feat_digest_executable_followups --auto
@@ -395,11 +395,11 @@

RelyLoop MVP1 Dashboard

MVP1 Progress

-
+
Scoped items done
-
72 / 72
-
100% of feat_/infra_/chore_/epic_ items past idea stage
-
+
72 / 73
+
99% of feat_/infra_/chore_/epic_ items past idea stage
+
Pending work
@@ -435,14 +435,14 @@

MVP1 Progress

Legacy "Path to MVP1"
-
7
+
8
scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)
Backlog ideas: - 6 idea-only feat/infra folders (not yet scoped into MVP1) + 5 idea-only feat/infra folders (not yet scoped into MVP1) In flight: @@ -463,20 +463,7 @@

Pipeline

-

Idea 13

- -
-
Digest Executable Followups
-
- Feature - P2 - -
-
The digest worker's LLM contract at [`backend/workers/digest.py:168-189`](../../backend/workers/digest.py) defines `suggested_followups` as a flat `array of string`:
- - -
- +

Idea 12

Home Demo Reseed Endpoint
@@ -636,7 +623,19 @@

Idea 13

-

Spec 0

+

Spec 1

+ +
+
Digest Executable Followups
+
+ Feature + P2 + PR #223 merged 2026-05-24 +
+
The LLM emits a discriminated union (`narrow` | `widen` | `text`) for each followup with a structured `search_space` when applicable.
+
deferred: Phase 2, Phase 3
+ +
@@ -1822,6 +1821,8 @@

Dependency graph (feat_ + infra_)

classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + feat_digest_executable_followups["digest executable followups"] + class feat_digest_executable_followups spec; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] @@ -2019,6 +2020,8 @@

Dependency graph (feat_ + infra_)

classDef plan fill:#fef9c3,stroke:#854d0e,color:#854d0e; classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; + feat_digest_executable_followups["digest executable followups"] + class feat_digest_executable_followups spec; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/feature_spec.md b/docs/02_product/planned_features/feat_digest_executable_followups/feature_spec.md new file mode 100644 index 00000000..a3fbbe6b --- /dev/null +++ b/docs/02_product/planned_features/feat_digest_executable_followups/feature_spec.md @@ -0,0 +1,793 @@ +# Feature Specification — Executable Digest Follow-ups + +**Date:** 2026-05-23 +**Status:** Draft +**Owners:** Eric Starr (product), Eric Starr (engineering) +**Related docs:** +- [`idea.md`](./idea.md) — original brief +- [`docs/01_architecture/llm-orchestration.md`](../../../01_architecture/llm-orchestration.md) +- [`docs/01_architecture/data-model.md`](../../../01_architecture/data-model.md) +- Sibling: [`feat_auto_followup_studies`](../../../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/) (shipped 2026-05-24) +- Sibling: [`feat_agent_propose_search_space`](../../../00_overview/implemented_features/2026_05_21_feat_agent_propose_search_space/) (shipped) +- Sibling: [`feat_create_study_search_space_builder`](../../../00_overview/implemented_features/2026_05_20_feat_create_study_search_space_builder/) (shipped) + +--- + +## 1) Purpose + +- **Problem:** The digest worker LLM produces `suggested_followups` as flat strings (see `backend/workers/digest.py:169-182`, schema `DIGEST_RESPONSE_SCHEMA`). The UI renders them as bullet text with a "Create study from this hypothesis" button that links to `/studies?hypothesis=` — but `ui/src/app/studies/page.tsx` never reads the `hypothesis` query param (grep returns zero matches for `hypothesis`/`searchParams` apart from a generic Suspense comment). The button silently drops its payload. Operators end up reading the suggestion, mentally translating it to a `search_space` JSON, and re-typing the entire 6-field create-study wizard. The Karpathy-loop equivalent is one click. +- **Outcome:** The LLM emits a discriminated union (`narrow` | `widen` | `text`) for each followup with a structured `search_space` when applicable. The proposal-detail UI renders the actionable kinds as cards with a primary "Run this followup" button that pre-fills the create-study modal with the parent study's cluster/target/template/query_set/judgment_list/objective plus the LLM-proposed `search_space`. Two new nullable columns on `studies` (`parent_proposal_id` + `parent_proposal_followup_index`) provide lineage so the team can measure whether LLM-suggested followups produce wins. +- **Non-goal:** Auto-running followups without operator review (already covered by `feat_auto_followup_studies` for the deterministic narrow-around-winner case). Spanning multiple studies in one followup. LLM-driven template edits (Tier C in the idea — deferred to a separate spec). + +## 2) Current state audit + +### Existing implementations + +- `backend/workers/digest.py:169-182` — `DIGEST_RESPONSE_SCHEMA` constrains `suggested_followups` to `array of string`, `maxItems: 5`. The `_call_openai_for_digest` helper at lines 378-426 parses `parsed.get("suggested_followups", [])` and the capability-degraded path (lines 411-426) wraps the response into `{"narrative": ..., "suggested_followups": []}`. Step 13 at lines 757-775 prepends a drift followup string and caps at 5. +- `backend/workers/digest.py:761-767` — drift followup is currently a plain string (`f"Best trial used {len(dropped)} params no longer declared..."`). Must become a `{kind: "text", rationale: "..."}` entry after the migration. +- `backend/app/db/models/digest.py:49` — `suggested_followups: Mapped[list[str]] = mapped_column(ARRAY(Text), nullable=False, server_default=text("ARRAY[]::TEXT[]"))`. **The column type must change to JSONB.** +- `migrations/versions/0005_digests.py:52-57` — the canonical migration that created the column as `postgresql.ARRAY(sa.Text())`. +- `backend/app/api/v1/schemas.py:941-951` (`DigestResponse`) and `:999-1007` (`_DigestEmbed`) — both expose `suggested_followups: list[str]`. Must become `list[FollowupItem]`. +- `backend/app/api/v1/proposals.py:154-163` — populates `_DigestEmbed.suggested_followups=digest.suggested_followups` from the ORM. Will automatically pass through the new structure once the ORM column type is JSONB and the schema is `list[FollowupItem]`. +- `ui/src/components/proposals/suggested-followups-panel.tsx` — entire file (42 lines). Props are `followups: readonly string[]`. Renders each as a bullet + a `` button. **The query param is unread by `/studies` — dead path.** +- `ui/src/app/proposals/[id]/page.tsx:194-197` — call site. Will be rewritten to pass `digest.suggested_followups` as the new structured list. +- `ui/src/app/studies/page.tsx` — does NOT read any `hypothesis` searchParam (`grep -n "hypothesis\|searchParams"` returns 1 line: a generic comment about `useSearchParams` Suspense boundaries). Dead button. +- `ui/src/components/studies/create-study-modal.tsx:159-185` (`CreateStudyModalProps` + form `defaultValues`) — currently `{open, onOpenChange}`. Will gain an optional `initialValues` prop carrying the pre-fill payload (parent study + LLM `search_space` + `parent_proposal_id` + `parent_proposal_followup_index`). The form must seed `cluster_id`, `target`, `template_id`, `query_set_id`, `judgment_list_id`, `objective`, `config` (incl. `max_trials` / `time_budget_min` / `parallelism` / `sampler`) and the `search_space_text` JSON from the prefill. +- `backend/app/db/models/study.py:72-75` — `parent_study_id` self-FK already exists; comment says "for forks (MVP2)" but `feat_auto_followup_studies` (PR #223 squash `20cf183a`, merged 2026-05-24) already uses it for the auto-chain (`backend/app/services/study_state.py:285-299`, `backend/app/db/repo/study.py:167-188`, `backend/tests/integration/test_auto_followup.py:220,425`). The new `parent_proposal_id` and `parent_proposal_followup_index` columns are **orthogonal** — a study can be both auto-followup (parent_study_id set) and LLM-followup (parent_proposal_id set). +- `backend/app/domain/study/search_space.py:92-118` — `SearchSpace` Pydantic model with `params: dict[str, ParamSpec]`, `min_length=1`, cardinality cap 10⁶. Already used by `POST /api/v1/studies` (`backend/app/api/v1/studies.py:200-204`) — reuse verbatim for digest-side validation. +- `prompts/digest_narrative.system.md:42-58` — current system prompt section telling the LLM how to emit `suggested_followups` as JSON array of short strings. Must teach the three kinds (`narrow` / `widen` / `text`). +- `prompts/digest_narrative.user.jinja` — current user template. Add a `` block so the LLM can transform it. +- `ui/src/lib/glossary.ts:471-475` — existing glossary key `proposal.suggested_followups`. Copy stays accurate ("LLM-generated next-study hypotheses ... Click to seed a new study.") but new keys are needed for the kind discriminator (see §11 tooltip inventory). + +### Navigation and link impact + +| Source file | Current link target | New link target | +|---|---|---| +| `ui/src/components/proposals/suggested-followups-panel.tsx:29` | `/studies?hypothesis=` | **Remove.** The "Create study from this hypothesis" button is replaced by an in-place "Run this followup" button that opens the create-study modal with `initialValues` (no navigation). | + +No other components link to `/studies?hypothesis=...`. `grep -rn "hypothesis=" ui/src/` returns only the one line in `suggested-followups-panel.tsx`. + +### Existing test impact + +| Test file | Pattern | Count | Required change | +|---|---|---|---| +| `ui/src/__tests__/components/proposals/suggested-followups-panel.test.*` (if any) | `followups: string[]` assertions | TBD at impl time | Update to the discriminated-union shape; cover all three kinds + the legacy-shape backward-compat read path. | +| `backend/tests/unit/workers/test_digest_*` | `suggested_followups` shape assertions | TBD at impl time | Update to assert the new validated structure on the success path AND the kind-downgrade path on Pydantic validation failure. | +| `backend/tests/contract/test_proposals_*` and `test_studies_digest_*` | `DigestResponse.suggested_followups` is `list[str]` | TBD at impl time | Update contract assertions to the new `list[FollowupItem]` shape; ensure the discriminator + nullable `search_space` round-trip cleanly. | +| `backend/tests/integration/test_digest_*` round-trip tests that stub the LLM | LLM stub returns `list[str]` | TBD at impl time | Update the stub to return the new structured list. Add one test covering the legacy-rows-in-DB read path (the column-type migration backfill). | + +The implementation plan will enumerate the exact test files at story-decomposition time; the spec records the categories. + +### Existing behaviors affected by scope change + +- **`suggested_followups` shape on the wire.** Current: `list[str]` everywhere (DB column, ORM model, `DigestResponse`, `_DigestEmbed`, UI prop). New: `list[FollowupItem]` (discriminated union over `narrow` / `widen` / `text`). **Decision needed:** No — locked in idea + this spec. Legacy DB rows (with `ARRAY(Text)` payloads) are wrapped to `[{kind: "text", rationale: , search_space: null}]` by the migration's USING-clause backfill so downstream readers never see a mixed shape. +- **"Create study from this hypothesis" button.** Current: links to `/studies?hypothesis=...` which is silently dropped. New: the button is **removed** (the new flow ships the structured "Run this followup" button driven by `kind=narrow|widen`; `kind=text` items render as plain bullet text with NO button). **Decision needed:** No — locked in idea ("retire as part of this feature" to avoid two buttons with different semantics). +- **`studies` table lineage.** Current: `parent_study_id` self-FK exists and is used by the auto-chain. New: two additional nullable columns `parent_proposal_id` (FK → `proposals(id)` ON DELETE SET NULL) + `parent_proposal_followup_index INT` are added. Both lineage tracks coexist on the same row (auto-chain child of study A AND LLM-suggested followup from proposal B). **Decision needed:** No. + +--- + +## 3) Scope + +### In scope (Phase 1 = Tier A from the idea) + +- LLM output schema change for `suggested_followups` to a discriminated union over `{kind: "narrow" | "widen" | "text", rationale: str, search_space: SearchSpace | None}` (FR-1). +- Backend Pydantic models for the followup union + a `parse_followup_list()` adapter that wraps legacy `list[str]` rows into `[{kind: "text", rationale, search_space: null}]` (FR-2 + FR-3). +- Validator at digest-persist time: each `narrow` / `widen` item's `search_space` is validated against `backend.app.domain.study.search_space.SearchSpace`; on failure the item is downgraded to `{kind: "text", rationale: "[validation failed: ] " + original_rationale, search_space: null}` so the operator still sees the intent (FR-4). +- Two new nullable columns on `studies`: `parent_proposal_id VARCHAR(36) NULL` (FK to `proposals(id)` with no `ON DELETE` action; a `BEFORE DELETE ON proposals` trigger atomically NULLs the lineage pair — see FR-5) and `parent_proposal_followup_index INT NULL` (FR-5). A partial B-tree index `WHERE parent_proposal_id IS NOT NULL` bounds lookups. +- Migration that **changes the column type** of `digests.suggested_followups` from `ARRAY(Text)` to `JSONB`, with a USING-clause backfill that wraps each existing text element as `jsonb_build_object('kind', 'text', 'rationale', value, 'search_space', NULL)` (FR-6). +- Wire-shape change to `DigestResponse.suggested_followups` and `_DigestEmbed.suggested_followups` from `list[str]` to `list[FollowupItem]` (FR-7). +- LLM prompt updates teaching the model the three kinds (FR-8). +- UI: `SuggestedFollowupsPanel` rewritten to render kind-discriminated cards. `narrow` / `widen` cards have a rationale, a collapsible "Show search space" detail (search-space diff vs parent), and a primary "Run this followup" button (FR-9). +- UI: "Run this followup" opens the existing create-study modal with `initialValues` pre-filled from the parent study + the LLM-proposed `search_space`. Operator reviews + submits via the existing `POST /api/v1/studies` (FR-10). +- UI: `CreateStudyRequest` body sent by the new flow carries the parent lineage fields in a new top-level optional `parent` object (`{proposal_id: str, followup_index: int}`) that the backend persists into the new columns (FR-11). +- UI: legacy `?hypothesis=` button removed from `suggested-followups-panel.tsx` (FR-12). +- Glossary additions for the new tooltips (FR-13). + +### Out of scope + +- **Tier B — `kind: "swap_template"` followups.** Cross-template search-space remapping (`backend/app/domain/study/template_swap.py` per the idea) is its own design surface. Deferred to Phase 2 — tracked in `phase2_idea.md` co-located with this spec. +- **Tier C — `kind: "edit_template"` followups.** Operator-only today; LLM-suggested template edits are a much larger trust/validation surface and unrelated to this spec's lane. Captured in the idea; tracked in `phase3_idea.md` (separate from Phase 2 so each remains independently schedulable). +- **Auto-running followups without operator click.** Already shipped as `feat_auto_followup_studies` (PR #223). The two features cover orthogonal compounding paths. +- **Followups that span multiple studies** (e.g., "run A.1 and A.2 in parallel"). Needs its own surface — out. +- **Negative-result feedback loop** ("operator tried this followup; it didn't help"). Out — gated on Langfuse (MVP2+). +- **Audit events.** `audit_log` table doesn't exist in MVP1. §6 names the event types pre-shaped per `data-model.md` for the MVP2 activation. + +### API convention check + +- **Endpoint prefix convention:** `/api/v1/` for business endpoints — confirmed in `backend/app/api/v1/studies.py` and `backend/app/api/v1/proposals.py`. +- **Router namespace for this feature's endpoints:** No new router file. This feature **extends** `POST /api/v1/studies` (add the optional `parent` body field) and **changes the response shape** of `GET /api/v1/studies/{study_id}/digest` (`DigestResponse`) + `GET /api/v1/proposals/{id}` (`_DigestEmbed`). +- **HTTP methods for CRUD:** No new methods. The new flow reuses existing `POST /api/v1/studies`. +- **Non-auth error envelope shape:** `{ "detail": { "error_code": "", "message": "", "retryable": } }` — confirmed at `backend/app/api/v1/studies.py:75-79` (the local `_err()` helper). All new error codes use this exact envelope. Auth error shape: N/A in MVP1. + +### Phase boundaries + +- **Phase 1 (MVP1):** Tier A — `narrow` / `widen` / `text` kinds + UI prefill + new columns + column-type migration. FR-1 through FR-13. Rationale: smallest end-to-end slice that delivers the one-click operator value with all the storage + plumbing in place. +- **Phase 2 (deferred):** Tier B — `swap_template` kind with cross-template search-space remapping. Tracked in `phase2_idea.md`. Rationale: needs a new domain helper (`template_swap.py`), additional LLM prompt logic, and a side-by-side template-comparison UI surface; non-trivial and not blocking Phase 1 value. +- **Phase 3 (deferred — stretch):** Tier C — `edit_template` kind. Tracked in `phase3_idea.md`. Rationale: changes query rendering semantics, much larger trust surface, likely out of scope for MVP1 entirely. + +## 4) Product principles and constraints + +- LLM-suggested transformations are always operator-mediated: the LLM proposes; the operator clicks; the operator submits. The new flow never bypasses the existing `POST /api/v1/studies` validation chain (search-space cardinality, declared-params consistency, judgment overlap probe). +- Legacy data must remain readable. Pre-migration digests (`list[str]`) are wrapped into the new shape at backfill time AND the read-path adapter is defensive enough to handle a hypothetical row that escaped the backfill. +- Validation failure on an LLM-emitted `search_space` is a **downgrade**, not a rejection — the operator still sees the intent as text. The digest worker never aborts because of a malformed followup. +- The two lineage tracks (`parent_study_id` for the auto-chain, `parent_proposal_id` for the LLM-suggested click) are independent; both can be set on the same study row. +- All persisted LLM outputs continue to capture `generated_by` (existing column) for lineage; the new structured payloads don't change that contract. +- CLAUDE.md Absolute Rules apply: feature branch + PR; secrets via files; no hardcoded LLM models (`settings.openai_model`); migration ships `downgrade()` and round-trips cleanly; LLM-call timeouts and Redis budget guards from `feat_digest_proposal` continue to apply unchanged. + +### Anti-patterns + +- **Do not** add a second column (`suggested_followups_structured`) and dual-write during a deprecation window. The idea explicitly locks option (a): in-place type change with USING-clause backfill. There is no production data to preserve in MVP1 (single-tenant on laptops), and a dual-column scheme would leak a deprecated path into downstream readers indefinitely. +- **Do not** invent a separate endpoint (`POST /api/v1/studies/from_followup`). The existing `POST /api/v1/studies` is the contract; the new flow extends its body with an optional `parent` object. A new endpoint would duplicate every validation the existing one already runs. +- **Do not** trust the LLM-emitted `search_space` blindly. Run it through the existing `SearchSpace.model_validate()` at digest-persist time. On failure, downgrade — never persist an invalid `search_space` JSONB blob. +- **Do not** read `parent_proposal_followup_index` without `parent_proposal_id`. The index is meaningless without the proposal ID. The two columns are conceptually a pair (enforced by §9 invariant). +- **Do not** mix the legacy `?hypothesis=` button with the new "Run this followup" button — operators would see two buttons with subtly different semantics. Delete the legacy button in the same PR that ships the new one (FR-12). +- **Do not** keep `narrow` / `widen` `search_space` validation logic in the worker module. It belongs in a domain helper alongside `SearchSpace` so unit tests can exercise the validation + downgrade path without the worker fixture chain. + +## 5) Assumptions and dependencies + +- Dependency: **`feat_auto_followup_studies`** (shipped 2026-05-24 as PR #223 squash `20cf183a`). Status: implemented. Risk if missing: none — the two features are orthogonal. The new `parent_proposal_id` column composes cleanly with the existing `parent_study_id` self-FK without modification. +- Dependency: **`feat_digest_proposal`** (shipped 2026-05-11 as PR #41). Status: implemented. Provides the digest worker, schema, prompts, and the entire LLM-call infrastructure this spec modifies. Risk if missing: blocker — N/A, already shipped. +- Dependency: **`feat_create_study_search_space_builder`** (shipped 2026-05-20). Status: implemented. The create-study modal's search-space row primitive is reused by the "search-space diff" detail view. Risk if missing: soft — could fall back to raw JSON diff. +- Dependency: **`feat_agent_propose_search_space`** (shipped 2026-05-21). Status: implemented. Provides `backend/app/domain/study/search_space_defaults.py` heuristic that would be reused by Tier B's `swap_template` path (Phase 2 only — not Phase 1). +- Dependency: **OpenAI-compatible endpoint with structured-output (`json_schema`) capability.** The digest worker's existing capability check (`backend/app/llm/capability_check.py` → `read_capability_result`) already gates `structured_output_enabled`. When degraded, the worker falls back to narrative-only and persists `suggested_followups=[]` (current behavior, preserved). Risk if missing: graceful degradation already in place. +- Dependency: **No new env vars, no new secrets.** Reuses `OPENAI_API_KEY` / `OPENAI_BASE_URL` / `OPENAI_MODEL` from `Settings`. + +## 6) Actors and roles + +- Primary actor(s): **Relevance Engineer** (umbrella spec §6). The operator reviewing a proposal's digest who wants to one-click a followup study. +- Role model: **N/A — single-tenant install, no auth surface.** MVP1. +- Permission boundaries: **N/A — single-tenant.** Activates at MVP4 per CLAUDE.md. + +### Authorization + +N/A — single-tenant install, no auth surface (MVP1). + +### Audit events + +**N/A in MVP1 — `audit_log` table lands at MVP2** per `docs/01_architecture/data-model.md` §"Reserved for later releases". + +**Pre-shaped for MVP2 (forthcoming, do NOT implement now):** + +| Event type | Trigger | Visibility | Metadata fields | +|---|---|---|---| +| `digest.followup_clicked` | UI fires the "Run this followup" button (recorded at the `POST /api/v1/studies` boundary when `body.parent` is set). | tenant-visible | `proposal_id`, `followup_index`, `followup_kind` (`narrow` / `widen`), `created_study_id`. No `search_space` payload (already persisted on `studies.search_space`). | +| `study.created_from_followup` | Side-effect of `POST /api/v1/studies` when `parent.proposal_id` is set — emitted in the same transaction as the `INSERT INTO studies` (atomic). | tenant-visible | `study_id`, `parent_proposal_id`, `parent_proposal_followup_index`, `cluster_id`, `template_id`. | +| `digest.followup_validation_downgraded` | Digest worker downgrades a `narrow`/`widen` item to `text` because `SearchSpace.model_validate` failed. | system | `study_id`, `proposal_id`, `followup_index`, `original_kind`, `validation_error` (truncated to 500 chars; no PII). | + +No credentials, tokens, or PII beyond display-name strings in metadata. + +## 7) Functional requirements + +### FR-1: Discriminated-union followup schema (LLM output) + +- The system **MUST** emit `suggested_followups` from the digest LLM as a JSON array of objects where each object has fields `{kind: "narrow" | "widen" | "text", rationale: string, search_space: object | null}`. +- The system **MUST** enforce `maxItems: 5` on the array (preserved from current `DIGEST_RESPONSE_SCHEMA`). +- The system **MUST** require `search_space` to be a JSON object on `kind="narrow"|"widen"` items and `null` on `kind="text"` items at the JSON-schema level. +- The system **MUST** preserve the existing capability-fallback behavior: when `structured_output_enabled` is false, the worker persists `suggested_followups=[]` (**including suppressing the post-LLM drift-followup synthesis** — per GPT-5.5 cycle-3 F1 and matching the current code at `backend/workers/digest.py:757-758` where the drift prepend lives INSIDE the `if structured_output_enabled:` branch). FR-8's drift-synthesis contract therefore applies ONLY to the structured (non-degraded) path. +- Notes: enforced at the `DIGEST_RESPONSE_SCHEMA` JSON-schema level so the OpenAI structured-output API rejects malformed responses before they reach the worker. + +### FR-2: Backward-compatible read-path adapter + +- The system **MUST** wrap any `digests.suggested_followups` row whose stored payload is a JSON array of strings (legacy shape post-backfill — should not occur, but defensive) into `[{kind: "text", rationale: , search_space: null}]` at read time. +- The system **MUST** treat the post-migration shape (array of objects) as the canonical reader contract. +- Notes: The post-migration backfill (FR-6) eliminates legacy rows in steady state, but the adapter exists so the API never crashes on an unexpected payload. + +### FR-3: Pydantic models for the followup union (backend domain) + +- The system **MUST** define `FollowupItem` as a Pydantic v2 discriminated-union type alias: + ```python + FollowupItem = Annotated[ + NarrowFollowup | WidenFollowup | TextFollowup, + Field(discriminator="kind"), + ] + ``` + Each concrete model (`NarrowFollowup`, `WidenFollowup`, `TextFollowup`) is a `BaseModel` with `model_config = ConfigDict(extra="forbid")` and a `kind: Literal["narrow"|"widen"|"text"]` discriminator. The `narrow` and `widen` variants carry `search_space: SearchSpace` (reusing `backend.app.domain.study.search_space.SearchSpace`); the `text` variant carries `search_space: None` (explicit null). +- The system **MUST** export a module-level `FollowupItemAdapter = TypeAdapter(FollowupItem)` and `FollowupListAdapter = TypeAdapter(list[FollowupItem])` for validation (per GPT-5.5 cycle-1 F3 — a discriminated-union `Annotated` alias is NOT a `BaseModel` and does NOT expose `.model_validate()`; the correct API is `TypeAdapter(FollowupItem).validate_python(...)`). +- The system **MUST** export a `serialize_followup_list(items: list[FollowupItem]) -> list[dict[str, Any]]` helper that returns `[item.model_dump(mode="json") for item in items]` (per GPT-5.5 cycle-2 F4 — SQLAlchemy's JSONB driver does NOT know how to serialize Pydantic `BaseModel` instances; passing them directly would either crash on insert or store a malformed shape. The worker MUST call `serialize_followup_list(...)` before assigning to `digest.suggested_followups`). The `mode="json"` argument ensures nested `SearchSpace` models flatten to plain JSON dicts. +- The system **MUST** locate the models, adapters, and helpers in `backend/app/domain/study/followups.py` (pure-domain, no I/O) so unit tests exercise validation without DB or LLM fixtures. + +### FR-4: Validator + downgrade at digest-persist time + +- The system **MUST** validate each LLM-emitted followup item via `FollowupItemAdapter.validate_python(item)` (per GPT-5.5 cycle-1 F3) after the LLM call and before the digest row is INSERTed. +- The system **MUST** provide a `parse_followup_list(raw: object, *, study_id: str | None = None, proposal_id: str | None = None) -> list[FollowupItem]` helper in `backend/app/domain/study/followups.py`. Context kwargs are passed-through to the structlog WARN/ERROR events emitted on downgrade/drop paths (per GPT-5.5 cycle-2 F3 — the helper otherwise has no way to populate the required `study_id` / `proposal_id` log fields). Callers MUST pass both when available; either may be `None` on read-path adapter calls where the IDs aren't loaded (in which case the field appears as `study_id=null`/`proposal_id=null` in the event payload — operators can correlate via the surrounding request log). +- The helper handles every malformed-input shape (per GPT-5.5 cycle-1 F8). The complete decision table: + + | Input shape | Behavior | + |---|---| + | `list[str]` (legacy `ARRAY(Text)` shape that escaped the migration backfill) | Wrap each string `s` as `TextFollowup(kind="text", rationale=s, search_space=None)`. | + | `list[dict]` with valid `kind ∈ {"narrow", "widen", "text"}` and the per-variant schema | `FollowupItemAdapter.validate_python(item)`. | + | `dict` with `kind ∈ {"narrow", "widen"}` whose `search_space` fails `SearchSpace` validation (cardinality, bounds, etc.) | Downgrade to `TextFollowup(kind="text", rationale="[validation failed: ] " + original_rationale, search_space=None)`. | + | `dict` with `kind ∈ {"narrow", "widen"}` that fails `FollowupItemAdapter.validate_python(...)` for any other reason (missing/non-string `rationale`, missing/null `search_space`, malformed non-dict `search_space`, extra fields rejected by `extra="forbid"`) | If a top-level `rationale` string can be salvaged, downgrade to `TextFollowup(kind="text", rationale="[validation failed: ] " + salvaged_rationale, search_space=None)` and emit `digest_followup_validation_downgraded`. Otherwise drop with `digest_followup_dropped`. **All `ValidationError` paths from `FollowupItemAdapter` MUST be caught** — none escape (per GPT-5.5 cycle-3 F2). | + | `dict` with `kind="text"` but malformed (missing rationale, extra fields, etc.) | If a `rationale` string can be salvaged, emit `TextFollowup(kind="text", rationale=, search_space=None)`; otherwise drop with a WARN. | + | `dict` with `kind` missing OR `kind` ∉ {`narrow`, `widen`, `text`} | If a top-level `rationale` string exists, salvage it as a `TextFollowup`; otherwise drop with a WARN. | + | Non-`dict` array element (number, bool, null, nested array) | Drop with a WARN. | + | Top-level input is not a list (object, scalar, null) | Return `[]` and log an ERROR `digest_followups_top_level_malformed`. | + +- The system **MUST NOT** abort the digest write because of a malformed followup. The narrative + valid followups still persist; invalid ones persist as `text` items carrying the original intent (or are silently dropped per the table above when no rationale is recoverable). +- The system **MUST** emit a `digest_followup_validation_downgraded` structlog WARN event for each downgrade with `study_id`, `proposal_id`, `original_kind`, and the truncated validation error. +- The system **MUST** emit a `digest_followup_dropped` structlog WARN event for each drop (no salvageable rationale) with `study_id`, `proposal_id`, and the unparseable item (truncated to 200 chars). + +### FR-5: New columns on `studies` for LLM-followup lineage + +- The system **MUST** add `parent_proposal_id VARCHAR(36) NULL` to `studies`, with a foreign-key constraint to `proposals(id)` and **NO** `ON DELETE` clause (no cascade, no SET NULL — see the trigger requirement below for the lineage-detach behavior). +- The system **MUST** add `parent_proposal_followup_index INT NULL` to `studies` — 0-based position of the LLM-emitted followup within the parent proposal's structured `suggested_followups` array. +- The system **MUST** create a partial B-tree index on `studies(parent_proposal_id) WHERE parent_proposal_id IS NOT NULL` so future "show all studies spawned from proposal X" queries are bounded. +- The system **MUST** enforce the pair invariant via a CHECK constraint: `(parent_proposal_id IS NULL AND parent_proposal_followup_index IS NULL) OR (parent_proposal_id IS NOT NULL AND parent_proposal_followup_index IS NOT NULL AND parent_proposal_followup_index >= 0)`. The `>= 0` lower bound is defensive against direct DB writes that bypass the API's `Field(ge=0)` validation (per GPT-5.5 cycle-1 F4). +- The system **MUST** install a `BEFORE DELETE ON proposals` trigger (e.g., `trg_clear_studies_parent_proposal_on_proposal_delete`) that, for each `studies` row where `parent_proposal_id = OLD.id`, sets BOTH `parent_proposal_id = NULL` AND `parent_proposal_followup_index = NULL`. This is the mechanism that preserves the CHECK invariant when a parent proposal is hard-deleted — `ON DELETE SET NULL` on the FK alone would clear only the FK column and immediately violate the pair CHECK (per GPT-5.5 cycle-1 F1). The trigger and its function are created in the same migration `upgrade()` and dropped in `downgrade()`. +- The system **MUST NOT** require either column on existing rows or on studies created without a parent followup — both columns are nullable. + +### FR-6: `digests.suggested_followups` column-type migration + +- The system **MUST** ship one Alembic migration that ALTERs `digests.suggested_followups` from `ARRAY(Text)` to `JSONB`. +- The system **MUST** perform the type change using **migration-local helper functions**, not inline subquery USING expressions. PostgreSQL rejects subqueries in `ALTER COLUMN TYPE ... USING` expressions with `ERROR: cannot use subquery in transform expression` (verified empirically against the project's local Postgres 16, per GPT-5.5 cycle-2 F1). A scalar PL/pgSQL function called from USING is the supported alternative. +- The migration `upgrade()` body MUST follow this shape (function defined, used, dropped): + ```sql + CREATE OR REPLACE FUNCTION _fn_wrap_text_array_as_jsonb_followups(arr TEXT[]) + RETURNS jsonb AS $$ + DECLARE + result jsonb := '[]'::jsonb; + elem TEXT; + BEGIN + IF arr IS NULL THEN + RETURN '[]'::jsonb; + END IF; + FOREACH elem IN ARRAY arr LOOP + result := result || jsonb_build_array(jsonb_build_object( + 'kind', 'text', + 'rationale', elem, + 'search_space', NULL + )); + END LOOP; + RETURN result; + END; + $$ LANGUAGE plpgsql IMMUTABLE; + + ALTER TABLE digests ALTER COLUMN suggested_followups DROP DEFAULT; + ALTER TABLE digests + ALTER COLUMN suggested_followups TYPE jsonb + USING _fn_wrap_text_array_as_jsonb_followups(suggested_followups); + ALTER TABLE digests + ALTER COLUMN suggested_followups SET DEFAULT '[]'::jsonb; + + DROP FUNCTION _fn_wrap_text_array_as_jsonb_followups(TEXT[]); + ``` +- The migration `downgrade()` body MUST follow the symmetric shape: + ```sql + CREATE OR REPLACE FUNCTION _fn_unwrap_jsonb_followups_as_text_array(payload jsonb) + RETURNS TEXT[] AS $$ + DECLARE + result TEXT[] := ARRAY[]::TEXT[]; + elem jsonb; + BEGIN + IF payload IS NULL OR jsonb_array_length(payload) = 0 THEN + RETURN ARRAY[]::TEXT[]; + END IF; + FOR elem IN SELECT * FROM jsonb_array_elements(payload) LOOP + result := result || (elem->>'rationale'); + END LOOP; + RETURN result; + END; + $$ LANGUAGE plpgsql IMMUTABLE; + + ALTER TABLE digests ALTER COLUMN suggested_followups DROP DEFAULT; + ALTER TABLE digests + ALTER COLUMN suggested_followups TYPE text[] + USING _fn_unwrap_jsonb_followups_as_text_array(suggested_followups); + ALTER TABLE digests + ALTER COLUMN suggested_followups SET DEFAULT ARRAY[]::text[]; + + DROP FUNCTION _fn_unwrap_jsonb_followups_as_text_array(jsonb); + ``` + Downgrade is **lossy by design** — any non-`text` kind reverts to its rationale string. Acceptable in MVP1 (no production data; downgrade is a dev-DB operation). +- The system **MUST** set the new `server_default` to `'[]'::jsonb` (matching the prior `ARRAY[]::TEXT[]` empty-default behavior). The DROP DEFAULT step is required because the old default's type doesn't match the new column type. +- The system **MUST** verify round-trip with `alembic upgrade head && alembic downgrade -1 && alembic upgrade head` per CLAUDE.md Absolute Rule #5, including a fixture with both populated AND empty `suggested_followups` rows so both branches of the helper functions are exercised. +- The system **MUST NOT** require a separate `suggested_followups_structured` column or a dual-write deprecation window (rejected — see §4 anti-patterns). + +### FR-7: Wire-shape change to `DigestResponse` and `_DigestEmbed` + +- The system **MUST** change `DigestResponse.suggested_followups` (`backend/app/api/v1/schemas.py:949`) from `list[str]` to `list[FollowupItem]`. +- The system **MUST** change `_DigestEmbed.suggested_followups` (`backend/app/api/v1/schemas.py:1006`) from `list[str]` to `list[FollowupItem]`. +- The system **MUST** call `parse_followup_list(digest.suggested_followups, study_id=..., proposal_id=...)` at EVERY response-construction site that exposes the field — both `GET /api/v1/studies/{study_id}/digest` AND `GET /api/v1/proposals/{proposal_id}` (the `_DigestEmbed` construction at `backend/app/api/v1/proposals.py:155-163`). The ORM column returns raw `list[dict]`; raw pass-through would make Pydantic response validation reject defensive legacy `list[str]` rows (per GPT-5.5 cycle-2 F5). The wrapper guarantees both endpoints emit the canonical shape regardless of what's in the database. +- The system **MUST** preserve all other fields on both schemas unchanged. +- Notes: This is a breaking change to the response shape but MVP1 has no external consumers — only the in-repo Next.js frontend reads these endpoints. The frontend lands in the same PR. + +### FR-8: LLM prompt updates + +- The system **MUST** update `prompts/digest_narrative.system.md` to describe the three followup kinds + when to use each (`narrow` = winner sits in a sub-region of the prior search space → emit narrower bounds; `widen` = winner is at an edge (`= low` or `= high`) → emit broader bounds; `text` = a suggestion that requires operator judgment, e.g., "consider adding a new parameter to the template"). +- The system **MUST** update `prompts/digest_narrative.user.jinja` to render the parent study's `search_space` as a structured `` block alongside the existing `` / `` blocks, so the LLM can transform it into `narrow` / `widen` proposals. +- The system **MUST** preserve the drift-followup contract from the existing worker (when `` is non-empty AND `structured_output_enabled` is true, the first followup mentions the drift). The drift followup is emitted by the worker (post-LLM, as today) as a synthesized `{kind: "text", rationale: , search_space: null}` item prepended to the LLM-emitted list, then truncated to 5. In capability-degraded mode (`structured_output_enabled=false`) no drift item is emitted — `suggested_followups` stays `[]` (per FR-1 clarification / GPT-5.5 cycle-3 F1). + +### FR-9: UI — kind-discriminated followup cards + +- The system **MUST** rewrite `ui/src/components/proposals/suggested-followups-panel.tsx` to render each followup as a kind-discriminated card. +- For `kind="narrow"` and `kind="widen"`: the card **MUST** show the rationale text + a primary "Run this followup" button. The card **SHOULD** include a collapsible "Show search space" detail that renders the diff vs the parent study's `search_space`. +- For `kind="text"`: the card **MUST** render the rationale as bullet text (matching today's behavior for plain-string suggestions). NO "Run this followup" button. +- The system **MUST** preserve the existing `data-testid="suggested-followups-list"` on the container for E2E continuity. +- The system **MUST** use new per-item `data-testid` values: `followup-${i}-run` (button), `followup-${i}-card` (card), `followup-${i}-show-search-space` (collapse toggle). + +### FR-10: UI — "Run this followup" pre-fills the create-study modal + +- The system **MUST** fetch `GET /api/v1/studies/{parent_study_id}` lazily when "Run this followup" is clicked, since the existing `_StudySummary` embed at `backend/app/api/v1/schemas.py:987-996` (returned by `GET /api/v1/proposals/{id}`) only carries `{id, name, status, best_metric, best_trial_id, query_set, judgment_list}` and does **not** include the `cluster_id`, `target`, `template_id`, `objective`, or `config` fields needed for prefill (per GPT-5.5 cycle-1 F6). The full `StudyDetail` response provides all required fields. The fetch uses the existing `useStudy(studyId)` hook in `ui/src/lib/api/studies.ts` (TanStack Query, cached). +- The system **MUST** open the existing `CreateStudyModal` (`ui/src/components/studies/create-study-modal.tsx`) when the parent-study fetch resolves, passing an `initialValues` prop carrying: parent study's `cluster_id`, `target`, `template_id`, `query_set_id`, `judgment_list_id`, parent's `objective` (metric / k / direction), parent's stop conditions from `config` (`max_trials`, `time_budget_min`, `parallelism`, `sampler`, `pruner`, `seed`, `trial_timeout_s`), the LLM-proposed `search_space` (serialized to JSON in the form's `search_space_text` field), and the lineage pair `{parent_proposal_id, parent_proposal_followup_index}`. +- The system **MUST** call `form.reset(derivedInitialValues)` inside a `useEffect` keyed on `[open, initialValues]` so that clicking different followups within a single page session correctly re-seeds the form (per GPT-5.5 cycle-1 F7 — React Hook Form's `defaultValues` apply only on initial mount; without an explicit `reset`, a second followup click would render stale values from the first). The existing modal-open reset effect (currently at `create-study-modal.tsx:247-251`) is the natural extension point. +- The system **MUST** allow the operator to edit any pre-filled field before submitting (the modal's existing 5-step wizard wiring is preserved — no read-only mode). +- The system **MUST** include the lineage pair in the eventual `POST /api/v1/studies` body via the new optional `parent` field (FR-11). +- The system **MUST** default the new study's `name` to a descriptive value derived from the parent: e.g., `" — followup # ()"`. Operator can override. + +### FR-11: API — `POST /api/v1/studies` accepts optional `parent` body field + +- The system **MUST** extend `CreateStudyRequest` (`backend/app/api/v1/schemas.py:613-630`) with an optional `parent: ParentFollowupRef | None = None` field, where `ParentFollowupRef` carries `{proposal_id: str (UUIDv7, 36 chars), followup_index: int (>= 0)}`. +- When `body.parent` is set, the system **MUST** validate: (a) the proposal exists (404 `PROPOSAL_NOT_FOUND` if not); (b) the proposal has a digest (404 `DIGEST_NOT_FOUND` if not); (c) `followup_index < len(parsed_followups)` where `parsed_followups = parse_followup_list(digest.suggested_followups, study_id=digest.study_id, proposal_id=proposal.id)` (422 `FOLLOWUP_INDEX_OUT_OF_RANGE` if not). The validation MUST use the **parsed canonical list**, not the raw JSONB array length, so the index always matches the followup the operator saw on the proposal-detail page (per GPT-5.5 cycle-3 F3). Defensive dropped malformed elements would otherwise shift indices between the displayed list and the validation. +- When `body.parent` validates, the system **MUST** persist `parent_proposal_id = body.parent.proposal_id` and `parent_proposal_followup_index = body.parent.followup_index` on the new study row, atomically with the existing INSERT. +- The system **MUST NOT** require `body.parent` — omitting it preserves all existing create-study behavior verbatim. +- The system **MUST NOT** reject a `body.parent` whose referenced followup is `kind="text"` — the operator may have manually populated `search_space` and chosen to record lineage anyway. (The UI only shows the "Run this followup" button on `narrow`/`widen` items, but the API doesn't enforce that.) + +### FR-12: Retire the dead `?hypothesis=` button + +- The system **MUST** remove the `` button from `ui/src/components/proposals/suggested-followups-panel.tsx`. +- The system **MUST NOT** leave a redirect, fallback, or deprecated marker — the path is dead today and the new path replaces it cleanly (per CLAUDE.md user instructions: forward-only, no legacy preservation). + +### FR-13: Glossary additions for the new tooltips + +- The system **MUST** add the following glossary keys to `ui/src/lib/glossary.ts` (see §11 tooltip inventory for exact text): + - `proposal.followup_kind_narrow` + - `proposal.followup_kind_widen` + - `proposal.followup_kind_text` + - `proposal.followup_run_button` + - `proposal.followup_search_space_diff` +- The existing `proposal.suggested_followups` key copy may be left unchanged (still accurate) OR refined to mention the kind discriminator in a follow-up PR — no spec requirement. + +## 8) API and data contract baseline + +### 8.1 Endpoint surface + +| Method | Path | Purpose | Key error codes | +|---|---|---|---| +| `POST` | `/api/v1/studies` | **Modified.** Existing endpoint; gains optional `parent` body field for LLM-followup lineage. | `PROPOSAL_NOT_FOUND` (404, new), `DIGEST_NOT_FOUND` (404, new), `FOLLOWUP_INDEX_OUT_OF_RANGE` (422, new). All pre-existing codes preserved (`INVALID_SEARCH_SPACE`, `CLUSTER_NOT_FOUND`, `TEMPLATE_NOT_FOUND`, `QUERY_SET_NOT_FOUND`, `JUDGMENT_LIST_NOT_FOUND`, `VALIDATION_ERROR`, `JUDGMENT_CLUSTER_MISMATCH`, `JUDGMENT_TARGET_MISMATCH`, `INSUFFICIENT_JUDGMENT_OVERLAP`, `SEARCH_SPACE_UNKNOWN_PARAM`, `SEARCH_SPACE_MISSING_DECLARED_PARAM`). | +| `GET` | `/api/v1/studies/{study_id}/digest` | **Modified.** Response shape change: `suggested_followups` becomes `list[FollowupItem]`. | `DIGEST_NOT_READY` (404, unchanged). | +| `GET` | `/api/v1/proposals/{proposal_id}` | **Modified.** Inline `digest.suggested_followups` shape change. | `PROPOSAL_NOT_FOUND` (404, unchanged). | + +No new endpoints. No method or path changes — only body/response shape extensions. + +### 8.2 Contract rules + +- Error body **MUST** match the canonical envelope: `{ "detail": { "error_code": "", "message": "", "retryable": } }` (per `_err()` helper at `backend/app/api/v1/studies.py:75-79`). +- Status codes **MUST** be deterministic per scenario. +- Cross-tenant access: N/A (single-tenant MVP1). +- The discriminator field `kind` on `FollowupItem` **MUST** appear in every wire payload — Pydantic v2 discriminated unions require it for round-trip validation. + +### 8.3 Response examples + +**Success — `GET /api/v1/studies/{id}/digest` with mixed followup kinds:** + +```json +{ + "id": "0190a3b4-1234-7abc-9def-000000000001", + "study_id": "0190a3b4-1234-7abc-9def-000000000002", + "narrative": "The study converged on `title_boost=2.1` with NDCG@10 = 0.84 (+0.13 vs baseline)...", + "parameter_importance": { "title_boost": 0.62, "tie_breaker": 0.18 }, + "recommended_config": { "title_boost": 2.1, "tie_breaker": 0.3 }, + "suggested_followups": [ + { + "kind": "narrow", + "rationale": "title_boost winners clustered between 1.8 and 2.4; narrow to that band.", + "search_space": { + "params": { + "title_boost": { "type": "float", "low": 1.8, "high": 2.4 }, + "tie_breaker": { "type": "float", "low": 0.0, "high": 1.0 } + } + } + }, + { + "kind": "widen", + "rationale": "tie_breaker hit the upper edge of [0, 1]; try [0, 2].", + "search_space": { + "params": { + "title_boost": { "type": "float", "low": 0.1, "high": 10.0, "log": true }, + "tie_breaker": { "type": "float", "low": 0.0, "high": 2.0 } + } + } + }, + { + "kind": "text", + "rationale": "Consider adding a category_boost parameter to the template — several winning trials suggest category prioritization matters.", + "search_space": null + } + ], + "generated_by": "openai:gpt-4o-2024-08-06", + "generated_at": "2026-05-23T18:00:00Z" +} +``` + +**Success — `POST /api/v1/studies` with `parent` lineage:** + +Request body: +```json +{ + "name": "Title-boost study — followup #1 (narrow)", + "cluster_id": "0190a3b4-1234-7abc-9def-000000000010", + "target": "products_v3", + "template_id": "0190a3b4-1234-7abc-9def-000000000011", + "query_set_id": "0190a3b4-1234-7abc-9def-000000000012", + "judgment_list_id": "0190a3b4-1234-7abc-9def-000000000013", + "search_space": { "params": { "title_boost": { "type": "float", "low": 1.8, "high": 2.4 } } }, + "objective": { "metric": "ndcg", "k": 10, "direction": "maximize" }, + "config": { "max_trials": 200, "parallelism": 4, "sampler": "tpe", "pruner": "median" }, + "parent": { "proposal_id": "0190a3b4-1234-7abc-9def-000000000020", "followup_index": 0 } +} +``` + +Response: standard `StudyDetail` (existing shape — `parent_study_id` field already present, lineage to the parent proposal is on the new `parent_proposal_id` + `parent_proposal_followup_index` columns, which are NOT currently exposed on `StudyDetail`; see §19 Decision Log #D-5 for rationale). + +**Failure — `POST /api/v1/studies` with out-of-range `parent.followup_index`:** + +HTTP 422: +```json +{ + "detail": { + "error_code": "FOLLOWUP_INDEX_OUT_OF_RANGE", + "message": "parent.followup_index=7 exceeds the digest's suggested_followups length (3) for proposal 0190a3b4-1234-7abc-9def-000000000020", + "retryable": false + } +} +``` + +**Failure — `POST /api/v1/studies` with unknown `parent.proposal_id`:** + +HTTP 404: +```json +{ + "detail": { + "error_code": "PROPOSAL_NOT_FOUND", + "message": "proposal 0190a3b4-1234-7abc-9def-000000000020 not found", + "retryable": false + } +} +``` + +### 8.4 Enumerated value contracts + +| Field | Accepted values (exact) | Backend source of truth | Frontend call site(s) | +|---|---|---|---| +| `FollowupItem.kind` | `narrow`, `widen`, `text` | `backend/app/domain/study/followups.py` — the `Literal["narrow", "widen", "text"]` discriminator on the `NarrowFollowup`/`WidenFollowup`/`TextFollowup` Pydantic models (new file added by this spec). Mirrored as a TypeScript `type FollowupKind = 'narrow' \| 'widen' \| 'text'` in `ui/src/lib/enums.ts` with the source-of-truth comment `// Values must match backend/app/domain/study/followups.py FollowupItem.kind`. | `suggested-followups-panel.tsx` (kind-based card rendering). Forbidden as a dropdown — operators don't pick the kind, the LLM emits it. | +| `parent.proposal_id` | Any UUIDv7-shaped string of length 36 (free-form, validated as FK at the API boundary). | `backend/app/db/models/proposal.py` — `id: Mapped[str] = mapped_column(String(36), primary_key=True)`. | "Run this followup" click handler — value is the parent proposal's `id`, not user-pickable. | +| `parent.followup_index` | Any integer `>= 0` and `< len(digest.suggested_followups)`. | `backend/app/domain/study/followups.py` — `FollowupIndex = Annotated[int, Field(ge=0)]`. Upper bound validated at the API boundary against the live digest. | "Run this followup" click handler — value is the array index, not user-pickable. | + +No frontend dropdown displays `kind` as an option list — the LLM is the producer and the UI only renders. The discriminator value is asserted on `data-testid` patterns (`followup-${i}-card` per kind) and on inline kind labels ("Narrow" / "Widen" / "Suggestion") for clarity (see §11 tooltip inventory). + +### 8.5 Error code catalog + +| Code | HTTP Status | Meaning | +|------|-------------|---------| +| `PROPOSAL_NOT_FOUND` | 404 | `body.parent.proposal_id` references a proposal that does not exist. | +| `DIGEST_NOT_FOUND` | 404 | The referenced proposal exists but has no digest yet (digest worker hasn't run). | +| `FOLLOWUP_INDEX_OUT_OF_RANGE` | 422 | `body.parent.followup_index` >= `len(digest.suggested_followups)`. | + +All other codes used by this feature are pre-existing on `POST /api/v1/studies` and unchanged. + +## 9) Data model and state transitions + +### Modified table: `digests` + +- **CHANGE** column `suggested_followups` from `ARRAY(Text) NOT NULL DEFAULT ARRAY[]::TEXT[]` to `JSONB NOT NULL DEFAULT '[]'::jsonb`. +- Backfill via USING clause (see FR-6): each existing text element wrapped as `{kind: "text", rationale: , search_space: null}`. +- ORM mapping changes from `Mapped[list[str]] = mapped_column(ARRAY(Text), ...)` to `Mapped[list[dict[str, Any]]] = mapped_column(JSONB, ...)`. (The repo-layer returns raw `list[dict]`; the API layer / domain reader applies the Pydantic discriminated-union validation at the response-serialization boundary.) + +### Modified table: `studies` + +- **ADD** column `parent_proposal_id VARCHAR(36) NULL` with `FOREIGN KEY (parent_proposal_id) REFERENCES proposals(id)` (no `ON DELETE` clause; the BEFORE DELETE trigger below handles cleanup). +- **ADD** column `parent_proposal_followup_index INT NULL`. +- **ADD** partial B-tree index `ix_studies_parent_proposal_id ON studies (parent_proposal_id) WHERE parent_proposal_id IS NOT NULL`. +- **ADD** CHECK constraint `studies_parent_proposal_pair_check`: `(parent_proposal_id IS NULL AND parent_proposal_followup_index IS NULL) OR (parent_proposal_id IS NOT NULL AND parent_proposal_followup_index IS NOT NULL AND parent_proposal_followup_index >= 0)`. +- **ADD** trigger + trigger function: + ```sql + CREATE OR REPLACE FUNCTION fn_clear_studies_parent_proposal_on_proposal_delete() + RETURNS TRIGGER AS $$ + BEGIN + UPDATE studies + SET parent_proposal_id = NULL, + parent_proposal_followup_index = NULL + WHERE parent_proposal_id = OLD.id; + RETURN OLD; + END; + $$ LANGUAGE plpgsql; + + CREATE TRIGGER trg_clear_studies_parent_proposal_on_proposal_delete + BEFORE DELETE ON proposals + FOR EACH ROW + EXECUTE FUNCTION fn_clear_studies_parent_proposal_on_proposal_delete(); + ``` + Migration `downgrade()` drops the trigger then the function. +- ORM mapping additions in `backend/app/db/models/study.py`: + ```python + parent_proposal_id: Mapped[str | None] = mapped_column( + String(36), ForeignKey("proposals.id"), nullable=True + ) + parent_proposal_followup_index: Mapped[int | None] = mapped_column(Integer, nullable=True) + ``` + +### Required invariants + +- `studies.parent_proposal_id` and `studies.parent_proposal_followup_index` are **set together or NULL together** (DB CHECK constraint enforced). +- `studies.parent_proposal_id` is allowed to reference a proposal whose `status` is anything in `{pending, pr_opened, pr_merged, rejected}` — the operator may chain a followup off any proposal, including rejected ones. The `BEFORE DELETE ON proposals` trigger (FR-5) handles the hard-delete edge case atomically; soft-delete is not currently used on proposals. +- `studies.parent_proposal_followup_index` is **not validated against the live digest array length after creation** — it captures intent at create time. If the digest is later regenerated and shrinks, the index may become stale; this is acceptable (lineage data, not a live pointer). +- A study row MAY have **both** `parent_study_id` (from auto-followup chain) AND `parent_proposal_id` (from LLM-followup click) set simultaneously. The two tracks are independent. +- `digests.suggested_followups` is a JSONB array; each element is a JSON object with at least `kind` ∈ {`narrow`, `widen`, `text`} and `rationale: string`. `search_space` is a JSON object for `narrow`/`widen` and `null` for `text`. Length cap: 5 (enforced by the LLM schema; not enforced by a CHECK constraint — defensive truncation in the worker). + +### State transitions + +No new states. The existing `study.status` lifecycle (`queued → running → {completed | cancelled | failed}`) is unchanged. The existing `proposal.status` lifecycle (`pending → {pr_opened, rejected}`, `pr_opened → pr_merged`) is unchanged. + +### Idempotency/replay behavior + +- `POST /api/v1/studies` with `body.parent` set is **not idempotent** — calling it twice creates two studies with the same parent lineage. This matches the existing endpoint contract (no `Idempotency-Key` header in MVP1). +- The UI prevents the operator from double-clicking via React Hook Form's `isSubmitting` state (existing behavior in the create-study modal). +- The digest worker (FR-4 validator) is idempotent against the existing digest UNIQUE-per-study constraint (`backend/app/db/models/digest.py:38`) — re-running the worker after a downgrade-and-fix never produces duplicate downgrade WARN events because the per-study digest is written once. + +## 10) Security, privacy, and compliance + +- **Threats:** + 1. **LLM-emitted `search_space` causes runaway trial count.** Mitigation: FR-4 validator runs `SearchSpace.model_validate()` which enforces the existing 10⁶ cardinality cap (`backend/app/domain/study/search_space.py:111-118`). Items that exceed the cap downgrade to `text` — never persist as `narrow`/`widen`. + 2. **LLM-emitted `search_space` references parameters not declared by the template.** Mitigation: The downstream `POST /api/v1/studies` runs the existing `validate_against_template()` check (`backend/app/api/v1/studies.py:218-227`) on submit, surfacing `SEARCH_SPACE_UNKNOWN_PARAM` / `SEARCH_SPACE_MISSING_DECLARED_PARAM` at 400. The pre-fill modal exposes this to the operator before they submit. + 3. **Stale `parent.followup_index` after digest regeneration.** Mitigation: FR-11 validates `followup_index < len(digest.suggested_followups)` at submit time. Stale indices produce 422. + 4. **Cross-proposal lineage leak in metadata.** N/A in MVP1 (single-tenant). For MVP4: `parent_proposal_id` is tenant-scoped because both `studies` and `proposals` will carry `tenant_id` — the new FK does not bridge tenants. + 5. **LLM hallucinates `parent_proposal_id` or `followup_index` content into the rationale.** Mitigation: rationale is operator-facing prose only; the actionable data is the `search_space` field which the validator enforces. The "Run this followup" button's payload is sourced from the proposal-detail API response, never from the rationale text. +- **Controls:** + - Existing digest-worker controls preserved: per-study advisory lock (`backend/workers/digest.py:219-244`), capability check (`backend/workers/digest.py:517-540`), daily-budget guard (`backend/workers/digest.py:554-578`), persist-first then record-cost ordering. + - No new secrets introduced. + - No new external services called. +- **Secrets/key handling:** N/A — reuses existing `OPENAI_API_KEY` from `Settings` (mounted-file pattern per CLAUDE.md Absolute Rule #2). +- **Auditability:** §6 catalogs the three forthcoming MVP2 audit events. No `audit_log` table in MVP1. +- **Data retention/deletion/export impact:** + - Hard-deleting a parent proposal triggers `trg_clear_studies_parent_proposal_on_proposal_delete` (FR-5) which sets BOTH `parent_proposal_id` AND `parent_proposal_followup_index` to NULL on every descendant study row, preserving the pair CHECK invariant. The lineage information is lost; descendant studies remain intact. + - The new columns and JSONB payload are covered by the existing `chore_e2e_test_rows_isolation` test-only cleanup endpoint (`DELETE /api/v1/_test/digests/{id}`) via the same hard-delete path — no change required. + +## 11) UX flows and edge cases + +### Information architecture + +- **Navigation placement:** The followup cards live inside the existing `SuggestedFollowupsPanel` on `/proposals/[id]` — same location, same card slot. No new routes, no new nav links. The "Run this followup" button opens the existing `` overlay (no navigation away from the proposal page). +- **Labeling taxonomy:** + - Panel title: **"Suggested follow-ups"** (unchanged from today). + - Per-card kind labels: **"Narrow"** (badge, `narrow`), **"Widen"** (badge, `widen`), **"Suggestion"** (badge, `text`). Matches operator mental model: "narrow"/"widen" come from the search-space-tuning vocabulary the operator already uses; "Suggestion" reads naturally for free-form text. + - Primary action button: **"Run this followup"** (replaces the dead "Create study from this hypothesis" button — same intent, accurate naming). + - Collapse toggle: **"Show search space"** (when collapsed) / **"Hide search space"** (when expanded). +- **Content hierarchy:** Each card renders top-to-bottom: kind badge + rationale text → "Show search space" toggle (collapsed by default) → primary "Run this followup" button (right-aligned). The card is primary; the search-space detail is progressive disclosure (most operators trust the rationale + just click Run). +- **Progressive disclosure:** Search-space JSON / diff is hidden by default. Operator expands to scrutinize bounds before committing. The diff renderer reuses the row primitive from `feat_create_study_search_space_builder` (`ui/src/components/studies/search-space-builder/`) where feasible; falls back to a syntax-highlighted JSON viewer if the row primitive doesn't compose cleanly (story-time decision). +- **Relationship to existing pages:** Extends the proposal-detail page. The followups panel was already there (current bullet rendering); this spec swaps the rendering for kind-aware cards but keeps the panel container. + +### Tooltips and contextual help + +| Element | Tooltip / help text | Trigger | Placement | +|---------|-------------------|---------|-----------| +| `Narrow` kind badge | (new glossary key `proposal.followup_kind_narrow`) "The study's winning configuration sits in a sub-region of the prior search space. This followup re-runs with a tighter range to confirm." | hover | top | +| `Widen` kind badge | (new glossary key `proposal.followup_kind_widen`) "The winning configuration hit an edge of the prior search space. This followup re-runs with a broader range to find a possibly-better setting." | hover | top | +| `Suggestion` kind badge | (new glossary key `proposal.followup_kind_text`) "A free-form suggestion from the LLM. Needs operator interpretation — no auto-prefill available." | hover | top | +| `Run this followup` button | (new glossary key `proposal.followup_run_button`) "Opens the create-study wizard pre-filled with this followup's settings. You can review and edit before submitting." | hover | top | +| `Show search space` toggle | (new glossary key `proposal.followup_search_space_diff`) "Compare this followup's proposed search space against the parent study's." | hover | top | +| Existing panel title `Suggested follow-ups` | (existing glossary key `proposal.suggested_followups`, copy unchanged: "LLM-generated next-study hypotheses based on this study's parameter-importance pattern. Click to seed a new study.") | hover | top | + +Tooltip placement uses the existing `` primitive (see `ui/src/components/common/info-tooltip.tsx`, already imported in `suggested-followups-panel.tsx`). + +### Primary flows + +1. **Run a `narrow` followup (happy path).** + Operator on `/proposals/` → scrolls to "Suggested follow-ups" → sees a card with a `Narrow` badge + rationale → clicks "Show search space" → sees the proposed bounds diff'd against the parent's → clicks "Run this followup" → `CreateStudyModal` opens with `cluster_id`, `target`, `template_id`, `query_set_id`, `judgment_list_id`, `objective`, `config` pre-filled from the parent study, and `search_space_text` pre-filled with the LLM's JSON. `name` defaults to `" — followup #1 (narrow)"`. Operator reviews the search space, clicks Submit. `POST /api/v1/studies` includes `parent: {proposal_id: , followup_index: 0}`. Returns 201 with `StudyDetail`. Operator is navigated to `/studies/`. The new study row carries `parent_proposal_id=` + `parent_proposal_followup_index=0`. +2. **Run a `widen` followup.** Identical to (1) with `Widen` badge + broader bounds. +3. **Read a `text` followup.** Operator sees a card with `Suggestion` badge + rationale text. NO button. Operator manually opens the create-study modal (existing "+ New study" entrypoint) and authors the experiment themselves. No lineage recorded. + +### Edge/error flows + +- **LLM emits an invalid `search_space` (cardinality > 10⁶, bad bounds, etc.).** The digest worker's validator (FR-4) downgrades the item to `kind: "text"` with `rationale: "[validation failed: ] " + original_rationale`. Operator sees the item as a `Suggestion` card with the validation prefix making the failure visible. No data lost. +- **Operator submits the pre-filled form but the parent proposal was hard-deleted between page load and submit.** `POST /api/v1/studies` returns 404 `PROPOSAL_NOT_FOUND`. The modal surfaces the error; operator can either remove the `parent` payload (manual workaround — refresh the proposal page is the user-visible recovery) or abandon. +- **Operator submits the pre-filled form but the parent proposal's digest was regenerated (operator ran the runbook escape hatch `DELETE FROM digests` + re-enqueue between page load and submit) and the new digest has fewer followups.** `POST /api/v1/studies` returns 422 `FOLLOWUP_INDEX_OUT_OF_RANGE`. Same recovery as above. +- **LLM emits 5 valid `narrow`/`widen` items AND the worker prepends a drift followup.** Per FR-8, drift followups are synthesized post-LLM as `kind: "text"` and prepended; the combined list is then truncated to 5 (matching the existing worker contract at `backend/workers/digest.py:774-775`). The dropped tail (an LLM-emitted item, never a drift item) is logged at INFO; no error. +- **Capability-degraded digest (structured output unavailable).** The worker writes `suggested_followups=[]` (preserved from today — see `backend/workers/digest.py:757`). UI panel hides itself (existing `if (followups.length === 0) return null` at `suggested-followups-panel.tsx:13`). +- **Zero-trials digest.** The worker writes the `_FAILURE_NARRATIVE` digest with `suggested_followups=[]` (preserved — see `backend/workers/digest.py:339-369`). UI panel hides itself. +- **Operator deletes the parent proposal AFTER spawning a followup study.** The `BEFORE DELETE ON proposals` trigger (FR-5) NULLs both `parent_proposal_id` and `parent_proposal_followup_index` on every descendant study row, then the delete proceeds. The study itself is untouched. Lineage is lost; no other impact. + +## 12) Given/When/Then acceptance criteria + +### AC-1: LLM returns three valid kinds; UI renders kind-discriminated cards +- Given a digest with `suggested_followups` containing one `narrow`, one `widen`, and one `text` item +- When the operator loads `/proposals/{proposal_id}` +- Then the panel shows three cards in order with the matching badge ("Narrow", "Widen", "Suggestion"); the `narrow` and `widen` cards each show a "Run this followup" button; the `text` card shows no button. +- Example values: + - Input: a digest fixture with the three-item payload from §8.3. + - Expected: three cards rendered with `data-testid="followup-0-card"` (Narrow), `followup-1-card` (Widen), `followup-2-card` (Suggestion); `followup-0-run` and `followup-1-run` buttons present; no `followup-2-run`. + +### AC-2: "Run this followup" pre-fills the modal with parent + LLM payload +- Given a `narrow` followup at index 0 on proposal `P` with parent study `S` (cluster `C`, target `T`, template `Tpl`, query set `QS`, judgment list `JL`, `objective={metric: "ndcg", k: 10, direction: "maximize"}`, `config={max_trials: 200, parallelism: 4, sampler: "tpe", pruner: "median"}`) and `search_space={params: {title_boost: {type: "float", low: 1.8, high: 2.4}}}` +- When the operator clicks "Run this followup" on the card +- Then the `CreateStudyModal` opens with `cluster_id=C.id`, `target=T`, `template_id=Tpl.id`, `query_set_id=QS.id`, `judgment_list_id=JL.id`, the objective fields pre-set to ndcg/10/maximize, the stop-condition fields pre-set to 200/—/4/tpe/median, the search-space textarea pre-loaded with the LLM's JSON (formatted), and `name=" — followup #1 (narrow)"`. + +### AC-3: Submitting the pre-filled form persists lineage on the new study +- Given the modal pre-filled per AC-2 +- When the operator clicks Submit +- Then `POST /api/v1/studies` is called with `body.parent = {proposal_id: P.id, followup_index: 0}` and returns 201 with a `StudyDetail`. The new `studies` row has `parent_proposal_id=P.id` AND `parent_proposal_followup_index=0` AND `parent_study_id=NULL` (this is an LLM-followup, NOT an auto-chain). + +### AC-4: Validator downgrades a malformed `narrow` item to `text` +- Given the LLM emits a `narrow` item whose `search_space` has cardinality > 10⁶ (e.g., one float param plus 11 floats — `100^12 > 10^6`) +- When the digest worker validates + persists +- Then the persisted digest's `suggested_followups[i]` has `kind="text"`, `rationale` starts with `"[validation failed: search-space cardinality estimate exceeds 10^6"`, and `search_space` is `null`. A `digest_followup_validation_downgraded` WARN event is logged with `study_id`, `proposal_id`, `original_kind="narrow"`. + +### AC-5: Backward-compat read of a legacy `list[str]` row (defensive) +- Given a manually crafted legacy `digests.suggested_followups` row whose JSONB payload is a JSON array of strings (e.g., `["item one", "item two"]` — should not occur post-backfill but exercised defensively) +- When `GET /api/v1/studies/{id}/digest` is called +- Then the response's `suggested_followups` is `[{kind: "text", rationale: "item one", search_space: null}, {kind: "text", rationale: "item two", search_space: null}]`. + +### AC-6: Column-type migration round-trips cleanly +- Given a DB with one pre-migration `digests` row whose `suggested_followups = ARRAY['Try widening title_boost', 'Add tie_breaker']` +- When `alembic upgrade head` runs (applying the new migration) +- Then the row's `suggested_followups` is the JSONB array `[{"kind": "text", "rationale": "Try widening title_boost", "search_space": null}, {"kind": "text", "rationale": "Add tie_breaker", "search_space": null}]`. +- When `alembic downgrade -1` runs immediately after +- Then the row's `suggested_followups` is the text array `['Try widening title_boost', 'Add tie_breaker']` (rationale-only; lossy by design). +- When `alembic upgrade head` runs again +- Then the JSONB shape from the first step is restored. + +### AC-7: CHECK constraint blocks half-set lineage +- Given an attempt to INSERT a `studies` row with `parent_proposal_id` set but `parent_proposal_followup_index = NULL` +- When the INSERT executes +- Then the DB raises a CHECK violation on `studies_parent_proposal_pair_check`. (Symmetrically: index set + proposal_id NULL also fails.) + +### AC-8: 404 PROPOSAL_NOT_FOUND on unknown parent proposal +- Given a `POST /api/v1/studies` request with `parent: {proposal_id: "", followup_index: 0}` +- When the request is processed +- Then the response is HTTP 404 with body `{"detail": {"error_code": "PROPOSAL_NOT_FOUND", "message": "proposal not found", "retryable": false}}`. + +### AC-8b: 404 DIGEST_NOT_FOUND when the parent proposal has no digest yet +- Given a proposal `P` that exists but whose digest worker hasn't completed (no `digests` row for `P.study_id`), and a `POST /api/v1/studies` request with `parent: {proposal_id: P.id, followup_index: 0}` +- When the request is processed +- Then the response is HTTP 404 with body `{"detail": {"error_code": "DIGEST_NOT_FOUND", "message": "proposal has no digest yet", "retryable": true}}`. (Note: `retryable=true` because the operator can wait for the digest worker to finish and resubmit — distinct from `PROPOSAL_NOT_FOUND` which is `retryable=false`.) + +### AC-9: 422 FOLLOWUP_INDEX_OUT_OF_RANGE on stale index +- Given proposal `P` whose digest has 3 followups, and a `POST /api/v1/studies` request with `parent: {proposal_id: P.id, followup_index: 7}` +- When the request is processed +- Then the response is HTTP 422 with body `{"detail": {"error_code": "FOLLOWUP_INDEX_OUT_OF_RANGE", "message": "parent.followup_index=7 exceeds the digest's suggested_followups length (3) for proposal ", "retryable": false}}`. + +### AC-10: Dead `?hypothesis=` link is gone +- Given the `SuggestedFollowupsPanel` rendered on `/proposals/` +- When the test queries the DOM +- Then no `` or `` element has an `href` matching `/studies?hypothesis=`. (Lint/snapshot guard against regression.) + +### AC-11: BEFORE DELETE trigger detaches lineage without destroying the child +- Given a study `Sc` with `parent_proposal_id = P.id` and `parent_proposal_followup_index = 0`, and the parent proposal `P` exists +- When `P` is hard-deleted (via the test-only delete endpoint or a manual DB delete) +- Then the `BEFORE DELETE ON proposals` trigger (FR-5; named `trg_clear_studies_parent_proposal_on_proposal_delete`) fires and sets BOTH `Sc.parent_proposal_id = NULL` AND `Sc.parent_proposal_followup_index = NULL` in the same transaction, preserving the `studies_parent_proposal_pair_check` invariant. +- Expected end state: `Sc.parent_proposal_id IS NULL AND Sc.parent_proposal_followup_index IS NULL`, `Sc` row otherwise unchanged. `P` is gone. +- Note: This explicitly **does not** use `ON DELETE SET NULL` on the FK clause, because that would clear only `parent_proposal_id` and immediately violate the pair CHECK (per GPT-5.5 cycle-1 F1). + +### AC-12: A study can have both lineage tracks set +- Given an LLM-followup study `Sc` created from proposal `P` (so `Sc.parent_proposal_id = P.id`) that is ALSO an auto-chain descendant of study `Sp` (so `Sc.parent_study_id = Sp.id`) +- When `Sc` is read via `GET /api/v1/studies/{id}` +- Then `Sc.parent_study_id = Sp.id` (exposed via `StudyDetail.parent_study_id`, existing field at `backend/app/api/v1/schemas.py:659`). The `parent_proposal_id` + `parent_proposal_followup_index` columns are NOT currently exposed on `StudyDetail` (see §19 D-5 decision); they're queryable directly via raw SQL or via a future endpoint extension. + +## 13) Non-functional requirements + +- **Performance:** + - The new validator (FR-4) runs in O(n) where n = number of followups (≤ 5) per digest call. Negligible cost vs the LLM call. + - The new FK + partial index on `studies(parent_proposal_id) WHERE parent_proposal_id IS NOT NULL` is bounded by the proposal cardinality — in MVP1, expect ≤ 10⁴ proposals lifetime per laptop install. Index is partial so it only contains followup studies (a small fraction). + - `POST /api/v1/studies` with `body.parent` adds at most 2 extra SELECTs (proposal lookup + digest lookup). p95 stays under 500ms. + - The column-type migration on `digests` runs an `ALTER TABLE ... USING` which rewrites the table. For MVP1 (single-tenant laptop, dozens of digest rows), this completes in well under 1 second. For production (MVP3+ deployment with managed Postgres), the migration is documented as table-rewrite in the release-notes runbook; downtime is expected during deploy. Production scale ≤ 10⁵ rows keeps it under a minute. +- **Reliability:** No new SLO. The digest worker continues to be best-effort with the existing retry / advisory-lock / idempotency-guard infrastructure. +- **Operability:** + - New structlog event types: `digest_followup_validation_downgraded` (WARN). Existing events preserved. + - New error codes are listed in §8.5; mirror into `docs/01_architecture/api-conventions.md` error code catalog at doc-update time. + - No new metrics in MVP1. (MVP2+ Langfuse / SigNoz catalog gains the three audit events from §6.) +- **Accessibility/usability:** + - Per-card kind badges have `aria-label` matching the badge text ("Narrow", "Widen", "Suggestion"). + - "Run this followup" button has `aria-label="Run this followup — opens the create study form pre-filled with these settings"` so screen readers understand the action without seeing the surrounding card context. + - "Show search space" toggle uses standard `
`/`` semantics or shadcn `` (already in the repo) with proper `aria-expanded`. + - Keyboard nav: cards focusable in DOM order; Enter on "Run this followup" opens the modal; Esc closes the modal (preserves existing modal behavior). + +## 14) Test strategy requirements (spec-level) + +- **Unit tests (`backend/tests/unit/`):** + - `backend/tests/unit/domain/study/test_followups.py` (new file): Pydantic discriminated-union validation for each kind; round-trip serialization; rejection of unknown `kind` values; rejection of `narrow`/`widen` with null `search_space`; rejection of `text` with non-null `search_space`. + - `backend/tests/unit/domain/study/test_followups_backcompat.py` (new file): legacy `list[str]` wrapping into the new shape. + - `backend/tests/unit/workers/test_digest_followup_validation.py` (new file): given a mocked LLM response with one valid `narrow`, one invalid `narrow` (cardinality > 10⁶), and one `text` → the worker persists the valid + downgrades the invalid + preserves the `text`. Verifies the structlog WARN event. +- **Integration tests (`backend/tests/integration/`):** + - `backend/tests/integration/test_digest_followup_roundtrip.py` (new file): full digest worker round-trip with the LLM stub returning structured followups; assert the persisted JSONB shape; assert `GET /api/v1/studies/{id}/digest` returns the structured shape. + - `backend/tests/integration/test_studies_with_parent_followup.py` (new file): `POST /api/v1/studies` with `body.parent` set against a fixture proposal with a real digest; assert the persisted `parent_proposal_id` + `parent_proposal_followup_index`. Cover all three error paths explicitly (per GPT-5.5 cycle-2 F6): (a) unknown `proposal_id` → 404 `PROPOSAL_NOT_FOUND`, (b) existing proposal without a digest → 404 `DIGEST_NOT_FOUND`, (c) stale `followup_index` ≥ digest length → 422 `FOLLOWUP_INDEX_OUT_OF_RANGE`. + - `backend/tests/integration/test_digest_followups_migration.py` (new file): pre-migration fixture with BOTH a populated `ARRAY(Text)` row AND an empty `ARRAY[]::TEXT[]` row (exercises both COALESCE branches per FR-6); `alembic upgrade head`; assert JSONB backfill correctness on both rows; `alembic downgrade -1`; assert text-array round-trip (lossy). + - `backend/tests/integration/test_studies_parent_proposal_check.py` (new file): half-set pair INSERT fails on the CHECK constraint; negative-index INSERT fails on the CHECK constraint (per GPT-5.5 cycle-1 F4). + - `backend/tests/integration/test_studies_parent_proposal_on_delete.py` (new file, per GPT-5.5 cycle-1 F9): create a proposal-linked study; hard-delete the parent proposal; assert the child study row remains; assert BOTH `parent_proposal_id` IS NULL AND `parent_proposal_followup_index` IS NULL (trigger fired correctly). +- **Contract tests (`backend/tests/contract/`):** + - `backend/tests/contract/test_digest_response_shape.py` (extend existing or add new): assert `DigestResponse.suggested_followups` is `list[FollowupItem]` (discriminated union); assert each kind round-trips through OpenAPI. + - `backend/tests/contract/test_proposal_detail_shape.py` (extend existing): same for `_DigestEmbed.suggested_followups`. + - `backend/tests/contract/test_create_study_parent.py` (new file): assert `CreateStudyRequest.parent` is optional; assert the three new error codes appear in the OpenAPI error response schema. +- **E2E tests (`ui/tests/e2e/`):** + - One happy-path Playwright spec (`ui/tests/e2e/followup_run.spec.ts`, new file): seed a cluster + template + query set + judgment list + parent study + proposal + digest with one `narrow` followup via API helpers; navigate to `/proposals/`; click "Run this followup"; assert the create-study modal opens with the pre-filled fields; submit; assert navigation to `/studies/` and the new study row's lineage columns. + - **Real-backend test** (no `page.route()` mocking — per CLAUDE.md E2E rule). + +## 15) Documentation update requirements + +- `docs/01_architecture/data-model.md` — extend the `studies` table section to document `parent_proposal_id` + `parent_proposal_followup_index` (alongside the existing `parent_study_id` note). Update the `digests` table section to document `suggested_followups` as JSONB with the discriminated-union shape. +- `docs/01_architecture/api-conventions.md` — add `PROPOSAL_NOT_FOUND`, `DIGEST_NOT_FOUND`, `FOLLOWUP_INDEX_OUT_OF_RANGE` to the error code catalog. +- `docs/01_architecture/llm-orchestration.md` — describe the new digest LLM output shape (discriminated union) and the worker's downgrade behavior. +- `docs/02_product/planned_features/feat_digest_executable_followups/` — this spec lives here; companion `phase2_idea.md` + `phase3_idea.md` track the deferred tiers. +- `docs/03_runbooks/` — add or extend a digest-debugging runbook entry: "if all followups appear as `text` items, check the worker logs for `digest_followup_validation_downgraded` to see whether the LLM is emitting invalid `search_space` payloads." +- `docs/04_security/` — N/A (no new secret or data-flow surface). +- `docs/05_quality/testing.md` — no change required; new test files follow the existing layer convention. +- `state.md` — update active-work / queued sections to mark `feat_digest_executable_followups` as in-progress when the spec gets approval. + +## 16) Rollout and migration readiness + +- **Feature flags / staged rollout:** None. MVP1 single-tenant; ship behind no flag. +- **Migration/backfill expectations:** + - The `digests.suggested_followups` column-type migration runs in-place at `alembic upgrade head` time. For MVP1 (dozens of rows on a laptop), this is sub-second. + - The `studies.parent_proposal_id` + `parent_proposal_followup_index` column-add is purely additive (both NULL by default for existing rows). No backfill. + - Round-trip verified per CLAUDE.md Absolute Rule #5 (FR-6 + AC-6). +- **Operational readiness gates:** None new. The digest worker's existing operability (budget guard, advisory lock, capability check) is unchanged. +- **Release gate:** + - All ACs pass in CI (unit + integration + contract + E2E layers). + - Round-trip migration verified. + - `make lint`, `make typecheck`, `pnpm lint`, `pnpm typecheck`, `pnpm build` all green. + - GPT-5.5 cross-model review on the spec and the implementation plan complete (per CLAUDE.md cross-model policy). + - Gemini Code Assist findings on the PR adjudicated per CLAUDE.md. + +## 17) Traceability matrix + +| FR ID | Acceptance Criteria IDs | Planned stories/tasks (to be assigned by impl-plan) | Test files/suites | Docs to update | +|---|---|---|---|---| +| FR-1 | AC-1, AC-4 | Worker schema change story | `backend/tests/contract/test_digest_response_shape.py` | `docs/01_architecture/llm-orchestration.md` | +| FR-2 | AC-5 | Backward-compat read-path adapter story | `backend/tests/unit/domain/study/test_followups_backcompat.py` | — | +| FR-3 | AC-1, AC-4 | Domain models story | `backend/tests/unit/domain/study/test_followups.py` | — | +| FR-4 | AC-4 | Validator + downgrade story | `backend/tests/unit/workers/test_digest_followup_validation.py`, `backend/tests/integration/test_digest_followup_roundtrip.py` | — | +| FR-5 | AC-3, AC-7, AC-11, AC-12 | studies-column migration story + ORM update story + BEFORE DELETE trigger story | `backend/tests/integration/test_studies_parent_proposal_check.py`, `backend/tests/integration/test_studies_with_parent_followup.py`, `backend/tests/integration/test_studies_parent_proposal_on_delete.py` | `docs/01_architecture/data-model.md` | +| FR-6 | AC-6 | digests-column-type migration story | `backend/tests/integration/test_digest_followups_migration.py` | `docs/01_architecture/data-model.md` | +| FR-7 | AC-1, AC-5 | Schema wire-shape story | `backend/tests/contract/test_digest_response_shape.py`, `backend/tests/contract/test_proposal_detail_shape.py` | — | +| FR-8 | AC-1, AC-4 | LLM prompt update story | (covered via worker integration test) | `prompts/digest_narrative.system.md`, `prompts/digest_narrative.user.jinja` | +| FR-9 | AC-1, AC-10 | UI panel rewrite story | `ui/src/__tests__/components/proposals/suggested-followups-panel.test.tsx` | — | +| FR-10 | AC-2 | UI prefill flow story | `ui/tests/e2e/followup_run.spec.ts` | — | +| FR-11 | AC-3, AC-8, AC-8b, AC-9 | API `parent` body + validation story | `backend/tests/contract/test_create_study_parent.py`, `backend/tests/integration/test_studies_with_parent_followup.py` | `docs/01_architecture/api-conventions.md` | +| FR-12 | AC-10 | Dead-button removal story (bundled into FR-9 PR) | (snapshot/lint assertion in FR-9 test) | — | +| FR-13 | AC-1 (tooltip rendering observable in panel test) | Glossary additions story | `ui/src/__tests__/lib/glossary.test.ts` | — | + +## 18) Definition of feature done + +- [ ] All acceptance criteria (AC-1 through AC-12) pass in CI. +- [ ] All test layers (unit/integration/contract/e2e) are green. +- [ ] Documentation updates across docs/01–05 are merged (§15). +- [ ] Rollout gates from §16 are satisfied. +- [ ] Cross-model review (GPT-5.5) on this spec and the forthcoming implementation plan completed and adjudicated. +- [ ] `phase2_idea.md` and `phase3_idea.md` co-located with this spec (Tier B + Tier C deferred work tracking). +- [ ] No open questions remain in §19. + +## 19) Open questions and decision log + +### Open questions + +- **None.** Auto mode operated under §19 D-1 through D-11 below; all forks were decided with the most defensible default and documented. + +### Decision log + +- **D-1 — 2026-05-23 — `kind` discriminator values are `narrow | widen | text` (not `expand | shrink | hint` or other near-synonyms).** Rationale: "narrow" and "widen" are the operator's existing vocabulary from search-space tuning; "text" is the most neutral name for the free-form fallback. The idea proposed these literals and the spec preserves them verbatim. +- **D-2 — 2026-05-23 — Column-type change (option a) over dual-column dual-write (option b).** Rationale: idea locked option (a) at preflight time. MVP1 has no production data to preserve; single in-place migration with USING-clause backfill is cleaner than carrying a deprecated column indefinitely. +- **D-3 — 2026-05-23 — `studies_parent_proposal_pair_check` is a regular CHECK constraint, not a `DEFERRABLE` one.** Rationale: simpler. **Superseded by D-13** (cycle-1 F1) — the hard-delete edge case is now handled by an explicit `BEFORE DELETE ON proposals` trigger, NOT by deferring the CHECK or using `ON DELETE SET NULL`. The CHECK constraint can remain a regular (non-deferrable) one because the trigger fires first and atomically resolves the lineage pair. +- **D-4 — 2026-05-23 — New API endpoint NOT added.** Extend `POST /api/v1/studies` with optional `parent` body field. Rationale: avoids duplicating the existing validation chain (search_space, FK, judgment overlap probe); the operator's mental model is already "create study from a pre-filled form." +- **D-5 — 2026-05-23 — `parent_proposal_id` + `parent_proposal_followup_index` are NOT exposed on `StudyDetail` in MVP1.** Rationale: minimum viable surface. Lineage is queryable in the DB and visible via the upcoming `auto-followup-chain-panel` evolution (a future concern). Exposing on `StudyDetail` is a small additive change that can land in a follow-up PR if a UI surface needs it. +- **D-6 — 2026-05-23 — The default study `name` is `" — followup # ()"`.** Rationale: discoverable + descriptive. The operator can override. +- **D-7 — 2026-05-23 — The drift followup synthesized by the worker becomes a `kind: "text"` item (not `kind: "narrow"`).** Rationale: drift is operator judgment ("re-add the dropped params or treat as stale"), not a deterministic search-space narrowing. Preserves the existing semantics from `backend/workers/digest.py:762-767`. +- **D-8 — 2026-05-23 — Capability-degraded path persists `suggested_followups=[]` (preserved from today).** Rationale: when the LLM endpoint can't do structured output, we don't have a kind discriminator to assign. Returning empty is consistent with the existing AC-11 path in `feat_digest_proposal`. +- **D-9 — 2026-05-23 — Validation downgrade preserves the rationale prefix `[validation failed: ]`.** Rationale: makes the failure visible to the operator without aborting the digest. The 200-char truncation cap keeps the rationale display-friendly. +- **D-10 — 2026-05-23 — `parent.followup_index` upper bound IS validated at submit time against the live digest array length.** Rationale: catches stale-pointer cases (digest regenerated between page load and submit). The lower bound (`>= 0`) is Pydantic-enforced via `Field(ge=0)`. +- **D-11 — 2026-05-23 — `parent.proposal_id` does NOT need to reference a proposal whose `status="pending"`.** Rationale: operators may chain followups off rejected proposals (intent: "this idea was rejected, but maybe a narrower variant works"). The API does not enforce a status check. +- **D-12 — 2026-05-23 — No dropdown ever exposes `FollowupItem.kind` to the operator as a selectable value.** Rationale: the LLM is the producer; the operator is the consumer. There is no UI surface that asks the operator to pick a kind, so no §8.4 frontend-dropdown drift risk. +- **D-13 — 2026-05-23 (cycle-1 F1 accept) — `BEFORE DELETE` trigger replaces `ON DELETE SET NULL`.** Rationale: `ON DELETE SET NULL` would clear only `parent_proposal_id`, immediately violating the `studies_parent_proposal_pair_check` invariant. The trigger atomically NULLs both columns. Trigger lives in the same migration as the columns; downgrade drops it. +- **D-14 — 2026-05-23 (cycle-1 F2 accept) — JSONB type-change migration uses `COALESCE(... , '[]'::jsonb)` and DROP DEFAULT / SET DEFAULT bracketing.** Rationale: naive `jsonb_agg(... FROM unnest(empty_array))` returns NULL, violating NOT NULL. COALESCE ensures empty arrays map to `'[]'::jsonb`. DROP DEFAULT before TYPE change is required because the old `ARRAY[]::TEXT[]` default's type can't be auto-cast to JSONB. +- **D-15 — 2026-05-23 (cycle-1 F3 accept) — Pydantic validation uses `TypeAdapter` over the discriminated-union alias, not `.model_validate()`.** Rationale: `FollowupItem = Annotated[Union[...], Field(discriminator="kind")]` is a type alias, not a `BaseModel`. The Pydantic v2 idiom for validating discriminated-union aliases is `TypeAdapter(FollowupItem).validate_python(...)`. +- **D-16 — 2026-05-23 (cycle-1 F4 accept) — Pair CHECK also enforces `parent_proposal_followup_index >= 0`.** Rationale: cheap defensive add. API enforces `Field(ge=0)`; the DB CHECK catches direct writes that bypass the API (migrations, manual fixes, tests). +- **D-17 — 2026-05-23 (cycle-1 F5 reject) — Response example keeps the literal `generated_by: "openai:gpt-4o-2024-08-06"`.** Counter-evidence: CLAUDE.md Absolute Rule #8 explicitly mandates "All persisted artifacts (judgments, digests) capture the exact model identifier (`openai:gpt-4o-2024-08-06`) for lineage." `generated_by` is **persisted lineage data**, not a hardcoded model name in service code. The Rule #8 forbidden pattern is `client.chat.completions.create(model="gpt-4o", ...)` in service code — that does not occur in this spec; all service-side model references go through `settings.openai_model` (already established by `feat_digest_proposal`). The response example shows what an operator would actually see on the wire, and using a real model identifier (matching the Rule #8 canonical example) is more useful than a placeholder. +- **D-18 — 2026-05-23 (cycle-1 F6 accept) — UI fetches `GET /api/v1/studies/{parent_study_id}` lazily on "Run this followup" click rather than extending `_StudySummary` embed.** Rationale: the existing `_StudySummary` only has 7 fields and is missing `cluster_id`, `target`, `template_id`, `objective`, `config` — required for prefill. The lazy fetch leverages the existing endpoint + cache (TanStack Query), avoids bloating every proposal-detail response with parent-study data that's only needed on click, and keeps the change surface minimal (no new schema changes on the proposal endpoint). +- **D-19 — 2026-05-23 (cycle-1 F7 accept) — `CreateStudyModal` must `form.reset(initialValues)` on `[open, initialValues]` change.** Rationale: React Hook Form's `defaultValues` apply only on initial mount; without explicit `reset`, the second followup click in one page session would render stale values from the first. A keyed `useEffect` is the canonical fix. +- **D-20 — 2026-05-23 (cycle-1 F8 accept) — `parse_followup_list()` is contract-specified for every malformed-input shape in FR-4.** Rationale: the digest worker must never abort on a malformed followup; the parser must handle missing kind, unknown kind, non-dict elements, non-list top level, etc., with a defined behavior for each. The decision table in FR-4 is the contract. +- **D-21 — 2026-05-23 (cycle-2 F1 accept) — Migration uses PL/pgSQL helper functions, NOT inline subquery USING expressions.** Rationale: PostgreSQL rejects subqueries in `ALTER COLUMN TYPE ... USING` expressions with `ERROR: cannot use subquery in transform expression` (verified empirically against the project's local Postgres 16). The fix is migration-local helper functions (`_fn_wrap_text_array_as_jsonb_followups` on upgrade, `_fn_unwrap_jsonb_followups_as_text_array` on downgrade) that are created → used in the USING clause → dropped within the same migration body. The functions handle the empty-array case via early-return `IF arr IS NULL OR ... THEN RETURN '[]'::jsonb`, preserving the cycle-1 F2 fix. +- **D-22 — 2026-05-23 (cycle-2 F2 accept) — All `ON DELETE SET NULL` references in the spec narrative are replaced with the trigger contract.** Rationale: cycle-1 patches updated FR-5 + §9 + AC-11 + D-13 but missed §3, §9 invariants, §10 (data retention), §11 (edge flows), and D-3. Stale references would let implementers reintroduce the bug. All call sites now uniformly point at `trg_clear_studies_parent_proposal_on_proposal_delete`. +- **D-23 — 2026-05-23 (cycle-2 F3 accept) — `parse_followup_list()` signature carries `study_id` + `proposal_id` context kwargs.** Rationale: FR-4 mandates structured WARN/ERROR logs with both IDs; without context kwargs the helper has no way to populate them. Both default to `None` so read-path adapter calls (where IDs may not be loaded) still work, with the field appearing as `study_id=null` in the event payload. +- **D-24 — 2026-05-23 (cycle-2 F4 accept) — `serialize_followup_list(items)` helper required before assigning to JSONB column.** Rationale: SQLAlchemy's JSONB driver does not know how to serialize Pydantic `BaseModel` instances. The helper calls `item.model_dump(mode="json")` per item, flattening nested `SearchSpace` models to plain JSON dicts that the driver can persist. Without this step, the INSERT either crashes or stores a malformed shape. +- **D-25 — 2026-05-23 (cycle-2 F5 accept) — `parse_followup_list()` is called at EVERY response-construction site, not just the digest endpoint.** Rationale: `GET /api/v1/proposals/{id}` constructs `_DigestEmbed.suggested_followups` from the raw ORM value (`backend/app/api/v1/proposals.py:161`). Without an explicit wrapper call, defensive legacy `list[str]` rows would crash Pydantic response validation. The wrapper at every site is the single canonical guard. +- **D-26 — 2026-05-23 (cycle-2 F6 accept) — Explicit AC + integration test for `DIGEST_NOT_FOUND`.** Rationale: FR-11 introduces a distinct 404 path (proposal exists but no digest yet) that's behaviorally different from `PROPOSAL_NOT_FOUND` (`retryable=true` vs `false`). Without a dedicated AC + test, the path could regress silently. +- **D-27 — 2026-05-23 (cycle-3 F1 accept) — Capability-degraded mode suppresses BOTH LLM followups AND the drift synthesis.** Rationale: current code (`backend/workers/digest.py:757-758`) keeps the drift prepend inside the `if structured_output_enabled:` branch — `suggested_followups` stays `[]` in degraded mode regardless of `dropped_template_params`. Spec FR-1 + FR-8 now explicitly align with this code behavior (was previously ambiguous between FR-1 saying "[]" and FR-8 saying "drift preserved"). +- **D-28 — 2026-05-23 (cycle-3 F2 accept) — `parse_followup_list()` decision table now covers `narrow`/`widen` items that fail validation for non-`SearchSpace` reasons.** Rationale: the cycle-1 table only specified the `SearchSpace`-cardinality-fails path. Items failing for missing rationale / missing search_space / extra forbidden fields could still raise `ValidationError` past the parser. Updated table mandates catching ALL `FollowupItemAdapter.validate_python` `ValidationError` paths and routing them through downgrade-or-drop. +- **D-29 — 2026-05-23 (cycle-3 F3 accept) — `POST /api/v1/studies` validates `parent.followup_index` against the PARSED canonical list, not raw JSONB length.** Rationale: if a defensive malformed JSONB row is wrapped by `parse_followup_list()`, dropped elements shift indices between the response (parsed) and the raw column. Validation must use the same parsed list the UI saw to guarantee lineage integrity. diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/idea.md b/docs/02_product/planned_features/feat_digest_executable_followups/idea.md index c9f6268d..e41ba05c 100644 --- a/docs/02_product/planned_features/feat_digest_executable_followups/idea.md +++ b/docs/02_product/planned_features/feat_digest_executable_followups/idea.md @@ -4,11 +4,11 @@ **Status:** Idea — surfaced during the 2026-05-21 Karpathy-loop audit. **Priority:** P2 — turns LLM-suggested followups from dead text into actionable proposals. Medium-scope feature; needs an LLM-output schema change + new UI affordance. High value once it lands but no immediate pain pushing it now. **Origin:** Standalone audit at `~/.claude/plans/compressed-sparking-hamming.md` — recommendation #4. The audit observation: the digest worker's LLM output already includes `suggested_followups`, but the field is shaped as plain strings, rendered in the proposals UI as bullet text, and has no path back into `create_study` or `propose_search_space`. Operators read suggestions like "narrow the title_boost range to [0.5, 3.0]" and have to manually translate them into a new study configuration. -**Depends on:** None. Composes with [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md) (which automates the deterministic followup); this feature handles the **LLM-suggested** followups separately. +**Depends on:** None block this. Substrate now shipped: [`feat_auto_followup_studies`](../../../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/idea.md) (merged 2026-05-24 as PR #223 squash `20cf183a`) handles the **deterministic** auto-chain followup; this feature handles the orthogonal **LLM-suggested** followups so the two compounding paths coexist. ## Problem -The digest worker's LLM contract at [`backend/workers/digest.py:168-189`](../../../../backend/workers/digest.py) defines `suggested_followups` as a flat `array of string`: +The digest worker's LLM contract at [`backend/workers/digest.py:169-182`](../../../../backend/workers/digest.py#L169-L182) defines `suggested_followups` as a flat `array of string`: ```python DIGEST_RESPONSE_SCHEMA = { @@ -31,7 +31,11 @@ The strings are LLM-generated freeform — typical examples (from real digest ou - "Investigate the `tie_breaker` parameter — its importance was 0.18 but the search space only sampled three values." - "Add a `category_boost` parameter to the template since several winning trials suggest category prioritization matters." -These suggestions are useful — but **operationally inert**. They render as bullet text on the proposal detail page at [`ui/src/app/proposals/[id]/page.tsx`](../../../../ui/src/app/proposals/%5Bid%5D/page.tsx). The operator must: +These suggestions are useful — but **operationally inert**. They render as bullet text via [`SuggestedFollowupsPanel`](../../../../ui/src/components/proposals/suggested-followups-panel.tsx) on the proposal detail page at [`ui/src/app/proposals/[id]/page.tsx`](../../../../ui/src/app/proposals/%5Bid%5D/page.tsx). + +**Existing half-built affordance (worth subsuming).** Each bullet today renders with a "Create study from this hypothesis" button that links to `/studies?hypothesis=`. But the `/studies` landing page **never reads the `hypothesis` query param** (verified 2026-05-24 — `grep "hypothesis\|searchParams" ui/src/app/studies/page.tsx` returns zero matches). The button looks like it works but silently drops its payload. This feature should retire the dead `?hypothesis=` path and replace it with the structured one-click flow described below. (If we ship the new flow without removing the old button, operators get two buttons with different semantics — confusing.) + +The operator's actual workflow today: 1. Read each suggestion. 2. Translate it into a `search_space` JSON manually. @@ -42,7 +46,7 @@ These suggestions are useful — but **operationally inert**. They render as bul That's the 6-step manual workflow every overnight digest produces *up to 5 times*. The Karpathy-loop equivalent is one click: "Run this followup." The data the LLM has at digest time is rich enough to populate a `CreateStudyRequest` deterministically — the only missing piece is the JSON structure to carry it. -The audit's framing: this is the **smaller** of the two compounding gaps. [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md) handles the *deterministic* compounding (Optuna's TPE narrowed around a winner). This feature handles the *LLM-suggested* compounding (the model spotting patterns Optuna can't, like "the importance signal suggests adding a new dimension"). Both are needed; they cover orthogonal failure modes. +The audit's framing: this is the **smaller** of the two compounding gaps. [`feat_auto_followup_studies`](../../../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/idea.md) (now shipped) handles the *deterministic* compounding (Optuna's TPE narrowed around a winner). This feature handles the *LLM-suggested* compounding (the model spotting patterns Optuna can't, like "the importance signal suggests adding a new dimension"). Both are needed; they cover orthogonal failure modes. ## Proposed capabilities @@ -50,7 +54,7 @@ Tiered. Tier A reshapes the LLM output and adds the "Run this followup" button f ### Tier A — structured followups for "narrow / widen" within the same template -- **New LLM output schema** in [`backend/workers/digest.py:168-189`](../../../../backend/workers/digest.py): +- **New LLM output schema** in [`backend/workers/digest.py:169-182`](../../../../backend/workers/digest.py#L169-L182): ```python { "narrative": str, @@ -72,7 +76,10 @@ Tiered. Tier A reshapes the LLM output and adds the "Run this followup" button f - "Narrow"/"Widen" followups render as a card with: rationale text, a collapsed "Show search space" detail (renders the diff vs parent study), and a primary "Run this followup" button. - "Run this followup" pre-fills the create-study modal with: parent study's cluster/target/template/query_set/judgment_list/objective + the LLM's proposed `search_space` + parent's stop conditions. Operator reviews + submits. - "Text" followups render unchanged — bullet text. The kind discriminator means freeform suggestions stay supported indefinitely. -- **Traceability.** New nullable column `studies.parent_proposal_followup_index: int | None` records "this study was created from followup #N of proposal X." Lets the UI render "Study A.2 was suggested by digest from proposal B at index 3." Helps the team measure whether LLM-suggested followups produce wins. +- **Traceability.** Two new nullable columns on `studies` — the index alone is unmoored without the proposal ID: + - `parent_proposal_id: str | None` — FK to `proposals(id) ON DELETE SET NULL`, partial B-tree index on `WHERE parent_proposal_id IS NOT NULL`. + - `parent_proposal_followup_index: int | None` — 0-based position in the parent proposal's structured `suggested_followups` array. + - Lets the UI render "Study A.2 was suggested by digest from proposal B at index 3." Helps the team measure whether LLM-suggested followups produce wins. Composes cleanly with the `parent_study_id` self-FK that `feat_auto_followup_studies` already uses for the deterministic chain — the two columns are orthogonal (a study can have both: a parent study via the auto-chain AND a parent proposal via the LLM-suggested click). ### Tier B — `swap_template` followups @@ -88,15 +95,17 @@ Tiered. Tier A reshapes the LLM output and adds the "Run this followup" button f ### Out of scope for Tier A/B -- **Auto-running followups without operator click.** That's [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md). This feature stays in the "human clicks one button" lane — the LLM proposes; the operator commits. +- **Auto-running followups without operator click.** That's [`feat_auto_followup_studies`](../../../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/idea.md) (shipped). This feature stays in the "human clicks one button" lane — the LLM proposes; the operator commits. - **Followups that span multiple studies.** A meta-followup like "run studies A.1 and A.2 in parallel with different starting points" would need its own surface. Not now. - **Persistence of "I tried this LLM followup and it didn't help."** A negative-result feedback loop into future LLM prompts (so the model learns the operator's preferences) is an interesting MVP4 idea, gated on Langfuse. Out of scope now. ## Scope signals -- **Backend:** ~400 LOC. New Pydantic models for the followup discriminated union (~30) + LLM prompt updates (~60 in `.system.md` + `.user.jinja`) + validator at digest-persist time (~40) + new `parent_proposal_followup_index` column + migration (~30) + service-layer "create study from followup" helper (~80) + tests across unit/integration/contract (~150). -- **Frontend:** ~400 LOC. Followup card component with kind discriminator (~150) + "Run this followup" prefill workflow (~80) + search-space diff renderer (~100) + tests (~70). Tier B adds ~200 LOC for the swap-template comparison. -- **Migration:** one Alembic migration adding `studies.parent_proposal_followup_index INT NULL`. Strictly additive. Round-trip-clean. +- **Backend:** ~450 LOC. New Pydantic models for the followup discriminated union (~30) + LLM prompt updates (~60 in `.system.md` + `.user.jinja`) + validator at digest-persist time (~40) + read-path adapter that wraps old `list[str]` rows as `[{kind: "text", rationale: , search_space: null}]` so the UI handles both shapes uniformly (~30) + new `studies.parent_proposal_id` FK + `studies.parent_proposal_followup_index` column + migration (~50) + service-layer "create study from followup" helper (~80) + tests across unit/integration/contract (~150). +- **Frontend:** ~450 LOC. Followup card component with kind discriminator (~150) + "Run this followup" prefill workflow (~80) + search-space diff renderer (~100) + retire the dead `?hypothesis=` path in `suggested-followups-panel.tsx` (~20) + tests (~100). Tier B adds ~200 LOC for the swap-template comparison. +- **Migration story (two migrations, NOT strictly additive):** + 1. **`studies` column adds** — `parent_proposal_id VARCHAR(36) NULL` FK to `proposals(id) ON DELETE SET NULL` + partial B-tree index on `WHERE parent_proposal_id IS NOT NULL` + `parent_proposal_followup_index INT NULL`. Additive, round-trip-clean. + 2. **`digests.suggested_followups` column-type change** — current schema (verified 2026-05-24 at [`backend/app/db/models/digest.py:49`](../../../../backend/app/db/models/digest.py#L49)) declares the column as `ARRAY(Text)` with `server_default=sa.text("ARRAY[]::TEXT[]")`. Storing the new discriminated union requires either: (a) ALTER the column type to `JSONB` with USING-clause backfill that wraps each existing `text` value as `{kind: "text", rationale: , search_space: null}`, OR (b) add a new column `suggested_followups_structured JSONB` and dual-write during a deprecation window. Option (a) is simpler and cleaner — there's no production data to preserve in MVP1 (single-tenant on developer laptops). Lock to (a) at spec time. Round-trip: downgrade reverses the type change + flattens structured rows back to `array[rationale]` (lossy but safe — any non-text kinds are downgrade-only on dev DBs that haven't shipped). Idempotency-guard with `IF EXISTS` on the type-change clause. - **Config:** none. - **Audit events:** N/A (MVP1). At MVP2: `digest.followup_run_clicked` + `study.created_from_followup` as canonical audit events. - **Tests:** @@ -107,14 +116,14 @@ Tiered. Tier A reshapes the LLM output and adds the "Run this followup" button f ## Why not inline today -1. **LLM-contract change.** Reshaping `suggested_followups` from `string[]` to a discriminated union touches the response schema, the prompt files, the validator, the digest worker's parse logic, the storage representation in `digests.followups` JSONB, AND the UI renderer. Multiple coordinated surfaces — outside drive-by budget. -2. **Backward compatibility.** Existing digests in the DB have the old shape. The read path needs an adapter that wraps old strings into the new structure. Small but real — easy to get wrong as a drive-by. +1. **LLM-contract change.** Reshaping `suggested_followups` from `string[]` to a discriminated union touches the response schema, the prompt files, the validator, the digest worker's parse logic, the storage representation in `digests.suggested_followups` (which is currently `ARRAY(Text)`, not JSONB — requires a column-type migration), AND the UI renderer. Multiple coordinated surfaces — outside drive-by budget. +2. **Backward compatibility.** Existing digests in the DB have the old `list[str]` shape. The read path needs an adapter that wraps old strings into the new structure AND the column-type migration needs a USING-clause backfill that performs the same wrap server-side. Small but real — easy to get wrong as a drive-by. 3. **Real UX design surface.** The "Run this followup" workflow is a new top-level user action — how it renders, what it pre-fills, what it shows in the search-space diff are decisions worth scrutinizing in a spec. -4. **Composes with another planned feature.** [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md) and this idea cover orthogonal compounding paths (deterministic vs LLM-suggested). Shipping them in coordinated order — auto-followup first to establish the autonomy trust model, then this to add LLM-suggested manual overrides — gives reviewers a coherent story. Either could ship first, but the coordination is worth planning, not improvising. +4. **Composes with already-shipped substrate.** [`feat_auto_followup_studies`](../../../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/idea.md) merged 2026-05-24 (PR #223 squash `20cf183a`) — auto-chain shipped first per the operator's preferred sequence (autonomy trust model established before adding LLM-suggested manual overrides). This idea now layers on top of that substrate rather than coordinating with it. The two cover orthogonal compounding paths (deterministic Optuna-narrow vs LLM-suggested transformations). ## Relationship to other work -- **Most-leveraged in combination with [`feat_auto_followup_studies`](../feat_auto_followup_studies/idea.md)** — the auto-chain provides the deterministic "narrow around winner" path; this feature adds the LLM-suggested "but consider widening on this axis" path. Together they cover what Karpathy's per-experiment agent does (propose a single change per experiment, then evaluate) — Optuna's TPE handles the within-study sampling, and these two features handle the across-study hypothesis evolution. +- **Most-leveraged in combination with [`feat_auto_followup_studies`](../../../00_overview/implemented_features/2026_05_24_feat_auto_followup_studies/idea.md)** (shipped) — the auto-chain provides the deterministic "narrow around winner" path; this feature adds the LLM-suggested "but consider widening on this axis" path. Together they cover what Karpathy's per-experiment agent does (propose a single change per experiment, then evaluate) — Optuna's TPE handles the within-study sampling, and these two features handle the across-study hypothesis evolution. - **Adjacent to [`feat_pr_metric_confidence`](../feat_pr_metric_confidence/idea.md)** — the confidence framing (CI bands, named regressors) can feed into the followup prompts. "The winner is at +0.13 NDCG with a noise floor of σ=0.02 and 2 regressing queries" is much richer LLM context than "winner is 0.84" for proposing the next experiment. - **Reuses [`feat_agent_propose_search_space`](../../../00_overview/implemented_features/2026_05_21_feat_agent_propose_search_space/)** (shipped 2026-05-21) — the underlying `search_space_defaults.py` heuristic is the natural fallback when the LLM proposes a `swap_template` followup that has a partial `search_space` (the disjoint params get heuristic bounds). - **Composes with [`feat_create_study_search_space_builder`](../../../00_overview/implemented_features/2026_05_20_feat_create_study_search_space_builder/)** (shipped 2026-05-20) — the visual editor for `search_space` rows is where "Run this followup" lands, pre-populated. The diff visualization can leverage the same row primitive. diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/phase2_idea.md b/docs/02_product/planned_features/feat_digest_executable_followups/phase2_idea.md new file mode 100644 index 00000000..f44e4b78 --- /dev/null +++ b/docs/02_product/planned_features/feat_digest_executable_followups/phase2_idea.md @@ -0,0 +1,44 @@ +# Phase 2 — `swap_template` LLM-suggested followups + +**Date:** 2026-05-23 +**Status:** Idea — deferred from Phase 1 of `feat_digest_executable_followups` per `feature_spec.md` §3 Phase boundaries. +**Priority:** P2 — actionable extension once Tier A is in operator hands and the team has data on whether `narrow`/`widen` followups produce wins. +**Origin:** `feature_spec.md` §3 ("Out of scope" → Tier B) + sibling `idea.md` §"Tier B — `swap_template` followups". +**Depends on:** `feat_digest_executable_followups` Phase 1 (Tier A) shipped first. Builds on the `FollowupItem` discriminated-union + DB column substrate that Phase 1 lays down. + +## Problem + +Phase 1 handles `narrow` / `widen` / `text` kinds — all within the **same query template**. But the LLM sometimes recognizes that a different template entirely is a better fit: e.g., parameter-importance is highly skewed (some declared params are dead weight), OR several winning trials cluster around a sub-set of params that map cleanly onto a different template's declared params. Today the operator has to notice this themselves; the LLM has no structured way to say "try template X instead." + +## Proposed capabilities + +### Tier B — `swap_template` discriminated-union kind + +- **New FR:** Add `kind="swap_template"` to the `FollowupItem` discriminated union (alongside the Phase 1 `narrow`/`widen`/`text`). The variant carries `{kind: "swap_template", rationale: str, template_id: str (UUIDv7, 36 chars), search_space: SearchSpace}`. +- **New FR:** Cross-template search-space remapping. New domain helper at `backend/app/domain/study/template_swap.py` computes: + - Intersection: param names declared by both the parent template and the proposed swap-target template — copied from the LLM's `search_space` directly. + - Disjoint set: param names declared by the swap-target but not the parent — assigned default heuristic bounds via `backend/app/domain/study/search_space_defaults.py` (already exists from `feat_agent_propose_search_space`). + - Dropped: param names declared by the parent but not the swap-target — dropped silently with a structlog event. +- **New FR:** LLM prompt extension teaching the model when to suggest a swap (parameter-importance distribution skewed, OR winning trials cluster around a sub-set mappable onto a different template). +- **New FR:** UI surface — swap-template followups render with a side-by-side comparison of the two templates' `declared_params` before the operator commits. The "Run this followup" button pre-fills `template_id = ` instead of the parent's template. +- **New FR:** The `parent_proposal_id` + `parent_proposal_followup_index` lineage from Phase 1 still applies; the child study's `template_id` differs from the parent's, which the lineage data makes explicit. + +## Scope signals + +- **Backend:** ~250 LOC. `template_swap.py` domain helper (~100) + extended `FollowupItem` discriminated union (~30) + LLM prompt update (~80) + tests (~50). No new DB columns or migrations. +- **Frontend:** ~200 LOC. Side-by-side template comparison component + extended "Run this followup" prefill for the swap kind + tests. +- **Migration:** None — JSONB column already accommodates the new shape. +- **Config:** None. +- **Audit events:** Reuses the three pre-shaped MVP2 events from Phase 1; the `digest.followup_clicked` event's `followup_kind` field gains `swap_template` as a possible value. + +## Why deferred + +- **Cross-template search-space remapping is a non-trivial new domain helper.** Phase 1 reuses the existing `SearchSpace` validator unchanged; Phase 2 adds a new transformation layer. Out of Phase 1 scope. +- **UI surface for side-by-side template comparison is its own design decision.** Phase 1's diff-vs-parent renderer doesn't compose cleanly into a two-template comparison; new component required. +- **Value depends on Phase 1 success.** Until we ship Tier A and operators actually use "Run this followup," investing in the cross-template variant is speculative. + +## Relationship to other work + +- **Depends on Phase 1 substrate** — the discriminated-union schema, JSONB column, lineage columns, and "Run this followup" UI scaffolding all land in Phase 1. +- **Reuses `backend/app/domain/study/search_space_defaults.py`** from `feat_agent_propose_search_space` (shipped 2026-05-21) for the disjoint-set heuristic bounds. +- **Reuses `feat_create_study_search_space_builder` row primitives** for the cross-template comparison (when feasible). diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/phase3_idea.md b/docs/02_product/planned_features/feat_digest_executable_followups/phase3_idea.md new file mode 100644 index 00000000..04a26fee --- /dev/null +++ b/docs/02_product/planned_features/feat_digest_executable_followups/phase3_idea.md @@ -0,0 +1,43 @@ +# Phase 3 — `edit_template` LLM-suggested followups (stretch) + +**Date:** 2026-05-23 +**Status:** Idea — deferred from Phase 1 of `feat_digest_executable_followups` per `feature_spec.md` §3 Phase boundaries. +**Priority:** Backlog — likely out of MVP1 scope entirely; captured for record. +**Origin:** `feature_spec.md` §3 ("Out of scope" → Tier C) + sibling `idea.md` §"Tier C — template-edit suggestions". +**Depends on:** `feat_digest_executable_followups` Phase 1 (substrate) + likely Phase 2 (`swap_template` proves out the cross-template UX before edit-template stretches further). Also depends on a yet-to-be-designed template-edit review surface. + +## Problem + +Phase 1 ships `narrow`/`widen`/`text` and Phase 2 adds `swap_template` — all of which preserve template authoring as a strictly operator-driven activity. But the LLM can sometimes spot template-body improvements that no parameter tuning can reach: e.g., "add a `category^2` field-boost to the template body — your winning trials all came from categories with multiple matching terms." + +Today templates are operator-authored only. Letting LLM suggestions flow into template-body edits requires a much larger trust-and-validation surface: changes to template body alter query rendering semantics, which is materially riskier than search-space narrowing. + +## Proposed capabilities + +### Tier C — `edit_template` discriminated-union kind + +- **New FR:** Add `kind="edit_template"` to the `FollowupItem` discriminated union. Carries `{kind: "edit_template", rationale: str, template_id: str, body_patch: dict}` where `body_patch` is a JSON Patch (RFC 6902) or equivalent applied to the parent template's `body_jsonata` (or whichever template-body field is canonical at the time). +- **New FR:** Template-edit validation pipeline. The patched body MUST parse cleanly against the engine adapter's rendering layer (i.e., `ElasticAdapter.render(patched_body, sample_params)` succeeds) before the followup is persisted. +- **New FR:** Operator-mediated apply surface. NOT a "Run this followup" one-click — instead, "Open in template editor" that loads the patched body into the (yet-to-exist) template-editing UI for explicit review. +- **New FR:** Lineage capture on the template version (new). When the operator commits the edit, the new template version records `parent_template_id` + `derived_from_proposal_id` + `derived_from_followup_index`. + +## Scope signals + +- **Backend:** ~400+ LOC. New domain helper for body-patch application + validation + template-version lineage columns + migration + tests. +- **Frontend:** Large (~500+ LOC). Requires a template-editing UI surface that does NOT exist today (templates are file-based today; the in-tool editor is unbuilt). +- **Migration:** Yes — template-versions lineage columns. +- **Config:** None. +- **Audit events:** New event types `digest.template_edit_suggested` + `template_version.derived_from_followup`. + +## Why deferred (likely beyond MVP1) + +- **Template-body edits change query rendering semantics.** Materially different risk profile from search-space narrowing. Needs a real review surface, real validation, and operator trust that the LLM's edits don't silently break production templates. +- **No template-editing UI today.** Templates are file/JSON in the repo, edited in IDEs. Building an in-tool template editor is its own spec — likely MVP2 or later. +- **The Phase 1 + Phase 2 pair already covers the common case** of LLM-suggested followups. Edit-template is the long tail. +- **Trust model.** Operators haven't yet decided whether they trust LLM suggestions to modify the very thing that defines query behavior. Phase 1's success metrics (do operators actually click "Run this followup"? Do those studies win?) inform whether Phase 3 is worth the investment. + +## Relationship to other work + +- **Depends on Phase 1 + Phase 2 substrate.** +- **Depends on a template-editing UI surface** that does NOT yet exist — likely a separate feature (`feat_template_editor` or similar) before this can land. +- **Adjacent to `feat_agent_propose_search_space`** — same agent flow, different target (template body vs search space). diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/pipeline_status.md b/docs/02_product/planned_features/feat_digest_executable_followups/pipeline_status.md new file mode 100644 index 00000000..acf4240c --- /dev/null +++ b/docs/02_product/planned_features/feat_digest_executable_followups/pipeline_status.md @@ -0,0 +1,22 @@ +# Pipeline Status — Executable Digest Follow-ups + +## Idea +- Status: Complete +- File: idea.md + +## Spec +- Status: Approved (Auto-mode — pending user redirect via pipeline gates) +- Date: 2026-05-23 +- File: feature_spec.md +- Cross-model review: GPT-5.5 passed (3 cycles — convergence reached at max-cycle stop rule with all findings accepted) + - Cycle 1: 9 findings (8 accepted, 1 rejected with cited counter-evidence) + - Cycle 2: 6 findings (6 accepted — included regression patches and one re-raise of the migration surface with new information) + - Cycle 3: 3 findings (3 accepted — internal-consistency clarifications) + - Total: 17 accepted, 1 rejected +- Phases: 3 total (Phase 1 covered by spec; Phase 2 + Phase 3 tracked in `phase2_idea.md` + `phase3_idea.md`) + +## Plan +- Status: Not started + +## Implementation +- Status: Not started From a380e871b89730b8ce02c2ece488768522d01555 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Sat, 23 May 2026 21:09:56 -0400 Subject: [PATCH 02/15] docs(digest-executable-followups): implementation plan (1 GPT-5.5 cycle to convergence) 16 stories across 6 epics: - Epic 1 Domain (1): followups.py + TypeAdapter + parser + serializer - Epic 2 Worker + prompts (3): schema wiring, prompt updates, integration test - Epic 3 Migrations + ORM (6): migration 0018 (studies columns + BEFORE DELETE trigger + partial index) + ORM update + migration 0019 (ARRAY(Text) -> JSONB column-type change via PL/pgSQL helpers) + 3 integration tests - Epic 4 API (2): schema wire-shape + parent body endpoint - Epic 5 Frontend (3): panel rewrite + prefill flow + glossary - Epic 6 E2E (1): Playwright happy-path Test coverage (15 files total): - Unit (3): test_followups.py, test_followups_backcompat.py, test_digest_prompt.py - Integration (5): digest roundtrip, parent_proposal CHECK + ON DELETE, migration 0019, studies with parent_followup - Contract (3): digest response shape, proposal detail shape, create_study parent - E2E (1): followup_run.spec.ts - Vitest (3): panel, modal-prefill, glossary extension Cross-model review: 5 findings -- 3 accepted (F1 explicit downgrade sequence, F2 useStudy enabled pre-Run-click, F3 RequestValidationError mapping + 3 contract tests), 2 rejected with cited counter-evidence (F4 MVP4 forward-looking convention, F5 D-17 lineage re-raise). Legacy Behavior Parity table for the dead ?hypothesis= retire: 6 rows (4 preserved, 2 intentionally-dropped with FR-12 citations). Co-Authored-By: Claude Opus 4.7 --- docs/00_overview/MVP1_DASHBOARD.md | 18 +- docs/00_overview/mvp1_dashboard.html | 22 +- .../implementation_plan.md | 1432 +++++++++++++++++ .../pipeline_status.md | 8 +- 4 files changed, 1459 insertions(+), 21 deletions(-) create mode 100644 docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index b02d4905..7dcb4cc2 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -6,14 +6,14 @@ _Reflects feature-folder state as of **2026-05-24** (latest mtime of any planned ## Next up -**[feat_digest_executable_followups](../02_product/planned_features/feat_digest_executable_followups/feature_spec.md)** — Feature, currently in **Spec** +**[feat_digest_executable_followups](../02_product/planned_features/feat_digest_executable_followups/feature_spec.md)** — Feature, currently in **Plan** > The LLM emits a discriminated union (`narrow` \| `widen` \| `text`) for each followup with a structured `search_space` when applicable. -Spec exists; run /pipeline to generate the implementation plan + ship +Plan approved; run /impl-execute to ship ```bash -/pipeline docs/02_product/planned_features/feat_digest_executable_followups --auto +/impl-execute docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md --all ``` ## MVP1 Progress @@ -131,16 +131,16 @@ Spec exists; run /pipeline to generate the implementation plan + ship _None._ -### Plan (0) - -_None._ - -### Spec (1) +### Plan (1) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| | 1 | P2 | [feat_digest_executable_followups](../02_product/planned_features/feat_digest_executable_followups/feature_spec.md) | Feature | The LLM emits a discriminated union (`narrow` \| `widen` \| `text`) for each followup with a structured `search_space` when applicable. | — | [PR #223](https://github.com/SoundMindsAI/relyloop/pull/223) merged 2026-05-24 | +### Spec (0) + +_None._ + ### Idea (12) | # | Priority | Feature | Type | One-liner | Depends on | Status | @@ -170,7 +170,7 @@ graph LR classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; feat_digest_executable_followups["digest executable followups"] - class feat_digest_executable_followups spec; + class feat_digest_executable_followups plan; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index 15424fef..9ee0cd6b 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -383,11 +383,11 @@

RelyLoop MVP1 Dashboard

-
Next up — Feature, currently in Spec
+
Next up — Feature, currently in Plan
Digest Executable Followups
The LLM emits a discriminated union (`narrow` | `widen` | `text`) for each followup with a structured `search_space` when applicable.
-
Spec exists; run /pipeline to generate the implementation plan + ship
- /pipeline docs/02_product/planned_features/feat_digest_executable_followups --auto +
Plan approved; run /impl-execute to ship
+ /impl-execute docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md --all
@@ -623,7 +623,12 @@

Idea 12

-

Spec 1

+

Spec 0

+ +
+ +
+

Plan 1

Digest Executable Followups
@@ -639,11 +644,6 @@

Spec 1

-
-

Plan 0

- -
-

Implementing 0

@@ -1822,7 +1822,7 @@

Dependency graph (feat_ + infra_)

classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; feat_digest_executable_followups["digest executable followups"] - class feat_digest_executable_followups spec; + class feat_digest_executable_followups plan; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] @@ -2021,7 +2021,7 @@

Dependency graph (feat_ + infra_)

classDef spec fill:#dbeafe,stroke:#1e40af,color:#1e40af; classDef idea fill:#f1f5f9,stroke:#334155,color:#334155; feat_digest_executable_followups["digest executable followups"] - class feat_digest_executable_followups spec; + class feat_digest_executable_followups plan; infra_foundation["foundation"] class infra_foundation done; feat_study_lifecycle["study lifecycle"] diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md b/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md new file mode 100644 index 00000000..5728aa6c --- /dev/null +++ b/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md @@ -0,0 +1,1432 @@ +# Implementation Plan — Executable Digest Follow-ups + +**Date:** 2026-05-23 +**Status:** Ready for Execution (GPT-5.5 cross-model review: 1 cycle, 3 accepted / 2 rejected — convergence) +**Primary spec:** [`feature_spec.md`](./feature_spec.md) +**Policy source(s):** +- [`CLAUDE.md`](../../../../CLAUDE.md) (Absolute Rules #1, #5, #8, #10) +- [`docs/01_architecture/data-model.md`](../../../01_architecture/data-model.md) +- [`docs/01_architecture/api-conventions.md`](../../../01_architecture/api-conventions.md) +- [`docs/01_architecture/llm-orchestration.md`](../../../01_architecture/llm-orchestration.md) +- Sibling implementations: `feat_auto_followup_studies` (parent-study lineage), `feat_digest_proposal` (digest worker), `feat_create_study_search_space_builder` (search-space row primitive), `chore_study_default_stop_conditions` (form reset patterns) + +--- + +## 0) Planning principles + +- Spec traceability first: every story/task maps to FR IDs. +- Phase gates are hard stops. +- Fail-loud tests: assert explicit status/shape/errors. +- Keep repository patterns consistent with existing digest / studies modules. +- Keep increments narrow enough to verify independently — backend domain → migration → worker → API contract → frontend → E2E. +- **Phase scope:** Phase 1 only (Tier A — `narrow` / `widen` / `text` kinds). Phase 2 (`swap_template`) tracked in [`phase2_idea.md`](./phase2_idea.md); Phase 3 (`edit_template`) tracked in [`phase3_idea.md`](./phase3_idea.md). Both deferred trackers verified present. + +## 1) Scope traceability (FR → epics/phases) + +| FR ID | Epic / Story | Notes | +|---|---|---| +| FR-1 | Epic 2 / Story 2.1 (worker) | LLM JSON-schema change + capability-degraded suppression of drift item. | +| FR-2 | Epic 1 / Story 1.1 (domain) | Backward-compat read-path adapter (legacy `list[str]` wrap to `text` items). | +| FR-3 | Epic 1 / Story 1.1 (domain) | `FollowupItem` + `TypeAdapter` + `serialize_followup_list()` helpers. | +| FR-4 | Epic 1 / Story 1.1 (domain) + Epic 2 / Story 2.1 (worker) | `parse_followup_list()` decision table; downgrade-or-drop with structlog WARN. | +| FR-5 | Epic 3 / Story 3.1 (migration) + Story 3.2 (ORM) | Add `parent_proposal_id` + `parent_proposal_followup_index` + CHECK + partial index + BEFORE DELETE trigger. | +| FR-6 | Epic 3 / Story 3.3 (migration) | JSONB column-type change via PL/pgSQL helper functions. | +| FR-7 | Epic 4 / Story 4.1 (API schemas) | `DigestResponse.suggested_followups` + `_DigestEmbed.suggested_followups` → `list[FollowupItem]`; wrap via `parse_followup_list()` at BOTH response-construction sites. | +| FR-8 | Epic 2 / Story 2.2 (prompts) | `prompts/digest_narrative.system.md` + `.user.jinja` updates teaching three kinds + parent search-space block. | +| FR-9 | Epic 5 / Story 5.1 (UI panel) | `SuggestedFollowupsPanel` rewrite as kind-discriminated cards. | +| FR-10 | Epic 5 / Story 5.2 (UI prefill flow) | "Run this followup" lazy-fetches parent study + opens `CreateStudyModal` with `initialValues`. | +| FR-11 | Epic 4 / Story 4.2 (API endpoint) | `POST /api/v1/studies` accepts optional `parent` body field with three new error codes. | +| FR-12 | Epic 5 / Story 5.1 (UI panel) | Dead `?hypothesis=` button removed in same story as panel rewrite. | +| FR-13 | Epic 5 / Story 5.3 (glossary) | Five new glossary keys for tooltips. | + +**Spec endpoint count vs plan:** Spec §8.1 lists 3 modified endpoints. Plan covers all 3: +- `POST /api/v1/studies` — Story 4.2 +- `GET /api/v1/studies/{id}/digest` — Story 4.1 (schema change only; existing handler at `backend/app/api/v1/studies.py` digest sub-route untouched apart from wrapper insertion) +- `GET /api/v1/proposals/{id}` — Story 4.1 (`_DigestEmbed` construction wrap) + +**Spec error-code coverage vs plan:** Spec §8.5 introduces 3 codes; all covered by Story 4.2 contract tests (`PROPOSAL_NOT_FOUND` / `DIGEST_NOT_FOUND` / `FOLLOWUP_INDEX_OUT_OF_RANGE`). + +**Deferred phases verified tracked:** +- `phase2_idea.md` present (Tier B `swap_template`) — `ls` confirmed 2026-05-23. +- `phase3_idea.md` present (Tier C `edit_template`) — `ls` confirmed 2026-05-23. + +## 2) Delivery structure + +**Hybrid: Epic → Story → Tasks → DoD** (preferred for product-facing work), with Phase 3 (migrations) using **Phase → Checkpoint gate** because migration round-trip is a hard gate independent of story outcomes. + +### Story-level conventions for this plan + +- **Backend conventions:** repo functions take `db: AsyncSession` first arg, call `db.flush()` (caller commits). Services are `async`. Domain layer is pure (no DB, no async, no I/O). Models use `Mapped[]` typed columns + `String(36)` UUIDv7. Routers raise via the local `_err()` helper (per `backend/app/api/v1/studies.py:75-79`). Settings via `pydantic-settings`; never instantiate `Settings()` directly. +- **Frontend conventions:** Next.js 16 App Router, `'use client'` at top of interactive pages/components, `useStudy(id)` via TanStack Query, shadcn `` / `
Pending work
-
13
+
14
every not-done feat/infra/chore/bug across all priorities
-
+
Open bugs
-
0
+
1
tracked bug_* idea files
@@ -425,7 +425,7 @@

MVP1 Progress

P2 (default)
-
12
+
13
important to file, not blocking
@@ -435,7 +435,7 @@

MVP1 Progress

Legacy "Path to MVP1"
-
8
+
9
scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)
@@ -463,7 +463,7 @@

Pipeline

-

Idea 12

+

Idea 13

Home Demo Reseed Endpoint
@@ -608,6 +608,19 @@

Idea 12

+
+
Markdown Doc Localstorage Undefined Jsdom
+
+ Bug + P2 + +
+
The afterEach hook unconditionally calls `window.localStorage.removeItem(...)` after each test, but `window.localStorage` is `undefined` in the test environment by the time the hook runs — either the
+ + +
+ +
E2E Seed Acme Helper Dead
diff --git a/docs/02_product/planned_features/bug_markdown_doc_localstorage_undefined_jsdom/idea.md b/docs/02_product/planned_features/bug_markdown_doc_localstorage_undefined_jsdom/idea.md new file mode 100644 index 00000000..65179484 --- /dev/null +++ b/docs/02_product/planned_features/bug_markdown_doc_localstorage_undefined_jsdom/idea.md @@ -0,0 +1,63 @@ +# Bug: `markdown-doc.test.tsx` fails on `localStorage.removeItem` (undefined `localStorage`) + +**Date:** 2026-05-24 +**Status:** Idea — captured during feat_digest_executable_followups implementation (Story 5.1 vitest sweep) + +## Origin + +Captured during feat_digest_executable_followups implementation. Running the +full UI test suite (`pnpm test`) surfaces three pre-existing failures in +`ui/src/__tests__/components/guides/markdown-doc.test.tsx`: + +- `wide-column toggle flips data-wide attribute` +- `hydrates text-size from localStorage on mount` +- `View on GitHub link points at the source path under docs/08_guides/` + +All three fail in the `afterEach` cleanup: + +``` +TypeError: Cannot read properties of undefined (reading 'removeItem') + ❯ src/__tests__/components/guides/markdown-doc.test.tsx:37:25 + 37| window.localStorage.removeItem('relyloop.guide-viewer.text-size'); +``` + +Verified pre-existing on `main` via `git stash` + re-run. + +## Problem + +The afterEach hook unconditionally calls +`window.localStorage.removeItem(...)` after each test, but `window.localStorage` +is `undefined` in the test environment by the time the hook runs — either the +jsdom env was torn down early OR the test setup never installed a +`localStorage` polyfill for this file. + +The check `if (typeof window !== 'undefined')` is necessary but not +sufficient — `window` exists in jsdom but `window.localStorage` can still +be undefined depending on the jsdom config / test ordering. + +## Why deferred + +Out of scope for current PR (feat_digest_executable_followups touches +digest follow-ups, not guide markdown rendering). Fix is small (one-line +defensive check) but unrelated to the active feature surface. + +## Proposed fix + +Wrap the removeItem call in a `typeof` guard: + +```ts +if (typeof window !== 'undefined' && window.localStorage) { + window.localStorage.removeItem('relyloop.guide-viewer.text-size'); +} +``` + +Or hoist the cleanup into a `beforeEach`/`afterEach` pair that uses +`vi.stubGlobal('localStorage', { ... })` for deterministic isolation. + +## Scope signals + +- Frontend test-only change +- No production code modified +- No migration +- No new dependency +- Verified by re-running `pnpm vitest run src/__tests__/components/guides/markdown-doc.test.tsx` diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md b/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md index 5728aa6c..144441dd 100644 --- a/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md +++ b/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md @@ -1316,7 +1316,7 @@ Plan testing explicitly by layer and map to stories. ## 9) Execution tracker ### Current sprint -- [ ] Story 1.1 — followups.py domain module +- [x] Story 1.1 — followups.py domain module - [ ] Story 2.1 — worker structured-output schema + validator wiring - [ ] Story 2.2 — LLM prompts - [ ] Story 2.3 — worker integration test diff --git a/ui/src/__tests__/app/proposals/[id]/page.test.tsx b/ui/src/__tests__/app/proposals/[id]/page.test.tsx index ef47c1e3..83e31f38 100644 --- a/ui/src/__tests__/app/proposals/[id]/page.test.tsx +++ b/ui/src/__tests__/app/proposals/[id]/page.test.tsx @@ -96,7 +96,11 @@ describe('Proposal detail page (Story 3.1 shell)', () => { narrative: '## Summary', parameter_importance: {}, recommended_config: {}, - suggested_followups: ['try BM25 tweak'], + // feat_digest_executable_followups Story 4.1: suggested_followups + // are now {kind, rationale, search_space} dicts. + suggested_followups: [ + { kind: 'text', rationale: 'try BM25 tweak', search_space: null }, + ], generated_at: '2026-05-12T00:00:00Z', }, }), @@ -112,10 +116,12 @@ describe('Proposal detail page (Story 3.1 shell)', () => { expect(screen.getByTestId('config-diff-row-slop')).toBeInTheDocument(); expect(screen.getByTestId('metric-delta-pct')).toHaveTextContent('(+55.0%)'); expect(screen.getByTestId('suggested-followups-list')).toBeInTheDocument(); - expect(screen.getByTestId('followup-0-create-study')).toHaveAttribute( - 'href', - '/studies?hypothesis=try%20BM25%20tweak', - ); + // feat_digest_executable_followups Story 5.1 / FR-12: legacy + // `?hypothesis=` link is retired; per-card test-ids replace it. + expect(screen.getByTestId('followup-0-card')).toBeInTheDocument(); + expect(screen.getByText('try BM25 tweak')).toBeInTheDocument(); + // Text-kind cards have NO Run button. + expect(screen.queryByTestId('followup-0-run')).toBeNull(); }); }); diff --git a/ui/src/__tests__/components/proposals/suggested-followups-panel.test.tsx b/ui/src/__tests__/components/proposals/suggested-followups-panel.test.tsx new file mode 100644 index 00000000..07fa27d0 --- /dev/null +++ b/ui/src/__tests__/components/proposals/suggested-followups-panel.test.tsx @@ -0,0 +1,158 @@ +import { render, screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { describe, expect, it, vi } from 'vitest'; + +import { + type FollowupItem, + SuggestedFollowupsPanel, +} from '@/components/proposals/suggested-followups-panel'; +import { TooltipProvider } from '@/components/ui/tooltip'; + +// feat_digest_executable_followups Story 5.1 — covers all 6 rows in the +// legacy-parity table plus the kind-discriminated card markup. + +const VALID_SEARCH_SPACE = { + params: { + title_boost: { type: 'float' as const, low: 0.5, high: 2.0, log: false }, + }, +}; + +const NARROW: FollowupItem = { + kind: 'narrow', + rationale: 'narrow around the winner', + search_space: VALID_SEARCH_SPACE, +}; + +const WIDEN: FollowupItem = { + kind: 'widen', + rationale: 'widen because winner hit edge', + search_space: VALID_SEARCH_SPACE, +}; + +const TEXT: FollowupItem = { + kind: 'text', + rationale: 'add brand-disambiguation queries', + search_space: null, +}; + +function renderPanel(props: Partial> = {}) { + const onRun = props.onRun ?? vi.fn(); + return { + onRun, + ...render( + + + , + ), + }; +} + +describe('', () => { + describe('legacy-parity (Story 5.1 §"Legacy behavior parity" table)', () => { + it('row 1 — hides when followups is empty', () => { + const onRun = vi.fn(); + render( + + + , + ); + expect(screen.queryByTestId('suggested-followups-list')).toBeNull(); + }); + + it('row 2 — preserves container data-testid="suggested-followups-list"', () => { + renderPanel(); + expect(screen.getByTestId('suggested-followups-list')).toBeInTheDocument(); + }); + + it('row 4 + AC-10 — no or with /studies?hypothesis= href', () => { + const { container } = renderPanel(); + const links = container.querySelectorAll('a[href*="/studies?hypothesis="]'); + expect(links.length).toBe(0); + }); + + it('row 5 — InfoTooltip on panel title is preserved', () => { + renderPanel(); + expect(screen.getByText('Suggested follow-ups')).toBeInTheDocument(); + }); + + it('row 6 — rationale text uses the small-text class on a

', () => { + renderPanel(); + // Each rationale is rendered as a

with text-sm class. + const paragraph = screen.getByText('narrow around the winner'); + expect(paragraph.tagName).toBe('P'); + expect(paragraph.className).toContain('text-sm'); + }); + }); + + describe('kind-discriminated cards', () => { + it('renders the Narrow badge + Run button on a narrow card', () => { + renderPanel({ followups: [NARROW], onRun: vi.fn() }); + expect(screen.getByText('Narrow')).toBeInTheDocument(); + expect(screen.getByTestId('followup-0-run')).toBeInTheDocument(); + }); + + it('renders the Widen badge + Run button on a widen card', () => { + renderPanel({ followups: [WIDEN], onRun: vi.fn() }); + expect(screen.getByText('Widen')).toBeInTheDocument(); + expect(screen.getByTestId('followup-0-run')).toBeInTheDocument(); + }); + + it('renders the Suggestion badge + NO Run button on a text card', () => { + renderPanel({ followups: [TEXT], onRun: vi.fn() }); + expect(screen.getByText('Suggestion')).toBeInTheDocument(); + expect(screen.queryByTestId('followup-0-run')).toBeNull(); + }); + + it('per-item data-testids land on every card', () => { + renderPanel(); + // narrow at index 0 + expect(screen.getByTestId('followup-0-card')).toBeInTheDocument(); + expect(screen.getByTestId('followup-0-run')).toBeInTheDocument(); + expect(screen.getByTestId('followup-0-show-search-space')).toBeInTheDocument(); + // widen at index 1 + expect(screen.getByTestId('followup-1-card')).toBeInTheDocument(); + expect(screen.getByTestId('followup-1-run')).toBeInTheDocument(); + // text at index 2 — no run, no show-search-space + expect(screen.getByTestId('followup-2-card')).toBeInTheDocument(); + expect(screen.queryByTestId('followup-2-run')).toBeNull(); + expect(screen.queryByTestId('followup-2-show-search-space')).toBeNull(); + }); + }); + + describe('interactions', () => { + it('clicking Run fires the onRun(index) callback', async () => { + const user = userEvent.setup(); + const { onRun } = renderPanel(); + await user.click(screen.getByTestId('followup-1-run')); + expect(onRun).toHaveBeenCalledTimes(1); + expect(onRun).toHaveBeenCalledWith(1); + }); + }); + + describe('parent search-space diff (Story 5.2 F2)', () => { + it('renders parent JSON block when parentSearchSpace is provided', () => { + const { container } = renderPanel({ + parentSearchSpace: { params: { tie_breaker: { type: 'float', low: 0, high: 1 } } }, + }); + expect( + container.querySelectorAll('[data-testid="followup-0-parent-search-space"]').length, + ).toBe(1); + }); + + it('renders loading message when parentStudyLoading is true', () => { + renderPanel({ parentStudyLoading: true }); + expect(screen.getByTestId('followup-0-search-space-loading')).toBeInTheDocument(); + }); + + it('renders error message when parentStudyError is non-null', () => { + renderPanel({ parentStudyError: new Error('boom') }); + expect(screen.getByTestId('followup-0-search-space-error')).toBeInTheDocument(); + }); + + it('always renders proposed search-space block on narrow/widen', () => { + renderPanel(); + expect(screen.getByTestId('followup-0-proposed-search-space')).toBeInTheDocument(); + expect(screen.getByTestId('followup-1-proposed-search-space')).toBeInTheDocument(); + }); + }); +}); diff --git a/ui/src/__tests__/components/studies/create-study-modal.followup-prefill.test.tsx b/ui/src/__tests__/components/studies/create-study-modal.followup-prefill.test.tsx new file mode 100644 index 00000000..8ff5fd86 --- /dev/null +++ b/ui/src/__tests__/components/studies/create-study-modal.followup-prefill.test.tsx @@ -0,0 +1,265 @@ +/** + * feat_digest_executable_followups Story 5.2 — modal prefill flow tests. + * + * Covers: + * (a) When `initialValues` is provided, the form fields populate from prefill. + * (b) Submitting attaches the `parent` lineage payload to the POST body. + * (c) When `initialValues` is omitted, the POST body has no `parent` field. + */ + +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import { http, HttpResponse } from 'msw'; +import type { ReactNode } from 'react'; + +import { TooltipProvider } from '@/components/ui/tooltip'; + +import { server } from '../../setup'; + +vi.mock('@/components/ui/select', async () => { + const { mockShadcnSelect } = await import('../../helpers/shadcn-select-mock'); + return mockShadcnSelect(); +}); + +vi.mock('sonner', () => ({ + toast: Object.assign(vi.fn(), { error: vi.fn(), success: vi.fn() }), + Toaster: () => null, +})); + +const { CreateStudyModal } = await import('@/components/studies/create-study-modal'); +type PrefillValues = import('@/components/studies/create-study-modal').PrefillValues; + +const API_BASE = 'http://api.test'; + +function wrap(node: ReactNode) { + const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render( + + {node} + , + ); +} + +interface PostBody { + parent?: { proposal_id: string; followup_index: number }; + [key: string]: unknown; +} + +function mockBackend() { + const postBodies: PostBody[] = []; + server.use( + http.get(`${API_BASE}/api/v1/clusters`, () => + HttpResponse.json( + { + data: [ + { + id: 'c1', + name: 'local-es', + engine_type: 'elasticsearch', + environment: 'dev', + base_url: 'http://localhost:9200', + auth_kind: 'es_apikey', + created_at: '2026-05-12T00:00:00Z', + health_check: { + status: 'green', + version: '9.4.0', + checked_at: '2026-05-12T00:00:00Z', + error: null, + }, + }, + ], + next_cursor: null, + has_more: false, + }, + { headers: { 'X-Total-Count': '1' } }, + ), + ), + http.get(`${API_BASE}/api/v1/clusters/c1/schema`, () => HttpResponse.json({ fields: [] })), + http.get(`${API_BASE}/api/v1/clusters/c1/targets`, () => + HttpResponse.json({ data: [{ name: 'products', doc_count: 42 }] }), + ), + http.get(`${API_BASE}/api/v1/query-sets`, () => + HttpResponse.json( + { + data: [{ id: 'qs1', name: 'demo', cluster_id: 'c1', created_at: '2026-05-12T00:00:00Z' }], + next_cursor: null, + has_more: false, + }, + { headers: { 'X-Total-Count': '1' } }, + ), + ), + http.get(`${API_BASE}/api/v1/judgment-lists`, () => + HttpResponse.json( + { + data: [ + { + id: 'jl1', + name: 'demo', + description: null, + query_set_id: 'qs1', + cluster_id: 'c1', + status: 'complete', + created_at: '2026-05-12T00:00:00Z', + }, + ], + next_cursor: null, + has_more: false, + }, + { headers: { 'X-Total-Count': '1' } }, + ), + ), + http.get(`${API_BASE}/api/v1/query-templates`, () => + HttpResponse.json( + { + data: [ + { + id: 'tpl1', + name: 'T1', + engine_type: 'elasticsearch', + version: 1, + created_at: '2026-05-12T00:00:00Z', + }, + ], + next_cursor: null, + has_more: false, + }, + { headers: { 'X-Total-Count': '1' } }, + ), + ), + http.get(`${API_BASE}/api/v1/query-templates/tpl1`, () => + HttpResponse.json({ + id: 'tpl1', + name: 'T1', + engine_type: 'elasticsearch', + body: '{}', + declared_params: { boost_title: 'float' }, + version: 1, + parent_id: null, + created_at: '2026-05-12T00:00:00Z', + }), + ), + http.post(`${API_BASE}/api/v1/studies`, async ({ request }) => { + const body = (await request.json()) as PostBody; + postBodies.push(body); + return HttpResponse.json({ id: 'st-new', name: 'demo', status: 'queued' }); + }), + ); + return { postBodies }; +} + +const PREFILL: PrefillValues = { + cluster_id: 'c1', + target: 'products', + template_id: 'tpl1', + query_set_id: 'qs1', + judgment_list_id: 'jl1', + name: 'parent-study — followup #1 (narrow)', + search_space_text: JSON.stringify( + { params: { boost_title: { type: 'float', low: 1.0, high: 2.0 } } }, + null, + 2, + ), + metric: 'ndcg', + k: 10, + direction: 'maximize', + max_trials: 200, + parallelism: 4, + sampler: 'tpe', + pruner: 'median', + parent: { proposal_id: 'p' + 'a'.repeat(35), followup_index: 0 }, +}; + +/** + * Walk the wizard from step 1 to step 4 (Search space), assuming all field + * values are already set (whether by prefill or test seed). Each call to + * step-next requires the prerequisite-step fields to be populated; we + * trust the prefill or seed to have done that. + * + * After this call, the wizard is at step 4 (search space + objective + + * config) and the Create study submit button is on-screen. + */ +async function walkToFinalStep(): Promise { + // Step 1 — pick cluster + target (prefill sets both; the user still has to + // confirm by clicking through, OR we can fire the change to nudge the + // controlled values into the DOM). + await waitFor(() => expect(screen.getByRole('option', { name: /local-es/ })).toBeInTheDocument()); + // The select onChange handler maps to the form, so trigger a change even + // if the prefill already set the value — confirms the dropdown surfaces the + // value as selected for the test. + fireEvent.change(screen.getByLabelText('Cluster'), { target: { value: 'c1' } }); + await waitFor(() => + expect(screen.queryAllByRole('option', { name: /products/ }).length).toBeGreaterThan(0), + ); + fireEvent.change(screen.getByLabelText('Target index / collection'), { + target: { value: 'products' }, + }); + fireEvent.click(screen.getByTestId('step-next')); + + // Step 2 — data sources (query set + judgment list). + await waitFor(() => expect(screen.getByTestId('step-2')).toBeInTheDocument()); + await waitFor(() => + expect(screen.queryAllByRole('option', { name: 'demo' }).length).toBeGreaterThan(0), + ); + fireEvent.change(screen.getByLabelText('Query set'), { target: { value: 'qs1' } }); + await waitFor(() => { + expect(screen.queryAllByRole('option', { name: 'demo' }).length).toBeGreaterThanOrEqual(2); + }); + fireEvent.change(screen.getByLabelText('Judgment list'), { target: { value: 'jl1' } }); + fireEvent.click(screen.getByTestId('step-next')); + + // Step 3 — template. + await waitFor(() => expect(screen.getByTestId('step-3')).toBeInTheDocument()); + await waitFor(() => expect(screen.getAllByRole('option').length).toBeGreaterThan(0)); + fireEvent.change(screen.getByLabelText('Query template (filtered by engine)'), { + target: { value: 'tpl1' }, + }); + fireEvent.click(screen.getByTestId('step-next')); + + // Step 4 — identity (study name). + await waitFor(() => expect(screen.getByTestId('step-4')).toBeInTheDocument()); +} + +describe('CreateStudyModal — followup prefill (Story 5.2)', () => { + afterEach(() => server.resetHandlers()); + + it('renders the prefilled study name when initialValues is passed', async () => { + mockBackend(); + wrap( {}} initialValues={PREFILL} />); + await walkToFinalStep(); + const nameInput = (await screen.findByLabelText('Study name')) as HTMLInputElement; + expect(nameInput.value).toBe('parent-study — followup #1 (narrow)'); + }); + + it('attaches the parent lineage payload to the POST body on submit', async () => { + const { postBodies } = mockBackend(); + wrap( {}} initialValues={PREFILL} />); + await walkToFinalStep(); + fireEvent.click(screen.getByTestId('step-next')); + await waitFor(() => expect(screen.getByTestId('step-5')).toBeInTheDocument()); + + fireEvent.click(screen.getByRole('button', { name: /Create study/i })); + + await waitFor(() => expect(postBodies.length).toBeGreaterThan(0)); + expect(postBodies[0]!.parent).toEqual({ + proposal_id: PREFILL.parent.proposal_id, + followup_index: 0, + }); + }); + + it('omits the parent field when initialValues is not provided (regression check)', async () => { + const { postBodies } = mockBackend(); + wrap( {}} />); + await walkToFinalStep(); + // The wizard's Step 4 (identity) needs a name when no prefill is set. + fireEvent.change(screen.getByLabelText('Study name'), { + target: { value: 'manual-no-parent' }, + }); + fireEvent.click(screen.getByTestId('step-next')); + await waitFor(() => expect(screen.getByTestId('step-5')).toBeInTheDocument()); + fireEvent.click(screen.getByRole('button', { name: /Create study/i })); + + await waitFor(() => expect(postBodies.length).toBeGreaterThan(0)); + expect('parent' in (postBodies[0] ?? {})).toBe(false); + }); +}); diff --git a/ui/src/__tests__/lib/glossary.test.ts b/ui/src/__tests__/lib/glossary.test.ts index 2503ef40..4b680f54 100644 --- a/ui/src/__tests__/lib/glossary.test.ts +++ b/ui/src/__tests__/lib/glossary.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it } from 'vitest'; import { AUTH_KIND_VALUES, ENVIRONMENT_VALUES, + FOLLOWUP_KIND_VALUES, HEALTH_STATUS_VALUES, JUDGMENT_SOURCE_VALUES, OBJECTIVE_DIRECTION_VALUES, @@ -87,6 +88,24 @@ describe('glossary parity against ui/src/lib/enums.ts (FR-4 / AC-5)', () => { }); }); +describe('feat_digest_executable_followups Story 5.3 — followup glossary keys', () => { + it('every FollowupKind value has a glossary key + no extras', () => { + // The kind-keys use a flat naming pattern (proposal.followup_kind_) + // rather than a dot prefix, so we check each key individually rather + // than via the expectGlossaryGroundedAgainstEnums helper. + for (const kind of FOLLOWUP_KIND_VALUES) { + const key = `proposal.followup_kind_${kind}` as const; + expect(glossary[key], `glossary[${key}] missing`).toBeDefined(); + expect(glossary[key]?.short).toBeTruthy(); + } + }); + + it('the Run button + search-space-diff glossary keys are populated', () => { + expect(glossary['proposal.followup_run_button']?.short).toBeTruthy(); + expect(glossary['proposal.followup_search_space_diff']?.short).toBeTruthy(); + }); +}); + describe('glossary content shape (FR-5)', () => { it('every `short` field is ≤140 characters', () => { for (const [key, entry] of Object.entries(glossary)) { diff --git a/ui/src/app/proposals/[id]/page.tsx b/ui/src/app/proposals/[id]/page.tsx index 31cf34e4..2eaed845 100644 --- a/ui/src/app/proposals/[id]/page.tsx +++ b/ui/src/app/proposals/[id]/page.tsx @@ -1,7 +1,7 @@ 'use client'; import Link from 'next/link'; import { useRouter, useSearchParams } from 'next/navigation'; -import { Suspense, use, useCallback, useEffect, useRef, useState } from 'react'; +import { Suspense, use, useCallback, useEffect, useMemo, useRef, useState } from 'react'; import { DetailPageShell } from '@/components/common/detail-page-shell'; import { InfoTooltip } from '@/components/common/info-tooltip'; @@ -12,8 +12,17 @@ import { PrPanel } from '@/components/proposals/pr-panel'; import { ProposalHeader } from '@/components/proposals/proposal-header'; import { RejectDialog } from '@/components/proposals/reject-dialog'; import { SuggestedFollowupsPanel } from '@/components/proposals/suggested-followups-panel'; +import { CreateStudyModal, type PrefillValues } from '@/components/studies/create-study-modal'; import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'; import { useOpenPR, useProposal } from '@/lib/api/proposals'; +import { useStudy } from '@/lib/api/studies'; +import type { + ObjectiveDirection, + ObjectiveK, + ObjectiveMetric, + PrunerKind, + SamplerKind, +} from '@/lib/enums'; interface RouteProps { params: Promise<{ id: string }>; @@ -108,6 +117,64 @@ export function ProposalDetailView({ proposalId }: { proposalId: string }) { }; }, []); + // feat_digest_executable_followups Story 5.2 — "Run this followup" orchestration. + // The panel reports the clicked index via `onRun`; the page lazily fetches + // the parent study (gated on actionable followups + Run click) and assembles + // the PrefillValues that open the CreateStudyModal. + const [runFollowupIndex, setRunFollowupIndex] = useState(null); + const proposal = proposalQ.data ?? null; + const followups = proposal?.digest?.suggested_followups ?? []; + const hasActionableFollowup = followups.some((f) => f.kind === 'narrow' || f.kind === 'widen'); + const parentStudyId = proposal?.study_id ?? null; + // GPT-5.5 cycle-1 F2: enable the parent fetch whenever the proposal has + // at least one actionable followup, not only on Run click, so the panel's + // "Show search space" detail can render the diff vs parent pre-click. + const parentStudy = useStudy(parentStudyId ?? '', { + enabled: parentStudyId !== null && hasActionableFollowup, + }); + + const prefillValues: PrefillValues | undefined = useMemo(() => { + if (runFollowupIndex === null) return undefined; + if (!proposal || !parentStudy.data || followups.length === 0) return undefined; + const f = followups[runFollowupIndex]; + if (!f || (f.kind !== 'narrow' && f.kind !== 'widen')) return undefined; + const s = parentStudy.data; + const objective = s.objective as { + metric: ObjectiveMetric; + k?: ObjectiveK; + direction: ObjectiveDirection; + }; + const config = s.config as { + max_trials?: number; + time_budget_min?: number; + parallelism?: number; + trial_timeout_s?: number; + sampler?: SamplerKind; + pruner?: PrunerKind; + seed?: number; + }; + return { + cluster_id: s.cluster_id, + target: s.target, + template_id: s.template_id, + query_set_id: s.query_set_id, + judgment_list_id: s.judgment_list_id, + name: `${s.name} — followup #${runFollowupIndex + 1} (${f.kind})`, + search_space_text: JSON.stringify(f.search_space, null, 2), + metric: objective.metric, + k: objective.k, + direction: objective.direction, + max_trials: config.max_trials ?? '', + time_budget_min: config.time_budget_min ?? '', + parallelism: config.parallelism ?? '', + trial_timeout_s: config.trial_timeout_s ?? '', + sampler: config.sampler, + pruner: config.pruner, + seed: config.seed ?? '', + parent: { proposal_id: proposal.id, followup_index: runFollowupIndex }, + }; + }, [runFollowupIndex, proposal, parentStudy.data, followups]); + // ?action=open_pr auto-trigger — fires fireOpenPR once per mount when the // URL carries the query param + the proposal is pending. Strips the param // via router.replace so a remount/back-nav with the same URL does NOT @@ -193,11 +260,27 @@ export function ProposalDetailView({ proposalId }: { proposalId: string }) {

{proposal.digest?.suggested_followups && proposal.digest.suggested_followups.length > 0 && ( - + | undefined) ?? + undefined + } + parentStudyLoading={parentStudy.isLoading} + parentStudyError={parentStudy.error ?? null} + /> )} )} + { + if (!o) setRunFollowupIndex(null); + }} + initialValues={prefillValues} + />
); } diff --git a/ui/src/components/proposals/suggested-followups-panel.tsx b/ui/src/components/proposals/suggested-followups-panel.tsx index 04682909..81aed287 100644 --- a/ui/src/components/proposals/suggested-followups-panel.tsx +++ b/ui/src/components/proposals/suggested-followups-panel.tsx @@ -1,15 +1,56 @@ 'use client'; -import Link from 'next/link'; import { InfoTooltip } from '@/components/common/info-tooltip'; +import { Badge } from '@/components/ui/badge'; import { Button } from '@/components/ui/button'; import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'; +import type { FollowupKind } from '@/lib/enums'; +import type { components } from '@/lib/types'; + +// feat_digest_executable_followups Story 5.1 — rewrite the panel to +// branch per discriminated-union ``FollowupItem`` kind. ``narrow`` and +// ``widen`` items carry a ``search_space`` body and surface as +// actionable "Run this followup" cards; ``text`` items render as +// rationale-only suggestion cards. The legacy ``?hypothesis=`` link is +// retired per FR-12. + +export type FollowupItem = components['schemas']['FollowupItem']; export interface SuggestedFollowupsPanelProps { - followups: readonly string[]; + followups: readonly FollowupItem[]; + /** + * Called with the 0-based index when the operator clicks "Run this followup". + * The page-level orchestrator (Story 5.2) is responsible for opening the + * CreateStudyModal with the appropriate ``initialValues``. + */ + onRun: (index: number) => void; + /** + * Parent study's ``search_space`` for the diff view. Optional — when omitted + * (e.g., still loading from a lazy useStudy call), the per-card "Show + * search space" expander falls back to rendering the proposed search-space + * JSON without a diff comparison. + */ + parentSearchSpace?: Record; + /** Loading flag for the lazy parent-study fetch. */ + parentStudyLoading?: boolean; + /** Non-null error from the lazy parent-study fetch. */ + parentStudyError?: unknown; } -export function SuggestedFollowupsPanel({ followups }: SuggestedFollowupsPanelProps) { +// Values must match backend/app/domain/study/followups.py FollowupItem.kind. +const KIND_LABELS: Record = { + narrow: 'Narrow', + widen: 'Widen', + text: 'Suggestion', +}; + +export function SuggestedFollowupsPanel({ + followups, + onRun, + parentSearchSpace, + parentStudyLoading = false, + parentStudyError = null, +}: SuggestedFollowupsPanelProps) { if (followups.length === 0) return null; return ( @@ -22,16 +63,80 @@ export function SuggestedFollowupsPanel({ followups }: SuggestedFollowupsPanelPr
    {followups.map((f, i) => ( -
  • - {f} - +
  • +
    + + {KIND_LABELS[f.kind]} + + +
    +

    {f.rationale}

    + {(f.kind === 'narrow' || f.kind === 'widen') && ( + <> +
    + + Show search space + + + + +
    + {parentStudyLoading && ( +

    + Loading parent search space... +

    + )} + {parentStudyError !== null && parentStudyError !== undefined && ( +

    + Could not load parent — showing proposed bounds only. +

    + )} + {parentSearchSpace !== undefined && ( +
    +

    Parent (current):

    + 
    +                            {JSON.stringify(parentSearchSpace, null, 2)}
+
    +
    + )} +
    +

    Proposed:

    + 
    +                          {JSON.stringify(f.search_space, null, 2)}
+
    +
    +
    +
    +
    + +
    + + )}
    • ))}
diff --git a/ui/src/components/studies/create-study-modal.tsx b/ui/src/components/studies/create-study-modal.tsx index 82adc46f..301fbbd8 100644 --- a/ui/src/components/studies/create-study-modal.tsx +++ b/ui/src/components/studies/create-study-modal.tsx @@ -156,12 +156,49 @@ const STEP_TITLES = [ 'Objective + config', ] as const; +/** + * feat_digest_executable_followups Story 5.2 — prefill payload for the + * "Run this followup" flow. Carries the parent-study-derived form-field + * values plus the lineage tuple that gets sent in the POST body's + * ``parent`` field. + */ +export interface PrefillValues { + cluster_id: string; + target: string; + template_id: string; + query_set_id: string; + judgment_list_id: string; + name: string; + search_space_text: string; + metric: ObjectiveMetric; + k?: ObjectiveK; + direction: ObjectiveDirection; + max_trials?: number | ''; + time_budget_min?: number | ''; + parallelism?: number | ''; + trial_timeout_s?: number | ''; + sampler?: SamplerKind; + pruner?: PrunerKind; + seed?: number | ''; + parent: { + proposal_id: string; + followup_index: number; + }; +} + export interface CreateStudyModalProps { open: boolean; onOpenChange: (open: boolean) => void; + /** + * feat_digest_executable_followups Story 5.2 — when provided, the modal + * opens with the form fields pre-populated from the parent study and the + * POST body carries the ``parent`` lineage payload. Re-renders with a + * different ``initialValues`` reset the form to the new values. + */ + initialValues?: PrefillValues; } -export function CreateStudyModal({ open, onOpenChange }: CreateStudyModalProps) { +export function CreateStudyModal({ open, onOpenChange, initialValues }: CreateStudyModalProps) { const create = useCreateStudy(); const [step, setStep] = useState(0); const [submitting, setSubmitting] = useState(false); @@ -250,6 +287,35 @@ export function CreateStudyModal({ open, onOpenChange }: CreateStudyModalProps) } }, [open]); + // feat_digest_executable_followups Story 5.2 — when ``initialValues`` is + // provided AND the modal is open, reset the form with the prefill payload. + // Runs AFTER the manualMode-reset effect above (React fires effects in + // declaration order). Re-renders with a different ``initialValues`` + // reset the form to the new values per AC-2 / D-19. + useEffect(() => { + if (open && initialValues) { + form.reset({ + cluster_id: initialValues.cluster_id, + target: initialValues.target, + template_id: initialValues.template_id, + query_set_id: initialValues.query_set_id, + judgment_list_id: initialValues.judgment_list_id, + name: initialValues.name, + search_space_text: initialValues.search_space_text, + metric: initialValues.metric, + k: initialValues.k, + direction: initialValues.direction, + max_trials: initialValues.max_trials, + time_budget_min: initialValues.time_budget_min, + parallelism: initialValues.parallelism, + trial_timeout_s: initialValues.trial_timeout_s, + sampler: initialValues.sampler, + pruner: initialValues.pruner, + seed: initialValues.seed, + }); + } + }, [open, initialValues, form]); + // FR-5 auto-engage: when the targets query fails with TARGETS_FORBIDDEN, // silently flip into manual mode. `open` is in BOTH the guard AND the // dependency list (cycle-2 GPT-5.5 review #3) — without it, a cached @@ -560,6 +626,18 @@ export function CreateStudyModal({ open, onOpenChange }: CreateStudyModalProps) search_space, objective, config, + // feat_digest_executable_followups Story 5.2 — when the modal was + // opened with prefill (the "Run this followup" flow), attach the + // lineage tuple so the backend records parent_proposal_id + + // parent_proposal_followup_index on the new study row. + ...(initialValues?.parent + ? { + parent: { + proposal_id: initialValues.parent.proposal_id, + followup_index: initialValues.parent.followup_index, + }, + } + : {}), }, { onSuccess: () => { diff --git a/ui/src/components/studies/digest-panel.tsx b/ui/src/components/studies/digest-panel.tsx index 20d4a16e..ed9183b4 100644 --- a/ui/src/components/studies/digest-panel.tsx +++ b/ui/src/components/studies/digest-panel.tsx @@ -94,7 +94,12 @@ export function DigestPanel({

    {followups.map((f, i) => ( -
  • {f}
    • + // feat_digest_executable_followups Story 4.1 — followups + // are now {kind, rationale, search_space} dicts; the + // study-page digest panel renders only the rationale as a + // simple bullet (the rich card UI lives on the proposal- + // detail page; this study summary is intentionally compact). +
  • {f.rationale}
    • ))}
diff --git a/ui/src/lib/api/studies.ts b/ui/src/lib/api/studies.ts index bcc02fbc..30b11895 100644 --- a/ui/src/lib/api/studies.ts +++ b/ui/src/lib/api/studies.ts @@ -55,6 +55,13 @@ type RefetchInterval = export interface UseStudyOptions { refetchInterval?: RefetchInterval; + /** + * feat_digest_executable_followups Story 5.2 — gate the fetch so the + * proposal-detail page can lazily load the parent study only when at + * least one ``narrow``/``widen`` followup is actionable, avoiding a + * wasteful request on proposals whose digests are all ``text`` items. + */ + enabled?: boolean; } export function useStudy( @@ -68,6 +75,7 @@ export function useStudy( return data; }, refetchInterval: options.refetchInterval ?? false, + enabled: options.enabled ?? true, }); } diff --git a/ui/src/lib/enums.ts b/ui/src/lib/enums.ts index 05445404..d6395f65 100644 --- a/ui/src/lib/enums.ts +++ b/ui/src/lib/enums.ts @@ -239,3 +239,12 @@ export type ProposalSortKey = (typeof PROPOSAL_SORT_VALUES)[number]; // Values must match backend/app/api/v1/schemas.py ProposalSourceWire. export const PROPOSAL_SOURCE_VALUES = ['study', 'manual'] as const; export type ProposalSource = (typeof PROPOSAL_SOURCE_VALUES)[number]; + +// feat_digest_executable_followups Story 5.1 — frontend mirror of the +// backend FollowupItem discriminator. The values flow back to the backend +// only indirectly (via persisted JSONB), but the frontend uses them to +// branch UI per card kind. +// Values must match backend/app/domain/study/followups.py FollowupItem.kind +// (Literal['narrow'] | Literal['widen'] | Literal['text']). +export const FOLLOWUP_KIND_VALUES = ['narrow', 'widen', 'text'] as const; +export type FollowupKind = (typeof FOLLOWUP_KIND_VALUES)[number]; diff --git a/ui/src/lib/glossary.ts b/ui/src/lib/glossary.ts index 2e4030d9..9981065c 100644 --- a/ui/src/lib/glossary.ts +++ b/ui/src/lib/glossary.ts @@ -473,6 +473,36 @@ export const glossary = { "LLM-generated next-study hypotheses based on this study's parameter-importance pattern. Click to seed a new study.", ariaLabel: 'More information about suggested follow-ups', }, + // feat_digest_executable_followups Story 5.3 — five glossary keys for + // the per-card kind badges, the Run button, and the search-space diff + // toggle on the rewritten SuggestedFollowupsPanel. + // Source-of-truth: backend/app/domain/study/followups.py NarrowFollowup + 'proposal.followup_kind_narrow': { + short: + "The study's winning configuration sits in a sub-region of the prior search space. This followup re-runs with a tighter range to confirm.", + ariaLabel: 'More information about narrow follow-ups', + }, + // Source-of-truth: backend/app/domain/study/followups.py WidenFollowup + 'proposal.followup_kind_widen': { + short: + 'The winning configuration hit an edge of the prior search space. This followup re-runs with a broader range to find a better setting.', + ariaLabel: 'More information about widen follow-ups', + }, + // Source-of-truth: backend/app/domain/study/followups.py TextFollowup + 'proposal.followup_kind_text': { + short: + 'A free-form suggestion from the LLM. Needs operator interpretation — no auto-prefill available.', + ariaLabel: 'More information about text follow-ups', + }, + 'proposal.followup_run_button': { + short: + 'Opens the create-study wizard pre-filled with this followup’s settings. You can review and edit before submitting.', + ariaLabel: 'More information about the run-followup button', + }, + 'proposal.followup_search_space_diff': { + short: "Compare this followup's proposed search space against the parent study's.", + ariaLabel: 'More information about the search-space diff', + }, 'proposal.status_filter': { short: 'Filter the proposals list by lifecycle state. "All" shows every proposal regardless of where it is in the open-PR / merge flow.', diff --git a/ui/src/lib/types.ts b/ui/src/lib/types.ts index fe67cc3c..56953d9f 100644 --- a/ui/src/lib/types.ts +++ b/ui/src/lib/types.ts @@ -383,12 +383,20 @@ export interface paths { put?: never; /** * Cancel Study - * @description Cancel a study (FR-1 + AC-3). + * @description Cancel a study (Story 2.3, FR-8 + AC-8/AC-9). * - * Routes through :func:`services.study_state.cancel_study`. The - * orchestrator detects the new status on its next poll tick and drains - * in-flight trials. Cancelling an already-terminal study (completed / - * cancelled / failed) raises ``InvalidStateTransition`` → 409. + * Optionally cascades to in-flight chain children. + * + * ``?cascade=true`` (default): routes through + * :func:`services.study_state.cancel_study_with_chain_cascade` — + * cancels the parent (if in-flight) AND recursively cancels in-flight + * descendants. Tolerates terminal parents (recurses through completed + * intermediates to reach an in-flight grandchild). + * + * ``?cascade=false``: routes through the original + * :func:`services.study_state.cancel_study` — single-study cancel, + * preserves the existing 409 error contract on terminal parents + * (AC-9 wire contract). */ post: operations['cancel_study_api_v1_studies__study_id__cancel_post']; delete?: never; @@ -397,6 +405,34 @@ export interface paths { patch?: never; trace?: never; }; + '/api/v1/studies/{study_id}/children': { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * List Study Children + * @description List direct child studies of a parent (FR-10 + D-13). + * + * Returns ``{"data": [], "next_cursor": null}`` for a study with no + * children — empty data array, NOT 404. 404 only fires when the parent + * study itself is missing. + * + * Per D-13 (direct-children-only): does NOT return transitive + * descendants. The chain panel renders parent ↑ + direct children ↓; + * operators walk lineage one hop per page navigation. + */ + get: operations['list_study_children_api_v1_studies__study_id__children_get']; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; '/api/v1/studies/{study_id}/trials': { parameters: { query?: never; @@ -765,6 +801,15 @@ export interface paths { /** * Get Config Repo Endpoint * @description Detail by id; 404 ``CONFIG_REPO_NOT_FOUND`` if missing. + * + * feat_config_repo_baseline_tracking FR-4 — when + * ``last_merged_proposal_id`` is set, embed the pointed-at proposal as a + * :class:`ProposalSummary` with ``is_currently_live=True``. The embed-side + * derivation uses the pointer context directly (NOT the generic + * ``proposals → clusters → config_repos`` JOIN used elsewhere) so the + * badge renders correctly even when the proposal's cluster was later + * unwired from this config_repo (spec §19 "Cluster-with-config_repo- + * rotated" decision-log entry). */ get: operations['get_config_repo_endpoint_api_v1_config_repos__config_repo_id__get']; put?: never; @@ -1113,6 +1158,22 @@ export interface components { /** Human Samples */ human_samples: components['schemas']['CalibrationSample'][]; }; + /** + * CategoricalParam + * @description Discrete choice parameter. + * + * Optuna ``suggest_categorical`` handles strings, ints, floats, and bools + * as choices. + */ + CategoricalParam: { + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + type: 'categorical'; + /** Choices */ + choices: (string | number | boolean)[]; + }; /** * ClusterAggregateHealth * @description Aggregate counts for the ``elasticsearch_clusters`` /healthz field (Story 3.5). @@ -1262,7 +1323,6 @@ export interface components { webhook_secret_ref: string | null; /** Webhook Registration Error */ webhook_registration_error: string | null; - /** Last Merged Proposal — feat_config_repo_baseline_tracking FR-4 */ last_merged_proposal?: components['schemas']['ProposalSummary'] | null; /** * Created At @@ -1513,6 +1573,10 @@ export interface components { * :class:`backend.app.domain.study.search_space.SearchSpace` so * :exc:`pydantic.ValidationError` produces the spec's 400 * ``INVALID_SEARCH_SPACE`` (per Story 3.3 task 2). + * + * feat_digest_executable_followups Story 4.2 — optional ``parent`` field + * records the parent proposal + followup-index lineage when the study + * was spawned from a digest "Run this followup" action (FR-11). */ CreateStudyRequest: { /** Name */ @@ -1533,10 +1597,17 @@ export interface components { }; objective: components['schemas']['ObjectiveSpec']; config: components['schemas']['StudyConfigSpec']; + parent?: components['schemas']['ParentFollowupRef'] | null; }; /** * DigestResponse * @description Body of ``GET /api/v1/studies/{id}/digest`` (FR-3 / AC-3). + * + * feat_digest_executable_followups Story 4.1 — ``suggested_followups`` is + * now a discriminated-union list (NarrowFollowup | WidenFollowup | + * TextFollowup), populated by the digest handler via + * ``parse_followup_list(digest.suggested_followups, ...)`` so legacy or + * malformed JSONB payloads never crash the response. */ DigestResponse: { /** Id */ @@ -1554,7 +1625,7 @@ export interface components { [key: string]: unknown; }; /** Suggested Followups */ - suggested_followups: string[]; + suggested_followups: components['schemas']['FollowupItem'][]; /** Generated By */ generated_by: string; /** @@ -1577,6 +1648,33 @@ export interface components { /** Doc Count */ doc_count?: number | null; }; + /** + * FloatParam + * @description Continuous float parameter. + * + * ``log=True`` enables log-uniform sampling + * (Optuna's ``suggest_float(..., log=True)``); requires ``low > 0``. + */ + FloatParam: { + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + type: 'float'; + /** Low */ + low: number; + /** High */ + high: number; + /** + * Log + * @default false + */ + log: boolean; + }; + FollowupItem: + | components['schemas']['NarrowFollowup'] + | components['schemas']['WidenFollowup'] + | components['schemas']['TextFollowup']; /** * GenerateJudgmentsResponse * @description Response of ``POST /api/v1/judgments/generate``. @@ -1701,6 +1799,21 @@ export interface components { /** Judgments */ judgments: components['schemas']['ImportJudgmentItem'][]; }; + /** + * IntParam + * @description Integer parameter inclusive of both bounds. + */ + IntParam: { + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + type: 'int'; + /** Low */ + low: number; + /** High */ + high: number; + }; /** * JudgmentListDetail * @description ``GET /api/v1/judgment-lists/{id}`` response. @@ -1883,6 +1996,20 @@ export interface components { */ created_at: string; }; + /** + * NarrowFollowup + * @description A 'narrow' followup — re-run with a tighter range than the parent. + */ + NarrowFollowup: { + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + kind: 'narrow'; + /** Rationale */ + rationale: string; + search_space: components['schemas']['SearchSpace']; + }; /** * ObjectiveSpec * @description Wire shape of ``studies.objective`` (write-side validated at create). @@ -1968,6 +2095,27 @@ export interface components { /** Notes */ notes?: string | null; }; + /** + * ParentFollowupRef + * @description Optional lineage payload on ``POST /api/v1/studies``. + * + * feat_digest_executable_followups FR-11 — when the operator clicks + * "Run this followup" on a proposal's digest card, the create-study + * payload carries the parent proposal's id + the 0-based index into + * the digest's ``suggested_followups`` array so the spawned study + * remembers where it came from. + * + * ``proposal_id`` is a UUIDv7 (36-char hex). The exact-length bound + * forces malformed strings to surface as 422 ``VALIDATION_ERROR`` + * rather than reach the DB FK check and emerge as a 404 + * ``PROPOSAL_NOT_FOUND``. + */ + ParentFollowupRef: { + /** Proposal Id */ + proposal_id: string; + /** Followup Index */ + followup_index: number; + }; /** * PerQueryOutcomesShape * @description Per-query outcome counts + the top-5 named regressors. @@ -2028,10 +2176,10 @@ export interface components { /** Rejected Reason */ rejected_reason: string | null; /** - * Is Currently Live — feat_config_repo_baseline_tracking FR-5 + * Is Currently Live * @default false */ - is_currently_live?: boolean; + is_currently_live: boolean; digest: components['schemas']['_DigestEmbed'] | null; /** * Created At @@ -2064,10 +2212,10 @@ export interface components { [key: string]: unknown; } | null; /** - * Is Currently Live — feat_config_repo_baseline_tracking FR-5 + * Is Currently Live * @default false */ - is_currently_live?: boolean; + is_currently_live: boolean; /** * Created At * Format: date-time @@ -2364,6 +2512,29 @@ export interface components { /** Fields */ fields: components['schemas']['FieldSpec'][]; }; + /** + * SearchSpace + * @description Pydantic model for the ``studies.search_space`` JSONB column. + * + * Wire format:: + * + * { + * "params": { + * "boost_title": {"type": "float", "low": 0.1, "high": 10.0, "log": true}, + * "min_should_match": {"type": "int", "low": 1, "high": 5}, + * "operator": {"type": "categorical", "choices": ["and", "or"]}, + * } + * } + */ + SearchSpace: { + /** Params */ + params: { + [key: string]: + | components['schemas']['FloatParam'] + | components['schemas']['IntParam'] + | components['schemas']['CategoricalParam']; + }; + }; /** * SeedCompletedStudyRequest * @description Payload for ``POST /api/v1/_test/studies/seed-completed``. @@ -2469,6 +2640,8 @@ export interface components { seed?: number | null; /** Secondary Metrics */ secondary_metrics?: string[] | null; + /** Auto Followup Depth */ + auto_followup_depth?: number | null; }; /** * StudyDetail @@ -2631,6 +2804,21 @@ export interface components { /** Data */ data: components['schemas']['TargetInfo'][]; }; + /** + * TextFollowup + * @description A free-form textual suggestion — no auto-prefill, operator interprets. + */ + TextFollowup: { + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + kind: 'text'; + /** Rationale */ + rationale: string; + /** Search Space */ + search_space?: null; + }; /** * TrialDetail * @description ``GET /api/v1/studies/{id}/trials`` response row. @@ -2729,6 +2917,20 @@ export interface components { /** Context */ ctx?: Record; }; + /** + * WidenFollowup + * @description A 'widen' followup — re-run with a broader range than the parent. + */ + WidenFollowup: { + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + kind: 'widen'; + /** Rationale */ + rationale: string; + search_space: components['schemas']['SearchSpace']; + }; /** * _ClusterEmbed * @description Inline cluster summary on proposal responses. @@ -2746,6 +2948,9 @@ export interface components { /** * _DigestEmbed * @description Inline digest summary on the proposal-detail response. + * + * feat_digest_executable_followups Story 4.1 — ``suggested_followups`` is + * now a discriminated-union list (see ``DigestResponse``). */ _DigestEmbed: { /** Id */ @@ -2761,7 +2966,7 @@ export interface components { [key: string]: unknown; }; /** Suggested Followups */ - suggested_followups: string[]; + suggested_followups: components['schemas']['FollowupItem'][]; /** * Generated At * Format: date-time @@ -3565,7 +3770,9 @@ export interface operations { }; cancel_study_api_v1_studies__study_id__cancel_post: { parameters: { - query?: never; + query?: { + cascade?: string; + }; header?: never; path: { study_id: string; @@ -3594,6 +3801,37 @@ export interface operations { }; }; }; + list_study_children_api_v1_studies__study_id__children_get: { + parameters: { + query?: never; + header?: never; + path: { + study_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + 'application/json': components['schemas']['StudyListResponse']; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + 'application/json': components['schemas']['HTTPValidationError']; + }; + }; + }; + }; list_study_trials_api_v1_studies__study_id__trials_get: { parameters: { query?: { @@ -3929,6 +4167,7 @@ export interface operations { source?: ('study' | 'manual') | null; template_id?: string | null; study_id?: string | null; + is_last_merged?: boolean | null; cursor?: string | null; limit?: number; sort?: From a9e2e453f88572b02766dcd1a18f2f76f1463da3 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Sat, 23 May 2026 22:34:42 -0400 Subject: [PATCH 10/15] test(ui): E2E happy path for Run-this-followup flow (Story 6.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - test_seeding: optional suggested_followups list[dict] kwarg; default unchanged. - _test router: new field on SeedCompletedStudyRequest passes through. - seed.ts helper: SeedFollowupItem type + suggestedFollowups arg. - followup_run.spec.ts: drives the full flow against the real backend — seeds a narrow followup, navigates to the proposal, clicks Run, walks the wizard (asserting the prefilled name), submits, and asserts a new study with the followup-derived name was created. Co-Authored-By: Claude Opus 4.7 (1M context) --- backend/app/api/v1/_test.py | 11 +++ backend/app/services/test_seeding.py | 25 +++++-- ui/tests/e2e/followup_run.spec.ts | 103 +++++++++++++++++++++++++++ ui/tests/e2e/helpers/seed.ts | 35 ++++++++- 4 files changed, 169 insertions(+), 5 deletions(-) create mode 100644 ui/tests/e2e/followup_run.spec.ts diff --git a/backend/app/api/v1/_test.py b/backend/app/api/v1/_test.py index 31fe6f71..f15da6f3 100644 --- a/backend/app/api/v1/_test.py +++ b/backend/app/api/v1/_test.py @@ -114,6 +114,16 @@ class SeedCompletedStudyRequest(BaseModel): "Optional per-query metrics for the runner-up trial; pairs with `winner_per_query`." ), ) + suggested_followups: list[dict[str, Any]] | None = Field( + default=None, + description=( + "feat_digest_executable_followups Story 6.1 — optional structured " + "FollowupItem list (`[{kind, rationale, search_space}]`) to seed " + "on the digest. When omitted, the seeder writes two default text-kind " + "items. The E2E Run-followup spec passes a `narrow` item so it can " + "drive the per-card Run button + modal prefill flow." + ), + ) class SeedCompletedStudyResponse(BaseModel): @@ -164,6 +174,7 @@ async def seed_completed_study( # pragma: no cover - integration only with_pending_proposal=body.with_pending_proposal, winner_per_query=body.winner_per_query, runner_up_per_query=body.runner_up_per_query, + suggested_followups=body.suggested_followups, ) await db.commit() return SeedCompletedStudyResponse( diff --git a/backend/app/services/test_seeding.py b/backend/app/services/test_seeding.py index 6a4b6701..59394459 100644 --- a/backend/app/services/test_seeding.py +++ b/backend/app/services/test_seeding.py @@ -48,6 +48,7 @@ async def seed_study_completed_with_digest( # pragma: no cover - integration o with_pending_proposal: bool = True, winner_per_query: dict[str, dict[str, Any]] | None = None, runner_up_per_query: dict[str, dict[str, Any]] | None = None, + suggested_followups: list[dict[str, Any]] | None = None, ) -> SeededStudyTriple: """Insert a complete study + 2 trials + digest (+ optional pending proposal). @@ -157,6 +158,23 @@ async def seed_study_completed_with_digest( # pragma: no cover - integration o ) digest_id = str(uuid_utils.uuid7()) + # feat_digest_executable_followups Story 6.1 — the column is now JSONB + # carrying ``[{kind, rationale, search_space}]`` dicts. Default to two + # text-kind items (preserving the pre-Story-3.3 visible behavior); the + # E2E happy-path test overrides via the new ``suggested_followups`` + # kwarg to inject a structured ``narrow`` item. + default_followups: list[dict[str, Any]] = [ + { + "kind": "text", + "rationale": "Try varying `description.boost` next.", + "search_space": None, + }, + { + "kind": "text", + "rationale": "Run with a larger query set to confirm the lift holds.", + "search_space": None, + }, + ] await repo.create_digest( db, id=digest_id, @@ -168,10 +186,9 @@ async def seed_study_completed_with_digest( # pragma: no cover - integration o ), parameter_importance={"title.boost": 1.0}, recommended_config={"title.boost": 2.5}, - suggested_followups=[ - "Try varying `description.boost` next.", - "Run with a larger query set to confirm the lift holds.", - ], + suggested_followups=( + suggested_followups if suggested_followups is not None else default_followups + ), generated_by="local:e2e_seed", ) diff --git a/ui/tests/e2e/followup_run.spec.ts b/ui/tests/e2e/followup_run.spec.ts new file mode 100644 index 00000000..3f6736a6 --- /dev/null +++ b/ui/tests/e2e/followup_run.spec.ts @@ -0,0 +1,103 @@ +/** + * E2E spec: Run-followup happy path (feat_digest_executable_followups Story 6.1). + * + * Drives the full "Run this followup" flow against the real backend with no + * `page.route()` mocking: + * 1. Seed cluster + template + query set + judgment list + completed study + * via the public `seedFullChain` helper, then promote it to a digest + * with a structured `narrow` followup via the test-only seed endpoint. + * 2. Navigate to `/proposals/` and assert the Narrow card renders. + * 3. Click "Run this followup" → assert the create-study modal opens + * with the parent study's name in the wizard's identity step. + * 4. Submit the wizard → assert navigation to `/studies/` (the + * `toast.success` triggers a router push). + * 5. Assert the new study's lineage columns via the API. + */ +import { expect, test } from '@playwright/test'; + +import { seedFullChain, seedStudyCompletedWithDigest } from './helpers/seed'; + +const API_BASE = process.env.PLAYWRIGHT_API_BASE_URL ?? 'http://127.0.0.1:8000'; + +const NARROW_SEARCH_SPACE = { + params: { + 'title.boost': { type: 'float', low: 1.5, high: 2.5, log: false }, + }, +}; + +test('Run this followup → modal opens prefilled → submit creates lineage-linked study', async ({ + page, + request, +}) => { + // 1. Seed the parent chain + a completed study + a structured `narrow` + // digest. The narrow followup carries an inline SearchSpace so the + // "Run this followup" button renders. + const chain = await seedFullChain(); + const completed = await seedStudyCompletedWithDigest({ + clusterId: chain.clusterId, + querySetId: chain.querySetId, + templateId: chain.templateId, + judgmentListId: chain.judgmentListId, + withPendingProposal: true, + suggestedFollowups: [ + { + kind: 'narrow', + rationale: 'Narrow title.boost around the winner', + search_space: NARROW_SEARCH_SPACE, + }, + ], + }); + expect(completed.proposalId).not.toBeNull(); + const proposalId = completed.proposalId as string; + + // 2. Navigate to the proposal detail page. + await page.goto(`/proposals/${proposalId}`); + await expect(page.getByTestId('followup-0-card')).toBeVisible({ timeout: 10_000 }); + // Narrow badge surfaces — scope to the card so the rationale text (which also + // starts with "Narrow") doesn't cause a strict-mode violation. + await expect( + page.getByTestId('followup-0-card').getByLabel('Narrow', { exact: true }), + ).toBeVisible(); + + // 3. Click "Run this followup". + await page.getByTestId('followup-0-run').click(); + + // 4. The CreateStudyModal opens. The form is prefilled — walk to the + // identity step (Step 4) and assert the study name is the + // followup-derived value. + // Step 1 (cluster + target) — click next. + await page.getByTestId('step-next').click(); + // Step 2 (data sources) — next. + await page.getByTestId('step-next').click(); + // Step 3 (template) — next. + await page.getByTestId('step-next').click(); + // Step 4 (identity) — Study name is prefilled to " — followup #1 (narrow)". + const nameInput = page.getByLabel('Study name'); + await expect(nameInput).toBeVisible(); + await expect(nameInput).toHaveValue(/followup #1 \(narrow\)/); + await page.getByTestId('step-next').click(); + // Step 5 (objective + config) — submit. + await page.getByRole('button', { name: /Create study/i }).click(); + + // 5. The mutation succeeds (toast.success fires). Verify a new study + // exists with the followup-derived name via the public API. + // + // Per spec D-5, parent_proposal_id is NOT exposed on StudyDetail — + // it's persisted (verified by the integration test + // test_studies_with_parent_followup.py::test_happy_path_persists_lineage) + // but not part of the public response surface. Asserting a new study + // exists with the prefill-derived name is sufficient end-to-end proof + // that the POST body (including the `parent` field) was accepted. + await page.waitForTimeout(2_000); + const studiesResp = await request.get(`${API_BASE}/api/v1/studies?limit=20`); + expect(studiesResp.ok()).toBe(true); + const body = (await studiesResp.json()) as { + data: Array<{ id: string; name: string }>; + }; + const newStudy = body.data.find( + (s) => /followup #1 \(narrow\)/.test(s.name) && s.id !== completed.studyId, + ); + expect(newStudy).toBeDefined(); + // Reference proposalId so TypeScript doesn't flag it as unused. + expect(proposalId).toMatch(/^[0-9a-f-]{36}$/); +}); diff --git a/ui/tests/e2e/helpers/seed.ts b/ui/tests/e2e/helpers/seed.ts index 8aeb737c..830253e9 100644 --- a/ui/tests/e2e/helpers/seed.ts +++ b/ui/tests/e2e/helpers/seed.ts @@ -685,14 +685,46 @@ export async function seedProposal(args: { * vitest component layer with mocked data — the orchestrator + digest * worker can't be reliably driven to completion in a Playwright timeout. */ +/** + * feat_digest_executable_followups Story 6.1 — discriminated-union + * FollowupItem shape that `seedStudyCompletedWithDigest` accepts for the + * digest's `suggested_followups`. Mirrors + * `backend/app/domain/study/followups.py` (FollowupItem). + */ +export type SeedFollowupItem = + | { + kind: 'narrow' | 'widen'; + rationale: string; + search_space: Record; + } + | { + kind: 'text'; + rationale: string; + search_space: null; + }; + export async function seedStudyCompletedWithDigest(args: { clusterId: string; querySetId: string; templateId: string; judgmentListId: string; withPendingProposal?: boolean; + /** + * Optional structured `FollowupItem` list to seed on the digest. When + * omitted, the backend seeder writes two default text-kind items. The + * Run-followup E2E spec passes a `narrow` item so it can drive the + * per-card Run button + modal prefill flow. + */ + suggestedFollowups?: SeedFollowupItem[]; }): Promise { - const { clusterId, querySetId, templateId, judgmentListId, withPendingProposal = true } = args; + const { + clusterId, + querySetId, + templateId, + judgmentListId, + withPendingProposal = true, + suggestedFollowups, + } = args; const result = await post<{ study_id: string; digest_id: string; proposal_id: string | null }>( '/api/v1/_test/studies/seed-completed', { @@ -701,6 +733,7 @@ export async function seedStudyCompletedWithDigest(args: { template_id: templateId, judgment_list_id: judgmentListId, with_pending_proposal: withPendingProposal, + ...(suggestedFollowups !== undefined ? { suggested_followups: suggestedFollowups } : {}), }, ); // Register all 3 IDs (or 2 if no proposal) so global-teardown drains them From 7407648528bf12c3ced3fc8af9a1c265c9a96ccd Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Sat, 23 May 2026 22:39:37 -0400 Subject: [PATCH 11/15] docs(digest-executable-followups): post-implementation docs updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - state.md: bump Alembic head to 0019; mention 0018 + 0019 owners. - architecture.md: add followups.py to domain map; extend migrations note. - api-conventions.md: document PROPOSAL_NOT_FOUND / DIGEST_NOT_FOUND / FOLLOWUP_INDEX_OUT_OF_RANGE on POST /api/v1/studies. - data-model.md: studies.parent_proposal_* columns + digests.suggested_followups type change to JSONB with the FollowupItem comment. - implementation_plan.md: mark all 16 stories complete in §9 tracker. Co-Authored-By: Claude Opus 4.7 (1M context) --- architecture.md | 13 +++++++- docs/01_architecture/api-conventions.md | 8 +++++ docs/01_architecture/data-model.md | 4 ++- .../implementation_plan.md | 32 +++++++++---------- state.md | 2 +- 5 files changed, 40 insertions(+), 19 deletions(-) diff --git a/architecture.md b/architecture.md index 38d0c484..d7ee1dd2 100644 --- a/architecture.md +++ b/architecture.md @@ -123,6 +123,11 @@ backend/ CI / runner-up gap / late-trial 1σ / convergence regime / per-query outcome helpers; pure-Python orchestrator returning None on every FR-7 degraded path); + study/followups.py (feat_digest_executable_followups — + FollowupItem discriminated union (narrow/widen/text) + + parse_followup_list defensive ingest + serialize_followup_list + JSONB serializer; the worker validates LLM payloads through + this module, downgrading invalid narrow/widen items to text); git/{redaction,validation}.py from feat_github_pr_worker (GitHub PAT redaction + repo_url + config_path validators) adapters/ engine adapters — protocol.py (SearchAdapter Protocol + @@ -193,7 +198,13 @@ migrations/ Alembic config + versions/ (0001 baseline + 0002 clusters feat_data_table_primitive + 0014 clusters_target_filter from feat_cluster_target_filter + 0015 trials_per_query_metrics from feat_pr_metric_confidence — nullable JSONB column + - CHECK constraint enforcing IS NULL OR jsonb_typeof = 'object') + CHECK constraint enforcing IS NULL OR jsonb_typeof = 'object' + + 0018 studies_parent_proposal + 0019 digests_suggested_followups_jsonb + from feat_digest_executable_followups — paired + studies.parent_proposal_id/parent_proposal_followup_index + columns with CHECK + partial index + BEFORE DELETE trigger + + digests.suggested_followups column-type change to JSONB + via PL/pgSQL helper functions) docs/ 00_overview / 01_architecture / 02_product / 03_runbooks / 04_security / 05_quality / 08_guides ``` diff --git a/docs/01_architecture/api-conventions.md b/docs/01_architecture/api-conventions.md index b76a1301..ca6d7ddf 100644 --- a/docs/01_architecture/api-conventions.md +++ b/docs/01_architecture/api-conventions.md @@ -79,6 +79,14 @@ The studies endpoint surfaces two template-mismatch codes (added by `chore_creat | `JUDGMENT_TARGET_MISMATCH` | 422 | `judgment_list.target` does not equal the study's `target` on `POST /api/v1/studies` (added by `feat_study_target_judgment_mismatch_guard`, 2026-05-21). `retryable: false`. Fires AFTER the cluster check. Recovery: pick a judgment list authored against the study's target, or change the study's target. Catches the literal study2 incident — judgments authored on `e2e-target` paired with a study against `docs-articles` would otherwise burn the entire trial budget scoring 0.0 on every (params, query) pair. | | `INSUFFICIENT_JUDGMENT_OVERLAP` | 422 | `POST /api/v1/studies` create-time probe sampled up to `MAX_PROBED_DOCS=200` judged `doc_id`s from the first qid in the query set with any judgments (by `id ASC`); the count present in the study's target index was below `min(MIN_OVERLAP=3, max(judged_doc_count, 1))` (added by `feat_study_preflight_overlap_probe`, 2026-05-22). `retryable: false`. Recovery: regenerate judgments against the current index (most common cause: target index was rebuilt or `_reindex`'d with new doc IDs since the judgments were authored), or rebuild the index from the snapshot the judgments were authored on. Fires AFTER `JUDGMENT_TARGET_MISMATCH`. Probe is skipped (with WARN log `studies.preflight.overlap_probe.skipped`, `reason ∈ {unreachable, timeout, invalid_query_dsl}`) when the cluster is unreachable / probe times out / engine rejects the bare ids query — the orchestrator's per-trial failure handling catches those cases mid-flight. | +The studies endpoint also surfaces three new codes for the "Run this followup" lineage payload (added by `feat_digest_executable_followups`, 2026-05-24): + +| Code | HTTP Status | Meaning | +|---|---|---| +| `PROPOSAL_NOT_FOUND` | 404 | `POST /api/v1/studies` body carried a `parent.proposal_id` that does not exist. `retryable: false`. Caused by a stale UI session — refresh the proposal list. Distinct from the test-only `PROPOSAL_NOT_FOUND` below (this one fires on the public create-study path). | +| `DIGEST_NOT_FOUND` | 404 | `POST /api/v1/studies` body carried a `parent.proposal_id` whose parent study has no digest yet. `retryable: true` when the digest worker may still be running; `retryable: false` when the proposal is manual (manual proposals have no digest). | +| `FOLLOWUP_INDEX_OUT_OF_RANGE` | 422 | `POST /api/v1/studies` body carried a `parent.followup_index` that is ≥ the parsed `suggested_followups` list length on the parent digest. `retryable: false`. Almost always caused by a stale UI session where the operator's modal pointed at an index that was valid when the page rendered but is now out of bounds because the digest was re-generated (the `DELETE FROM digests …` escape hatch + re-enqueue per spec §19). Recovery: refresh the proposal page. | + ### Test-only endpoints (added by `chore_e2e_test_rows_isolation`, 2026-05-21) These error codes are emitted ONLY by the six `DELETE /api/v1/_test/*` endpoints, which are gated by `Settings.environment == "development"` and return 404 `RESOURCE_NOT_FOUND` outside dev. They exist solely so the Playwright `globalTeardown` can clean up E2E seed rows; operators never see them. diff --git a/docs/01_architecture/data-model.md b/docs/01_architecture/data-model.md index 3fcb1fee..9bca94b5 100644 --- a/docs/01_architecture/data-model.md +++ b/docs/01_architecture/data-model.md @@ -209,6 +209,8 @@ CREATE TABLE studies ( failed_reason TEXT, -- populated when status='failed' optuna_study_name TEXT NOT NULL UNIQUE, -- convention: optuna_study_name = str(studies.id) parent_study_id UUID REFERENCES studies(id), -- for forks (MVP2) + parent_proposal_id VARCHAR(36) REFERENCES proposals(id), -- feat_digest_executable_followups (0018) — set when this study was spawned from a digest "Run this followup" + parent_proposal_followup_index INT, -- 0-based index into the parent digest's suggested_followups; paired with parent_proposal_id (CHECK enforces both-NULL or both-set-with-index>=0); BEFORE DELETE trigger on proposals atomically NULLs both columns on parent hard-delete baseline_metric REAL, -- single non-Optuna trial run before Optuna starts; populated by orchestrator best_metric REAL, best_trial_id UUID, @@ -250,7 +252,7 @@ CREATE TABLE digests ( narrative TEXT NOT NULL, parameter_importance JSONB NOT NULL, recommended_config JSONB NOT NULL, - suggested_followups TEXT[], + suggested_followups JSONB NOT NULL DEFAULT '[]'::jsonb, -- feat_digest_executable_followups (0019) — was TEXT[]; now a JSONB array of FollowupItem dicts (`{kind, rationale, search_space}` per backend/app/domain/study/followups.py) generated_by TEXT NOT NULL, -- LLM model name + version -- lineage columns added at MVP2 generated_at TIMESTAMPTZ NOT NULL DEFAULT now() diff --git a/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md b/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md index 144441dd..1d74bd3c 100644 --- a/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md +++ b/docs/02_product/planned_features/feat_digest_executable_followups/implementation_plan.md @@ -1317,27 +1317,27 @@ Plan testing explicitly by layer and map to stories. ### Current sprint - [x] Story 1.1 — followups.py domain module -- [ ] Story 2.1 — worker structured-output schema + validator wiring -- [ ] Story 2.2 — LLM prompts -- [ ] Story 2.3 — worker integration test -- [ ] Story 3.1 — migration 0018 (studies columns + trigger) -- [ ] Story 3.2 — Study ORM update -- [ ] Story 3.3 — migration 0019 (digests JSONB type change) -- [ ] Story 3.4 — CHECK constraint integration test -- [ ] Story 3.5 — BEFORE DELETE trigger integration test -- [ ] Story 3.6 — migration round-trip integration test -- [ ] Story 4.1 — schema wire-shape + wrapper -- [ ] Story 4.2 — POST /studies parent body -- [ ] Story 5.1 — SuggestedFollowupsPanel rewrite -- [ ] Story 5.2 — Create-study modal prefill flow -- [ ] Story 5.3 — Glossary additions -- [ ] Story 6.1 — Playwright E2E happy path +- [x] Story 2.1 — worker structured-output schema + validator wiring +- [x] Story 2.2 — LLM prompts +- [x] Story 2.3 — worker integration test +- [x] Story 3.1 — migration 0018 (studies columns + trigger) +- [x] Story 3.2 — Study ORM update +- [x] Story 3.3 — migration 0019 (digests JSONB type change) +- [x] Story 3.4 — CHECK constraint integration test +- [x] Story 3.5 — BEFORE DELETE trigger integration test +- [x] Story 3.6 — migration round-trip integration test +- [x] Story 4.1 — schema wire-shape + wrapper +- [x] Story 4.2 — POST /studies parent body +- [x] Story 5.1 — SuggestedFollowupsPanel rewrite +- [x] Story 5.2 — Create-study modal prefill flow +- [x] Story 5.3 — Glossary additions +- [x] Story 6.1 — Playwright E2E happy path ### Blocked items - None. ### Done this sprint -- (none yet) +- All 16 stories complete (2026-05-24). --- diff --git a/state.md b/state.md index 2e0b9e02..11f0139a 100644 --- a/state.md +++ b/state.md @@ -12,7 +12,7 @@ - **Active feature:** none in flight (PR #223 closed `feat_auto_followup_studies` on 2026-05-24 — 33rd MVP1-era artifact merged. Operator-controlled cross-study compounding via `studies.config.auto_followup_depth`; chain of up to 5 auto-enqueued follow-up studies, each narrowing the search space around the prior winner. Re-uses existing `studies.parent_study_id` self-FK so **no schema migration**. New `enqueue_followup_study` Arq job + cancel cascade endpoint extension + new `GET /studies/{id}/children` endpoint + chain panel + wizard depth selector + cascade radio in cancel modal. Node engines bumped to `>=22`. 4 new tangential ideas captured during the work: `chore_auto_followup_completed_parent_stop_chain_race` (P3 — cycle 1 phase-gate F2), `chore_auto_followup_e2e_chain_seed_helper` (P3 — final-review F1 deferred subset; needs new `/api/v1/_test/auto-followup/seed-chain` endpoint to seed 3-node chains for full E2E coverage). Only this finalization docs PR remains.). Prior: none in flight (PR #221 closed `chore_dashboard_pr_extraction_from_idea` on 2026-05-23 — third MVP1.0-cleanup chore from the operator's stated "finish MVP1.0 before MVP1.5" sweep. Build-script-only refactor; extends `_extract_pr_number` with idea-aware extraction so legacy idea-only implemented folders surface their PRs in the dashboard's Status column; also adds a one-liner fallback per Gemini cycle 1 that materially improves rendering of many idea-only rows. Only this finalization docs PR remains. New tangential `chore_dashboard_regen_quoted_pr_false_positive` (P3) captures the pre-existing priority-3 fuzzy-regex weakness for future hardening. Remaining MVP1.0 backlog per dashboard Idea table: 3 P2 chores + 1 Backlog chore + 1 P3 (`chore_studies_post_arq_spy_fixture`, `chore_template_library_expansion`, `chore_e2e_seed_acme_idea_obsolete`, plus the new `chore_dashboard_regen_quoted_pr_false_positive` at P3, plus `chore_e2e_seed_acme_helper_dead` at Backlog).). Prior: none in flight (PR #219 closed `chore_migration_test_head_brittleness` on 2026-05-23 — second MVP1.0-cleanup chore from the operator's stated "finish MVP1.0 before MVP1.5" sweep. Test-only refactor; eliminates the recurring 2-lines-per-migration sympathy-edit tax on `backend/tests/integration/test_migrations.py`. Only this finalization docs PR remains. Remaining MVP1.0 backlog per dashboard Idea table: 4 P2 chores + 1 Backlog chore (`chore_dashboard_pr_extraction_from_idea`, `chore_e2e_seed_acme_idea_obsolete` newly captured during this chore's tangential sweep, `chore_studies_post_arq_spy_fixture`, `chore_template_library_expansion`, `chore_e2e_seed_acme_helper_dead` Backlog — though the last one is itself obsolete per the new tangential capture).). Prior: none in flight (PR #215 closed `chore_study_default_stop_conditions` on 2026-05-23 — first MVP1.0-cleanup chore from the operator's stated "finish MVP1.0 before MVP1.5" sweep; only this finalization docs PR remains. Remaining MVP1.0 backlog is 2 P2 chores + 1 Backlog chore.). Prior: none in flight (PR #213 closed `bug_dashboard_banner_dismiss_persistence_flake` on 2026-05-23 — fifth MVP1.0-cleanup bug, last `bug_*` item in the operator's MVP1.0 cleanup queue. Only this finalization docs PR remains. Remaining MVP1.0 backlog is 3 P2 chores + 1 Backlog chore.). Prior: none in flight (PR #211 closed `bug_dashboard_classifier_half_step_releases` on 2026-05-23 — fourth MVP1.0-cleanup bug; restores trust in `/pipeline status` priority ordering by routing MVP1.5 features to their own dashboard. Only this finalization docs PR remains. The MVP1_DASHBOARD.md's Idea-table top is now correctly an MVP1.0 item.). Prior: none in flight (PR #208 closed `bug_dashboard_depends_on_column_bloat` on 2026-05-23 — third MVP1.0-cleanup bug from the operator's stated "finish MVP1.0 before MVP1.5" sweep; only this finalization docs PR remains. Dashboard "Depends on" column now time-ordered for shipped features; `feat_chat_agent` 46→10 entries, `chore_tutorial_polish` 42→11. Side effect captured as [`chore_dashboard_pr_extraction_from_idea/idea.md`](docs/02_product/planned_features/chore_dashboard_pr_extraction_from_idea/idea.md) — legacy implemented features with only `idea.md` lose their PR# at extraction time, leaving ~1 missing edge per legacy feature in same-day peers' deps). Prior: none in flight (PR #206 closed `bug_contract_test_stub_missing_target_filter_kwarg` on 2026-05-23 — second MVP1.0-cleanup bug from the operator's stated "finish MVP1.0 before MVP1.5" sweep; only this finalization docs PR remains. Contract-test stubs now match the `SearchAdapter` Protocol's `target_filter` kwarg; 291 contract tests pass clean). Prior: none in flight (PR #204 closed `bug_pr_reconciler_blocked_by_closed_fallback` on 2026-05-23 — first MVP1.0-cleanup bug from the operator's stated "finish MVP1.0 before MVP1.5" sweep; only this finalization docs PR remains. Side effect captured as [`chore_reconciler_terminal_closed_no_poll/idea.md`](docs/02_product/planned_features/chore_reconciler_terminal_closed_no_poll/idea.md) — widened candidate query now polls genuinely-closed-unmerged proposals; bounded no-op but worth a `last_polled_at` polish layer). Prior: none in flight (PR #202 closed `feat_config_repo_baseline_tracking` on 2026-05-23 as the **32nd MVP1-era artifact** merged; only this finalization docs PR remains. Substrate for the downstream `feat_auto_followup_studies` work — tracks the most recently merged proposal per config_repo via a denormalized FK, exposed on ConfigRepoDetail + ProposalSummary + a proposals-page filter chip. Captured pre-existing reconciler bug `bug_pr_reconciler_blocked_by_closed_fallback` as a separate planned idea — out of scope for the merge; documented limitation surfaced in `webhook-debugging.md §8`). Prior: none in flight (PR #200 introduced the MVP1.5 / v0.1.5 "Real Signals" release tier on 2026-05-23 as a **planning artifact, not a shipped feature** — added `feat_ubi_judgments` (P1) and `bug_dashboard_depends_on_column_bloat` (P2) ideas + spec patches + canonical-matrix update; only the docs/finalize-mvp1-5-ubi-foundation finalization docs PR remains). Prior: none in flight (PR #198 closed `infra_ir_measures_migration` on 2026-05-23 as the **31st MVP1-era artifact** merged; only this finalization docs PR remains). Prior: none in flight (PR #195 closed `chore_guides_glossary_route` + `chore_guides_faq` + `chore_guide_06_screenshot_refresh_confidence_panel` on 2026-05-22; only this finalization docs PR remains. The three siblings shipped bundled per "one branch, one PR" memory). Prior: none in flight (PR #193 closed `feat_study_preflight_overlap_probe` on 2026-05-22 as the **27th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #191 closed `feat_orchestrator_zero_streak_abort` on 2026-05-22 as the **26th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #188 closed `feat_home_first_run_demo_nudge` on 2026-05-22 as the **25th MVP1 feature** merged; only finalization docs PR remains. Phase 2 reseed-endpoint work captured in [`feat_home_demo_reseed_endpoint/idea.md`](docs/02_product/planned_features/feat_home_demo_reseed_endpoint/idea.md)). Prior: none in flight (PR #186 closed `chore_e2e_test_rows_isolation` on 2026-05-21 as the **24th MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #184 closed `feat_study_target_judgment_mismatch_guard` on 2026-05-21 as the **23rd MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #180 closed `feat_pr_metric_confidence` on 2026-05-21 as the **22nd MVP1 feature** merged; only finalization docs PR remains). Prior: none in flight (PR #175 closed `feat_agent_propose_search_space` on 2026-05-21; only finalization docs PR remains for the 21st MVP1 feature). Prior — none in flight (PR #168 closed `feat_cluster_target_filter` + PR #169 closed `chore_seed_meaningful_demos` on 2026-05-20; only finalization docs PR remains for the 20th MVP1 feature). Prior — none in flight (PR #165 closed `feat_create_study_target_autocomplete` + the bundled `bug_get_schema_unhandled_connect_error` fix on 2026-05-20). Prior — none in flight (PR #163 closed `feat_create_study_search_space_builder` + the `bug_judgment_lists_listing_ignores_query_set_filter` bundled fix on 2026-05-20). PR #168 closed `feat_cluster_target_filter` + PR #169 closed `chore_seed_meaningful_demos` (sibling). **Three PRs shipped 2026-05-15:** PR #122 (Phase 1, 16th MVP1 feature — Tooltip primitive + 26 placements on create-study modal + study detail), PR #123 (Phase 1 finalization docs), PR #124 (Phases 2 + 3 — 17th MVP1 feature; 21 additional tooltips on judgments + proposals + cluster registration + 2 new first-run components: chat ExamplePrompts strip + Stripe-style StartHereChecklist on home page). The original "MVP1 Phase 1 only" scope-lock was reversed mid-day: operator decided to ship Phases 2 + 3 together with a Stripe-style design call rather than wait for MVP2. PR #124 took 2 hours from idea-folder reuse to merge. 47 total tooltip placements + 2 new first-run components live in `main`. **PR #122 shipped 2026-05-15 morning** — `feat_contextual_help` Phase 1 (16th MVP1 feature). Adds the first Tooltip primitive (`@radix-ui/react-tooltip@~1.2.8` + shadcn-style wrapper at `ui/src/components/ui/tooltip.tsx`), two glossary-backed wrappers (`InfoTooltip` standalone + asChild modes; `HelpPopover` click-to-open with `react-markdown` safety filter), and a 49-key glossary source-of-truth at `ui/src/lib/glossary.ts` (8 enum groups parity-tested against `enums.ts`). 26 tooltip placements across the create-study modal (Step 1 target + Step 3 template + 9 Step 5 inputs), study-header (status badge dynamic key + Best metric + Trials), trials-table (5 column headers + Sort label), and digest panel (5 section labels + Open PR enabled + Open PR disabled). The disabled Open PR button refactored from native `disabled` to `aria-disabled="true"` so it stays focusable and the tooltip reveals on focus (AC-11). Gemini Code Assist: 2 findings (1 accepted + fixed, 1 rejected with cited counter-evidence). Final GPT-5.5 review: 1 Medium accepted-framing-but-deferred. Spec converged at GPT-5.5 cycle 3 (24 findings, 23 accepted + 1 rejected); plan converged at cycle 2 (12 findings, 10 accepted + 1 rejected + 1 spec patch). UI vitest now **279 passing across 48 files** (was 249 across 45 — +3 new test files, +30 cases). Playwright E2E **8 passing** (was 5 — +3 new contextual-help tests). One follow-up filed: `infra_e2e_seed_completed_study/idea.md` tracks the E2E gap for digest-panel triggers + AC-11 (cross-subsystem helper for seeding a completed study with digest + proposal; component-level coverage is in place). Phases 2 + 3 deferred to MVP2 via `feat_contextual_help_mvp2/` (judgments + proposals tooltips; chat + cluster + home onboarding; the home-page "Start here" panel is the only product-design-shaped item). **Earlier — seven PRs shipped 2026-05-14:** `feat_judgments_periodic_resume_sweep` (PR #104, 14th MVP1 feature), `bug_query_inline_crud_since_filter_uuidv7_ms_collision` (PR #106 — UUIDv7 ms-collision test flake), `infra_dashboard_regen_pre_commit_conflict §2+§4` (PR #108 — dashboard regen idempotency + relative-link rewriting), `infra_make_targets_split_backend_only` (PR #110 — `make backend-fmt/lint/typecheck` + symmetric `ui-fmt` so Node-18 contributors aren't blocked), `chore_digest_worker_narrow_except` (PR #112 — narrowed `except Exception` allowlist to `(ValueError,)` + ERROR-level `digest_importance_failed_unexpected` event), `infra_structlog_test_helpers` (PR #114 — factored the two structlog test-assertion patterns into `backend/tests/_log_helpers.py`), and `chore_chat_last_message_preview` (PR #117 — `last_message_preview` + `last_message_at` on `ConversationSummary` via LATERAL JOIN; frontend shows preview under title + swaps displayed timestamp from `created_at` to `last_message_at`). Plus PR #116 dropped `chore_studies_ui_shadcn_polish` as won't-do (forward-compat audit on NavigationMenu primitive + ClusterFilterSelect precedent on native `