From f49c94b6087141e56b962e9793d725b6ab4c53d1 Mon Sep 17 00:00:00 2001 From: SoundMindsAI Date: Mon, 1 Jun 2026 19:34:18 -0400 Subject: [PATCH 1/9] =?UTF-8?q?docs(planned-features):=20infra=5Fsolr=5Fsm?= =?UTF-8?q?oke=5Fstability=20=E2=80=94=20idea=20preflight=20+=20spec=20+?= =?UTF-8?q?=20plan=20+=20pipeline=5Fstatus=20+=20dashboards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Planning artifacts for the smoke-job Solr stability work (Phase 2 of infra_solr_ci_readiness, extracted to its own standalone folder per PR #381). - idea.md: preflighted (5 patches applied via /idea-preflight before /pipeline) - feature_spec.md: 4 FRs, 5 ACs, 6 decisions; GPT-5.5 reviewed 3 cycles (15 -> 16 -> 10 findings, all accepted) - implementation_plan.md: 7 stories across 2 epics (6 in feature PR + 1 in follow-up state.md PR); GPT-5.5 reviewed 3 cycles (13 -> 10 -> 6 findings, all accepted) - pipeline_status.md: Spec + Plan stages approved - dashboards: regenerated by pre-commit hook to surface the new folder Signed-off-by: SoundMindsAI --- docs/00_overview/DASHBOARD.md | 2 +- docs/00_overview/MVP2_DASHBOARD.md | 74 +- docs/00_overview/dashboard.html | 2 +- docs/00_overview/mvp2_dashboard.html | 44 +- .../feature_spec.md | 327 ++++++++ .../infra_solr_smoke_stability/idea.md | 19 +- .../implementation_plan.md | 770 ++++++++++++++++++ .../pipeline_status.md | 24 + 8 files changed, 1196 insertions(+), 66 deletions(-) create mode 100644 docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/feature_spec.md create mode 100644 docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/implementation_plan.md create mode 100644 docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/pipeline_status.md diff --git a/docs/00_overview/DASHBOARD.md b/docs/00_overview/DASHBOARD.md index 6df1ea26..18d15b1c 100644 --- a/docs/00_overview/DASHBOARD.md +++ b/docs/00_overview/DASHBOARD.md @@ -7,7 +7,7 @@ _Top-level index across MVP1 → GA v1+ as of **2026-06-01**. Click a release na | Release | Theme | Progress | Status | |---|---|---|---| | [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 94 / 94 scoped done | **Complete** | -| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 8 / 17 scoped done · 27 remaining | **In progress** | +| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 8 / 18 scoped done · 28 remaining | **In progress** | | MVP3 / v0.3 | Observable | — | **Not yet scoped** | | GA v1 / v1.0 | Production-ready | — | **Not yet scoped** | diff --git a/docs/00_overview/MVP2_DASHBOARD.md b/docs/00_overview/MVP2_DASHBOARD.md index 5b408862..e3a8506a 100644 --- a/docs/00_overview/MVP2_DASHBOARD.md +++ b/docs/00_overview/MVP2_DASHBOARD.md @@ -21,15 +21,15 @@ Plan approved; run /impl-execute to ship | Metric | Value | |---|---| | Filed under MVP2 | **39** folders total (done + specced not-done + idea backlog + bugs) | -| Specced features done | **8 / 17** (47%) — of features *past the idea stage* (those with a spec); the idea backlog below is NOT in this denominator, so 100% ≠ release complete | +| Specced features done | **8 / 18** (44%) — of features *past the idea stage* (those with a spec); the idea backlog below is NOT in this denominator, so 100% ≠ release complete | | Pending work | **31** items (every not-done feat/infra/chore/bug across all priorities) | | → P0 — do next | **0** unblocking / paying daily cost | | → P1 | **2** high-value, ready when P0 clears | | → P2 (default) | 25 important to file, not blocking | | → Backlog | 4 captured for record, not planned | | Open bugs | 10 | -| Legacy "Path to MVP2" | 27 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | -| Backlog ideas | 4 idea-only feat/infra (not yet scoped into MVP2) | +| Legacy "Path to MVP2" | 28 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) | +| Backlog ideas | 3 idea-only feat/infra (not yet scoped into MVP2) | | In flight | 0 feature(s) actively shipping | ## Pipeline @@ -51,50 +51,50 @@ Plan approved; run /impl-execute to ship _None._ -### Plan (9) +### Plan (10) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P2 | [feat_apply_path_normalizer_declaration](planned_features/02_mvp2/feat_apply_path_normalizer_declaration/feature_spec.md) | Feature | The winning normalizer ships as a **structured, language-agnostic manifest** in the config-repo PR — not just prose. | — | — | -| 2 | P2 | [feat_overnight_studies_summary_card](planned_features/02_mvp2/feat_overnight_studies_summary_card/feature_spec.md) | Feature | A "ran while you were away" card surfaces at the top of `/studies` when at least one overnight chain has completed since the operator's last visit. | — | [PR #343](https://github.com/SoundMindsAI/relyloop/pull/343) | -| 3 | P2 | [feat_query_normalization_tuning](planned_features/02_mvp2/feat_query_normalization_tuning/feature_spec.md) | Feature | A template that opts in by declaring `query_normalizer` as a Categorical param gets the Optuna loop deciding empirically — on the operator's judgment set — whether lowercasing, trimming, or contractio | — | — | -| 4 | P2 | [feat_query_normalizer_typed_pipeline](planned_features/02_mvp2/feat_query_normalizer_typed_pipeline/feature_spec.md) | Feature | A new typed search-space member `NormalizerPipelineParam` lets a template declare an **ordered list of normalization steps**; the Optuna loop samples over the powerset of declared steps and proposes t | — | — | -| 5 | P2 | [feat_ubi_llm_study_comparison](planned_features/02_mvp2/feat_ubi_llm_study_comparison/feature_spec.md) | Feature | A single dedicated route `/studies/compare?a={id}&b={id}` renders the two studies side-by-side with a per-panel diff column: a sentence-level digest-narrative diff, a best-trial parameter table with s | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) | -| 6 | P2 | [infra_generated_artifact_freshness_gate](planned_features/02_mvp2/infra_generated_artifact_freshness_gate/feature_spec.md) | Infra | CI fails a PR whose committed `types.ts` does not match what the live OpenAPI schema would produce, and whose `ui/public/docs/*` copies do not match their `docs/08_guides/*` sources. | — | — | -| 7 | P2 | [chore_arq_pool_aclose_deprecation](planned_features/02_mvp2/chore_arq_pool_aclose_deprecation/feature_spec.md) | Chore | Both call sites use `await arq_pool.aclose()`; no `DeprecationWarning` on shutdown; a regression guard asserts the async-correct form on both paths so a future edit cannot silently reintroduce `close( | — | — | -| 8 | P2 | [chore_cluster_detail_rung_badge](planned_features/02_mvp2/chore_cluster_detail_rung_badge/feature_spec.md) | Chore | The cluster-detail page surfaces a `` for the cluster, scoped by a user-selected (or auto-seeded) query set + target. | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) | -| 9 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/feature_spec.md) | Chore | The 9 skipped cases are rewritten to the async "POST + poll-until-terminal" shape, the timeout case is re-homed to the worker layer, a new `AC-Async` case asserts the `running → complete` polling tran | — | [PR #286](https://github.com/SoundMindsAI/relyloop/pull/286) | +| 1 | P1 | [infra_solr_smoke_stability](planned_features/02_mvp2/infra_solr_smoke_stability/feature_spec.md) | Infra | The `pr.yml` `smoke (operator-path tutorial flow)` job is red on every branch. | — | [PR #367](https://github.com/SoundMindsAI/relyloop/pull/367) merged 2026-06-01 | +| 2 | P2 | [feat_apply_path_normalizer_declaration](planned_features/02_mvp2/feat_apply_path_normalizer_declaration/feature_spec.md) | Feature | The winning normalizer ships as a **structured, language-agnostic manifest** in the config-repo PR — not just prose. | — | — | +| 3 | P2 | [feat_overnight_studies_summary_card](planned_features/02_mvp2/feat_overnight_studies_summary_card/feature_spec.md) | Feature | A "ran while you were away" card surfaces at the top of `/studies` when at least one overnight chain has completed since the operator's last visit. | — | [PR #343](https://github.com/SoundMindsAI/relyloop/pull/343) | +| 4 | P2 | [feat_query_normalization_tuning](planned_features/02_mvp2/feat_query_normalization_tuning/feature_spec.md) | Feature | A template that opts in by declaring `query_normalizer` as a Categorical param gets the Optuna loop deciding empirically — on the operator's judgment set — whether lowercasing, trimming, or contractio | — | — | +| 5 | P2 | [feat_query_normalizer_typed_pipeline](planned_features/02_mvp2/feat_query_normalizer_typed_pipeline/feature_spec.md) | Feature | A new typed search-space member `NormalizerPipelineParam` lets a template declare an **ordered list of normalization steps**; the Optuna loop samples over the powerset of declared steps and proposes t | — | — | +| 6 | P2 | [feat_ubi_llm_study_comparison](planned_features/02_mvp2/feat_ubi_llm_study_comparison/feature_spec.md) | Feature | A single dedicated route `/studies/compare?a={id}&b={id}` renders the two studies side-by-side with a per-panel diff column: a sentence-level digest-narrative diff, a best-trial parameter table with s | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) | +| 7 | P2 | [infra_generated_artifact_freshness_gate](planned_features/02_mvp2/infra_generated_artifact_freshness_gate/feature_spec.md) | Infra | CI fails a PR whose committed `types.ts` does not match what the live OpenAPI schema would produce, and whose `ui/public/docs/*` copies do not match their `docs/08_guides/*` sources. | — | — | +| 8 | P2 | [chore_arq_pool_aclose_deprecation](planned_features/02_mvp2/chore_arq_pool_aclose_deprecation/feature_spec.md) | Chore | Both call sites use `await arq_pool.aclose()`; no `DeprecationWarning` on shutdown; a regression guard asserts the async-correct form on both paths so a future edit cannot silently reintroduce `close( | — | — | +| 9 | P2 | [chore_cluster_detail_rung_badge](planned_features/02_mvp2/chore_cluster_detail_rung_badge/feature_spec.md) | Chore | The cluster-detail page surfaces a `` for the cluster, scoped by a user-selected (or auto-seeded) query set + target. | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) | +| 10 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/feature_spec.md) | Chore | The 9 skipped cases are rewritten to the async "POST + poll-until-terminal" shape, the timeout case is re-homed to the worker layer, a new `AC-Async` case asserts the `running → complete` polling tran | — | [PR #286](https://github.com/SoundMindsAI/relyloop/pull/286) | ### Spec (0) _None._ -### Idea (22) +### Idea (21) | # | Priority | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---|---|---| -| 1 | P1 | [infra_solr_smoke_stability](planned_features/02_mvp2/infra_solr_smoke_stability/idea.md) | Infra | After Phase 1 ships, the `pr.yml` `backend` job goes green (the heavy-lane reseed test no longer requires Solr). But the `smoke-test` job is independent — it runs `make up` which brings up the full Co | — | Idea — deferred from `infra_solr_ci_readiness` Phase 1 because the right stabilization lever depends on log evidence from a smoke-runner failure | -| 2 | P1 | [bug_backend_suite_nondeterministic_caplog_isolation](planned_features/02_mvp2/bug_backend_suite_nondeterministic_caplog_isolation/idea.md) | Bug | Many backend unit tests assert on captured log records (`caplog` / a structlog capture fixture) and fail with empty-capture shapes (`assert []`, `assert 'x' in []`) when run in the full randomized sui | — | Idea — bug surfaced by CI on PR #363 | -| 3 | P2 | [infra_openapi_types_freshness_gate](planned_features/02_mvp2/infra_openapi_types_freshness_gate/idea.md) | Infra | Idea — Phase 2 of [`infra_generated_artifact_freshness_gate`](../infra_generated_artifact_freshness_gate/feature_spec.md), extracted to its own folder | — | Idea — Phase 2 of [`infra_generated_artifact_freshness_gate`](../infra_generated_artifact_freshness_gate/feature_spec.md), extracted to its own folder | -| 4 | P2 | [chore_demo_reseed_partial_completion_fast_test](planned_features/02_mvp2/chore_demo_reseed_partial_completion_fast_test/idea.md) | Chore | `infra_solr_ci_readiness` made the demo reseed engine-tolerant: when an engine is unreachable, its scenario is skipped, the reseed completes with `status="complete"` and a non-empty `scenarios_skipped | — | Idea — tangential discovery during `infra_solr_ci_readiness` Story 1.2 implementation | -| 5 | P2 | [chore_solr_cred_backfill_needs_api_restart](planned_features/02_mvp2/chore_solr_cred_backfill_needs_api_restart/idea.md) | Chore | `scripts/install.sh` step 5a backfills a `local-solr:` entry into a **pre-existing** `./secrets/cluster_credentials.yaml`. That write is correct and idempotent. But the `api` (and `worker`) process re | — | — | -| 6 | P2 | [chore_solr_post_pipeline_followups](planned_features/02_mvp2/chore_solr_post_pipeline_followups/idea.md) | Chore | The 13-story `infra_adapter_solr` execution surfaced several follow-on items that fit neither the original spec nor any sister feature folder. None block the MVP2 Solr release — they're operator-exper | — | Idea — tangential observations from `infra_adapter_solr` end-to-end | -| 7 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | -| 8 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | -| 9 | P2 | [chore_ubi_hybrid_template_render](planned_features/02_mvp2/chore_ubi_hybrid_template_render/idea.md) | Chore | Idea — contract decision deferred (NOT a worker bug) | — | Idea — contract decision deferred (NOT a worker bug) | -| 10 | P2 | [chore_ubi_reader_search_after_pagination](planned_features/02_mvp2/chore_ubi_reader_search_after_pagination/idea.md) | Chore | `UbiReader._scan_ubi_events` / `_scan_ubi_queries` each issue ONE `search_batch` (a `size`-limited query). To stay under the engine result-window they now cap at 10000 rows per (target, window).… | — | Idea — deferred from `feat_ubi_judgments` (found during the rung-3 E2E) | -| 11 | P2 | [bug_baseline_phase_test_isolation](planned_features/02_mvp2/bug_baseline_phase_test_isolation/idea.md) | Bug | `backend/tests/unit/workers/test_orchestrator_baseline_phase.py::TestComputeBaselineWaitS::*` fail when run in isolation with: | — | Idea — pre-existing bug surfaced during `feat_ubi_judgments` PR #317 | -| 12 | P2 | [bug_contract_allowlists_outdated_after_mvp2_features](planned_features/02_mvp2/bug_contract_allowlists_outdated_after_mvp2_features/idea.md) | Bug | Three separate contract-test allowlists were not updated as features shipped through MVP2. Each is a "hand-maintained canonical list of valid values" that drifts when a feature adds new entries to the | — | Idea — tangential discovery during `feat_study_convergence_indicator` pre-push gate | -| 13 | P2 | [bug_e2e_teardown_chain_node_delete_500](planned_features/02_mvp2/bug_e2e_teardown_chain_node_delete_500/idea.md) | Bug | The E2E global-teardown deletes seeded rows in a fixed order (per `chore_e2e_test_rows_isolation` Story 1.2 cleanup registration). For auto-followup **chains**, the seeded nodes are `queued` studies c | — | Idea — tangential discovery during `feat_overnight_autopilot` (Story 4.2 E2E, PR forthcoming) | -| 14 | P2 | [bug_judgment_header_omits_click_bucket](planned_features/02_mvp2/bug_judgment_header_omits_click_bucket/idea.md) | Bug | A judgment list generated from UBI (or hybrid UBI+LLM) carries non-zero `source_breakdown.click`, but the header's source-breakdown card silently drops it. Operators reviewing a UBI list see only the | — | Idea — tangential discovery during `feat_overnight_autopilot` (Story 2.1, PR forthcoming) | -| 15 | P2 | [bug_relyloop_spec_ubi_section_drift](planned_features/02_mvp2/bug_relyloop_spec_ubi_section_drift/idea.md) | Bug | [`docs/00_overview/relyloop-spec.md`](relyloop-spec.md) §"Click-derived judgments — OpenSearch UBI as the engine-neutral primary path" (line ~706) carries two staleness bugs from the 2026-05-27 releas | — | Idea — captured during `feat_ubi_judgments` preflight (2026-05-29) | -| 16 | P2 | [bug_reseed_failure_blocks_retry_arq_singleton_dedup](planned_features/02_mvp2/bug_reseed_failure_blocks_retry_arq_singleton_dedup/idea.md) | Bug | `run_demo_reseed` is enqueued with a fixed Arq job id `demo_reseed:singleton` (the singleton concurrency guard). When a run reaches a terminal state, Arq stores its **result** under `arq:result:demo_r | — | Idea — tangential discovery while verifying `fix(demo): add Solr (8983) to the reseed engine host-URL mapping` (branch `feat_demo_reseed_solr_and_steplog`) | -| 17 | P2 | [bug_seed_meaningful_demos_silent_bulk_errors](planned_features/02_mvp2/bug_seed_meaningful_demos_silent_bulk_errors/idea.md) | Bug | [`scripts/seed_meaningful_demos.py:917-935`](../../scripts/seed_meaningful_demos.py#L917-L935) bulk-indexes 1000 Amazon ESCI products into a dedicated index per demo scenario: | — | Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep | -| 18 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | -| 19 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | -| 20 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | -| 21 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | -| 22 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | +| 1 | P1 | [bug_backend_suite_nondeterministic_caplog_isolation](planned_features/02_mvp2/bug_backend_suite_nondeterministic_caplog_isolation/idea.md) | Bug | Many backend unit tests assert on captured log records (`caplog` / a structlog capture fixture) and fail with empty-capture shapes (`assert []`, `assert 'x' in []`) when run in the full randomized sui | — | Idea — bug surfaced by CI on PR #363 | +| 2 | P2 | [infra_openapi_types_freshness_gate](planned_features/02_mvp2/infra_openapi_types_freshness_gate/idea.md) | Infra | Idea — Phase 2 of [`infra_generated_artifact_freshness_gate`](../infra_generated_artifact_freshness_gate/feature_spec.md), extracted to its own folder | — | Idea — Phase 2 of [`infra_generated_artifact_freshness_gate`](../infra_generated_artifact_freshness_gate/feature_spec.md), extracted to its own folder | +| 3 | P2 | [chore_demo_reseed_partial_completion_fast_test](planned_features/02_mvp2/chore_demo_reseed_partial_completion_fast_test/idea.md) | Chore | `infra_solr_ci_readiness` made the demo reseed engine-tolerant: when an engine is unreachable, its scenario is skipped, the reseed completes with `status="complete"` and a non-empty `scenarios_skipped | — | Idea — tangential discovery during `infra_solr_ci_readiness` Story 1.2 implementation | +| 4 | P2 | [chore_solr_cred_backfill_needs_api_restart](planned_features/02_mvp2/chore_solr_cred_backfill_needs_api_restart/idea.md) | Chore | `scripts/install.sh` step 5a backfills a `local-solr:` entry into a **pre-existing** `./secrets/cluster_credentials.yaml`. That write is correct and idempotent. But the `api` (and `worker`) process re | — | — | +| 5 | P2 | [chore_solr_post_pipeline_followups](planned_features/02_mvp2/chore_solr_post_pipeline_followups/idea.md) | Chore | The 13-story `infra_adapter_solr` execution surfaced several follow-on items that fit neither the original spec nor any sister feature folder. None block the MVP2 Solr release — they're operator-exper | — | Idea — tangential observations from `infra_adapter_solr` end-to-end | +| 6 | P2 | [chore_studies_post_arq_spy_fixture](planned_features/02_mvp2/chore_studies_post_arq_spy_fixture/idea.md) | Chore | The studies POST handler at [`backend/app/api/v1/studies.py:307`](../../backend/app/api/v1/studies.py#L307) calls `await _enqueue_start_study(request, study_id)` after a successful create. The helper | — | Idea — surfaced during `feat_study_preflight_overlap_probe` (PR ___) phase-gate review | +| 7 | P2 | [chore_template_library_expansion](planned_features/02_mvp2/chore_template_library_expansion/idea.md) | Chore | Three connected gaps: | — | Idea — surfaced during a UX review of parameter-tuning ergonomics on 2026-05-19. | +| 8 | P2 | [chore_ubi_hybrid_template_render](planned_features/02_mvp2/chore_ubi_hybrid_template_render/idea.md) | Chore | Idea — contract decision deferred (NOT a worker bug) | — | Idea — contract decision deferred (NOT a worker bug) | +| 9 | P2 | [chore_ubi_reader_search_after_pagination](planned_features/02_mvp2/chore_ubi_reader_search_after_pagination/idea.md) | Chore | `UbiReader._scan_ubi_events` / `_scan_ubi_queries` each issue ONE `search_batch` (a `size`-limited query). To stay under the engine result-window they now cap at 10000 rows per (target, window).… | — | Idea — deferred from `feat_ubi_judgments` (found during the rung-3 E2E) | +| 10 | P2 | [bug_baseline_phase_test_isolation](planned_features/02_mvp2/bug_baseline_phase_test_isolation/idea.md) | Bug | `backend/tests/unit/workers/test_orchestrator_baseline_phase.py::TestComputeBaselineWaitS::*` fail when run in isolation with: | — | Idea — pre-existing bug surfaced during `feat_ubi_judgments` PR #317 | +| 11 | P2 | [bug_contract_allowlists_outdated_after_mvp2_features](planned_features/02_mvp2/bug_contract_allowlists_outdated_after_mvp2_features/idea.md) | Bug | Three separate contract-test allowlists were not updated as features shipped through MVP2. Each is a "hand-maintained canonical list of valid values" that drifts when a feature adds new entries to the | — | Idea — tangential discovery during `feat_study_convergence_indicator` pre-push gate | +| 12 | P2 | [bug_e2e_teardown_chain_node_delete_500](planned_features/02_mvp2/bug_e2e_teardown_chain_node_delete_500/idea.md) | Bug | The E2E global-teardown deletes seeded rows in a fixed order (per `chore_e2e_test_rows_isolation` Story 1.2 cleanup registration). For auto-followup **chains**, the seeded nodes are `queued` studies c | — | Idea — tangential discovery during `feat_overnight_autopilot` (Story 4.2 E2E, PR forthcoming) | +| 13 | P2 | [bug_judgment_header_omits_click_bucket](planned_features/02_mvp2/bug_judgment_header_omits_click_bucket/idea.md) | Bug | A judgment list generated from UBI (or hybrid UBI+LLM) carries non-zero `source_breakdown.click`, but the header's source-breakdown card silently drops it. Operators reviewing a UBI list see only the | — | Idea — tangential discovery during `feat_overnight_autopilot` (Story 2.1, PR forthcoming) | +| 14 | P2 | [bug_relyloop_spec_ubi_section_drift](planned_features/02_mvp2/bug_relyloop_spec_ubi_section_drift/idea.md) | Bug | [`docs/00_overview/relyloop-spec.md`](relyloop-spec.md) §"Click-derived judgments — OpenSearch UBI as the engine-neutral primary path" (line ~706) carries two staleness bugs from the 2026-05-27 releas | — | Idea — captured during `feat_ubi_judgments` preflight (2026-05-29) | +| 15 | P2 | [bug_reseed_failure_blocks_retry_arq_singleton_dedup](planned_features/02_mvp2/bug_reseed_failure_blocks_retry_arq_singleton_dedup/idea.md) | Bug | `run_demo_reseed` is enqueued with a fixed Arq job id `demo_reseed:singleton` (the singleton concurrency guard). When a run reaches a terminal state, Arq stores its **result** under `arq:result:demo_r | — | Idea — tangential discovery while verifying `fix(demo): add Solr (8983) to the reseed engine host-URL mapping` (branch `feat_demo_reseed_solr_and_steplog`) | +| 16 | P2 | [bug_seed_meaningful_demos_silent_bulk_errors](planned_features/02_mvp2/bug_seed_meaningful_demos_silent_bulk_errors/idea.md) | Bug | [`scripts/seed_meaningful_demos.py:917-935`](../../scripts/seed_meaningful_demos.py#L917-L935) bulk-indexes 1000 Amazon ESCI products into a dedicated index per demo scenario: | — | Idea — captured during `bug_smoke_seed_es_unavailable_shards_race` Phase 2.5 tangential sweep | +| 17 | P2 | [bug_webhook_concurrent_merge_race_timing_sensitive](planned_features/02_mvp2/bug_webhook_concurrent_merge_race_timing_sensitive/idea.md) | Bug | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | — | Idea — surfaced during `bug_demo_clusters_unreachable_in_healthz` PR #236 CI. | +| 18 | Backlog | [feat_fts_rank_ordering](planned_features/02_mvp2/feat_fts_rank_ordering/idea.md) | Feature | `feat_data_table_primitive` shipped filter-only FTS — `?q=foo` matches rows where `search_vector @@ plainto_tsquery('english', 'foo')` is true but orders results by `created_at DESC, id DESC` (the def | — | Idea — deferred from `feat_data_table_primitive` (MVP1) per spec §16. | +| 19 | Backlog | [infra_arq_subprocess_test](planned_features/02_mvp2/infra_arq_subprocess_test/idea.md) | Infra | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; | — | Idea (deferred from `feat_study_lifecycle` Phase 2 / PR #25 final GPT-5.5 review). Still applicable as of 2026-05-14: the three in-process tests cited below still cover the resume contract correctly; a subprocess test would add a narrow Arq-version-regression guard. | +| 20 | Backlog | [chore_auto_followup_parent_advisory_lock](planned_features/02_mvp2/chore_auto_followup_parent_advisory_lock/idea.md) | Chore | The shipped `feat_auto_followup_studies` worker uses a two-layer idempotency scheme: | — | Idea — captured as a standalone file to resolve broken cross-references in `feat_auto_followup_studies` D-11 + plan F2 + `bug_auto_followup_completed_parent_stop_chain_race/idea.md`. The slug was coined 2026-05-24 in D-11 but only existed as descriptive prose across other documents until now. | +| 21 | Backlog | [bug_chat_long_conversation_truncation](planned_features/02_mvp2/bug_chat_long_conversation_truncation/idea.md) | Bug | [`backend/app/services/agent_chat.send_user_message`](../../backend/app/services/agent_chat.py) defensively caps the OpenAI history at the most recent `HISTORY_MAX_MESSAGES = 100` messages… | — | Held for MVP2 (decided 2026-05-13). Folder renamed with `_mvp2` suffix to make the deferral visible at-a-glance in `ls docs/00_overview/planned_features/`. Resume work when MVP2 starts — no technical dependency on MVP2 infra (audit_log is N/A; Langfuse is convenience only); the deferral is scope discipline + zero current impact (latent bug, no operator has hit the 100-message cap). | ## Dependency graph @@ -125,6 +125,8 @@ graph LR class feat_ubi_llm_study_comparison plan; infra_generated_artifact_freshness_gate["generated artifact freshness gate"] class infra_generated_artifact_freshness_gate plan; + infra_solr_smoke_stability["solr smoke stability"] + class infra_solr_smoke_stability plan; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done; feat_study_sub_warmup_guard["study sub warmup guard"] diff --git a/docs/00_overview/dashboard.html b/docs/00_overview/dashboard.html index 7cbbaf9e..17d5a557 100644 --- a/docs/00_overview/dashboard.html +++ b/docs/00_overview/dashboard.html @@ -392,7 +392,7 @@

Releases

MVP2 / v0.2
Three-Engine + Real Signals
-
8 / 17 scoped done · 27 remaining
+
8 / 18 scoped done · 28 remaining
In progress
diff --git a/docs/00_overview/mvp2_dashboard.html b/docs/00_overview/mvp2_dashboard.html index 48a1e495..72969fbf 100644 --- a/docs/00_overview/mvp2_dashboard.html +++ b/docs/00_overview/mvp2_dashboard.html @@ -397,9 +397,9 @@

MVP2 Progress

Specced features done
-
8 / 17
-
47% specced · 39 filed under MVP2
-
+
8 / 18
+
44% specced · 39 filed under MVP2
+
Pending work
@@ -435,14 +435,14 @@

MVP2 Progress

Legacy "Path to MVP2"
-
27
+
28
scoped not-done + bugs + chore-ideas only (excludes feat/infra ideas)
Backlog ideas: - 4 idea-only feat/infra folders (not yet scoped into MVP2) + 3 idea-only feat/infra folders (not yet scoped into MVP2) In flight: @@ -463,20 +463,7 @@

Pipeline

-

Idea 22

- -
-
Solr Smoke Stability
-
- Infra - P1 - -
-
After Phase 1 ships, the `pr.yml` `backend` job goes green (the heavy-lane reseed test no longer requires Solr). But the `smoke-test` job is independent — it runs `make up` which brings up the full Co
- - -
- +

Idea 21

Backend Suite Nondeterministic Caplog Isolation
@@ -758,7 +745,20 @@

Spec 0

-

Plan 9

+

Plan 10

+ +
+
Solr Smoke Stability
+
+ Infra + P1 + PR #367 merged 2026-06-01 +
+
The `pr.yml` `smoke (operator-path tutorial flow)` job is red on every branch.
+ + +
+
Apply Path Normalizer Declaration
@@ -1020,6 +1020,8 @@

Dependency graph (feat_ + infra_)

class feat_ubi_llm_study_comparison plan; infra_generated_artifact_freshness_gate["generated artifact freshness gate"] class infra_generated_artifact_freshness_gate plan; + infra_solr_smoke_stability["solr smoke stability"] + class infra_solr_smoke_stability plan; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done; feat_study_sub_warmup_guard["study sub warmup guard"] @@ -1061,6 +1063,8 @@

Dependency graph (feat_ + infra_)

class feat_ubi_llm_study_comparison plan; infra_generated_artifact_freshness_gate["generated artifact freshness gate"] class infra_generated_artifact_freshness_gate plan; + infra_solr_smoke_stability["solr smoke stability"] + class infra_solr_smoke_stability plan; feat_contextual_help_mvp2["contextual help mvp2"] class feat_contextual_help_mvp2 done; feat_study_sub_warmup_guard["study sub warmup guard"] diff --git a/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/feature_spec.md b/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/feature_spec.md new file mode 100644 index 00000000..1f79bfce --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/feature_spec.md @@ -0,0 +1,327 @@ +# Feature Specification — `infra_solr_smoke_stability` + +**Date:** 2026-06-01 +**Status:** Draft +**Owners:** Eric Starr (engineering lead) +**Related docs:** +- [`idea.md`](./idea.md) +- [`infra_solr_ci_readiness` Phase 1 (shipped, PR #367)](../../../implemented_features/2026_06_01_infra_solr_ci_readiness/feature_spec.md) +- [`docs/03_runbooks/demo-reseed-engine-tolerance.md`](../../../../03_runbooks/demo-reseed-engine-tolerance.md) — Phase 1 runbook (the backend-half story) + +--- + +## 1) Purpose + +- **Problem:** The `pr.yml` `smoke (operator-path tutorial flow)` job is red on every branch. It runs `make up`, which brings up the full Compose stack including the `solr:10.0` service; on the GHA `ubuntu-24.04` runner the Solr container crashes during boot (`relyloop-solr-1 exited (1)`), failing the smoke job at [`Makefile:109`](../../../../../Makefile#L109) (`bash scripts/install.sh`). The `pr.yml` failure-diagnostics step ([`.github/workflows/pr.yml:716-728`](../../../../../.github/workflows/pr.yml#L716-L728)) collects Compose logs for `api worker postgres redis elasticsearch ui` but neither `solr` nor `opensearch`, so the actual crash reason is invisible from CI artifacts. `infra_solr_ci_readiness` Phase 1 (PR #367, merged 2026-06-01) addressed only the `backend` job's reseed failure — the smoke job stays red until this work ships. +- **Outcome (two halves, sequenced for honesty):** **(A) Diagnostics in place — guaranteed.** Compose log artifacts on smoke failure include all three engines (Solr + OpenSearch added; ES already covered), so future runner regressions are diagnosable from CI alone. This half lands no matter what. **(B) Optimistic heap-cap fix — attempted.** Lever 1 (Solr heap cap to 256m) is applied to the smoke job in the same PR; if it fixes the crash, the smoke job goes green and the "every `pr.yml` job green" contract that started with Phase 1 is complete. If it doesn't (e.g., non-heap memory failure), the now-captured Solr logs feed an FR-3 follow-up that picks Lever 2 or 3 from evidence. The smoke debt is marked resolved in `state.md` only if (B) succeeds; if only (A) lands, the debt entry is updated with the captured evidence + the follow-up spec link. +- **Non-goal:** This spec does NOT change the live-Solr integration tests scaffolded but not exercised by `infra_adapter_solr` (those live in [`chore_solr_post_pipeline_followups`](../chore_solr_post_pipeline_followups/idea.md)). It does NOT add a Solr service container to the backend job — Phase 1's engine-tolerant reseed already made that unnecessary. It does NOT change the `solr.UBIComponent`-not-in-stock-image story or any product behavior. + +## 2) Current state audit + +### Existing implementations + +- [`.github/workflows/pr.yml:487-738`](../../../../../.github/workflows/pr.yml#L487-L738) — `smoke-test` job (the `docker:` job follows at line 739). Runs `make up` (line 629), waits for `/healthz` (lines 631-642), runs migrate + seed + Playwright/pytest. Has no Solr-specific health wait — the api's `/healthz` probes Solr as one of five parallel subsystem checks (with a 200ms timeout per CLAUDE.md), so a missing Solr surfaces as a subsystem-status field, not a job failure. +- [`.github/workflows/pr.yml:719`](../../../../../.github/workflows/pr.yml#L719) — `docker compose logs --no-color api worker postgres redis elasticsearch ui > smoke-logs.txt 2>&1`. The smoke job's failure-diagnostics collect step; missing `solr` and `opensearch`. +- [`docker-compose.yml:271-285`](../../../../../docker-compose.yml#L271-L285) — `solr` service block. Reads `SOLR_HEAP: ${SOLR_HEAP_SIZE:-512m}` (line 274); healthcheck has `start_period: 30s` (line 285), `interval: 10s`, `timeout: 5s`, `retries: 6`. Total tolerance before the container is marked unhealthy: up to ~95s, depending on how Docker schedules the first probe and whether failing probes consume the full 5s timeout vs. fail-fast. +- [`.github/workflows/pr.yml:287`](../../../../../.github/workflows/pr.yml#L287) — the backend job sets `ES_JAVA_OPTS: "-Xms256m -Xmx256m"` for its Elasticsearch service container. This is the precedent for the smoke job's Solr heap override. + +### Navigation and link impact + +| Source file | Current reference | New reference | +|---|---|---| +| `CLAUDE.md` (Key Runbooks table) | — | Add row → [`docs/03_runbooks/smoke-solr-stability.md`](../../../../03_runbooks/smoke-solr-stability.md) | +| `state.md` (Known debt) | "Smoke half still open (Phase 2)" entry | Strike (Phase 2 has now shipped) | + +No UI, no API, no operator-facing URL repointing. The two doc edits above are the only navigation changes. + +### Existing test impact + +| Test file | Pattern | Count | Required change | +|---|---|---|---| +| `.github/workflows/pr.yml` (`smoke-test` job) | `docker compose logs ... api worker postgres redis elasticsearch ui` | 1 | Add `solr` + `opensearch` to the service list | +| `.github/workflows/pr.yml` (`smoke-test` job) | `env:` block on the `make up` step | 1 | Add `SOLR_HEAP_SIZE: "256m"` | + +No backend pytest, frontend vitest, or Playwright spec changes. Verification routes through AC-1 (smoke RAN + non-smoke jobs green), AC-2 (diagnostics artifact on red runs), AC-4 (static workflow assertions — covers diagnostics correctness on green runs) — not "green smoke is required." + +### Existing behaviors affected by scope change + +- **Smoke job's tolerance to Solr boot.** Current: implicit, undocumented — depends on whether the `solr` healthcheck reports healthy before any step that needs the api to have probed Solr. New: explicit — the smoke job caps Solr heap at 256m to reduce JVM memory pressure on the runner. The cap is a hypothesis; the runner's exact memory budget is not cited because it's not load-bearing for the decision (the rationale is "cheapest available lever," not "sized to a specific budget"). Decision needed: **no** — the lever choice is locked in §19. + +--- + +## 3) Scope + +### In scope + +- **FR-1:** Failure-diagnostics fold-in. The smoke-test job's "Collect docker compose logs on failure" step ([`pr.yml:716-728`](../../../../../.github/workflows/pr.yml#L716-L728)) is extended to capture `solr` AND `opensearch` Compose logs. +- **FR-2:** Lever 1 (heap-cap) applied. `SOLR_HEAP_SIZE: "256m"` is set in the smoke job's `make up` step env block, alongside `RELYLOOP_SKIP_AUTO_SEED: "1"`. This mirrors the backend job's `ES_JAVA_OPTS: -Xms256m -Xmx256m` precedent. +- **FR-3:** Verification — the smoke job must RUN on the same PR that ships FR-1 + FR-2. If smoke is GREEN, AC-4's static workflow check is the proof that FR-1's diagnostics half is correctly applied (the actual log artifact is not produced on a green run). If smoke is RED, AC-2 becomes the runtime gating check (the artifact must contain `relyloop-solr-1` + `relyloop-opensearch-1` sections). Green smoke is the desired outcome; if Lever 1 doesn't fix the crash, the PR can still merge (`main` no longer enforces heavy-CI checks per `state.md`), but a follow-up spec MUST be filed AND linked from this PR's body BEFORE merge (not after), so the forcing function is mechanical rather than time-based. +- **FR-4:** Runbook entry. A new file at `docs/03_runbooks/smoke-solr-stability.md` (canonical path, not "extend existing") documents the heap-cap rationale, the lever cascade (1 → 2 → 3) with explicit escalation triggers, and the diagnostic-first workflow ("read the smoke-logs artifact's `relyloop-solr-1` section before picking a lever"). + +### Out of scope + +- Lever 2 (`start_period` bump) and Lever 3 (smoke-tolerance of Solr-down). These are escalation paths if Lever 1 doesn't resolve the crash; the runbook (FR-4) documents the **starting-point edit** for each, but per D-4 neither is genuinely one-line in scope — Lever 2 needs a CI-only env override for the healthcheck (so local dev isn't slowed), Lever 3 needs to audit the tutorial-path smoke pytest's assumptions about Solr availability. If triggered, either gets its own spec scoped to the evidence captured by FR-1. +- Live-Solr integration tests in CI ([`chore_solr_post_pipeline_followups`](../chore_solr_post_pipeline_followups/idea.md)). +- The `solr.UBIComponent`-not-in-stock-image story (no change; Phase 1 made the demo reseed already handle this). +- Any change to the `backend` job's Solr posture (already handled by Phase 1 — backend is engine-tolerant via reachability probe). +- Any change to local dev — `docker-compose.yml`'s Solr block is untouched. The heap override is GHA-only, set via env at job scope. + +### API convention check + +N/A — this spec adds no API endpoints. No router, no auth, no error envelope. + +### Phase boundaries + +**Single-phase spec.** The work splits naturally into two commits within one PR (FR-1 lands first as a one-line YAML edit; FR-2 lands second alongside it) but ships in a single PR. The idea's "two PRs" framing was the cautious sequence for the case where Lever 1 might not work; choosing Lever 1 as the locked default (D-1) plus the relaxed AC-1 / D-6 stance ("smoke red is allowed to merge because the diagnostics half is the durable value") collapses the sequence into one PR without sacrificing safety. The worst case is a smoke job that's no greener than before — at which point the new Solr+OpenSearch diagnostics artifact tells the FR-3 follow-up which lever to try next, AND the diagnostics half has already landed (so the next iteration is not blind). + +**Deferred phase tracking:** No deferred phase; the escalation paths (Lever 2, Lever 3) are documented in §19's decision log + runbook, not as a `phase2_idea.md`. Captured in §19 below as the explicit follow-up path. + +## 4) Product principles and constraints + +- **Local dev never carries CI-only env vars.** `SOLR_HEAP_SIZE=256m` is a GHA-runner tuning value — it lives in `.github/workflows/pr.yml`, never in `.env.example` or `docker-compose.yml`. The Compose default of 512m stays correct for the operator laptop. +- **CI failures must be diagnosable from artifacts alone.** Operators investigating a red smoke job should not need to re-run with extra logging — every service the smoke job depends on must be in the failure-diagnostics collect list. Phase 1 established this contract for the backend job (engine-reachability probe ⇒ skip + WARN log); this work extends it to the smoke job's diagnostics step. +- **Match the precedent.** The backend job already caps ES heap at `-Xms256m -Xmx256m` ([`pr.yml:287`](../../../../../.github/workflows/pr.yml#L287)) on the same `ubuntu-24.04` runner class. Solr 10's default heap (512m) is roughly twice that on a runner already running api + worker + postgres + redis + elasticsearch + opensearch + ui. Heap-cap is the most likely root cause and the cheapest, lowest-risk lever. + +### Anti-patterns + +- **Do not** change `docker-compose.yml`'s `${SOLR_HEAP_SIZE:-512m}` default to `256m`. That would silently make local Solr slower for every operator. The override slot was added precisely so CI can choose differently from the operator default. +- **Do not** skip FR-1 (the diagnostics fold-in) and apply only FR-2. If the heap cap doesn't fix the crash, the next failure must produce Solr logs in the smoke-logs artifact — otherwise the next iteration is blind. Diagnostics-first is a hard ordering invariant. +- **Do not** wrap Solr boot in a custom health-poll step in the smoke job. The api's `/healthz` already probes Solr; if Solr is down post-Lever-1, the smoke job will surface that via the api's `subsystems.solr` field. Adding a redundant poll obscures the path of evidence (operator reads `/healthz` first, then docker logs second). +- **Do not** apply Levers 1 + 2 + 3 all at once "just in case." Levers 2 and 3 mask different failure modes than Lever 1 — applying them blindly hides whatever the real root cause is, and the next regression (when Solr starts crashing again for a different reason) is harder to debug. + +## 5) Assumptions and dependencies + +- **Dependency:** `infra_solr_ci_readiness` Phase 1 (PR #367, merged 2026-06-01). + - **Why required:** Phase 1 established the contract that "Solr is allowed to be missing in CI" on the backend job. The smoke job's posture changes here are a parallel statement of the same contract: Solr boot is allowed to be unreliable, and CI must produce evidence either way. + - **Status:** Merged. + - **Risk if missing:** None now that it's shipped. +- **Dependency:** GHA standard `ubuntu-24.04` / `ubuntu-latest` runner class. The heap-cap is a hypothesis sized against the published runner memory budget (per the GitHub docs at the time of writing); we are NOT citing a measured Solr-on-the-runner OOM trace because that evidence is exactly what FR-1 produces. If the captured logs show a non-heap memory failure mode (metaspace exhaustion, native memory, total-runner pressure from sibling JVMs), Lever 1 may not be sufficient and the follow-up per FR-3 must address that. + - **Status:** All `pr.yml` jobs use `ubuntu-24.04` or `ubuntu-latest` (verified via grep). No self-hosted runner in scope. + - **Risk if missing:** A future migration to ARM runners or self-hosted runners with different memory budgets would re-trigger the smoke flake; the runbook entry calls this out. + +## 6) Actors and roles + +- **Primary actor:** RelyLoop maintainer landing a PR. +- **Role model:** N/A — pre-MVP4, single-tenant, no auth surface. +- **Permission boundaries:** N/A. + +### Authorization + +N/A — single-tenant install, no auth surface (per [`tech-stack.md` §"Canonical release matrix"](../../../../01_architecture/tech-stack.md)). + +### Audit events + +N/A — this work mutates no tenant-visible state. The only writes are to the GHA workflow YAML, which is source-controlled (every change is a git commit, which is the audit trail for CI changes). + +## 7) Functional requirements + +### FR-1: Smoke-job failure-diagnostics fold-in (`solr` + `opensearch`) + +- Requirement: + - The smoke-test job's "Collect docker compose logs on failure" step ([`.github/workflows/pr.yml:716-728`](../../../../../.github/workflows/pr.yml#L716-L728)) **MUST** include `solr` and `opensearch` in the `docker compose logs` service list. Final form: `docker compose logs --no-color api worker postgres redis elasticsearch opensearch solr ui > smoke-logs.txt 2>&1 || true` (the trailing `|| true` is mandatory because the diagnostics step is intentionally best-effort — adding services must not cause the step to fail if one service's logs can't be collected). + - To diagnose total-runner-memory failures (where Solr is OOM-killed by the kernel rather than throwing a Solr-internal OOM trace), the diagnostics step **SHOULD** also append `docker compose ps -aq | xargs -r docker inspect --format '{{.Name}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}}' >> smoke-logs.txt 2>&1 || true` so per-container `OOMKilled` flags + exit codes are captured. (`docker compose ps --format json` does NOT include `OOMKilled` — that field lives in `docker inspect`'s `State` struct.) The `|| true` keeps the diagnostics best-effort. +- Notes: This is a one-line YAML edit. Pre-existing — `opensearch` is also missing from the list today (the smoke job ships an OpenSearch container per [`docker-compose.yml:235`](../../../../../docker-compose.yml#L235)) — folding both in costs nothing. + +### FR-2: Lever 1 applied — cap Solr heap at 256m for the smoke job + +- Requirement: + - The smoke-test job **MUST** set `SOLR_HEAP_SIZE: "256m"` in the env block of the "Bring up the stack" step ([`.github/workflows/pr.yml:620-629`](../../../../../.github/workflows/pr.yml#L620-L629) — `env:` at line 621, `run: make up` at line 629), alongside the existing `RELYLOOP_GIT_SHA`, `RELYLOOP_SKIP_BUILD`, and `RELYLOOP_SKIP_AUTO_SEED` entries. + - The smoke-test job **MUST** also set `COMPOSE_PROJECT_NAME: "relyloop"` at **job-level `env`** (not step-level — GHA step env does NOT persist to later steps, and the failure-diagnostics step at line 716-728 must see the same project name as `make up` so `docker compose logs` finds the same stack). This pins the container-name prefix (`relyloop-solr-1`, `relyloop-opensearch-1`) so the AC-2 diagnostics grep is deterministic. + - The `docker-compose.yml` `solr` service block **MUST NOT** change — the Compose default of `${SOLR_HEAP_SIZE:-512m}` is correct for local dev and is what the env-var override slots into. + - The change **MUST NOT** alter the heap for any other Compose service (postgres, redis, elasticsearch, opensearch, api, worker, ui). +- Notes: The 256m value mirrors the backend job's ES_JAVA_OPTS precedent at [`pr.yml:287`](../../../../../.github/workflows/pr.yml#L287). This is a **hypothesis** — we do not yet have a measured Solr-on-the-runner OOM trace (FR-1 produces that evidence). The hypothesis: Solr 10 + the `ltr` module fits in 256m for the synthetic CI workload, matching the same pattern that works for Elasticsearch. If the captured logs show metaspace OOM, native memory exhaustion, or a non-memory crash, Lever 1 will not be sufficient and FR-3's follow-up applies. + +### FR-3: Smoke job runs; outcome triages cleanly; follow-up filed if needed + +- Requirement: + - On the PR that ships FR-1 + FR-2, the `pr.yml` `smoke (operator-path tutorial flow)` job **MUST** RUN to completion (success OR failure — not cancelled / skipped). + - All other `pr.yml` jobs **MUST** pass on the same run (the full required-job list is enumerated in AC-1). + - **If smoke is green:** Lever 1 fixed the crash. Merge normally. + - **If smoke is red:** AC-2 MUST be satisfied (the diagnostics artifact contains Solr + OpenSearch logs proving FR-1 works). The PR may still merge on the fast lane (per D-6 + `state.md`'s no-heavy-CI-required posture), AND a **follow-up artifact for Lever 2 (or Lever 3)** — at minimum an idea-stage `idea.md` in its own planned-features folder — MUST be filed AND linked from this PR's body **BEFORE merge** (per cycle-3 finding #7 — a mechanical forcing function rather than a 48-hour promise). The forcing function is the PR-body link, not the artifact's depth — a tiny idea file is acceptable for a one-line lever-YAML follow-up; a full spec is overkill for that scope. +- Notes: This requirement replaces the original "green smoke is the merge gate" framing per cycle-1 cross-model finding #10. The diagnostics half (FR-1) is the durable value of this work — it lands regardless of lever outcome. The lever half (FR-2) is the optimistic attempt at fixing the crash with the cheapest known lever; if it fails, the evidence it produces is what unblocks the next iteration. + +### FR-4: Runbook entry + +- Requirement: + - A new runbook file at `docs/03_runbooks/smoke-solr-stability.md` (canonical path; do NOT extend an existing runbook — keep concerns separate) **MUST** be created documenting: (a) the heap-cap rationale and the GHA-runner-only scope; (b) the lever cascade (Lever 1 → Lever 2 → Lever 3) for future smoke-job Solr failures, with **evidence-mapped escalation triggers** (not symptom-mapped): "slow boot that later becomes healthy (logs show Solr listening but past the healthcheck window) → Lever 2 (start_period bump); JVM heap/metaspace/native-memory OOM or kernel `OOMKilled: true` → memory tuning revisit (cap sibling JVMs ES + OpenSearch, not just bump start_period — the cycle-2 NFR known risk); Solr unavailable but tutorial smoke path doesn't actually depend on Solr → Lever 3 (smoke-tolerance)"; (c) the diagnostic-first workflow — "read smoke-logs artifact's `relyloop-solr-1` section AND the `docker inspect` exit-state line BEFORE picking a lever." + - The CLAUDE.md "Key Runbooks" table **MUST** be updated with a row linking to `docs/03_runbooks/smoke-solr-stability.md`, similar to the existing "Demo reseed engine tolerance" row. +- Notes: The runbook section is the durable artifact — without it, a future maintainer hitting the next Solr CI flake has no documented path through the levers. + +## 8) API and data contract baseline + +### 8.1 Endpoint surface + +N/A — no API endpoints. + +### 8.2 Contract rules + +N/A. + +### 8.3 Response examples + +N/A. + +### 8.4 Enumerated value contracts + +N/A — no filters, badges, or enums introduced. + +### 8.5 Error code catalog + +N/A. + +## 9) Data model and state transitions + +N/A — no schema changes. Alembic head stays `0022_solr_engine_auth_check`. + +## 10) Security, privacy, and compliance + +- **Threats:** None — the change is GHA-config-only. +- **Controls:** GHA workflow file is source-controlled; every edit is a reviewable git commit. +- **Secrets/key handling:** No new secrets. `SOLR_HEAP_SIZE` is non-sensitive tuning data and ships in the workflow YAML cleartext. +- **Auditability:** Git history is the audit trail. +- **Data retention/deletion/export impact:** None. + +## 11) UX flows and edge cases + +N/A — no UI surface. The "user" here is the maintainer reading CI status on the PR page. + +## 12) Given/When/Then acceptance criteria + +### AC-1: Smoke job runs; outcome triages cleanly + +- Given the PR branches off `main` (which carries the `infra_solr_ci_readiness` Phase 1 merge). +- When the PR's `pr.yml` workflow runs. +- Then the `smoke (operator-path tutorial flow)` job runs to completion (either `success` or `failure`, NOT `cancelled` or `skipped`), AND every other `pr.yml` job — enumerated exactly as the workflow declares (no ellipses) — completes with `success`: + - `backend (unit tests — fast lane)` + - `license-headers` + - `license-inventory` + - `static-checks (backend — ruff + mypy + guards, always-run)` + - `static-checks (frontend — prettier + eslint + tsc + vitest, always-run)` + - `backend (lint + typecheck + tests + coverage)` + - `frontend (lint + typecheck + tests + build)` + - `docker buildx (relyloop/api)` + - `docker buildx (relyloop/ui)` +- Any `skipped` outcome on the non-smoke jobs (e.g., due to a path filter or `if:` guard not satisfied) MUST be treated as a failed AC-1 unless explicitly documented as expected behavior for that PR's diff (e.g., a docs-only PR may skip backend tests — but this spec's PR touches `.github/workflows/pr.yml`, so no path filter skip should fire). +- **Re-derive before implementing.** The job list above is captured at spec time (2026-06-01). Before /impl-execute starts, re-derive the list from the actual `pr.yml` HEAD with `python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); [print(j.get('name', jid)) for jid, j in wf['jobs'].items()]"` — if any job has been added, removed, or renamed since spec time, update AC-1's list before merge. +- Outcome triage: + - **Smoke green:** Lever 1 fixed the crash — happy path; PR merges normally. + - **Smoke red:** Lever 1 did not fix the crash; AC-2 then becomes the gating verification (the diagnostics artifact MUST contain Solr logs). PR is still mergeable on the fast lane (`main` no longer enforces heavy-CI checks per `state.md`); FR-3 requires a follow-up spec within 48 hours. +- Example values (use the exact GHA job display names, NOT slugs): + - Verification: `gh run view --json jobs -q '.jobs[] | select(.name != "smoke (operator-path tutorial flow)") | "\(.conclusion)\t\(.name)"'` shows every non-smoke job as `success`. + +### AC-2: Failure-diagnostics artifact contains Solr + OpenSearch logs (when smoke is red) + +- Given a smoke-job failure on a run of this PR's branch (which carries FR-1 + FR-2). **Pre-PR-#367 artifacts are NOT acceptable evidence** — they were produced by the old workflow before FR-1 applied. AC-2 must be verified on a post-FR-1 run. +- When the "Upload failure diagnostics" step uploads `smoke-logs.txt`. +- Then the file contains both Solr AND OpenSearch container log lines. The smoke job MUST set `COMPOSE_PROJECT_NAME=relyloop` in its env block alongside `SOLR_HEAP_SIZE` (FR-2) — this pins the container-name prefix to `relyloop-solr-1` / `relyloop-opensearch-1` so the verification grep is deterministic (Docker Compose's default project name is the working-dir basename, which is `relyloop` on every GHA checkout today but is not contractually guaranteed; pinning removes the dependency). +- Example values: + - Verification (in CI artifact post-download): `grep -Eq '^relyloop-solr-1[[:space:]]+\|' smoke-logs.txt && grep -Eq '^relyloop-opensearch-1[[:space:]]+\|' smoke-logs.txt`. +- **If the smoke job happens to go green on the run (Lever 1 worked):** AC-2's runtime check is NOT required — AC-5's static workflow verification (the `docker compose logs` command in pr.yml includes `solr` and `opensearch`) proves the diagnostics half is correctly applied. A scratch failure is not forced because (a) it is operationally noisy and (b) injecting a failure before `make up` succeeds would produce zero Solr containers and therefore zero Solr logs — falsely failing AC-2. + +### AC-3: Local `make up` is unaffected (verified by `docker compose config`, not runtime) + +- Given a checkout of this PR's branch. +- When `docker compose config` runs (no env overrides set — i.e., no `SOLR_HEAP_SIZE` in the shell environment, no `.env` override). +- Then the rendered `solr` service block contains `SOLR_HEAP: 512m` (the Compose default), AND `docker-compose.yml` is unchanged versus `main` (the heap override lives only in `.github/workflows/pr.yml`). +- Example values: + - Verification (structural — robust against re-ordering of the `environment:` block): `docker compose config --format json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['services']['solr']['environment']['SOLR_HEAP'])"` returns `512m`. + - And: `git diff main -- docker-compose.yml` returns no diff. + +### AC-4: GHA workflow env interpolation resolves to `SOLR_HEAP=256m` under the smoke job + +- Given the smoke-test job's env block on this PR's branch. +- When the workflow renders (verifiable statically without running CI by parsing the YAML). +- Then the "Bring up the stack" step's env block contains `SOLR_HEAP_SIZE: "256m"` (step-level — affects `make up`), AND the smoke-test job's job-level `env:` contains `COMPOSE_PROJECT_NAME: "relyloop"` (job-level — persists to every later step including failure-diagnostics per cycle-3 finding #1), AND a `SOLR_HEAP_SIZE=256m docker compose config` invocation against the unchanged `docker-compose.yml` renders the `solr` service's `SOLR_HEAP` as `256m`. +- Example values: + - YAML parse — step-level: `python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); steps=wf['jobs']['smoke-test']['steps']; step=next(s for s in steps if (s.get('run') or '').strip()=='make up'); assert (step.get('env') or {}).get('SOLR_HEAP_SIZE')=='256m', step.get('env')"` succeeds (selects the `make up` step specifically — does NOT merge across all steps). + - YAML parse — job-level: `python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); job=wf['jobs']['smoke-test']; assert (job.get('env') or {}).get('COMPOSE_PROJECT_NAME')=='relyloop', job.get('env')"` succeeds. + - Structural Compose check: `SOLR_HEAP_SIZE=256m docker compose config --format json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['services']['solr']['environment']['SOLR_HEAP'])"` returns `256m`. + +### AC-5: Runbook lever cascade is reachable from CLAUDE.md + +- Given a maintainer reading [CLAUDE.md](../../../../../CLAUDE.md) "Key Runbooks" table. +- When they look up "smoke job Solr stability" / "GHA Solr crash" / similar phrasing. +- Then a row in the table links to `docs/03_runbooks/smoke-solr-stability.md` which documents Levers 1/2/3 with escalation triggers and the diagnostic-first workflow. +- Example values: + - Verification: `grep -q 'smoke-solr-stability.md' CLAUDE.md && test -f docs/03_runbooks/smoke-solr-stability.md`. + +## 13) Non-functional requirements + +- **Performance:** No new runtime cost. The heap cap reduces Solr's JVM memory footprint by ~256MB on every CI run, which marginally reduces overall runner memory pressure (downstream side benefit: api/worker/postgres/redis have more headroom). +- **Reliability:** If Lever 1 is the correct lever, smoke pass rate on `pr.yml` should rise toward 100%. We do not have historical pass-rate metrics — Phase 1's merge timeline means baseline is "100% red on smoke since Solr added on 2026-05-31." +- **Known risk (not addressed in this spec):** The smoke Compose stack runs **three JVM services concurrently** (Elasticsearch, OpenSearch, Solr). The backend job caps ES heap (`ES_JAVA_OPTS: -Xms256m -Xmx256m`) but that env applies to GHA service containers, not to the smoke job's Compose stack. So the smoke stack's ES and OpenSearch run at their Compose defaults today. If the captured Solr logs reveal total-runner memory pressure (e.g., kernel-level OOM-killer signal rather than a Solr-internal OOM trace), the fix is not Solr-specific — it requires capping ES + OpenSearch heap in the smoke Compose path too. That work is OUT OF SCOPE here but is flagged for the follow-up spec triggered by FR-3. +- **Operability:** No change to operator-visible behavior. CI maintainers gain diagnosis-time evidence (Solr + OpenSearch logs in failure artifacts). +- **Accessibility/usability:** N/A. + +## 14) Test strategy requirements + +This work has no backend, frontend, or domain-logic surface — the existing test pyramid is unchanged. The smoke-job itself is the integration test for FR-2 and FR-3; the static-checks YAML lint + the workflow validation steps in GHA exercise FR-1. + +| Layer | Coverage | +|---|---| +| Unit (`backend/tests/unit/`) | N/A — no Python code changes | +| Integration (`backend/tests/integration/`) | N/A — no Python code changes | +| Contract (`backend/tests/contract/`) | N/A — no API changes | +| E2E (`ui/tests/e2e/`) | N/A — no UI changes | +| Smoke job (`.github/workflows/pr.yml` `smoke-test`) | The smoke job's own green-on-green outcome is the verification (AC-1). No new pytest spec; the existing tutorial-path smoke spec runs as-is. | +| Workflow YAML static validation | YAML parse + structural assertion of the `SOLR_HEAP_SIZE` / `COMPOSE_PROJECT_NAME` env keys per AC-4 (run as part of pre-push gate, not a CI job). GHA's implicit YAML-load check on push is a backstop but is NOT a lint job — it only prevents totally invalid workflows from being scheduled. | +| Runbook verification | `test -f docs/03_runbooks/smoke-solr-stability.md && grep -q 'smoke-solr-stability.md' CLAUDE.md` — verified at PR review (AC-5). | + +## 15) Documentation update requirements + +- `docs/01_architecture/`: no change. The Compose Solr block is unchanged; the architecture doc's Solr description stays accurate. +- `docs/02_product/`: no change (CI infra, not a product feature). +- `docs/03_runbooks/`: create new `smoke-solr-stability.md` per FR-4 (canonical path locked in §3 / FR-4). +- `docs/04_security/`: no change. +- `docs/05_quality/`: no change (CI infra outcome is captured in `testing.md`'s pyramid; the smoke layer is unchanged). +- `CLAUDE.md`: add a "Key Runbooks" row pointing at the new runbook section per AC-5. +- `state.md`: update "Last 5 merges" + "Known debt" entries. **The "smoke half still open" debt entry is marked RESOLVED only if smoke went green on the merge run (Lever 1 worked).** If smoke merged red on the fast lane (D-6 path), the debt entry is UPDATED to record the captured evidence (Solr exit reason from the smoke-logs artifact) + a link to the FR-3 follow-up spec, NOT marked resolved. Update at finalization. + +## 16) Rollout and migration readiness + +- **Feature flags / staged rollout:** None — CI infra, ships atomically with the PR's merge. +- **Migration/backfill expectations:** None. +- **Operational readiness gates:** The smoke job must RUN to a non-cancelled/non-skipped conclusion on this PR's own run (AC-1). If it passes, Lever 1 worked — happy path. If it fails, AC-2 becomes the gating check (the diagnostics artifact must contain Solr + OpenSearch logs proving FR-1 worked); merge is still allowed on the fast lane per D-6 (mutable-branch-protection caveat below), and the captured evidence drives the FR-3 follow-up spec within 48 hours. +- **Branch-protection caveat for the red-merge path:** D-6 relies on `main` not enforcing the smoke job as a required status check (the rule was removed 2026-05-31 per `state.md`). Before merging in the smoke-red case, the implementer MUST re-verify branch protection is still permissive: `gh api repos/SoundMindsAI/relyloop/branches/main/protection 2>/dev/null` returns 404 (no protection) OR the `required_status_checks` does not list the smoke job. If protection has been re-enabled to require smoke, the red-merge path is BLOCKED and the implementer must escalate per FR-3 within the same PR (apply Lever 2 as a follow-up commit) or split into two PRs. +- **Release gate:** + - All non-smoke `pr.yml` jobs pass; smoke job RAN (success or failure, not cancelled/skipped) — per AC-1. + - `SOLR_HEAP_SIZE` correctly resolves to `256m` under the smoke job (AC-5; verifiable statically via `docker compose config` + YAML parse — no need to wait for CI). + - Smoke-logs artifact contains `relyloop-solr-1` + `relyloop-opensearch-1` sections **only on the smoke-red path** (AC-2). On the smoke-green path, AC-4's static workflow assertion covers FR-1 correctness (no scratch failure is forced — that would be noisy and could false-fail per cycle-2 finding #5). + - Runbook file + CLAUDE.md row added (AC-4). + - If smoke is red on the merge run: follow-up artifact (idea-stage file at minimum, full spec if scope warrants) for Lever 2/3 **filed AND linked from PR body BEFORE merge** (FR-3 forcing function — mechanical, not time-based). + +## 17) Traceability matrix + +| FR ID | Acceptance Criteria IDs | Planned stories/tasks | Test files/suites | Docs to update | +|---|---|---|---|---| +| FR-1 (diagnostics fold-in) | AC-2 | Story 1.1 (YAML edit: services + `docker compose ps`) | smoke-logs artifact inspection on a post-FR-1 run | — | +| FR-2 (Lever 1: heap cap + COMPOSE_PROJECT_NAME) | AC-3, AC-4 | Story 1.2 (env-block edit) | `docker compose config --format json` structural check; YAML parse | — | +| FR-3 (smoke ran; triage; follow-up forcing function) | AC-1, AC-2 | Story 1.2 + 1.3 (verify run + branch-protection re-check) | full `pr.yml` matrix outcome triage | — | +| FR-4 (runbook + CLAUDE.md) | AC-5 | Story 2.1 (runbook), Story 2.2 (CLAUDE.md row) | grep + `test -f` verification | `docs/03_runbooks/smoke-solr-stability.md`, `CLAUDE.md` | + +## 18) Definition of feature done + +- [ ] AC-1 — smoke job RAN on the merge PR + all other required `pr.yml` jobs (full list in AC-1) green. +- [ ] AC-2 — failure-diagnostics artifact verifiably contains `relyloop-solr-1` + `relyloop-opensearch-1` log lines (required only in the smoke-red case; in smoke-green case, AC-4's static YAML check covers FR-1 correctness). +- [ ] AC-3 — `docker compose config` renders `SOLR_HEAP: 512m` (Compose default unchanged); `git diff main -- docker-compose.yml` is empty. +- [ ] AC-4 — `.github/workflows/pr.yml` smoke-test step has `SOLR_HEAP_SIZE: "256m"` AND `COMPOSE_PROJECT_NAME: "relyloop"` env (static YAML parse); `SOLR_HEAP_SIZE=256m docker compose config --format json | jq .services.solr.environment.SOLR_HEAP` returns `"256m"`. +- [ ] AC-5 — runbook file `docs/03_runbooks/smoke-solr-stability.md` exists; CLAUDE.md "Key Runbooks" table links to it. +- [ ] No open questions remain in §19. +- [ ] If AC-1's outcome is "smoke red," a follow-up artifact for Lever 2 (or Lever 3) — at minimum an idea-stage `idea.md` in its own planned-features folder — **is filed AND linked from this PR's body BEFORE merge** — this is a mechanical forcing function (the PR can't merge without the link in the body) rather than a time-based "within 48 hours" promise. Artifact depth is a judgement call: a one-line lever-YAML follow-up rates an idea file; a multi-file scope (e.g., Lever 3 smoke-tolerance audit) rates a full spec. +- [ ] Branch-protection state on `main` re-verified before merge in the smoke-red case (per §16 branch-protection caveat). +- [ ] `state.md` updated: **only if smoke green**, strike the "Smoke half still open (Phase 2)" item entirely; **if smoke red**, update the same item to record the captured evidence + the follow-up spec path/link. + +## 19) Open questions and decision log + +### Open questions + +_None at spec time._ The lever-choice question (the idea's central open fork) is locked below. + +### Decision log + +- **2026-06-01 — D-1: Lever 1 (heap-cap) is the locked default for this spec.** Rationale: it is the cheapest, lowest-risk lever; it has the strongest precedent (the backend job's `ES_JAVA_OPTS: -Xms256m -Xmx256m`); and the worst case if it doesn't fix the crash is a no-change-in-redness smoke job that now produces Solr logs in the failure-diagnostics artifact — feeding the next iteration. Lever 2 (`start_period` bump from 30s to 60s/90s) and Lever 3 (smoke-tolerance of Solr-down) are documented as escalation paths in the runbook (FR-4) and explicitly out of scope here. +- **2026-06-01 — D-2: Single-PR delivery, not the two-PR sequence the idea proposed.** Rationale: the idea's two-PR sequence was a hedge against bundling a "we don't know what the fix is" lever PR with a clean diagnostics PR. Locking Lever 1 by D-1 collapses that risk — Lever 1 is the cleanest, lowest-risk fix, the diagnostics fold-in is one line, and a single PR is faster to land. If Lever 1 doesn't work, the next PR is just as cheap to write because the diagnostics are now in place. +- **2026-06-01 — D-3: Fold `opensearch` in alongside `solr` in the failure-diagnostics list.** Rationale: OpenSearch is also a smoke-job dependency (Compose line 235) and is also missing from the log collect line at `pr.yml:719`. Folding both in is one line of diff and removes a pre-existing diagnostic gap. The idea explicitly named this as a worthwhile fold-in. +- **2026-06-01 — D-4: Escalation path documented in the runbook, NOT as a `phase2_idea.md`.** Rationale: Levers 2 and 3 have documented triggers and known starting-point edits (Lever 2 = bump `start_period` from 30s to 60s/90s; Lever 3 = make smoke job tolerant of Solr-down via tutorial-path test adjustment). However, NEITHER lever is genuinely "one line" — Lever 2 changes the Compose healthcheck which affects local dev, so it requires a CI-only env override slot (similar to `SOLR_HEAP_SIZE`) plus a docker-compose.yml edit; Lever 3 requires auditing the tutorial-path smoke pytest's implicit assumptions about Solr availability, which is multi-file. The runbook documents the **starting point** for each lever; the **scope** is determined when the lever is actually applied, driven by the captured log evidence. Either lever, if triggered by FR-3, gets its own spec (one file, ~50-100 lines per the pattern of this one) — not a `phase2_idea.md` pre-written today against unknown evidence. +- **2026-06-01 — D-6: Smoke job is allowed to merge red on this PR.** Rationale: GPT-5.5 caught (cycle 1 finding #10) that the single-PR delivery (D-2) plus a "smoke must be green" DoD created a deadlock — if Lever 1 doesn't fix the crash, the diagnostics half (FR-1) cannot land either, and the next iteration is still blind. Resolution: AC-1 is relaxed to "smoke job RAN" (success OR failure, not cancelled/skipped); if smoke is red, AC-2 becomes the gating verification (the artifact must contain Solr+OpenSearch logs). This is safe because `main` no longer enforces heavy-CI required-status-checks (per `state.md`, the operator removed that rule 2026-05-31), so the PR is mergeable on the fast lane in the Lever-1-fails case. FR-3's follow-up clause (file a follow-up spec within 48 hours) is the forcing function that ensures the lever-half work doesn't get lost. +- **2026-06-01 — D-5: `docker-compose.yml` is NOT touched.** Rationale: the heap override slot already exists (`${SOLR_HEAP_SIZE:-512m}`). Changing the 512m default to anything else would silently affect every operator's local laptop, which is the wrong scope of fix. The env var goes in the GHA workflow only. diff --git a/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/idea.md b/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/idea.md index e3032e0f..ecffd09a 100644 --- a/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/idea.md +++ b/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/idea.md @@ -14,18 +14,20 @@ The diagnostic step that gates this work is reading the actual `docker compose l ## Proposed capabilities -### Capability B — Solr smoke-runner stability - Concretely, the work is: -1. **Capture logs.** Rerun the smoke job on a branch that intentionally lets Solr come up, capture `docker compose logs solr` from the failure surface. The existing failure-diagnostics upload at [`pr.yml:716-728`](../../../../../.github/workflows/pr.yml#L716-L728) covers `api worker postgres redis elasticsearch ui` but NOT `solr` — first sub-task is to ADD `solr` to that list so future failures auto-capture. +1. **Capture logs.** Rerun the smoke job on a branch that intentionally lets Solr come up, capture `docker compose logs solr` from the failure surface. The existing failure-diagnostics upload at [`pr.yml:716-728`](../../../../../.github/workflows/pr.yml#L716-L728) (the smoke-test job's "Collect docker compose logs on failure" step) covers `api worker postgres redis elasticsearch ui` but is missing **both** `solr` (line 271 in compose, the one this work is about) **and** `opensearch` (line 235 in compose, a pre-existing gap worth fixing in the same edit). First sub-task is to add both to that list so future smoke failures auto-capture. 2. **Pick the lever, in priority order** (cheapest first): - **Lever 1 (heap-cap):** Set `SOLR_HEAP_SIZE=256m` in the smoke job's `env:` block (matching the backend job's `ES_JAVA_OPTS: -Xms256m -Xmx256m` at [`pr.yml:287`](../../../../../.github/workflows/pr.yml#L287)). Compose's `solr` service reads `${SOLR_HEAP_SIZE:-512m}` ([`docker-compose.yml:274`](../../../../../docker-compose.yml#L274)) — the override slot already exists. - **Lever 2 (start_period):** Bump the Solr healthcheck `start_period` from `30s` ([`docker-compose.yml:285`](../../../../../docker-compose.yml#L285)) to `60s` or `90s`. Solr 10 + LTR module first-load on a cold runner can take longer than ES/OS on the same hardware. - **Lever 3 (tolerance):** Make the smoke job tolerant of Solr being down — drop `solr` from any healthcheck the smoke step gates on, drop it from the `make up` boot expectation, accept "Solr crashed, other engines green" as smoke success. This is the last-resort lever because it gives up the per-PR signal that Solr can boot. -3. **Verify on a green smoke run** before declaring Phase 2 done. +3. **Verify on a green smoke run** before declaring the work done. + +The lever choice is the unresolved decision — it should be locked in `feature_spec.md` based on the log evidence. + +### Recommended sequencing — two PRs, not one -The lever choice is the unresolved decision — it should be locked in the Phase 2 spec based on the log evidence. +Sub-task 1 (adding `solr` + `opensearch` to the failure-diagnostics collect list) is a one-line YAML edit with zero risk. It can ship in a tiny standalone PR before the lever-choice work even begins — and it MUST ship first, because the lever choice depends on Solr logs from a failing smoke run that this fix is what produces. Lever PR is then driven by the captured evidence. Don't bundle them: a one-line diagnostic-capture PR can merge today; the lever PR waits on data. ## Scope signals @@ -35,12 +37,13 @@ The lever choice is the unresolved decision — it should be locked in the Phase - **Config:** smoke job environment (one new env var, or a YAML edit to the healthcheck block). - **Audit events:** N/A (no state mutations, pre-MVP2 anyway). -## Why deferred +## Why log evidence gates the lever choice -The lever choice depends on log evidence that doesn't exist yet. Bundling a "we don't know what the fix is" item into Phase 1 would have delayed the unblock for the backend job (which has a clean, known fix). Splitting lets Phase 1 ship with high confidence and lets Phase 2 be driven by data. +The three levers address three different failure modes: heap-cap addresses OOM-kill, start_period bump addresses healthcheck-races-startup, and tolerance bypasses the issue entirely. They are NOT interchangeable — picking the wrong one ships a fix that doesn't fix anything. Until we read what `docker compose logs solr` actually says when the smoke runner crashes the container, the lever choice is guesswork. That is why sub-task 1 (diagnostics capture) is sequenced first. ## Relationship to other work - Pairs with `infra_solr_ci_readiness` Phase 1 ([shipped spec](../../../implemented_features/2026_06_01_infra_solr_ci_readiness/feature_spec.md)). Phase 1 fixed the `backend` job; this work fixes the `smoke` job. Together they take `pr.yml` from "red on every branch" to "green on every branch." -- Sibling [`chore_solr_post_pipeline_followups`](../chore_solr_post_pipeline_followups/idea.md) tracks the live-Solr integration tests scaffolded but not exercised when Solr shipped. Phase 2 doesn't touch those — it's purely about runner stability. +- Sibling [`chore_solr_post_pipeline_followups`](../chore_solr_post_pipeline_followups/idea.md) tracks the live-Solr integration tests scaffolded but not exercised when Solr shipped. This work doesn't touch those — it's purely about runner stability. +- **Coordinate with** [`chore_solr_cred_backfill_needs_api_restart`](../chore_solr_cred_backfill_needs_api_restart/idea.md). That chore is in the `make up` boot path: `scripts/install.sh` step 5a backfills the Solr cred into a running stack, but the `api` / `worker` settings cache memoizes the YAML at process start. The smoke job runs `make up` on a fresh runner (no pre-existing cred file, no pre-running api), so the cached-stale-creds failure mode does NOT trip there — but if smoke-stabilization work ever changes the boot ordering or pre-stages a cred file, re-check this. Today the two are independent. - Independent of [`bug_reseed_failure_blocks_retry_arq_singleton_dedup`](../bug_reseed_failure_blocks_retry_arq_singleton_dedup/idea.md) — that bug is in the reseed Arq-dedup path, not the smoke/CI path. diff --git a/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/implementation_plan.md b/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/implementation_plan.md new file mode 100644 index 00000000..829448ee --- /dev/null +++ b/docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_stability/implementation_plan.md @@ -0,0 +1,770 @@ +# Implementation Plan — `infra_solr_smoke_stability` + +**Date:** 2026-06-01 +**Status:** Draft +**Primary spec:** [`feature_spec.md`](./feature_spec.md) +**Policy source(s):** [CLAUDE.md](../../../../../CLAUDE.md) (Absolute Rules + Common Pitfalls), [`docs/03_runbooks/demo-reseed-engine-tolerance.md`](../../../../03_runbooks/demo-reseed-engine-tolerance.md) (Phase 1 precedent — companion runbook to the one this work creates) + +--- + +## 0) Planning principles + +- This is an infra/CI plan. Most template sections (backend domain/service/repo, frontend, migration, API contracts, audit events) are N/A — the deliverable is GHA workflow YAML + a new runbook + a CLAUDE.md edit. +- Two epics: **Epic 1** (workflow changes, sequential within one PR), **Epic 2** (documentation). +- Phase 1 (the spec's only phase) covers FR-1, FR-2, FR-3, FR-4. No deferred phase — Levers 2/3 escalate into their own spec when evidence triggers them (D-4). +- **Soft sequencing recommendation** (per cycle-2 plan finding #1; NOT a hard gate): Epic 1 → Epic 2. The runbook's lever cascade is generic/forward-looking (it documents Levers 1/2/3 + memory-pressure escalation in the abstract, NOT this PR's specific outcome), so Epic 2 can run before Story 1.3 watches CI. But running Epic 2 second has a small benefit: if smoke goes red, the Story 1.3 evidence can sharpen wording. Either order is fine. Story 2.3 finalization re-verifies AC-1 on the FINAL HEAD SHA regardless, so the merge-target run is always captured. + +## 1) Scope traceability (FR → epics/stories) + +| FR ID | Epic / Story | Notes | +|---|---|---| +| FR-1 (diagnostics fold-in: `solr` + `opensearch` + `docker inspect` exit-state) | Epic 1 / Story 1.1 | One-line YAML edit + the `docker inspect` adjunct line. | +| FR-2 (Lever 1 heap-cap + `COMPOSE_PROJECT_NAME` pin) | Epic 1 / Story 1.2 | `SOLR_HEAP_SIZE` is step-level on "Bring up the stack"; `COMPOSE_PROJECT_NAME` is JOB-level so it persists to the later failure-diagnostics step (per cycle-3 finding #1). | +| FR-3 (smoke ran; triage; before-merge forcing function if red) | Epic 1 / Story 1.3 | Verification + outcome triage + branch-protection re-check + (if smoke red) file the follow-up spec stub AND link it from the PR body before merge. | +| FR-4 (new runbook + CLAUDE.md row) | Epic 2 / Stories 2.1 + 2.2 | Story 2.1 = new `smoke-solr-stability.md`; Story 2.2 = CLAUDE.md "Key Runbooks" row. | + +**Phase coverage:** spec defines a single phase; this plan covers all of it. No `phase2_idea.md` required (D-4). + +## 2) Delivery structure + +**Format:** Epic → Story → Tasks → DoD. The spec is infra-shaped so the stories are smaller than a typical product story; that's intentional — each story is its own small commit on a single feature branch, all merging in one PR. + +### Conventions (project-specific) + +- **Single feature branch + single PR.** Branch: `feature/infra-solr-smoke-stability`. Each story = one commit (or one logical group). +- **Conventional Commits + DCO.** Every commit message follows the project regex (`infra(...)` for workflow edits, `docs(...)` for runbook/CLAUDE.md). Every commit signed with `git commit -s`. +- **No skip-hooks.** Never `--no-verify` or `--no-gpg-sign`. +- **Pre-push gate before push.** Run `make fmt && make lint && make typecheck && make test-unit && make pre-commit` per CLAUDE.md (this project's standard set; for infra-only PRs, lint + the pre-commit hooks cover the YAML and markdown surface). +- **Static workflow validation.** For YAML edits to `.github/workflows/pr.yml`, parse with Python's `yaml.safe_load` and assert the structural properties per AC-4 before pushing — don't rely on GHA's implicit load to catch typos. + +### AI Agent Execution Protocol + +0. **Load context first.** Read `architecture.md`, `state.md`, and `feature_spec.md` §19 decision log before starting Story 1.1. +1. **Read scope** for the story being executed. +2. **Edit the workflow / compose / runbook file.** +3. **Run the static validation** specific to that story (YAML parse, `docker compose config`, `test -f` on the new runbook file). +4. **Commit** with `-s`. +5. After all Epic 1 stories: push, open PR, watch CI. +6. After CI runs: execute Story 1.3's outcome triage. +7. After Epic 1 verified: execute Epic 2 (runbook + CLAUDE.md row). +8. After all 5 stories: update `state.md` per §4.0; finalize. + +--- + +## Epic 1 — Workflow + Compose changes + +**Goal:** Land the diagnostics half (FR-1) and the optimistic-lever half (FR-2), then triage the resulting smoke-job outcome (FR-3) and either merge clean (smoke green) or merge with a pre-filed follow-up spec linked from the PR body (smoke red, per D-6). + +### Story 1.1 — Failure-diagnostics fold-in (FR-1) + +**Outcome:** The smoke-test job's "Collect docker compose logs on failure" step captures `solr` + `opensearch` (in addition to the existing api/worker/postgres/redis/elasticsearch/ui), AND appends a `docker inspect` adjunct line that captures per-container `OOMKilled` flags + exit codes. The step remains best-effort (`|| true`). + +**New files** + +| File | Purpose | +|---|---| +| _(none)_ | This story modifies an existing workflow only. | + +**Modified files** + +| File | Change | +|---|---| +| `.github/workflows/pr.yml` (smoke-test job, around lines 716-728) | (a) Extend the `docker compose logs` service list from `api worker postgres redis elasticsearch ui` to `api worker postgres redis elasticsearch opensearch solr ui`. (b) Append a second line: `docker compose ps -aq \| xargs -r docker inspect --format '{{.Name}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}}' >> smoke-logs.txt 2>&1 \|\| true`. Both lines must keep `\|\| true` (best-effort). | + +**Endpoints / Key interfaces / Pydantic schemas** + +N/A — workflow YAML only. + +**Tasks** + +1. Read [`.github/workflows/pr.yml`](../../../../../.github/workflows/pr.yml) lines 716-728 to confirm the current "Collect docker compose logs on failure" step shape. +2. Edit the `run:` block to: + ```yaml + run: | + docker compose logs --no-color api worker postgres redis elasticsearch opensearch solr ui > smoke-logs.txt 2>&1 || true + docker compose ps -aq | xargs -r docker inspect --format '{{.Name}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}} health={{with .State.Health}}{{.Status}}{{end}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}' >> smoke-logs.txt 2>&1 || true + curl -s http://127.0.0.1:8000/healthz >> smoke-logs.txt 2>&1 || true + ``` + (Preserve the existing `curl /healthz` tail line. The extended `--format` includes `health` + `started` + `finished` fields per cycle-3 plan finding #4, so the Lever-2 runbook trigger ("Solr booted but past healthcheck tolerance") can be diagnosed from artifact evidence alone — the prior template only had exit/oom/error.) +3. Run YAML parse validation: + ```bash + python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); steps=wf['jobs']['smoke-test']['steps']; step=next(s for s in steps if s.get('name')=='Collect docker compose logs on failure'); assert 'opensearch solr ui' in step['run'], step['run']; assert 'docker inspect' in step['run'], step['run']" + ``` +4. Commit: `infra(smoke-ci): capture solr + opensearch logs + docker inspect on smoke failure`. + +**Definition of Done (DoD)** + +- [ ] `.github/workflows/pr.yml` smoke-test job's "Collect docker compose logs on failure" step lists `solr` and `opensearch` in the logs command. +- [ ] The same step appends `docker inspect` output (capturing `OOMKilled` + exit codes) to `smoke-logs.txt`, with `|| true`. +- [ ] YAML parse validation script (Task 3) succeeds. +- [ ] All three lines retain `|| true` — best-effort diagnostics (cycle-3 finding #2 wording: "intentionally best-effort"). +- [ ] Commit message follows Conventional Commits + DCO. + +--- + +### Story 1.2 — Lever 1 heap-cap + `COMPOSE_PROJECT_NAME` pin (FR-2) + +**Outcome:** The smoke-test job's "Bring up the stack" step sets `SOLR_HEAP_SIZE: "256m"` (step-level) so Solr boots with a 256m heap on the GHA runner. The smoke-test job's job-level `env:` sets `COMPOSE_PROJECT_NAME: "relyloop"` so the container-name prefix is deterministic (`relyloop-solr-1`, `relyloop-opensearch-1`) for both the `make up` step AND the later failure-diagnostics step (GHA step-level env does NOT persist to later steps — per cycle-3 finding #1, this MUST be job-level). + +**New files** + +| File | Purpose | +|---|---| +| _(none)_ | This story modifies an existing workflow only. | + +**Modified files** + +| File | Change | +|---|---| +| `.github/workflows/pr.yml` (smoke-test job — line 487 `smoke-test:` declaration block) | (a) Add a job-level `env:` block immediately after `permissions:` (line 502-503) containing `COMPOSE_PROJECT_NAME: "relyloop"`. (b) Add `SOLR_HEAP_SIZE: "256m"` to the "Bring up the stack" step's existing env block (line 621-628), alongside `RELYLOOP_GIT_SHA`, `RELYLOOP_SKIP_BUILD`, and `RELYLOOP_SKIP_AUTO_SEED`. | + +**Endpoints / Key interfaces / Pydantic schemas** + +N/A. + +**Tasks** + +1. Read `.github/workflows/pr.yml` lines 487-510 (job header + `permissions:` block) and lines 620-629 ("Bring up the stack" step) to confirm insertion points. **Critical sanity check (per cycle-1 finding #12):** verify NO job-level `env:` block already exists between `permissions:` and `steps:` on the `smoke-test:` job — duplicate YAML keys at the same level produce undefined merge behavior. If a job-level `env:` block has been added since spec time, APPEND `COMPOSE_PROJECT_NAME: "relyloop"` to it rather than creating a second block. Check with: `python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); print('existing job-env keys:', list((wf['jobs']['smoke-test'].get('env') or {}).keys()))"`. +2. Insert a job-level `env:` block after `permissions:` (typical GHA layout: `permissions:` then `env:` then `steps:`) — OR extend the existing one per the sanity check above: + ```yaml + smoke-test: + name: smoke (operator-path tutorial flow) + if: ${{ vars.SKIP_HEAVY_CI != 'true' }} + runs-on: ubuntu-24.04 + timeout-minutes: 15 + needs: [docker, docker-ui] + permissions: + contents: read + env: + # Pin Compose project name so container-name prefixes (relyloop-solr-1, + # relyloop-opensearch-1) are deterministic for diagnostics grep across + # both `make up` and the later failure-diagnostics step (GHA step-level + # env does NOT persist to later steps). + # See infra_solr_smoke_stability spec FR-2 + AC-2. + COMPOSE_PROJECT_NAME: "relyloop" + steps: + ... + ``` +3. Edit the "Bring up the stack" step's `env:` block to add `SOLR_HEAP_SIZE: "256m"`: + ```yaml + - name: Bring up the stack + env: + RELYLOOP_GIT_SHA: ${{ github.sha }} + RELYLOOP_SKIP_BUILD: "1" + RELYLOOP_SKIP_AUTO_SEED: "1" + # Cap Solr heap to reduce JVM memory pressure on the GHA runner. + # Compose's solr service reads ${SOLR_HEAP_SIZE:-512m}; the 512m default + # is correct for local dev, the 256m cap is CI-only. + # See infra_solr_smoke_stability spec FR-2 + decision log D-1, D-5. + SOLR_HEAP_SIZE: "256m" + run: make up + ``` +4. Run AC-4 validation: + ```bash + # Step-level SOLR_HEAP_SIZE: + python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); steps=wf['jobs']['smoke-test']['steps']; step=next(s for s in steps if (s.get('run') or '').strip()=='make up'); assert (step.get('env') or {}).get('SOLR_HEAP_SIZE')=='256m', step.get('env')" + # Job-level COMPOSE_PROJECT_NAME: + python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); job=wf['jobs']['smoke-test']; assert (job.get('env') or {}).get('COMPOSE_PROJECT_NAME')=='relyloop', job.get('env')" + # Structural Compose render: + SOLR_HEAP_SIZE=256m docker compose config --format json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['services']['solr']['environment']['SOLR_HEAP']=='256m'; print('ok')" + ``` +5. Verify Story 1.1's edits + Story 1.2's edits don't conflict — run AC-3 to confirm local `make up` still resolves Solr heap to 512m (no env set): + ```bash + docker compose config --format json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['services']['solr']['environment']['SOLR_HEAP']=='512m', d['services']['solr']['environment']; print('local default unchanged')" + git diff main -- docker-compose.yml # must be empty + ``` +6. Commit: `infra(smoke-ci): cap solr heap to 256m + pin COMPOSE_PROJECT_NAME for diagnostics`. + +**Definition of Done (DoD)** + +- [ ] `.github/workflows/pr.yml` smoke-test job has a job-level `env:` block with `COMPOSE_PROJECT_NAME: "relyloop"`. +- [ ] The "Bring up the stack" step's step-level `env:` block contains `SOLR_HEAP_SIZE: "256m"` alongside the existing entries. +- [ ] AC-4 YAML parse checks (both step-level + job-level) succeed. +- [ ] AC-4 structural Compose check succeeds (Solr renders as `SOLR_HEAP: 256m` with the env var set). +- [ ] AC-3 local-default check succeeds (Solr renders as `SOLR_HEAP: 512m` without the env var set; `git diff main -- docker-compose.yml` empty). +- [ ] Both env entries carry inline comments explaining the rationale + linking to the spec. +- [ ] Commit message follows Conventional Commits + DCO. + +--- + +### Story 1.3 — Push, watch CI, triage outcome, file follow-up if red (FR-3) + +**Outcome:** The PR is opened with both Story 1.1 + Story 1.2 commits. CI runs to a definitive outcome on `pr.yml`. Outcome triage: +- **Smoke green** → AC-1 / AC-3 / AC-4 all satisfied; proceed to Epic 2. +- **Smoke red** → AC-2 verified from the artifact (Solr + OpenSearch sections present, `docker inspect` exit-state present); branch protection re-verified; follow-up spec stub for Lever 2 (or Lever 3, based on captured evidence) filed AND linked from this PR's body; THEN proceed to Epic 2. + +**New files** + +| File | Purpose | +|---|---| +| _(conditional)_ `docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_lever2/idea.md` (or `..._lever3/`, depending on evidence) | Created ONLY if smoke is red. Tiny idea-stage file (10-30 lines) scoped to the next lever per the FR-4 runbook escalation triggers. Linked from this PR's body. | + +**Modified files** + +| File | Change | +|---|---| +| _(conditional)_ This PR's body on GitHub | Add a "Follow-up" section linking the lever-2/3 idea file path. ONLY in the smoke-red case. | + +**Tasks** + +1. Push the branch: `git push -u origin feature/infra-solr-smoke-stability`. +2. Open the PR with `gh pr create`. PR title: `infra(smoke-ci): cap Solr heap + capture solr/opensearch logs on smoke failure`. PR body lists Story 1.1 + 1.2 + Epic 2 outcomes and explicitly notes the D-6 red-merge-allowed posture. +3. Identify the PR run constrained to `pr.yml` AND the current branch HEAD SHA (per cycle-1 finding #9, the unconstrained `gh run list --branch ...` can return a stale or wrong-workflow run): + ```bash + HEAD_SHA=$(git rev-parse HEAD) + RUN_ID=$(gh run list --workflow pr.yml --branch feature/infra-solr-smoke-stability --limit 5 --json databaseId,headSha -q ".[] | select(.headSha == \"$HEAD_SHA\") | .databaseId" | head -1) + echo "RUN_ID=$RUN_ID for HEAD_SHA=$HEAD_SHA" + gh run watch "$RUN_ID" + ``` +4. Once CI completes, derive the job-by-job outcome: + ```bash + gh run view "$RUN_ID" --json jobs -q '.jobs[] | "\(.conclusion)\t\(.name)"' + ``` +5. **AC-1 check — mechanical (per cycle-2 plan finding #4):** assert every non-smoke job is `success` AND smoke is `success` or `failure` (NOT `cancelled`/`skipped`). Exits non-zero on failure: + ```bash + python3 - <<'PY' "$RUN_ID" + import json, subprocess, sys + run_id = sys.argv[1] + wf = __import__('yaml').safe_load(open('.github/workflows/pr.yml')) + expected_jobs = {j.get('name', jid) for jid, j in wf['jobs'].items()} + smoke_name = 'smoke (operator-path tutorial flow)' + gh = subprocess.run(['gh', 'run', 'view', run_id, '--json', 'jobs'], capture_output=True, text=True, check=True) + jobs = json.loads(gh.stdout)['jobs'] + actual = {j['name']: j['conclusion'] for j in jobs} + bad = [] + for name in expected_jobs: + if name not in actual: + bad.append(f'MISSING: {name}') + continue + c = actual[name] + if name == smoke_name: + if c not in ('success', 'failure'): + bad.append(f'SMOKE_UNTRIAGED: {c}') + elif c != 'success': + bad.append(f'NON_SMOKE_NOT_SUCCESS: {name} = {c}') + if bad: + print('AC-1 FAILED:', '; '.join(bad), file=sys.stderr); sys.exit(1) + print(f'AC-1 ok (smoke={actual[smoke_name]})') + PY + ``` +6. **Branch on smoke outcome:** + + **(a) Smoke green path:** + - AC-1 + AC-3 + AC-4 are all green. Proceed to Epic 2. + - No follow-up artifact needed. The Outcome (B) half delivered. + + **(b) Smoke red path:** + - Verify the failure-diagnostics artifact name matches the upload step (per cycle-1 finding #10): + ```bash + python3 -c "import yaml; wf=yaml.safe_load(open('.github/workflows/pr.yml')); steps=wf['jobs']['smoke-test']['steps']; step=next(s for s in steps if s.get('name')=='Upload failure diagnostics'); print('artifact name:', step['with']['name'])" + # Expected: smoke-logs + ``` + - Download the failure-diagnostics artifact to a known dir (robust against `gh run download`'s extract-to-subdir behavior): + ```bash + rm -rf /tmp/smoke-artifacts && mkdir -p /tmp/smoke-artifacts + gh run download "$RUN_ID" --name smoke-logs --dir /tmp/smoke-artifacts + LOGS=$(find /tmp/smoke-artifacts -name smoke-logs.txt -print -quit) + test -n "$LOGS" || (echo "smoke-logs.txt not in artifact" && exit 1) + ``` + - Run AC-2 verification on the downloaded `$LOGS` — mechanical assertion (per cycle-2 plan findings #5): + ```bash + grep -Eq '^relyloop-solr-1[[:space:]]+\|' "$LOGS" || { echo "AC-2 FAILED: no relyloop-solr-1 log lines" >&2; exit 1; } + grep -Eq '^relyloop-opensearch-1[[:space:]]+\|' "$LOGS" || { echo "AC-2 FAILED: no relyloop-opensearch-1 log lines" >&2; exit 1; } + grep -Eq 'relyloop-solr-1 exit=[0-9-]+ oom=(true|false) error=' "$LOGS" || { echo "FAILED: missing solr docker inspect exit-state line (Story 1.1 adjunct)" >&2; exit 1; } + echo "AC-2 ok" + ``` + (Note: the inspect line emits `oom=true` / `oom=false` per the `{{.State.OOMKilled}}` template in Story 1.1 — NOT `OOMKilled=true`. Runbook prose in Story 2.1 uses the same `oom=...` wording for consistency, per cycle-1 finding #11.) + - **Re-verify branch protection — mechanical (per cycle-2 plan finding #6 + cycle-3 plan finding #6):** explicitly inspect the HTTP status to distinguish 404 (no protection — pass) from 403 (auth/permission failure — escalate, do NOT proceed); parse the JSON and fail if smoke is required. + ```bash + python3 - <<'PY' + import json, subprocess, sys + # Use `gh api --include` to get the HTTP status line, not just the body. + r = subprocess.run(['gh', 'api', '-i', 'repos/SoundMindsAI/relyloop/branches/main/protection'], capture_output=True, text=True) + # `-i` puts status on stdout (header), body after; on non-2xx gh still exits non-zero. + status_line = r.stdout.splitlines()[0] if r.stdout else '' + if '404' in status_line: + print('branch-protection: HTTP 404 (no protection) — red-merge path open') + sys.exit(0) + if '403' in status_line or '401' in status_line: + print(f'branch-protection: AUTH FAILURE ({status_line}) — cannot verify; ESCALATE (do not assume no protection)', file=sys.stderr) + sys.exit(1) + if r.returncode != 0 and not r.stdout: + print(f'branch-protection: unexpected gh failure: {r.stderr}', file=sys.stderr); sys.exit(1) + # Parse the body (after the blank line that separates headers from body) + body = r.stdout.split('\n\n', 1)[1] if '\n\n' in r.stdout else r.stdout + d = json.loads(body) + required = (d.get('required_status_checks') or {}).get('contexts', []) + [c['context'] for c in (d.get('required_status_checks') or {}).get('checks', [])] + if any('smoke' in c.lower() for c in required): + print(f'branch-protection: smoke IS required ({required}) — RED-MERGE PATH CLOSED. Apply Lever 2 as follow-up commit BEFORE merge.', file=sys.stderr) + sys.exit(1) + print(f'branch-protection: smoke NOT in required checks ({required or "[]"}) — red-merge path open') + PY + ``` + If the script exits non-zero, STOP. Auth-failure case: re-authenticate (`gh auth status` / `gh auth refresh`); only proceed once status is unambiguous. Smoke-required case: apply Lever 2 as a follow-up commit on this same branch BEFORE merge, OR split into two PRs. + - **Read the captured Solr logs** to determine the failure mode (per FR-4 runbook trigger mapping in Story 2.1 §3): + - `oom=true` in the inspect line OR a JVM heap/metaspace/native-memory OOM trace in `^relyloop-solr-1 |` lines → **memory-pressure escalation** (revisit heap sizing across ES + OpenSearch + Solr together — the §13 Known Risk path; NOT just reapplying the Solr cap). + - Solr listening successfully (boot succeeded per `^relyloop-solr-1 |` log lines) but the artifact's inspect line shows `health=unhealthy` or `health=starting` past the effective healthcheck tolerance (`start_period: 30s` + `interval: 10s` × `retries: 6` with `timeout: 5s` per docker-compose.yml — up to ~95s; do NOT use a flat 30s cutoff per cycle-2 plan finding #8 + cycle-3 plan finding #4) → **Lever 2** (healthcheck-timing escalation via a CI-only override mechanism — details locked in the Lever 2 follow-up spec; per cycle-1 finding #7, do NOT presume the exact YAML shape now). + - Solr unavailable AND the tutorial-path smoke test genuinely doesn't depend on Solr (requires a full smoke-path audit, NOT just a one-file grep, per cycle-1 finding #8) → **Lever 3** (smoke-tolerance — multi-file scope, full spec). + - **File the follow-up artifact** at `docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_/idea.md` (the slug picks per the chosen escalation: `_memory_pressure_revisit`, `_start_period_lever2`, or `_tolerance_lever3`). Per the spec's relaxed FR-3 / DoD wording: an **idea-stage file** is the minimum bar (acceptable for one-line lever YAML edits); a full spec is warranted if the scope is multi-file (e.g., Lever 3 tolerance audit). The idea file should be ~30-50 lines: Origin (this PR run URL + captured Solr exit reason + `oom=` value), Problem (one paragraph), Proposed lever (one of the three above), Why deferred (waiting on the data we just captured = now captured), Relationship to this work (cite this PR). + - **Link the new artifact from this PR's body** by editing the PR via `gh pr edit --body-file `. The link is the mechanical forcing function — the PR body MUST contain a "Follow-up" section pointing at the new artifact's path BEFORE merge. **After editing, verify mechanically (per cycle-2 plan finding #7):** + ```bash + gh pr view --json body -q .body | grep -F "docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_" || { echo "PR body missing follow-up link" >&2; exit 1; } + ``` + +7. Commit the follow-up artifact (if smoke red) on this same feature branch — that keeps the PR + follow-up reference in one place rather than across branches. Commit message: `docs(planned-features): capture infra_solr_smoke_ follow-up`. + +8. **Re-trigger CI** (since steps 6-7 may have added a commit). Re-derive `RUN_ID` per step 3 with the new `HEAD_SHA`, then re-watch. The merge-target run is the LATEST `pr.yml` run on the final HEAD SHA, not the run from step 3 (per cycle-1 finding #2). + +**Definition of Done (DoD)** + +- [ ] PR is open on GitHub. +- [ ] CI completed; smoke job's outcome is one of `success` or `failure` (NOT `cancelled` / `skipped`). +- [ ] All non-smoke `pr.yml` jobs are `success` per AC-1. +- [ ] **If smoke green:** Story 1.3 is complete; no follow-up needed. +- [ ] **If smoke red:** AC-2 grep on the failure-diagnostics artifact succeeds AND branch protection re-check confirms the smoke job is NOT a required status check AND a follow-up idea file exists for the next lever AND the PR body links it under a "Follow-up" section. + +--- + +## Epic 2 — Documentation + +**Goal:** Write the runbook so future maintainers have an evidence-driven escalation path, and surface it in CLAUDE.md so it's reachable from the project's canonical conventions doc. Runs AFTER Epic 1 verification (Story 1.3 outcome triage) so the runbook documents what was observed empirically, not what was predicted. + +### Story 2.1 — Write `smoke-solr-stability.md` runbook (FR-4) + +**Outcome:** A new runbook file exists at `docs/03_runbooks/smoke-solr-stability.md` documenting the heap-cap rationale, the evidence-mapped lever cascade (Lever 1 → Lever 2 → Lever 3 → sibling-JVM-cap escalation), and the diagnostic-first workflow. Modeled structurally on the existing [`demo-reseed-engine-tolerance.md`](../../../../03_runbooks/demo-reseed-engine-tolerance.md) (the Phase 1 sibling runbook) for visual consistency. + +**New files** + +| File | Purpose | +|---|---| +| `docs/03_runbooks/smoke-solr-stability.md` | Runbook documenting (a) heap-cap rationale + GHA-only scope; (b) evidence-mapped lever cascade with explicit triggers (not symptom-mapped); (c) diagnostic-first workflow — "read smoke-logs `relyloop-solr-1` section + `docker inspect` exit state BEFORE picking a lever." Includes a "What to do if smoke goes red after Lever 1" decision tree. Carries the SPDX header pattern used by all RelyLoop runbooks. | + +**Modified files** + +| File | Change | +|---|---| +| _(none)_ | Story 2.2 handles CLAUDE.md separately. | + +**Endpoints / Key interfaces / Pydantic schemas** + +N/A. + +**Tasks** + +1. Read [`docs/03_runbooks/demo-reseed-engine-tolerance.md`](../../../../03_runbooks/demo-reseed-engine-tolerance.md) lines 1-30 for the SPDX header + Owner/Audience pattern. +2. Create the new runbook file with sections: + - **Header:** SPDX-FileCopyrightText + SPDX-License-Identifier (per the REUSE convention; the freshness gate passes 1655/1655 currently). + - **Owner:** `infra_solr_smoke_stability` (this folder, once finalized: `implemented_features/2026_06_01_infra_solr_smoke_stability/`). + - **Audience:** maintainers diagnosing a red `smoke` job on a PR. + - **§1: Why Solr heap is capped at 256m in CI.** Reference the ES `ES_JAVA_OPTS` precedent at [`pr.yml:287`](../../../../../.github/workflows/pr.yml#L287); call out the hypothesis nature (spec D-1 + §13 Known Risk); the GHA-runner-only scope (Compose default of 512m stays for local dev). + - **§2: When smoke goes red — the diagnostic workflow.** Step-by-step: (a) `gh run download --name smoke-logs`; (b) `grep -E '^relyloop-solr-1' smoke-logs.txt` to read Solr container output; (c) `grep 'relyloop-solr-1 exit=' smoke-logs.txt` to read the `docker inspect` exit-state line (FR-1 adjunct); (d) classify the failure per §3 below. + - **§3: The lever cascade (evidence-mapped, not symptom-mapped).** Lever 1 is the CURRENT baseline (already shipped by the PR this runbook ships with); Levers 2 / 3 / memory-pressure are FUTURE escalations triggered if the smoke job goes red AFTER Lever 1 is in place. Each escalation entry: trigger evidence + the lever + the rationale + the file edit shape. + - **Lever 1 (CURRENT, this PR):** Solr heap capped to 256m via `SOLR_HEAP_SIZE: "256m"` step-env on the smoke job's "Bring up the stack" step. This is the baseline state. Do NOT re-apply. + - **IF Lever 1 didn't fix the crash, escalate based on the captured evidence:** + - **Memory-pressure escalation (heap, metaspace, native OOM, OR kernel `oom=true`):** the spec's §13 Known Risk path. The smoke Compose stack runs three JVM services (Solr + ES + OpenSearch) concurrently; the backend job's `ES_JAVA_OPTS` env applies to GHA service containers NOT to the smoke job's Compose stack, so ES + OpenSearch run at their Compose defaults today. Edit: cap ES + OpenSearch heap in the smoke Compose path too (via `ES_JAVA_OPTS` env override on smoke + Compose env-var slot for OpenSearch heap; details locked in the follow-up spec). Multi-file scope. This is the unified escalation for both Solr-internal OOM traces and kernel-side `oom=true` (per cycle-1 plan finding #6 — Lever 1 was previously listed separately for these; that's wrong because the lever is what changed, the failure mode is what triggers the next change). + - **Lever 2 (healthcheck-timing escalation):** Trigger: Solr boots successfully (logs show "Started SolrJetty" or equivalent) but the container's effective healthcheck tolerance window (`start_period: 30s` + `interval: 10s` × `retries: 6` with `timeout: 5s` per [`docker-compose.yml:280-285`](../../../../../docker-compose.yml#L280-L285) — total tolerance up to ~95s, NOT a flat 30s window per cycle-2 plan finding #8) elapsed before the healthcheck went green. Read Solr log timestamps and compare to the healthcheck transitions in `docker inspect` to confirm — don't assume "boot > 30s = Lever 2." Edit: add a CI-only override mechanism for the Solr healthcheck timing, preserving the local-dev defaults. The exact YAML shape (env-var interpolation in healthcheck values vs. a docker-compose override file vs. a step-level `docker compose ... --wait-timeout` flag) is locked in the Lever-2 follow-up spec. + - **Lever 3 (smoke-tolerance):** Trigger: Solr genuinely unavailable AND the smoke path genuinely doesn't depend on Solr. **Important:** Lever 3 selection requires a FULL smoke-path audit, not a single-file grep. The smoke path can depend on Solr transitively via `/healthz` (the api probes Solr as a subsystem), seed/migrate steps, fixtures, or backend service calls. The audit covers: `backend/tests/smoke/test_tutorial_path.py` + all seed scripts (`make seed-clusters`, `make seed-es`) + the api's `/healthz` Solr-probe behavior (currently treats a missing Solr as a `subsystems.solr` field, not a job failure — per spec §2 audit) + the tutorial walkthrough flow. Only if Solr is genuinely unreferenced across that surface is Lever 3 viable. Edit: change the smoke job's success criteria to tolerate `subsystems.solr: down` in `/healthz` AND adjust any tutorial-path assertions that touch Solr (per cycle-2 plan finding #9 — there is no dedicated "Solr healthcheck wait" step to drop; the dependency is via `/healthz` subsystem semantics + tutorial-path implicit assumptions). Multi-file scope, full spec. + - **§4: Why each lever stays GHA-only.** Local dev (`make up`) runs without `SOLR_HEAP_SIZE`/`COMPOSE_PROJECT_NAME` so Compose defaults apply: Solr 512m heap, project name derived from working directory. Don't change `docker-compose.yml` defaults (per spec D-5). +3. Verify SPDX gate stays green: `uv run reuse lint` → 1655+/1655+. +4. Commit: `docs(runbooks): smoke-solr-stability — heap-cap rationale + evidence-mapped lever cascade`. + +**Definition of Done (DoD)** + +- [ ] `docs/03_runbooks/smoke-solr-stability.md` exists with the four sections (§1 rationale, §2 diagnostic workflow, §3 lever cascade with evidence-mapped triggers, §4 GHA-only scope). +- [ ] SPDX header present (matching the demo-reseed-engine-tolerance.md pattern). +- [ ] `uv run reuse lint` still reports `Congratulations! Your project is compliant`. +- [ ] All cited file references resolve (`test -f` passes for `.github/workflows/pr.yml`, the existing sibling runbook). +- [ ] Lever cascade in §3 uses **evidence-mapped triggers** (not symptom-mapped) per cycle-3 finding #3. +- [ ] Commit message follows Conventional Commits + DCO. + +--- + +### Story 2.2 — Surface the runbook in CLAUDE.md (FR-4) + + + +**Outcome:** CLAUDE.md "Key Runbooks" table contains a row linking to `docs/03_runbooks/smoke-solr-stability.md`, so a maintainer hitting a red smoke job can find the lever cascade from the project's canonical conventions doc. + +**New files** + +| File | Purpose | +|---|---| +| _(none)_ | This story modifies an existing file only. | + +**Modified files** + +| File | Change | +|---|---| +| [`CLAUDE.md`](../../../../../CLAUDE.md) "Key Runbooks" table | Add a row after the "Demo reseed engine tolerance" row (the structural sibling) linking to `docs/03_runbooks/smoke-solr-stability.md`. Row format: `\| Smoke job Solr stability — lever cascade (heap-cap / start_period / smoke-tolerance) for a red `smoke` CI job \| [`docs/03_runbooks/smoke-solr-stability.md`](docs/03_runbooks/smoke-solr-stability.md) (`infra_solr_smoke_stability`) \|`. | + +**Tasks** + +1. Open [`CLAUDE.md`](../../../../../CLAUDE.md), find the "Key Runbooks" table (search for `## Key Runbooks` or the "Demo reseed engine tolerance" row added by `infra_solr_ci_readiness` Phase 1). +2. Add the new row immediately after the "Demo reseed engine tolerance" row — these two are sibling runbooks (backend half + smoke half of the same Solr CI debt). +3. Run AC-5 verification: + ```bash + grep -q 'smoke-solr-stability.md' CLAUDE.md && test -f docs/03_runbooks/smoke-solr-stability.md && echo "AC-5 ok" + ``` +4. Commit: `docs(claude-md): link smoke-solr-stability runbook from Key Runbooks table`. + +**Definition of Done (DoD)** + +- [ ] `CLAUDE.md` "Key Runbooks" table contains a row matching the pattern in "Modified files" above. +- [ ] AC-5 verification command (Task 3) succeeds. +- [ ] Row placement is immediately after the "Demo reseed engine tolerance" row. +- [ ] Commit message follows Conventional Commits + DCO. + +--- + +### Story 2.3a — Pre-merge final CI verification + cleanup of stale red→green follow-ups (cycle-1 plan findings #2 + #3; cycle-3 plan finding #1 sequencing fix; cycle-3 plan finding #2 red→green cleanup) + +**Outcome:** The FINAL `pr.yml` run on the FINAL HEAD SHA (post-Epic-2 commits, post any smoke-red follow-up commit) is re-verified to satisfy AC-1 + (if final-smoke red) AC-2. If Story 1.3 created a follow-up artifact + PR-body link because Story 1.3's run was red, but the final run is green, the stale artifact is deleted and the PR body's "Follow-up" section is removed. Sequencing matters: state.md is NOT updated in this story (state.md updates need the post-merge squash SHA, so they happen in Story 2.3b as a separate post-merge PR per the project's established pattern — see PR #368 finalizing PR #367). + +**New files / Modified files** + +| File | Change | +|---|---| +| _(conditional)_ `docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_/` | DELETED if Story 1.3 created it AND the final run is green (red→green divergence cleanup). | +| _(conditional)_ This PR's body on GitHub | "Follow-up" section REMOVED if Story 1.3 added it AND the final run is green. | + +(Note: `state.md` updates do NOT live in this story — see Story 2.3b. Pre-merge state.md edits would force a chicken-and-egg loop because they change HEAD which invalidates the just-verified CI run, per cycle-3 plan finding #1. The squash-merge SHA is also unknown until merge, per cycle-3 plan finding #5.) + +**Endpoints / Key interfaces / Pydantic schemas** + +N/A. + +**Tasks** + +1. Determine the FINAL HEAD SHA (post-Epic-2 commits, post any smoke-red follow-up commit): `HEAD_SHA=$(git rev-parse HEAD)`. +2. Re-derive the merge-target `pr.yml` run for that SHA: + ```bash + FINAL_RUN_ID=$(gh run list --workflow pr.yml --branch feature/infra-solr-smoke-stability --limit 5 --json databaseId,headSha -q ".[] | select(.headSha == \"$HEAD_SHA\") | .databaseId" | head -1) + gh run watch "$FINAL_RUN_ID" + ``` +3. Re-run the mechanical AC-1 assertion script from Story 1.3 step 5 against `$FINAL_RUN_ID`. +4. **Re-verify AC-2 on the FINAL run whenever smoke is red** (per cycle-2 plan finding #2 — not just when the outcome changed from Story 1.3). If the final smoke conclusion is `failure`: + - Re-download the smoke-logs artifact from `$FINAL_RUN_ID`: `gh run download "$FINAL_RUN_ID" --name smoke-logs --dir /tmp/smoke-artifacts-final`. + - Re-run the AC-2 mechanical assertions on the FINAL artifact (per Story 1.3 step 6(b)'s grep/exit-1 commands). + - Re-read the FINAL artifact's Solr exit reason + `oom=` value. + - **If the final evidence diverges from Story 1.3's evidence**, re-execute Story 1.3 step 6(b)'s "File the follow-up artifact" + "Link the new artifact from this PR's body" tasks against the FINAL evidence, AMENDING (not duplicating) the Story-1.3 idea file. The PR body link must point at the artifact that reflects the merge-target evidence. +5. **Red→green cleanup (per cycle-3 plan finding #2):** If Story 1.3 was triggered (smoke red on the initial run) AND the FINAL run is `success`: + - Delete the follow-up artifact folder created by Story 1.3 (e.g., `rm -rf docs/00_overview/planned_features/02_mvp2/infra_solr_smoke_/`). + - Edit the PR body to REMOVE the "Follow-up" section: `gh pr edit --body-file ` and verify with `gh pr view --json body -q .body | grep -F 'infra_solr_smoke_' && { echo "PR body still references the deleted artifact" >&2; exit 1; } || true`. + - Commit on the feature branch: `chore(planned-features): drop infra_solr_smoke_ follow-up — final smoke run went green`. + - **Iterate Task 2-4 once more** for the new HEAD SHA created by this cleanup commit — verify the new final run is also `success` (or if it goes red, re-execute step 4 with the new evidence). One iteration max in practice; if smoke flakes between green and red across iterations, escalate to the user. +6. Verify the PR is ready to merge: + ```bash + gh pr view --json mergeStateStatus,statusCheckRollup -q '{state: .mergeStateStatus, checks: [.statusCheckRollup[] | {name, conclusion}]}' + ``` + Expect `state: CLEAN` or `state: UNSTABLE` (the latter if smoke red but other jobs green — D-6 allowed merge posture). + +**Definition of Done (DoD)** + +- [ ] Final `pr.yml` run on the final HEAD SHA satisfies AC-1 (mechanical assertion script exits 0). +- [ ] If smoke red: AC-2 mechanical assertions pass on the FINAL artifact; follow-up artifact reflects FINAL evidence (not stale Story-1.3 evidence); PR body link verified mechanically. +- [ ] If smoke red → green divergence: stale follow-up artifact deleted + PR body "Follow-up" section removed + the post-cleanup HEAD's `pr.yml` run also verified. +- [ ] `gh pr view --json mergeStateStatus` returns `CLEAN` or `UNSTABLE` (depending on smoke outcome). +- [ ] Commit message (if any cleanup commit was made) follows Conventional Commits + DCO. + +--- + +### Story 2.3b — Post-merge state.md finalization (separate follow-up PR) + +**Outcome:** A small follow-up PR updates `state.md` to reflect the merge outcome with the actual squash-merge SHA. This matches the project's established pattern (e.g., PR #368 `docs(state): finalize infra_solr_ci_readiness Phase 1 (PR #367 merged)` was a separate finalization PR on the same day as PR #367). It exists as a separate PR because (a) the squash SHA isn't known until merge (cycle-3 plan finding #5) and (b) committing state.md pre-merge would create a chicken-and-egg HEAD invalidation loop (cycle-3 plan finding #1). + +**Branch / PR structure** + +- Branch: `chore/finalize-infra-solr-smoke-stability` (off the post-merge `main`). +- PR title: `docs(state): finalize infra_solr_smoke_stability (PR # merged)`. +- Scope: `state.md` only. + +**New files** + +| File | Purpose | +|---|---| +| _(none)_ | `state.md` only. | + +**Modified files** + +| File | Change | +|---|---| +| [`state.md`](../../../../../state.md) "Last 5 merges" + "Known debt" sections | Prepend to "Last 5 merges" + UPDATE "Smoke half still open (Phase 2)" entry per the conditional below. | + +**Tasks** + +1. After the feature PR merges, capture the squash-merge SHA from `gh pr view --json mergeCommit -q .mergeCommit.oid`. +2. Branch off updated `main`: `git checkout main && git pull && git checkout -b chore/finalize-infra-solr-smoke-stability`. +3. Edit `state.md`: + - Prepend to "Last 5 merges" (newest first; drop the now-6th row): + - **Smoke green:** `**2026-MM-DD** — \`infra_solr_smoke_stability\` (PR #XXX, squash-merged \`\`). Capped Solr heap to 256m in the smoke job (\`SOLR_HEAP_SIZE: "256m"\` step-env on \`make up\`) + pinned \`COMPOSE_PROJECT_NAME=relyloop\` at job-env + added \`solr\` + \`opensearch\` + \`docker inspect\` exit-state to failure-diagnostics. Smoke job is now green on every branch — completes the \`pr.yml\`-green-on-every-branch contract that started with Phase 1. New runbook at \`docs/03_runbooks/smoke-solr-stability.md\` with the evidence-mapped lever cascade for future Solr CI debt.` + - **Smoke red:** `**2026-MM-DD** — \`infra_solr_smoke_stability\` (PR #XXX, squash-merged \`\`, smoke RED on merge per D-6 fast-lane posture). Diagnostics half landed: smoke-logs artifact now contains Solr + OpenSearch + \`docker inspect\` exit-state. Heap-cap half did NOT fix the crash — captured evidence: . Follow-up: \`>.` + - UPDATE "Known debt" entry "Smoke half still open (Phase 2)": + - **Smoke green:** Strike the entry entirely. Smoke debt resolved. + - **Smoke red:** Update the entry to: `**Smoke half: lever 1 (heap-cap) shipped 2026-MM-DD but did not fix the crash.** Captured evidence: . Next escalation: . Tracked in \`02_mvp2/infra_solr_smoke_\`.` +4. Commit + push + open follow-up PR. + +**Definition of Done (DoD)** + +- [ ] Follow-up PR opened with title `docs(state): finalize infra_solr_smoke_stability (PR # merged)`. +- [ ] `state.md` "Last 5 merges" prepended with the appropriate (green vs red) one-liner including the actual squash-merge SHA. +- [ ] `state.md` "Known debt" entry updated per the conditional logic above. +- [ ] No `state.md` size-gate violation (`wc -c state.md` < 60 KB). +- [ ] Follow-up PR's CI passes (it's docs-only; the same `pr.yml` matrix runs). +- [ ] Follow-up PR merged. + +--- + +## UI Guidance + +**N/A — no frontend scope in this plan.** No `