infra(smoke-ci): cap Solr heap + capture solr/opensearch logs on smoke failure

.github/workflows/pr.yml

@@ -491,7 +491,14 @@ jobs:
     # those skip too, so this stays consistent either way.)
     if: ${{ vars.SKIP_HEAVY_CI != 'true' }}
     runs-on: ubuntu-24.04
-    timeout-minutes: 15
+    # Bumped 15 → 25 min after infra_solr_smoke_stability PR #383: the
+    # demo-ubi.spec.ts beforeAll hook drives a reseed that AC-8 bounds at
+    # 24 min, and with Solr actually booting now (PR #383's Lever-0 perms
+    # fix unblocked it) the reseed actually exercises the Solr scenario
+    # — so total wall-clock for the smoke job's Playwright E2E step
+    # crossed 15 min and started hitting the job-level cap. 25 covers
+    # the reseed bound + setup/teardown headroom.
+    timeout-minutes: 25
     # Depend on the parallel `docker` (API) + `docker-ui` jobs so both image
     # artifacts are ready before `make up`. Pre-bumps this PR was paying ~10min
     # for `docker compose up -d` (image pulls + API + UI builds inside the
@@ -501,6 +508,13 @@ jobs:
     needs: [docker, docker-ui]
     permissions:
       contents: read
+    env:
+      # Pin Compose project name so container-name prefixes (relyloop-solr-1,
+      # relyloop-opensearch-1) are deterministic for the failure-diagnostics
+      # grep across both `make up` and the later log-collect step. GHA
+      # step-level env does NOT persist to later steps, so this MUST be
+      # job-level. See infra_solr_smoke_stability spec FR-2 + AC-2.
+      COMPOSE_PROJECT_NAME: "relyloop"
     steps:
       - uses: actions/checkout@v6
 
@@ -626,7 +640,27 @@ jobs:
           # (chore_drop_demo_seed_from_ci). Without this skip install.sh
           # would still auto-seed ~5min of demo data on every CI run.
           RELYLOOP_SKIP_AUTO_SEED: "1"
-        run: make up
+          # Cap Solr heap to reduce JVM memory pressure on the GHA runner.
+          # Compose's solr service reads ${SOLR_HEAP_SIZE:-512m}; the 512m
+          # default is correct for local dev (docker-compose.yml:274) — this
+          # 256m cap is CI-only. Mirrors the backend job's
+          # ES_JAVA_OPTS: "-Xms256m -Xmx256m" precedent at line 287.
+          # See infra_solr_smoke_stability spec FR-2 + decision log D-1, D-5.
+          SOLR_HEAP_SIZE: "256m"
+        # Pre-create ./data/solr with Solr's container UID (8983) so the
+        # bind-mount target is writable from inside the container. The
+        # solr:10.0 image runs as UID/GID 8983; on the GHA runner the
+        # bind-mount target defaults to root ownership, which makes Solr's
+        # boot script fail at "Cannot write to /var/solr as 8983:8983" and
+        # exit(1) within ~500ms (verified empirically on PR #383 run
+        # 26788939356 — diagnostics fold-in caught this; lever cascade did
+        # not anticipate it because all three locked levers assumed
+        # JVM-internal crash modes, not filesystem permissions).
+        # CI-only; never edit docker-compose.yml's defaults for this.
+        run: |
+          mkdir -p ./data/solr
+          sudo chown 8983:8983 ./data/solr
+          make up
 
       - name: Wait for /healthz
         run: |
@@ -715,8 +749,16 @@ jobs:
 
       - name: Collect docker compose logs on failure
         if: failure()
+        # `infra_solr_smoke_stability` FR-1: include solr + opensearch in the
+        # log collect list (both were missing — opensearch is a pre-existing
+        # gap), and append per-container `docker inspect` exit-state with
+        # OOMKilled / Health.Status / StartedAt / FinishedAt so total-runner
+        # memory pressure and slow-boot vs healthcheck-window failures are
+        # diagnosable from the artifact alone. All three lines `|| true` —
+        # intentionally best-effort.
         run: |
-          docker compose logs --no-color api worker postgres redis elasticsearch ui > smoke-logs.txt 2>&1 || true
+          docker compose logs --no-color api worker postgres redis elasticsearch opensearch solr ui > smoke-logs.txt 2>&1 || true
+          docker compose ps -aq | xargs -r docker inspect --format '{{.Name}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}} health={{with .State.Health}}{{.Status}}{{end}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}' >> smoke-logs.txt 2>&1 || true
           curl -s http://127.0.0.1:8000/healthz >> smoke-logs.txt 2>&1 || true
 
       - name: Upload failure diagnostics

CLAUDE.md

@@ -605,3 +605,4 @@ Run `/pipeline status` for the live view from spec dependencies.
 | Parallel-worktree workflow — sibling checkouts, `make test-worktree`, leak prevention | [`docs/03_runbooks/parallel-worktrees.md`](docs/03_runbooks/parallel-worktrees.md) (`infra_agent_sibling_worktree_isolation` Phase 2) |
 | Interpreting the convergence verdict (converged / still_improving / too_few_trials) | [`docs/03_runbooks/convergence-verdict.md`](docs/03_runbooks/convergence-verdict.md) (`feat_study_convergence_indicator`) |
 | Demo reseed engine tolerance — partial completion (`scenarios_skipped`), all-engines-unreachable failure, re-seed after starting an engine | [`docs/03_runbooks/demo-reseed-engine-tolerance.md`](docs/03_runbooks/demo-reseed-engine-tolerance.md) (`infra_solr_ci_readiness`) |
+| Smoke job Solr stability — lever cascade (heap-cap / start_period / smoke-tolerance) for a red `smoke` CI job, diagnostic workflow from the smoke-logs artifact | [`docs/03_runbooks/smoke-solr-stability.md`](docs/03_runbooks/smoke-solr-stability.md) (`infra_solr_smoke_stability`) |

docs/00_overview/BACKLOG_DASHBOARD.md

@@ -2,7 +2,7 @@
 
 # RelyLoop BACKLOG Dashboard
 
-_Reflects feature-folder state as of **2026-06-01** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`backlog_dashboard.html`](backlog_dashboard.html) in a browser._
+_Reflects feature-folder state as of **2026-06-02** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`backlog_dashboard.html`](backlog_dashboard.html) in a browser._
 
 ## Next up
 

docs/00_overview/DASHBOARD.md

@@ -1,13 +1,13 @@
 # RelyLoop — Release Roadmap
 
-_Top-level index across MVP1 → GA v1+ as of **2026-06-01**. Click a release name to drill into the per-release dashboard. Theme labels sourced from [`docs/01_architecture/tech-stack.md` §"Canonical release matrix"](../01_architecture/tech-stack.md). For the rich local view, open [`dashboard.html`](dashboard.html) in a browser._
+_Top-level index across MVP1 → GA v1+ as of **2026-06-02**. Click a release name to drill into the per-release dashboard. Theme labels sourced from [`docs/01_architecture/tech-stack.md` §"Canonical release matrix"](../01_architecture/tech-stack.md). For the rich local view, open [`dashboard.html`](dashboard.html) in a browser._
 
 ## Releases
 
 | Release | Theme | Progress | Status |
 |---|---|---|---|
 | [MVP1 / v0.1](MVP1_DASHBOARD.md) | The Loop | 94 / 94 scoped done | **Complete** |
-| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 8 / 17 scoped done · 27 remaining | **In progress** |
+| [MVP2 / v0.2](MVP2_DASHBOARD.md) | Three-Engine + Real Signals | 8 / 18 scoped done · 28 remaining | **In progress** |
 | MVP3 / v0.3 | Observable | — | **Not yet scoped** |
 | GA v1 / v1.0 | Production-ready | — | **Not yet scoped** |
 

docs/00_overview/MVP1_DASHBOARD.md

@@ -2,7 +2,7 @@
 
 # RelyLoop MVP1 Dashboard
 
-_Reflects feature-folder state as of **2026-06-01** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp1_dashboard.html`](mvp1_dashboard.html) in a browser._
+_Reflects feature-folder state as of **2026-06-02** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp1_dashboard.html`](mvp1_dashboard.html) in a browser._
 
 ## Next up
 

docs/00_overview/MVP2_DASHBOARD.md

@@ -2,7 +2,7 @@
 
 # RelyLoop MVP2 Dashboard
 
-_Reflects feature-folder state as of **2026-06-01** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp2_dashboard.html`](mvp2_dashboard.html) in a browser._
+_Reflects feature-folder state as of **2026-06-02** (latest mtime of any planned/implemented feature `.md` file). Regenerated by `make dashboard` and the `mvp1-dashboard-regen` pre-commit hook. For the rich local view (filter chips, type colors), open [`mvp2_dashboard.html`](mvp2_dashboard.html) in a browser._
 
 ## Next up
 
@@ -20,15 +20,15 @@ Plan approved; run /impl-execute to ship
 
 | Metric | Value |
 |---|---|
-| Filed under MVP2 | **39** folders total (done + specced not-done + idea backlog + bugs) |
-| Specced features done | **8 / 17** (47%) — of features *past the idea stage* (those with a spec); the idea backlog below is NOT in this denominator, so 100% ≠ release complete |
-| Pending work | **31** items (every not-done feat/infra/chore/bug across all priorities) |
+| Filed under MVP2 | **40** folders total (done + specced not-done + idea backlog + bugs) |
+| Specced features done | **8 / 18** (44%) — of features *past the idea stage* (those with a spec); the idea backlog below is NOT in this denominator, so 100% ≠ release complete |
+| Pending work | **32** items (every not-done feat/infra/chore/bug across all priorities) |
 | → P0 — do next | **0** unblocking / paying daily cost |
-| → P1 | **2** high-value, ready when P0 clears |
+| → P1 | **3** high-value, ready when P0 clears |
 | → P2 (default) | 25 important to file, not blocking |
 | → Backlog | 4 captured for record, not planned |
 | Open bugs | 10 |
-| Legacy "Path to MVP2" | 27 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) |
+| Legacy "Path to MVP2" | 28 items — scoped-not-done + bugs + chore-ideas only (excludes feat/infra ideas) |
 | Backlog ideas | 4 idea-only feat/infra (not yet scoped into MVP2) |
 | In flight | 0 feature(s) actively shipping |
 
@@ -51,19 +51,20 @@ Plan approved; run /impl-execute to ship
 
 _None._
 
-### Plan (9)
+### Plan (10)
 
 | # | Priority | Feature | Type | One-liner | Depends on | Status |
 |---|---|---|---|---|---|---|
-| 1 | P2 | [feat_apply_path_normalizer_declaration](planned_features/02_mvp2/feat_apply_path_normalizer_declaration/feature_spec.md) | Feature | The winning normalizer ships as a **structured, language-agnostic manifest** in the config-repo PR — not just prose. | — | — |
-| 2 | P2 | [feat_overnight_studies_summary_card](planned_features/02_mvp2/feat_overnight_studies_summary_card/feature_spec.md) | Feature | A "ran while you were away" card surfaces at the top of `/studies` when at least one overnight chain has completed since the operator's last visit. | — | [PR #343](https://github.com/SoundMindsAI/relyloop/pull/343) |
-| 3 | P2 | [feat_query_normalization_tuning](planned_features/02_mvp2/feat_query_normalization_tuning/feature_spec.md) | Feature | A template that opts in by declaring `query_normalizer` as a Categorical param gets the Optuna loop deciding empirically — on the operator's judgment set — whether lowercasing, trimming, or contractio | — | — |
-| 4 | P2 | [feat_query_normalizer_typed_pipeline](planned_features/02_mvp2/feat_query_normalizer_typed_pipeline/feature_spec.md) | Feature | A new typed search-space member `NormalizerPipelineParam` lets a template declare an **ordered list of normalization steps**; the Optuna loop samples over the powerset of declared steps and proposes t | — | — |
-| 5 | P2 | [feat_ubi_llm_study_comparison](planned_features/02_mvp2/feat_ubi_llm_study_comparison/feature_spec.md) | Feature | A single dedicated route `/studies/compare?a={id}&b={id}` renders the two studies side-by-side with a per-panel diff column: a sentence-level digest-narrative diff, a best-trial parameter table with s | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) |
-| 6 | P2 | [infra_generated_artifact_freshness_gate](planned_features/02_mvp2/infra_generated_artifact_freshness_gate/feature_spec.md) | Infra | CI fails a PR whose committed `types.ts` does not match what the live OpenAPI schema would produce, and whose `ui/public/docs/*` copies do not match their `docs/08_guides/*` sources. | — | — |
-| 7 | P2 | [chore_arq_pool_aclose_deprecation](planned_features/02_mvp2/chore_arq_pool_aclose_deprecation/feature_spec.md) | Chore | Both call sites use `await arq_pool.aclose()`; no `DeprecationWarning` on shutdown; a regression guard asserts the async-correct form on both paths so a future edit cannot silently reintroduce `close( | — | — |
-| 8 | P2 | [chore_cluster_detail_rung_badge](planned_features/02_mvp2/chore_cluster_detail_rung_badge/feature_spec.md) | Chore | The cluster-detail page surfaces a `<UbiRungBadge>` for the cluster, scoped by a user-selected (or auto-seeded) query set + target. | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) |
-| 9 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/feature_spec.md) | Chore | The 9 skipped cases are rewritten to the async "POST + poll-until-terminal" shape, the timeout case is re-homed to the worker layer, a new `AC-Async` case asserts the `running → complete` polling tran | — | [PR #286](https://github.com/SoundMindsAI/relyloop/pull/286) |
+| 1 | P1 | [infra_solr_smoke_stability](planned_features/02_mvp2/infra_solr_smoke_stability/feature_spec.md) | Infra | The `pr.yml` `smoke (operator-path tutorial flow)` job is red on every branch. | — | [PR #367](https://github.com/SoundMindsAI/relyloop/pull/367) merged 2026-06-01 |
+| 2 | P2 | [feat_apply_path_normalizer_declaration](planned_features/02_mvp2/feat_apply_path_normalizer_declaration/feature_spec.md) | Feature | The winning normalizer ships as a **structured, language-agnostic manifest** in the config-repo PR — not just prose. | — | — |
+| 3 | P2 | [feat_overnight_studies_summary_card](planned_features/02_mvp2/feat_overnight_studies_summary_card/feature_spec.md) | Feature | A "ran while you were away" card surfaces at the top of `/studies` when at least one overnight chain has completed since the operator's last visit. | — | [PR #343](https://github.com/SoundMindsAI/relyloop/pull/343) |
+| 4 | P2 | [feat_query_normalization_tuning](planned_features/02_mvp2/feat_query_normalization_tuning/feature_spec.md) | Feature | A template that opts in by declaring `query_normalizer` as a Categorical param gets the Optuna loop deciding empirically — on the operator's judgment set — whether lowercasing, trimming, or contractio | — | — |
+| 5 | P2 | [feat_query_normalizer_typed_pipeline](planned_features/02_mvp2/feat_query_normalizer_typed_pipeline/feature_spec.md) | Feature | A new typed search-space member `NormalizerPipelineParam` lets a template declare an **ordered list of normalization steps**; the Optuna loop samples over the powerset of declared steps and proposes t | — | — |
+| 6 | P2 | [feat_ubi_llm_study_comparison](planned_features/02_mvp2/feat_ubi_llm_study_comparison/feature_spec.md) | Feature | A single dedicated route `/studies/compare?a={id}&b={id}` renders the two studies side-by-side with a per-panel diff column: a sentence-level digest-narrative diff, a best-trial parameter table with s | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) |
+| 7 | P2 | [infra_generated_artifact_freshness_gate](planned_features/02_mvp2/infra_generated_artifact_freshness_gate/feature_spec.md) | Infra | CI fails a PR whose committed `types.ts` does not match what the live OpenAPI schema would produce, and whose `ui/public/docs/*` copies do not match their `docs/08_guides/*` sources. | — | — |
+| 8 | P2 | [chore_arq_pool_aclose_deprecation](planned_features/02_mvp2/chore_arq_pool_aclose_deprecation/feature_spec.md) | Chore | Both call sites use `await arq_pool.aclose()`; no `DeprecationWarning` on shutdown; a regression guard asserts the async-correct form on both paths so a future edit cannot silently reintroduce `close( | — | — |
+| 9 | P2 | [chore_cluster_detail_rung_badge](planned_features/02_mvp2/chore_cluster_detail_rung_badge/feature_spec.md) | Chore | The cluster-detail page surfaces a `<UbiRungBadge>` for the cluster, scoped by a user-selected (or auto-seeded) query set + target. | — | [PR #320](https://github.com/SoundMindsAI/relyloop/pull/320) |
+| 10 | P2 | [chore_demo_seeding_integration_tests_rewrite](planned_features/02_mvp2/chore_demo_seeding_integration_tests_rewrite/feature_spec.md) | Chore | The 9 skipped cases are rewritten to the async "POST + poll-until-terminal" shape, the timeout case is re-homed to the worker layer, a new `AC-Async` case asserts the `running → complete` polling tran | — | [PR #286](https://github.com/SoundMindsAI/relyloop/pull/286) |
 
 ### Spec (0)
 
@@ -73,7 +74,7 @@ _None._
 
 | # | Priority | Feature | Type | One-liner | Depends on | Status |
 |---|---|---|---|---|---|---|
-| 1 | P1 | [infra_solr_smoke_stability](planned_features/02_mvp2/infra_solr_smoke_stability/idea.md) | Infra | After Phase 1 ships, the `pr.yml` `backend` job goes green (the heavy-lane reseed test no longer requires Solr). But the `smoke-test` job is independent — it runs `make up` which brings up the full Co | — | Idea — deferred from `infra_solr_ci_readiness` Phase 1 because the right stabilization lever depends on log evidence from a smoke-runner failure |
+| 1 | P1 | [infra_smoke_reseed_runtime_budget](planned_features/02_mvp2/infra_smoke_reseed_runtime_budget/idea.md) | Infra | Idea — captured at `infra_solr_smoke_stability` PR #383 merge time | — | Idea — captured at `infra_solr_smoke_stability` PR #383 merge time |
 | 2 | P1 | [bug_backend_suite_nondeterministic_caplog_isolation](planned_features/02_mvp2/bug_backend_suite_nondeterministic_caplog_isolation/idea.md) | Bug | Many backend unit tests assert on captured log records (`caplog` / a structlog capture fixture) and fail with empty-capture shapes (`assert []`, `assert 'x' in []`) when run in the full randomized sui | — | Idea — bug surfaced by CI on PR #363 |
 | 3 | P2 | [infra_openapi_types_freshness_gate](planned_features/02_mvp2/infra_openapi_types_freshness_gate/idea.md) | Infra | Idea — Phase 2 of [`infra_generated_artifact_freshness_gate`](../infra_generated_artifact_freshness_gate/feature_spec.md), extracted to its own folder | — | Idea — Phase 2 of [`infra_generated_artifact_freshness_gate`](../infra_generated_artifact_freshness_gate/feature_spec.md), extracted to its own folder |
 | 4 | P2 | [chore_demo_reseed_partial_completion_fast_test](planned_features/02_mvp2/chore_demo_reseed_partial_completion_fast_test/idea.md) | Chore | `infra_solr_ci_readiness` made the demo reseed engine-tolerant: when an engine is unreachable, its scenario is skipped, the reseed completes with `status="complete"` and a non-empty `scenarios_skipped | — | Idea — tangential discovery during `infra_solr_ci_readiness` Story 1.2 implementation |
@@ -125,6 +126,8 @@ graph LR
   class feat_ubi_llm_study_comparison plan;
   infra_generated_artifact_freshness_gate["generated artifact freshness gate"]
   class infra_generated_artifact_freshness_gate plan;
+  infra_solr_smoke_stability["solr smoke stability"]
+  class infra_solr_smoke_stability plan;
   feat_contextual_help_mvp2["contextual help mvp2"]
   class feat_contextual_help_mvp2 done;
   feat_study_sub_warmup_guard["study sub warmup guard"]

docs/00_overview/backlog_dashboard.html

@@ -369,7 +369,7 @@
   <div class="back-link"><a href="dashboard.html">← Roadmap overview</a></div>
   <h1>RelyLoop BACKLOG Dashboard</h1>
   <div class="meta">
-    Reflects feature-folder state as of 2026-06-01 (latest mtime of any
+    Reflects feature-folder state as of 2026-06-02 (latest mtime of any
     <code>docs/00_overview/planned_features/</code> or
     <code>docs/00_overview/implemented_features/</code> file).
     See <a href="../../state.md">state.md</a> for the active branch context,