feat(worker): wire ConfidenceShape into digest narrative prompt (Story 1.6)

- prompts/digest_narrative.user.jinja gains two new XML blocks inserted
  after <baseline_vs_achieved>: <confidence> (CI band + aggregate
  signals — each sub-line independently {% if %}-gated per FR-7) and
  <per_query_outcomes> (improved/unchanged/regressed counts +
  comparison_against label + up to 5 named regressor rows).
- prompts/digest_narrative.system.md extends the XML-block inventory to
  document blocks 8 + 9 with conditional-inclusion semantics, and
  replaces the narrative opening-guidance sentence with the FR-6
  string ("Open with the headline metric delta, immediately followed by
  a one-sentence confidence framing that mentions the CI band [when
  <confidence> is present], the per-query outcome counts [when
  <per_query_outcomes> is present], and the worst-regressed query by
  name [when <per_query_outcomes> has regressors]").
- render_digest_user_prompt gains optional confidence: Mapping | None =
  None kwarg; passes through to the jinja render verbatim. None default
  preserves the existing one-callsite contract (worker is the only
  caller).
- backend/workers/digest.py awaits fetch_study_confidence(db, study)
  immediately before render_digest_user_prompt, serializes via
  ConfidenceShape.model_dump() (so jinja consumes a plain dict, per
  cycle-1 GPT-5.5 F3), and threads through the new kwarg.
- backend/tests/unit/workers/test_digest_prompt_render.py adds 5 new
  cases covering AC-14: (1) <confidence> block present + all sub-lines
  populated; (2) absent when confidence=None; (3) <per_query_outcomes>
  present + regressor rows render; (4) absent when nested
  per_query_outcomes is None; (5) system prompt carries the FR-6
  opening-guidance string (whitespace-normalized assertion tolerates
  the markdown soft-wraps) plus the documented block list entries.

Verification: 1039 backend unit tests pass (+5 new); 189 contract;
526/526 in-container integration; backend fmt + lint + typecheck +
ruff format --check parity clean. No regressions in
test_digest_generate / test_digest_zero_trials /
test_digest_capability_fallback — the new fetch_study_confidence call
returns None for the seeded happy-path study (no per_query_metrics on
its single trial), so the jinja blocks skip silently and the existing
prompt-render assertions stay valid.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: SoundMindsAI

backend/app/llm/digest_prompt.py

@@ -80,6 +80,7 @@ def render_digest_user_prompt(
     recommended_config: Mapping[str, Any],
     dropped_template_params: Sequence[str],
     include_recommendation: bool = True,
+    confidence: Mapping[str, Any] | None = None,
 ) -> str:
     """Render the per-study user message for the digest narrative call.
 
@@ -105,6 +106,12 @@ def render_digest_user_prompt(
         include_recommendation: cycle-3 F3 toggle. ``True`` (default) emits
             the full structured prompt; ``False`` emits the degraded /
             narrative-only variant for the capability-fallback path.
+        confidence: serialized ``ConfidenceShape`` (via
+            ``ConfidenceShape.model_dump()``) per feat_pr_metric_confidence
+            FR-6. ``None`` (default) skips both the ``<confidence>`` and
+            ``<per_query_outcomes>`` jinja blocks; a partial shape (some
+            sub-fields ``None``) emits only the populated sub-lines via the
+            template's per-sub-field ``{% if %}`` guards.
 
     Returns:
         The rendered user message string, ready to send as the OpenAI
@@ -127,6 +134,7 @@ def render_digest_user_prompt(
         recommended_config=recommended_config,
         dropped_template_params=dropped_template_params,
         include_recommendation=include_recommendation,
+        confidence=confidence,
     )
 
 

backend/tests/unit/workers/test_digest_prompt_render.py

@@ -24,6 +24,7 @@
     _SANDBOX_ENV,
     DigestPromptBundle,
     load_digest_prompts,
+    render_digest_system_prompt,
     render_digest_user_prompt,
 )
 
@@ -155,6 +156,139 @@ def test_autoescape_neutralizes_adversarial_study_name() -> None:
     assert "</study><inject>malicious-instruction" not in output
 
 
+# ---------------------------------------------------------------------------
+# feat_pr_metric_confidence Story 1.6 — <confidence> + <per_query_outcomes>
+# ---------------------------------------------------------------------------
+
+
+def _make_test_confidence_dict(**overrides: object) -> dict[str, object]:
+    """Build a fully-populated serialized ConfidenceShape for the jinja blocks.
+
+    Mirrors what ``ConfidenceShape.model_dump()`` emits at the digest-worker
+    call site. Tests override sub-fields by passing them as kwargs.
+    """
+    base: dict[str, object] = {
+        "headline": {"metric": "ndcg", "value": 0.840, "k": 10, "n_queries": 20},
+        "ci_95": {"low": 0.780, "high": 0.890, "method": "bootstrap_n1000", "n_samples": 20},
+        "runner_up_gap": {
+            "value": 0.002,
+            "classification": "robust_plateau",
+            "top10_within": 0.004,
+            "runner_up_metric": 0.838,
+        },
+        "late_trial_stddev": {"value": 0.012, "window_size": 20, "min_window_required": 10},
+        "convergence": {"best_at_trial": 387, "total_trials": 1000, "regime": "early_held"},
+        "per_query_outcomes": {
+            "improved": 14,
+            "unchanged": 4,
+            "regressed": 2,
+            "comparison_against": "runner_up",
+            "top_regressors": [
+                {
+                    "query_id": "q1",
+                    "query_text": "vintage acoustic guitar",
+                    "winner_score": 0.41,
+                    "comparison_score": 0.92,
+                    "delta": -0.51,
+                },
+                {
+                    "query_id": "q2",
+                    "query_text": "leather wallet",
+                    "winner_score": 0.55,
+                    "comparison_score": 0.78,
+                    "delta": -0.23,
+                },
+            ],
+        },
+    }
+    base.update(overrides)
+    return base
+
+
+def test_user_prompt_includes_confidence_block_when_data_present() -> None:
+    """FR-6 / AC-14: full confidence dict produces the <confidence> XML block."""
+    kwargs = dict(CANONICAL_KWARGS)
+    kwargs["confidence"] = _make_test_confidence_dict()
+    output = render_digest_user_prompt(**kwargs)  # type: ignore[arg-type]
+    assert "<confidence>" in output
+    assert "</confidence>" in output
+    # Headline + CI sub-lines.
+    assert "ci_low: 0.78" in output
+    assert "ci_high: 0.89" in output
+    assert "n_queries: 20" in output
+    # Aggregate signals.
+    assert "runner_up_gap: 0.002 (robust_plateau)" in output
+    assert "late_trial_stddev: 0.012" in output
+    assert "convergence: early_held (best at trial 387 of 1000)" in output
+
+
+def test_user_prompt_omits_confidence_block_when_none() -> None:
+    """FR-7 / AC-12: confidence=None skips both blocks entirely."""
+    output = render_digest_user_prompt(**CANONICAL_KWARGS)  # type: ignore[arg-type]
+    # Canonical kwargs don't set `confidence` — defaults to None.
+    assert "<confidence>" not in output
+    assert "<per_query_outcomes>" not in output
+
+
+def test_user_prompt_includes_per_query_outcomes_block_when_nested_data_present() -> None:
+    """The <per_query_outcomes> block surfaces nested counts + named regressors."""
+    kwargs = dict(CANONICAL_KWARGS)
+    kwargs["confidence"] = _make_test_confidence_dict()
+    output = render_digest_user_prompt(**kwargs)  # type: ignore[arg-type]
+    assert "<per_query_outcomes>" in output
+    assert "</per_query_outcomes>" in output
+    assert "improved: 14" in output
+    assert "unchanged: 4" in output
+    assert "regressed: 2" in output
+    assert "comparison_against: runner_up" in output
+    # Each regressor row: text + winner → comparison + delta in parens.
+    assert "- vintage acoustic guitar: 0.41" in output
+    assert "0.92" in output
+    assert "(-0.51)" in output
+    assert "- leather wallet: 0.55" in output
+
+
+def test_user_prompt_omits_per_query_outcomes_block_when_subfield_is_none() -> None:
+    """FR-7: confidence present but per_query_outcomes=None → outer block only."""
+    kwargs = dict(CANONICAL_KWARGS)
+    kwargs["confidence"] = _make_test_confidence_dict(per_query_outcomes=None)
+    output = render_digest_user_prompt(**kwargs)  # type: ignore[arg-type]
+    # The <confidence> block still renders (CI + aggregate signals).
+    assert "<confidence>" in output
+    # <per_query_outcomes> stays suppressed.
+    assert "<per_query_outcomes>" not in output
+
+
+def test_system_prompt_has_fr6_opening_guidance_and_block_inventory() -> None:
+    """AC-14 system-prompt half: the opening guidance + block list are updated.
+
+    The replacement string from spec FR-6 is in the prompt file but
+    soft-wrapped at ~80 columns. We collapse whitespace before asserting so
+    the test tolerates wrap location while still proving the substring
+    contract — the LLM sees newlines as whitespace too.
+    """
+    system = render_digest_system_prompt()
+    # Collapse all runs of whitespace (incl. newlines + indents) into single
+    # spaces so soft-wrapped sentences match continuous-string assertions.
+    flat = " ".join(system.split())
+    # Opening-guidance replacement (FR-6 line edit). Backticks around
+    # `<confidence>` / `<per_query_outcomes>` tag names are load-bearing —
+    # they signal to the LLM that these are XML block names, not English.
+    assert (
+        "Open with the headline metric delta, immediately followed by a one-sentence "
+        "confidence framing that mentions the CI band (when `<confidence>` is present), "
+        "the per-query outcome counts (when `<per_query_outcomes>` is present), and the "
+        "worst-regressed query by name (when `<per_query_outcomes>` has regressors)."
+    ) in flat
+    # The original "Open with the headline metric delta. Then explain" sentence
+    # must NOT exist verbatim — the replacement superseded it.
+    assert "headline metric delta. Then explain" not in flat
+    # Block inventory must document the two new XML blocks (these appear on
+    # their own lines so a direct substring check is fine).
+    assert "8. `<confidence>`" in system
+    assert "9. `<per_query_outcomes>`" in system
+
+
 def test_sandbox_rejects_attribute_access() -> None:
     """Defense in depth: SandboxedEnvironment blocks dunder-access from template authors.
 

backend/workers/digest.py

@@ -72,6 +72,7 @@
     known_models,
 )
 from backend.app.llm.digest_prompt import load_digest_prompts, render_digest_user_prompt
+from backend.app.services.study_confidence import fetch_study_confidence
 
 logger = structlog.get_logger(__name__)
 
@@ -686,6 +687,14 @@ async def generate_digest(ctx: dict[str, Any], study_id: str) -> None:
                 rubric_summary = rubric_text[:280] + ("..." if len(rubric_text) > 280 else "")
                 if not rubric_summary:
                     rubric_summary = "(see judgment list rubric)"
+                # feat_pr_metric_confidence Story 1.6 (FR-6): assemble the
+                # per-study ConfidenceShape and serialize for the jinja
+                # ``<confidence>`` + ``<per_query_outcomes>`` blocks. Returns
+                # ``None`` on degraded paths (FR-7) so the blocks skip cleanly.
+                confidence_shape = await fetch_study_confidence(db, study)
+                confidence_payload = (
+                    confidence_shape.model_dump() if confidence_shape is not None else None
+                )
                 user_prompt = render_digest_user_prompt(
                     study_name=study.name,
                     cluster_name=cluster_name,
@@ -701,6 +710,7 @@ async def generate_digest(ctx: dict[str, Any], study_id: str) -> None:
                     recommended_config=recommended_config,
                     dropped_template_params=dropped,
                     include_recommendation=structured_output_enabled and not all_dropped,
+                    confidence=confidence_payload,
                 )
                 bundle = load_digest_prompts()
                 openai_client = AsyncOpenAI(api_key=api_key, base_url=settings.openai_base_url)

docs/02_product/planned_features/feat_pr_metric_confidence/implementation_plan.md

@@ -1055,7 +1055,7 @@ None planned. The feature is purely additive across all surfaces.
 - [ ] Story 1.3 — Domain module `confidence.py`
 - [x] Story 1.4 — `ConfidenceShape` + StudyDetail enrichment
 - [x] Story 1.5 — PR body section + worker plumbing
-- [ ] Story 1.6 — Digest narrative prompt extension
+- [x] Story 1.6 — Digest narrative prompt extension
 - [ ] **Epic 1 gate**
 - [ ] Story 2.1 — TypeScript types + enums
 - [ ] Story 2.2 — `<ConfidencePanel>` component + glossary + page mount

prompts/digest_narrative.system.md

@@ -24,15 +24,32 @@ The user message contains XML-delimited blocks:
 7. `<degraded_mode>` (only when `include_recommendation=False`) — the operator's
    OpenAI endpoint failed the structured-output capability probe. Return free-
    form prose narrative only — no JSON, no recommendations, no follow-ups.
+8. `<confidence>` (only when the orchestrator computed a non-null
+   `ConfidenceShape` for the study) — bootstrap 95% CI on the headline metric
+   (`ci_low`/`ci_high`/`n_queries`) plus aggregate signals (`runner_up_gap`,
+   `late_trial_stddev`, `convergence`). Each sub-line is omitted independently
+   when its sub-field is null (FR-7 graceful-degradation contract). For
+   studies still running, or studies whose winner trial predates the
+   `per_query_metrics` migration, the block may be absent or partial.
+9. `<per_query_outcomes>` (only when both the winner trial and the runner-up
+   trial have per-query metrics) — `improved` / `unchanged` / `regressed`
+   counts, the `comparison_against` reference (`runner_up` in MVP1; `baseline`
+   when Phase 2 ships), and up to 5 named regressor rows
+   (`query_text: winner_score → comparison_score (delta)`). Omitted entirely
+   when the comparison data isn't available.
 
 For the **structured** path (default, `include_recommendation=True`), return a
 JSON object with exactly two fields:
 
 - `narrative` — a markdown string (~200–600 words). Open with the headline
-  metric delta. Then explain *why* the recommendation works, citing the
-  `<parameter_importance>` map and 2–3 top trials. Reference the
-  `<recommended_config>` literal params + values where useful, but do NOT
-  reprint the full config — the data layer already has it.
+  metric delta, immediately followed by a one-sentence confidence framing that
+  mentions the CI band (when `<confidence>` is present), the per-query outcome
+  counts (when `<per_query_outcomes>` is present), and the worst-regressed
+  query by name (when `<per_query_outcomes>` has regressors). Then explain
+  *why* the recommendation works, citing the `<parameter_importance>` map and
+  2–3 top trials. Reference the `<recommended_config>` literal params + values
+  where useful, but do NOT reprint the full config — the data layer already
+  has it.
 - `suggested_followups` — a JSON array of at most 5 short strings, each a
   concrete next action the engineer can take (e.g. "Re-run with a wider
   `tie_breaker` range", "Add a judgment for query 'wireless headphones' to

prompts/digest_narrative.user.jinja

@@ -12,7 +12,24 @@ baseline_metric: {{ baseline_metric if baseline_metric is not none else 'N/A (no
 achieved_metric: {{ achieved_metric }}
 </baseline_vs_achieved>
 
-<top_trials>
+{% if confidence %}<confidence>
+{% if confidence.ci_95 %}ci_low: {{ confidence.ci_95.low }}
+ci_high: {{ confidence.ci_95.high }}
+{% endif %}n_queries: {{ confidence.headline.n_queries }}
+{% if confidence.runner_up_gap %}runner_up_gap: {{ confidence.runner_up_gap.value }} ({{ confidence.runner_up_gap.classification or 'unclassified' }})
+{% endif %}{% if confidence.late_trial_stddev %}late_trial_stddev: {{ confidence.late_trial_stddev.value }}
+{% endif %}{% if confidence.convergence %}convergence: {{ confidence.convergence.regime }} (best at trial {{ confidence.convergence.best_at_trial }} of {{ confidence.convergence.total_trials }})
+{% endif %}</confidence>
+
+{% endif %}{% if confidence and confidence.per_query_outcomes %}<per_query_outcomes>
+improved: {{ confidence.per_query_outcomes.improved }}
+unchanged: {{ confidence.per_query_outcomes.unchanged }}
+regressed: {{ confidence.per_query_outcomes.regressed }}
+comparison_against: {{ confidence.per_query_outcomes.comparison_against }}
+{% for r in confidence.per_query_outcomes.top_regressors %}- {{ r.query_text }}: {{ r.winner_score }} → {{ r.comparison_score }} ({{ r.delta }})
+{% endfor %}</per_query_outcomes>
+
+{% endif %}<top_trials>
 {% for t in top_trials %}trial #{{ t.number }} — primary_metric={{ t.primary_metric }}, params={{ t.params }}
 {% endfor %}</top_trials>