feat(canary): fail loud when the collector is blind on a provably live fleet (#1813)

docs/memory/architecture.md

@@ -723,7 +723,9 @@ Outbound phone calls from agents via Twilio Media Streams + Gemini Live (details
 
 Continuous orchestration-invariant watcher. Deterministic library (`src/backend/canary/`) shared between the 5-min watcher service and the on-demand admin endpoint — the library reads state (Redis × the configured SQL backend, SQLite OR PostgreSQL via `DATABASE_URL`, #300/#1093 × agent registries) but writes nothing; the service persists violations to `canary_violations` and classifies green→red transitions. All SQL-tier collector reads route through the `get_engine()`/`DATABASE_URL` seam (#1540) so the harness is backend-consistent on PostgreSQL — previously the raw-sqlite collectors read a stale `/data/trinity.db` on PG and the SQL checks went vacuously green. No LLM reasoning anywhere — the canary's value depends on determinism. Disabled by default; `CANARY_ENABLED=1` on staging/dev. **Alert sink:** one Slack Block Kit webhook POST per green→red transition (`CANARY_SLACK_WEBHOOK_URL` env; unset = silent sink — cycles still run, violations persist; continuing-red doesn't re-post). Every registry invariant must carry **all four** per-invariant surfaces in `services/canary_alerts.py` (name, runbook, `_render_message` branch, `_render_forensic` branch) or its alert degrades to a bare-id fallback; CI-guarded bidirectionally by `tests/unit/test_1880_canary_alert_parity.py` (#1880). Names come from each invariant module's docstring title — **not** the catalog, whose ids are not the registry's. Both render fallbacks are deliberately state-free so an un-rendered invariant cannot echo `observed_state` (E-04/G-04 scrub at the check, reporting a reason code / pattern name only). **Operational:** a rendered alert carries agent names, execution/schedule ids, and — for G-04 — which agent and row hold a credential-shaped value and that it is stored plaintext. No secret *value* is ever sent, but that is a precise pointer, and it reaches the `text` fallback too (mobile push, connected integrations). Channel membership behind `CANARY_SLACK_WEBHOOK_URL` is not tied to the Trinity admin role, so **point it at a restricted channel** — before #1880 the same information required an admin query against `canary_violations`. **Fleet:** `config/canary-fleet.yaml` deploys synthetic load generators (`canary-fleet-burst`, `canary-fleet-long`) via the systems-deploy API — without traffic the checks are trivially green.
 
-Lookup keys: S-01/E-02/L-03 shipped via #653; S-02/E-01/E-05/B-01 (Phase 2) and S-03/B-02/R-01 (Phase 3) via #882; E-03/G-03/E-04/G-04 (Phase 4) via #1077 (E-04/G-04 stacked on #1450's queued-read rework).
+Lookup keys: S-01/E-02/L-03 shipped via #653; S-02/E-01/E-05/B-01 (Phase 2) and S-03/B-02/R-01 (Phase 3) via #882; E-03/G-03/E-04/G-04 (Phase 4) via #1077 (E-04/G-04 stacked on #1450's queued-read rework); H-01 (Phase 5) via #1813.
+
+**`H-` is the harness-health family (#1813)** — the only invariants whose violation means *the observer is blind*, not *the observed system is broken*. Filed apart from `G-` (global/cross-cutting) on purpose: a detector outage triaged as a platform defect sends the on-call after the wrong bug, and an H-01 violation invalidates every other green in that cycle.
 
 | ID | Tier | Severity | Invariant (bug class guarded) |
 |----|------|----------|-------------------------------|
@@ -742,6 +744,7 @@ Lookup keys: S-01/E-02/L-03 shipped via #653; S-02/E-01/E-05/B-01 (Phase 2) and
 | B-02 | B | critical | No queued without slots-full: queued > 0 ⇒ slots full OR a drain tick fired <60s ago (`canary:drain_tick_at` heartbeat) |
 | L-03 | A | crit/major | Delete cascades: no live row in any cross-cutting table (sharing, schedules, non-terminal executions, skills, tags, shared files, public links, pending operator queue/access requests, agent-scoped MCP keys, active chat sessions) referencing an `agent_name` absent from `agent_ownership`; no orphan `agent:slots:{name}` (critical for orphaned executions/slots, major otherwise; #129 class) |
 | R-01 | A | critical | No zombie Claude processes: per running agent container, `ps -eo stat,comm` shows zero `^Z.*claude` (anchored `^Z` — procps-ng emits STAT left-aligned; guards PR #407). Docker-exec source; per-container failures land in `sources_unavailable` so one unhealthy container doesn't kill the cycle |
+| H-01 | A | crit/major | **Collector blindness (harness self-check, #1813):** the SQL roster read (`_collect_known_agents`) returned zero rows — or raised — while an **independent, non-SQL** source proves the fleet is alive. #1540 repointed the collectors at the live engine but left the failure *shape*: zero rows is indistinguishable from a clean fleet, so a re-blinded collector reports all-clear (verified: a 2-agent fleet on a diverged backend yields `known_agents=∅`, `sources_unavailable=[]`, **zero violations** across all invariants). Prior coverage was worse than none — L-03 fires only when an execution happens to hold a slot, and then blames a *ghost agent*, misdiagnosing a blind detector as a delete-cascade bug. Evidence: `docker_agent_names` (running agent containers, taken from the container LIST — not `zombie_counts`, which is keyed by `exec_run` success and thins on a degraded container) ∪ `orphan_redis_slots`. Reasons: `roster_read_failed` / `roster_empty_contradicted` (both critical) / `roster_empty_unverifiable` (major — evidence source itself unreachable; the "dead smoke detector chirps" branch). **Confirmed on elapsed wall-clock** (`CONFIRMATION_MIN_SECONDS`, marker `canary:h01:suspect_since` — E-02's cross-cycle-state precedent) so the last-agent delete race — DB row gone, container still tearing down — cannot false-fire; cost is at most one extra cycle to alarm. Deliberately **not** "a second cycle": prod runs `--workers 2` and `canary_service` holds no leader lease, so both loops share the marker and worker B would confirm worker A's sighting seconds later, collapsing the gate inside the very window it exists to ride out. An unreadable **or unwritable** marker fires **unconfirmed** rather than skipping (a guard that cannot self-check must say so). Scoped to the roster read ONLY: on a live-but-quiet fleet `terminal_rows`/`enabled_schedules`/`orphan_refs`/`terminal_exec_statuses` are all legitimately zero, so a general "any collector reads zero" rule would false-alarm on every idle install. Dual-track by construction (pure function over the Snapshot, issues no SQL). **Residual:** an entirely *stopped* fleet holds no containers and no slots, so it reaches `roster_empty_unverifiable` at most; partial blindness (roster returns 1 of 20) is out of scope — a count comparison would false-fire on create/stop races |
 
 ### Agent Compatibility Validation (#668)