Canary Phase 2 + 3: add invariants S-02, E-01, E-05, B-01, S-03, B-02, R-01

[obasilakis]

Summary

Phases 2 and 3 of the Canary Invariant Harness (#411), shipped together in PR #884. Follow-up to PR #653 (Phase 1 — S-01, E-02, L-03).

Adds seven invariants in two waves. The split reflects internal complexity — Phase 2 is the four single-source SQL/Redis checks, Phase 3 adds the three "moderate" ones that each need one new piece of plumbing.

Context

Phase 1 landed the canary library + service + fleet manifest + admin endpoint. The orchestration invariant catalog (docs/testing/orchestration-invariant-catalog.md) identifies 12 high-impact invariants for the starting subset; 3 are now in production. This issue covers the next 7. After it lands, the 2 remaining catalog invariants (E-06, G-01) require either HTTP fan-out or one-shot restart probes — different shape, deferred to a future ticket.

Acceptance Criteria

Phase 2 — single-source SQL/Redis (no new sources)

• S-02 — No overbooking added: ZCARD(agent:slots:A) ≤ agent_ownership.max_parallel_tasks. Critical. Tier A.
• E-01 — Terminal-state closure added: status='running' AND started_at < now() - (timeout_seconds + 300s) must be 0. Critical. Tier B. Uses iso_cutoff() (architecture invariant #16).
• E-05 — Dispatched rows have session added: status='running' AND started_at < now() - 60s ⇒ claude_session_id IS NOT NULL. Major. Tier B. Guards bug: Cleanup service misses 'skipped' executions and slow to detect no-session failures #106.
• B-01 — Queue-status coherence added: per agent, db.get_queued_count matches snapshot's independently-collected queued id-list count. Critical. Tier A. Regression guard against future cache layer / status-filter drift.

Phase 3 — moderate complexity (one new piece of plumbing each)

• S-03 — Slot TTL ≥ execution timeout added: for every member of agent:slots:A, the companion agent:slot:A:{eid} HASH has TTL ≥ execution_timeout_seconds + 300s (SLOT_TTL_BUFFER). Critical. Tier A. Guards bug: Stale slot cleanup uses fixed 20-min TTL regardless of agent timeout #226.
• B-02 — No queued without slots-full added: if any agent has len(queued) > 0, then either len(slots excluding sentinels) == max_parallel OR a drain-tick fired in the last 60s. Critical. Tier B. Catches stalled drain callbacks. Requires CapacityManager.run_maintenance() to write a canary:drain_tick_at heartbeat Redis key on each successful sweep.
• R-01 — No zombie Claude processes added: for every running trinity.platform=agent container, ps -eo stat,comm | grep ' Z.*claude' | wc -l returns 0. Critical. Tier A. Guards bug: agent-server.py spins at ~83% CPU after claude CLI subprocess becomes defunct #407. New source type (Docker exec) — failures recorded in sources_unavailable so an unhealthy container doesn't kill the cycle.

Common

• Each invariant wired into deterministic library and registered with CanaryService.run_cycle().
• For each: a manual reproduction in PR description — mutate code or state to provoke the violation, observe POST /api/canary/run-cycle reports it, revert, observe green.
• docs/memory/architecture.md CANARY-001 section updated with Phase 2 + Phase 3 invariant blocks.

Technical Notes

• Phase 2 stays scope-pure to canary/ (+ architecture doc). Phase 3's B-02 plumbing touches services/capacity_manager.py:run_maintenance — a one-line heartbeat write at the END of the sweep (so a crash mid-sweep doesn't claim success).
• R-01's docker exec per cycle is bounded by agent count (not execution count). Cost ≈ 1 exec per running agent per 5 min — negligible.

Out of scope

• Phase 4 — E-06 (HTTP fan-out across agents to /api/executions/running) and G-01 (one-shot post-restart probe — different shape, doesn't fit the 5-min loop).

Refs

• Catalog: docs/testing/orchestration-invariant-catalog.md Recommended Starting Subset
• Phase 1: Research and implement orchestration invariant catalog + continuous canary harness #411, PR feat(#411): canary invariant harness Phase 1 (S-01, E-02, L-03) #653
• Architecture: CANARY-001 section in docs/memory/architecture.md

[vybe]

Shipped in v0.6.0 (released via #998).