feat: Redis Streams event bus for WebSocket delivery (RELIABILITY-003) - #438
Merged
Conversation
Adds CLAUDE.md pointer to the orchestration reliability plan and records the 2026-04-20 revision: pause #294/#291 pending #306, treat Redis Streams event bus as the keystone for Tier 2.5 simplification, and gate cleanup collapse on a 2-week push-path soak. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the in-process ConnectionManager + `except: pass` broadcast with a Redis Streams transport. Reconnects no longer drop events: clients send `?last-event-id=<stream_id>` and receive missed events via XRANGE catchup (capped at 5000 entries; trimmed cursors trigger a `resync_required` marker that the frontend answers with a REST refetch). Key pieces: - services/event_bus.py — EventBus publisher (fire-and-forget XADD, bounded outbound queue, 10000 MAXLEN env-tunable) + StreamDispatcher (one XREAD BLOCK per process, in-memory fan-out, per-client asyncio.Queue(256), 3-failure eviction, supervised reader with exponential backoff, 2s graceful drain on shutdown). - main.py — ConnectionManager / FilteredWebSocketManager kept as thin shims over the bus so the 33 legacy broadcast call sites don't change. /ws and /ws/events accept an optional last-event-id query param (regex-validated to `^\d+-\d+$`). - Frontend WS clients capture `_eid` and replay on reconnect; resync handlers refetch authoritative state (agents + activity history + pending notifications). - 23 unit tests: id validation, scope visibility, XADD envelope, fallback buffer, eviction, slow-consumer resync, monotonic cursor guard, invalid-cursor resync. Plus live roundtrip + reconnect replay verified against localhost stack. Keystone for Tier 2.5 simplification per `docs/planning/ORCHESTRATION_RELIABILITY_2026-04.md`: #428 (capacity consolidate), #429 (cleanup collapse), #307 (heartbeat push) will reuse the same stream primitive; #408 dissolves once agent-push completion retires the 1h blocking HTTP call. Those are follow-ups — this PR is WebSocket delivery only. Closes #306 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes warning W1 from PR #438 validation — records the `?last-event-id=` query param, regex gate, REPLAY_GAP_LIMIT ceiling, services/event_bus.py entry, and updates invariant #10 to name the new transport so future broadcast sites don't bypass the manager shims. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closed
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the in-process
ConnectionManager.broadcast()+except: passpattern with a Redis Streams transport so short WebSocket disconnects no longer drop events. Keystone for Tier 2.5 simplification perdocs/planning/ORCHESTRATION_RELIABILITY_2026-04.md— #428/#429/#307 reuse the same primitive; #408 dissolves once agent-push completion rides on top of it.EventBus— fire-and-forgetXADDwithMAXLEN ~10000(env-tunable viaREDIS_STREAM_MAXLEN), bounded outbound queue so Redis latency never blocks broadcast call sites.StreamDispatcher— singleXREAD BLOCKper backend process, in-memory fan-out, per-clientasyncio.Queue(256)with drop-and-resync on slow consumers, 3-failure eviction, supervised reader with exponential backoff, 2s graceful drain on SIGTERM.?last-event-id=<stream_id>; dispatcher runsXRANGEcatchup capped at the reader's position at register time (prevents catchup/live overlap). Trimmed cursors or gaps >5000 →{type: "resync_required"}→ frontend full-refetches authoritative state via REST.utils/websocket.jsandstores/network.jscapture_eidfrom incoming messages and append&last-event-id=on reconnect.manager.broadcast(...)/filtered_manager.broadcast_filtered(...)call sites are untouched — the managers are now thin shims over the bus.Scope discipline
This PR is WebSocket delivery only. Deferred per plan:
CAPACITY-CONSOLIDATE— agent-push completion + queue unification (gated on 2-week soak)CLEANUP-COLLAPSE— retires the 9-path reconciliation pyramidRELIABILITY-004— heartbeat push (reuses this stream)Changes
src/backend/services/event_bus.py— new module (EventBus + StreamDispatcher + helpers)src/backend/main.py— manager shims, lifespan start/stop,/ws+/ws/eventsacceptlast-event-idsrc/frontend/src/utils/websocket.js—_eidcapture, reconnect URL,resync_requiredhandlersrc/frontend/src/stores/network.js— same wiring for the collaboration dashboard WStests/test_event_bus.py— 23 unit testsdocs/memory/feature-flows/websocket-event-bus.md— new flow doc + index row + xref fromactivity-stream.mdTest plan
pytest tests/test_event_bus.py -v→ 23/23 passing (envelope shape, id validation, scope visibility, Redis-unavailable fallback, 3-failure eviction, slow-consumer resync marker, monotonic cursor guard, invalid-cursor resync, agent_name inference,update_accessible_agentsmutation)_eidlast-event-id→ all 3 replayed in orderlast-event-id→ connection accepted gracefully, bad id ignorednetwork.jsreconnect URL includes capturedlast-event-idisConnected: true, no console errors from the new pathFollow-ups (non-blocking, captured in
/reviewinformational findings)stream_offlineto clients (low priority)console.warnin catch blocksCloses #306
🤖 Generated with Claude Code