Skip to content

feat: Redis Streams event bus for WebSocket delivery (RELIABILITY-003) - #438

Merged
vybe merged 3 commits into
mainfrom
feature/306-redis-streams-event-bus
Apr 21, 2026
Merged

feat: Redis Streams event bus for WebSocket delivery (RELIABILITY-003)#438
vybe merged 3 commits into
mainfrom
feature/306-redis-streams-event-bus

Conversation

@vybe

@vybe vybe commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the in-process ConnectionManager.broadcast() + except: pass pattern with a Redis Streams transport so short WebSocket disconnects no longer drop events. Keystone for Tier 2.5 simplification per docs/planning/ORCHESTRATION_RELIABILITY_2026-04.md#428/#429/#307 reuse the same primitive; #408 dissolves once agent-push completion rides on top of it.

  • Publisher: EventBus — fire-and-forget XADD with MAXLEN ~10000 (env-tunable via REDIS_STREAM_MAXLEN), bounded outbound queue so Redis latency never blocks broadcast call sites.
  • Consumer: StreamDispatcher — single XREAD BLOCK per backend process, in-memory fan-out, per-client asyncio.Queue(256) with drop-and-resync on slow consumers, 3-failure eviction, supervised reader with exponential backoff, 2s graceful drain on SIGTERM.
  • Reconnect replay: clients send ?last-event-id=<stream_id>; dispatcher runs XRANGE catchup capped at the reader's position at register time (prevents catchup/live overlap). Trimmed cursors or gaps >5000 → {type: "resync_required"} → frontend full-refetches authoritative state via REST.
  • Frontend: both utils/websocket.js and stores/network.js capture _eid from incoming messages and append &last-event-id= on reconnect.
  • Public API preserved: 33 existing manager.broadcast(...) / filtered_manager.broadcast_filtered(...) call sites are untouched — the managers are now thin shims over the bus.

Scope discipline

This PR is WebSocket delivery only. Deferred per plan:

Changes

  • src/backend/services/event_bus.py — new module (EventBus + StreamDispatcher + helpers)
  • src/backend/main.py — manager shims, lifespan start/stop, /ws + /ws/events accept last-event-id
  • src/frontend/src/utils/websocket.js_eid capture, reconnect URL, resync_required handler
  • src/frontend/src/stores/network.js — same wiring for the collaboration dashboard WS
  • tests/test_event_bus.py — 23 unit tests
  • docs/memory/feature-flows/websocket-event-bus.md — new flow doc + index row + xref from activity-stream.md

Test plan

  • Unit: pytest tests/test_event_bus.py -v → 23/23 passing (envelope shape, id validation, scope visibility, Redis-unavailable fallback, 3-failure eviction, slow-consumer resync marker, monotonic cursor guard, invalid-cursor resync, agent_name inference, update_accessible_agents mutation)
  • Live smoke against localhost stack (playwright):
    • Publish → browser receives payload with injected _eid
    • Close socket, publish 3 events offline, reconnect with last-event-id → all 3 replayed in order
    • Malformed last-event-id → connection accepted gracefully, bad id ignored
    • Dashboard network.js reconnect URL includes captured last-event-id
  • End-to-end integration test (Redis + backend + client) deferred — the 2-week soak in the plan is the real signal
  • Manual: dashboard loads with 10 agents visible, isConnected: true, no console errors from the new path

Follow-ups (non-blocking, captured in /review informational findings)

  • I1: Redis-down silently pauses live updates — signal stream_offline to clients (low priority)
  • I3: Frontend refetch errors swallowed silently — add console.warn in catch blocks

Closes #306

🤖 Generated with Claude Code

vybe and others added 3 commits April 21, 2026 11:03
Adds CLAUDE.md pointer to the orchestration reliability plan and
records the 2026-04-20 revision: pause #294/#291 pending #306, treat
Redis Streams event bus as the keystone for Tier 2.5 simplification,
and gate cleanup collapse on a 2-week push-path soak.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the in-process ConnectionManager + `except: pass` broadcast with
a Redis Streams transport. Reconnects no longer drop events: clients send
`?last-event-id=<stream_id>` and receive missed events via XRANGE catchup
(capped at 5000 entries; trimmed cursors trigger a `resync_required`
marker that the frontend answers with a REST refetch).

Key pieces:
- services/event_bus.py — EventBus publisher (fire-and-forget XADD,
  bounded outbound queue, 10000 MAXLEN env-tunable) + StreamDispatcher
  (one XREAD BLOCK per process, in-memory fan-out, per-client
  asyncio.Queue(256), 3-failure eviction, supervised reader with
  exponential backoff, 2s graceful drain on shutdown).
- main.py — ConnectionManager / FilteredWebSocketManager kept as thin
  shims over the bus so the 33 legacy broadcast call sites don't change.
  /ws and /ws/events accept an optional last-event-id query param
  (regex-validated to `^\d+-\d+$`).
- Frontend WS clients capture `_eid` and replay on reconnect; resync
  handlers refetch authoritative state (agents + activity history +
  pending notifications).
- 23 unit tests: id validation, scope visibility, XADD envelope, fallback
  buffer, eviction, slow-consumer resync, monotonic cursor guard,
  invalid-cursor resync. Plus live roundtrip + reconnect replay verified
  against localhost stack.

Keystone for Tier 2.5 simplification per
`docs/planning/ORCHESTRATION_RELIABILITY_2026-04.md`: #428 (capacity
consolidate), #429 (cleanup collapse), #307 (heartbeat push) will reuse
the same stream primitive; #408 dissolves once agent-push completion
retires the 1h blocking HTTP call. Those are follow-ups — this PR is
WebSocket delivery only.

Closes #306

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes warning W1 from PR #438 validation — records the `?last-event-id=`
query param, regex gate, REPLAY_GAP_LIMIT ceiling, services/event_bus.py
entry, and updates invariant #10 to name the new transport so future
broadcast sites don't bypass the manager shims.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Redis Streams event bus for reliable WebSocket delivery (RELIABILITY-003)

1 participant