bug: Slack Socket Mode dies every ~4 hours — watchdog recovers but root cause unknown

[pavshulin]

Summary

Slack Socket Mode WebSocket session dies every ~4 hours on a healthy
instance. The application-level watchdog from #278 / PR #281 is working —
it detects the dead session and reconnects within 60–180 s — but the
root cause of the frequent disconnects is unknown and worth fixing:

• Each disconnect creates a 60–180 s window where Slack messages may be missed
• High frequency suggests a fixable upstream cause (NAT timeout,
  Slack LB cycling, SDK heartbeat config)
• The watchdog masks the symptom so we never noticed how often it happens

Evidence (local Trinity backend, 2026-05-05 → 2026-05-06)

When (UTC)	Event	Recovery
2026-05-05 22:24:12	Watchdog detected dead session	Reconnect attempt 1 failed (DNS)
2026-05-05 22:24:13	Cannot connect to host slack.com:443: No address associated with hostname	—
2026-05-05 22:26:51	Reconnect succeeded (attempt 2)	159 s outage
2026-05-05 23:06:50	SDK: "session stale, disconnected for 42+ seconds"	Reconnected in 1 attempt
2026-05-05 23:10:51	Dead session	Reconnected in 1 attempt
2026-05-06 09:25:56	Dead session	Reconnected in 1 attempt

4 confirmed silent disconnects in ~18 hours.

Hypotheses to investigate

• NAT/firewall idle timeout — many NATs drop idle TCP after ~30 min;
  WebSocket pings may not be frequent enough on this path.
• Slack-side load-balancer cycling — Slack rotates Socket Mode
  endpoints periodically; SDK should follow gracefully.
• SDK heartbeat config — slack_sdk.socket_mode.aiohttp ping interval
  may be too long.
• Concurrent reconnects — SDK's current_session_monitor and our
  watchdog could both call connect_to_new_endpoint; the SDK lock makes
  it safe but may produce thrash.

Watchdog status

The application-level watchdog from #278 is the safety net for these
silent disconnects and is performing as designed. This issue is about
identifying and removing the root cause so the watchdog has nothing
to do.

Files

• src/backend/adapters/transports/slack_socket.py
• tests/unit/test_slack_watchdog.py

Acceptance criteria

• Identify the dominant cause of disconnects (network capture,
  SDK-level instrumentation, or correlation with Slack status)
• Prove or disprove each hypothesis above
• If fixable: ship the fix and verify < 1 disconnect / 24 h on the
  same instance
• If not fixable: document the root cause and confirm the watchdog's
  60–180 s recovery SLA is acceptable

Original report

Preserved for context — see commit history.

[pavshulin]

/claim

[github-actions]

Claimed by @pavshulin. Label status-in-progress added.

Next steps:

1. Create branch: feature/244-<short-slug>
2. Implement the feature
3. Open a PR with Fixes #244