Skip to content

fix(slack): wrap connect_to_new_endpoint() in asyncio.wait_for to prevent watchdog stall #683

Description

@pavshulin

Summary

The Slack Socket Mode watchdog calls await self.client.connect_to_new_endpoint()
without a timeout. If that SDK call hangs (DNS lookup blocked, TCP SYN
no-response, slack.com unreachable), the watchdog task is blocked
indefinitely inside the await. No subsequent ticks. The watchdog effectively
dies silently — the very failure mode #244 was meant to prevent.

Discovery

Surfaced during the /autoplan review of #244 (multi-connection mode).
The original start() path wraps client.connect() in
asyncio.wait_for(..., timeout=10), but the reconnect path in
_attempt_reconnect() does not have an analogous guard.

Location

src/backend/adapters/transports/slack_socket.py — inside
_attempt_reconnect():

try:
    await ctx.client.connect_to_new_endpoint()  # ← no timeout
    ...

Proposed fix

Wrap the call with a generous timeout (e.g. 30 s) and treat timeout as a
reconnect failure (which feeds the existing exponential-backoff path):

try:
    await asyncio.wait_for(
        ctx.client.connect_to_new_endpoint(),
        timeout=30,
    )
    ...
except asyncio.TimeoutError:
    logger.error(
        f"Socket Mode watchdog [c={ctx.index}]: reconnect timed out (30s)"
    )

Blast radius

After #244 (multi-connection N=2 default): a single client's stuck reconnect
no longer takes down all of Slack — siblings keep absorbing traffic. So this
went from "P1 cliff" to "P2 quality-of-life." Still worth fixing.

Acceptance

  • _attempt_reconnect() calls connect_to_new_endpoint() under
    asyncio.wait_for with a configurable timeout (default 30 s)
  • Test: when the SDK reconnect call hangs, the watchdog logs the timeout
    and the next tick still runs
  • No regression in existing reconnect tests

Related

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions