bug: Scheduler loses HTTP connection to backend during long-running executions, reporting false failures

[vybe]

Summary

The scheduler's HTTP connection to the backend drops during long-running agent executions (10-20+ minutes), causing Server disconnected without sending a response errors. The agent tasks actually complete successfully on the container side, but the scheduler never receives the response. This results in zero execution records for affected agents and misleading error logs.

Component

Scheduler / Backend (TaskExecutionService)

Priority

P2

Error

[ERROR] scheduler.service: Schedule Heartbeat execution failed: Server disconnected without sending a response.

This is an httpx RemoteProtocolError — the TCP connection from the scheduler to the backend is reset by the server before a response is returned.

Location

• File: src/scheduler/service.py (line ~713, _call_backend_execute_task)
• File: src/backend/routers/internal.py (line ~185, execute_task_internal)
• Function: _call_backend_execute_task → POST /api/internal/execute-task

Root Cause

The scheduler calls POST /api/internal/execute-task with httpx and waits for the backend to return. The backend's endpoint is a synchronous HTTP request that blocks until the agent finishes (via agent_post_with_retry → agent container's /api/task). For long-running tasks (10-20+ minutes), the underlying TCP connection between scheduler and backend is reset.

Possible causes:

1. Uvicorn worker recycling — Multiple uvicorn workers (5 observed) may recycle during long requests
2. TCP keepalive defaults — The httpx client in the scheduler creates a new AsyncClient per request (no connection pooling or keepalive configuration)
3. Docker network timeouts — The trinity-network bridge may have default TCP timeout settings that close idle-appearing connections

The key issue is the architecture: the scheduler makes a blocking HTTP request that must stay open for the entire duration of the agent's execution (potentially 30-60 minutes). Any connection instability causes a false failure.

Observed Impact

• False failure logging: Scheduler logs errors even though agent work completes successfully
• Zero execution records: schedule_executions table has 0 rows for affected agents — the scheduler can't record results it never receives
• Stale slot cleanups: Every execution cycle triggers a stale slot cleanup warning because the previous slot wasn't released cleanly
• No actual data loss: Agent work (predictions, commits, dashboard updates) completes normally on the container side

Reproduction Steps

1. Create an agent with a scheduled task that takes >10 minutes to complete
2. Configure the schedule with timeout_seconds: 3600 (1 hour)
3. Wait for the schedule to trigger
4. Observe scheduler logs: Server disconnected without sending a response
5. Verify the agent actually completed the work (check session logs in the agent container)
6. Check schedule_executions table — no records for the agent

Suggested Fix

Option A (Recommended): Fire-and-forget with polling

Instead of blocking on the HTTP response, the scheduler should:

1. POST to /api/internal/execute-task which returns immediately with an execution_id
2. Poll GET /api/internal/execution/{id}/status periodically until completion
3. This eliminates the long-lived HTTP connection entirely

# In scheduler/service.py _call_backend_execute_task:
async with httpx.AsyncClient() as client:
    # Step 1: Start execution (returns immediately)
    response = await client.post(
        f"{config.backend_url}/api/internal/execute-task",
        headers=headers,
        json={**payload, "async": True},
        timeout=30.0,  # Short timeout for dispatch
    )
    execution_id = response.json()["execution_id"]

    # Step 2: Poll for completion
    deadline = time.monotonic() + request_timeout
    while time.monotonic() < deadline:
        await asyncio.sleep(10)
        status_resp = await client.get(
            f"{config.backend_url}/api/internal/execution/{execution_id}/status",
            headers=headers,
            timeout=10.0,
        )
        result = status_resp.json()
        if result["status"] in ("completed", "failed"):
            return result

Option B (Quick fix): Increase TCP keepalive and connection resilience

# In scheduler/service.py _call_backend_execute_task:
transport = httpx.AsyncHTTPTransport(
    retries=1,
    keepalive_expiry=30.0,
)
async with httpx.AsyncClient(
    transport=transport,
    timeout=httpx.Timeout(request_timeout, connect=10.0, pool=10.0),
) as client:
    ...

Environment

• Trinity version: d46281f
• Scheduler: Python/APScheduler with httpx client
• Backend: FastAPI/Uvicorn (multiple workers)
• Affected agents: Any agent with scheduled tasks running >10 minutes

Related

• src/scheduler/service.py — _call_backend_execute_task method
• src/scheduler/agent_client.py — execute_task method
• src/backend/routers/internal.py — /execute-task endpoint
• src/backend/services/task_execution_service.py — execute_task method

[vybe]

Implementation Plan: Issue #101 — Scheduler Async Fire-and-Forget with DB Polling

Problem

The scheduler makes a blocking HTTP call to POST /api/internal/execute-task that holds a TCP connection open for the entire task duration (10-60+ minutes). The connection drops due to TCP idle timeouts, Docker bridge networking, or uvicorn worker recycling, causing RemoteProtocolError: Server disconnected without sending a response. The agent completes its work, but the scheduler never receives the response.

Three interrelated bugs:

1. TCP Connection Drop — Long-lived HTTP connection drops after 15-30 min
2. Status Overwrite — Scheduler catches the exception and blindly marks execution as failed, even if TaskExecutionService already updated it to success in the shared DB
3. Cleanup Interference — CleanupService marks any execution still running after 30 min as failed, hitting legitimate long tasks

Solution

Replace the blocking HTTP call with fire-and-forget dispatch + DB polling, mirroring the existing async_mode pattern in the /task endpoint.

BEFORE: Scheduler → POST (blocks 15-60 min) → Backend → Agent
                  ← Result (or TCP drop)

AFTER:  Scheduler → POST async=true (30s max) → Backend spawns background task
                  ← {"status": "accepted"} (immediate)
        Scheduler → poll DB every 10s until execution status != "running"

Changes

1. Backend: Add async mode to internal endpoint

File: src/backend/routers/internal.py

• Add async_mode: bool = False to InternalTaskExecutionRequest model (line 174-182)
• When async_mode=True in execute_task_internal() (line 186-223):
  • Spawn asyncio.create_task(_execute_task_internal_background(task_service, request))
  • Return immediately: {"status": "accepted", "execution_id": request.execution_id, "async_mode": True}
• Add _execute_task_internal_background() function:
  • Calls task_service.execute_task() (same as sync path)
  • TaskExecutionService handles all lifecycle: slot, activity, DB updates, cleanup
  • Logs completion/error (TaskExecutionService already updates the execution record on both success and failure, so no additional DB writes needed)
• Add import asyncio at top of file

2. Scheduler: Switch to async dispatch + DB polling

File: src/scheduler/service.py

Rewrite _call_backend_execute_task() (lines 664-720):

• POST to backend with async_mode: True and a 30-second timeout (dispatch only)
• If backend returns {"status": "accepted"}, enter polling loop:
  • Poll self.db.get_execution(execution_id) every 10 seconds
  • Total polling deadline: timeout_seconds + 60 (buffer beyond task timeout)
  • When execution.status != "running" → return result dict
  • If deadline exceeded → raise timeout exception
• Backward compatibility: if backend returns a non-accepted response (old backend), treat it as a sync result and return directly

3. Scheduler: Guard against status overwrite

File: src/scheduler/service.py

Modify the except block in _execute_schedule_with_lock() (lines 645-662):

• Before calling self.db.update_execution_status(failed), check current status:

  current = self.db.get_execution(execution.id)
  if current and current.status != ExecutionStatus.RUNNING:
      # Backend already finalized — don't overwrite
  else:
      # Genuinely failed — mark as failed

• Adjust the WebSocket event to reflect the actual status

4. Cleanup service: Increase stale timeout

File: src/backend/services/cleanup_service.py

• Change EXECUTION_STALE_TIMEOUT_MINUTES from 30 to 120 (line 24)
• Change ACTIVITY_STALE_TIMEOUT_MINUTES from 30 to 120 (line 25)
• This prevents the cleanup service from marking legitimate long-running tasks (up to 1 hour) as failed

5. Scheduler config: Add poll_interval setting

File: src/scheduler/config.py

• Add poll_interval: int field (default 10, from POLL_INTERVAL env var)
• Used in _call_backend_execute_task() polling loop

Files Changed

File	Change
src/backend/routers/internal.py	Add async_mode field, async dispatch, background function
src/scheduler/service.py	Rewrite _call_backend_execute_task(), add status overwrite guard
src/backend/services/cleanup_service.py	Increase stale timeouts from 30 to 120 min
src/scheduler/config.py	Add poll_interval config
docs/memory/requirements.md	Update §12.9 cleanup service timeout note
docs/memory/changelog.md	Add timestamped entry

Edge Cases

Scenario	Behavior
Task completes in < 30s	Scheduler polls DB, finds success on first poll
Task takes 15-60 min	Scheduler polls every 10s, connection-drop-proof
Backend crashes mid-task	Execution stays running, cleanup marks failed at 120 min
Scheduler crashes mid-poll	Lock expires (10 min), execution record already tracked by backend
Old backend (no async support)	Scheduler fallback: treat response as sync result
Old scheduler (no async flag)	Backend defaults async_mode=False, sync behavior unchanged
Cleanup service vs long task	120-min threshold prevents premature cleanup

Deploy Order

1. Backend first — Add async mode (backward compatible, default async_mode=False)
2. Scheduler second — Switch to async dispatch + polling

Verification

1. Deploy changes, restart backend and scheduler services
2. Create a schedule with a 5+ minute task on an agent
3. Trigger manually via Tasks tab or scheduler API
4. Monitor scheduler logs — should see:
   • POST /api/internal/execute-task accepted in < 1s
   • Polling messages every 10s
   • Final result logged when complete
5. Check schedule_executions table — verify record shows success with response, cost, duration
6. Check Dashboard Timeline — verify execution appears with correct status
7. Long task test: Schedule a 20+ minute task, verify it completes without false failed status

GitHub Issue Update

Attach this plan as a comment on issue #101 with implementation steps and file list.

[vybe]

Implementation Complete

All changes from the approved plan have been implemented:

Changes Made

File	Change
src/backend/routers/internal.py	Added async_mode field; when True, spawns background task and returns immediately
src/scheduler/service.py	Rewrote _call_backend_execute_task() with async dispatch + DB polling; added status overwrite guard
src/scheduler/config.py	Added poll_interval config (default 10s, POLL_INTERVAL env var)
src/backend/services/cleanup_service.py	Increased stale timeouts from 30 → 120 minutes
docs/memory/requirements.md	Added §10.6 SCHED-ASYNC-001, updated §12.9

Tests

tests/scheduler_tests/test_async_dispatch.py — 11 tests, all passing:

• Async dispatch sends async_mode=True with 30s timeout
• DB polling finds success/failed/delayed completion
• Polling timeout raises exception
• Status overwrite guard prevents overwriting finalized executions
• Backward compatibility with sync (old backend) responses
• Full end-to-end async flow

Deploy Order

1. Backend first — async_mode defaults to False, fully backward compatible
2. Scheduler second — switches to async dispatch + polling

[vybe]

Closing — fixed by SCHED-ASYNC-001 (commit 747ec4a). Scheduler now uses async fire-and-forget dispatch + DB polling, eliminating TCP connection drops on long-running tasks.