Summary
When agent executions time out at the backend level, the backend marks the execution as failed in the database but does not kill the spawned claude process inside the agent container. Over time, orphaned processes accumulate, consuming memory and inflating the "running" execution count in stats.
Component
Backend / Execution Manager
Priority
P2
Observed Behavior
- Backend sends a task to an agent container by spawning a
claude process (via docker exec)
- If the execution exceeds the backend's timeout, the backend marks the execution as
failed in schedule_executions
- The
claude OS process inside the container continues running indefinitely
- The
/api/agents/{name}/executions/{id}/terminate endpoint returns "Execution not found in agent" because the backend has already lost track of the process
- Failed executions show
cost: null, context_used: null — confirming the backend abandoned them without receiving a response
Real-world impact observed: 6 concurrent claude processes found in a single agent container (5 orphaned), each consuming ~160MB RAM (~800MB wasted). One "running" execution had been stuck for 3+ hours. The orphaned processes had to be manually killed via docker exec ... kill.
Root Cause
The backend does not maintain a mapping from execution ID to the container PID of the spawned claude process. When a timeout occurs, there is no handle to send a SIGTERM/SIGKILL to the child process. The backend simply stops waiting and writes failed to the database.
Additionally, the execution stats API (/api/agents/execution-stats) counts these zombie executions as running, since their status was never updated in all tracking layers.
Reproduction Steps
- Send a long-running task to an agent (e.g., a command that takes >10 minutes like installing large Python packages)
- Wait for the backend execution timeout to fire
- Observe the execution is marked
failed in the database
- Run
docker exec agent-{name} ps aux | grep claude — the process is still alive
- Repeat several times — orphaned processes accumulate
- Check
/api/agents/execution-stats — running_count is inflated
Suggested Fix
Several approaches (not mutually exclusive):
Option A: Track PID and kill on timeout
When spawning docker exec agent-{name} claude ..., capture the PID. On timeout, issue docker exec agent-{name} kill {pid}.
Option B: Container-side reaper
Add a lightweight watchdog process inside agent containers that periodically checks for claude processes with no active parent/session and kills them.
Option C: Backend cleanup on status check
When the backend detects executions stuck in running status past their timeout + grace period, proactively exec into the container to kill stale processes and update the DB status.
Option D: Process group management
Spawn claude processes in their own process group so the entire tree can be killed with a single signal.
Environment
- Trinity version:
9f910e9
- Docker version: 29.2.1
- OS: Ubuntu 22.04.5 LTS
Related
- Execution spawning logic in the backend execution manager
/api/agents/{name}/executions/{id}/terminate endpointschedule_executions table (status field lifecycle)/api/agents/execution-stats endpoint (running count accuracy)
Summary
When agent executions time out at the backend level, the backend marks the execution as
failedin the database but does not kill the spawnedclaudeprocess inside the agent container. Over time, orphaned processes accumulate, consuming memory and inflating the "running" execution count in stats.Component
Backend / Execution Manager
Priority
P2
Observed Behavior
claudeprocess (viadocker exec)failedinschedule_executionsclaudeOS process inside the container continues running indefinitely/api/agents/{name}/executions/{id}/terminateendpoint returns"Execution not found in agent"because the backend has already lost track of the processcost: null,context_used: null— confirming the backend abandoned them without receiving a responseReal-world impact observed: 6 concurrent
claudeprocesses found in a single agent container (5 orphaned), each consuming ~160MB RAM (~800MB wasted). One "running" execution had been stuck for 3+ hours. The orphaned processes had to be manually killed viadocker exec ... kill.Root Cause
The backend does not maintain a mapping from execution ID to the container PID of the spawned
claudeprocess. When a timeout occurs, there is no handle to send a SIGTERM/SIGKILL to the child process. The backend simply stops waiting and writesfailedto the database.Additionally, the execution stats API (
/api/agents/execution-stats) counts these zombie executions asrunning, since their status was never updated in all tracking layers.Reproduction Steps
failedin the databasedocker exec agent-{name} ps aux | grep claude— the process is still alive/api/agents/execution-stats—running_countis inflatedSuggested Fix
Several approaches (not mutually exclusive):
Option A: Track PID and kill on timeout
When spawning
docker exec agent-{name} claude ..., capture the PID. On timeout, issuedocker exec agent-{name} kill {pid}.Option B: Container-side reaper
Add a lightweight watchdog process inside agent containers that periodically checks for
claudeprocesses with no active parent/session and kills them.Option C: Backend cleanup on status check
When the backend detects executions stuck in
runningstatus past their timeout + grace period, proactively exec into the container to kill stale processes and update the DB status.Option D: Process group management
Spawn
claudeprocesses in their own process group so the entire tree can be killed with a single signal.Environment
9f910e9Related
/api/agents/{name}/executions/{id}/terminateendpointschedule_executionstable (statusfield lifecycle)/api/agents/execution-statsendpoint (running count accuracy)