bug: agent-server.py spins at ~83% CPU after claude CLI subprocess becomes defunct

[vybe]

Summary

After a scheduled task runs, the claude --print --output-format stream-json --verbose … subprocess inside an agent container can exit leaving a <defunct> zombie child. The parent python3 /app/agent-server.py does not reap it and begins spinning at ~83% CPU indefinitely. The FastAPI server inside the agent container then stops responding to every HTTP request (including /, /api/session, /api/files/download), even though the container remains in running state.

The backend's services.agent_client opens its circuit breaker for the affected agent and keeps re-opening it roughly every 35 s. Scheduled and ad-hoc tasks against the agent then get watchdog-terminated and recorded as failed — often misclassified downstream as "Auth failure (fallback detection): Subscription token may be expired or revoked" when the real cause is the SIGINT from the watchdog.

Other agents on the same host — rebuilt from the same base image — are unaffected.

Component

Agent Runtime / Base Image (docker/base-image/agent_server)

Priority

P1 — major feature broken for a single agent (all HTTP endpoints hang), no in-agent recovery. Workaround: restart the container.

Error

No explicit error is logged by the agent-server at the moment of failure — the process simply stops making forward progress. The last agent log line before the hang is:

INFO:agent_server.services.process_registry:[ProcessRegistry] Registered execution <exec-id>

On the backend side the symptom is a repeating pattern:

[Slots] Agent '<agent>' acquired slot 1/3 for execution <exec-id> (TTL=2100s)
[TaskExecService] Calling agent <agent> /api/task (timeout=1810.0s, ...)
Circuit OPENED for agent <agent> after N failures
Circuit CLOSED for agent <agent> (recovered)
Circuit OPENED for agent <agent> after N+1 failures
... (every ~35s indefinitely, until the container is restarted)

And, after the watchdog terminates the stale execution, a misleading error surfaces in the agent-server log:

ERROR:agent_server.services.claude_code:[Headless Task] Auth failure (fallback detection): Subscription token may be expired or revoked. Generate a new one with 'claude setup-token'.

— which is a false positive caused by SIGINT during stream-json reading, not an actual auth issue.

Location

• File: docker/base-image/agent_server/services/claude_code.py — subprocess / stream-json reader loop; does not appear to drain stdout/stderr and reap the child on all exit paths
• File: docker/base-image/agent_server/services/process_registry.py — verify cleanup hook fires on unexpected subprocess exit
• File: docker/base-image/startup.sh — recently changed; verify nothing regressed around Python launch / SIGCHLD handling
• Also suspect: the 50-turn cap termination path — if the task hits the cap, confirm the child is torn down cleanly

Process evidence

Process table inside the container while hung:

PID 1     Ss    /bin/bash /app/startup.sh
PID 17    S     sudo /usr/sbin/sshd -D
PID 23    S      \_ sshd: /usr/sbin/sshd -D [listener]
PID 18   82.8% Sl python3 /app/agent-server.py     <-- spinning
PID 35    0.6% Z   \_ [claude] <defunct>           <-- zombie, never reaped
PID 22    S     tail -f /dev/null

Container resource state:

CPU: 100.79%
Memory: 60.8 MiB / 4 GiB

Root Cause (suspected)

When the claude CLI subprocess exits (normal completion, cap hit, or killed mid-stream) without its stdout/stderr pipes being fully drained, the agent-server's async subprocess reader enters a tight loop reading from an already-closed pipe without detecting EOF, and never .wait()s the child. Result: parent spins, child becomes a zombie, asyncio event loop is starved, FastAPI stops serving.

Reproduction

Not yet reliably reproduced. Trigger appears to be: claude CLI subprocess exiting without its stdout/stderr being fully drained during stream-json parsing. Suggested repro angle:

1. Launch a scheduled task on an agent that will invoke a long claude --print --output-format stream-json --verbose run (e.g. any skill that produces many tool calls / approaches the 50-turn cap).
2. While the subprocess is streaming, forcibly kill the claude child (kill -9 <claude-pid>) or have it exit non-zero mid-stream.
3. Observe the parent python3 /app/agent-server.py CPU climb and stay pinned; observe the child become <defunct>; observe all agent endpoints stop responding.

Suggested Fix

• In claude_code.py, ensure the subprocess reader:
  • Treats EOF on stdout as a terminal condition and breaks out of the read loop.
  • Always awaits process.wait() in a finally: so the child is reaped regardless of exit path (success, cap, SIGINT, non-zero exit).
  • Distinguishes "child was SIGINT'd by watchdog" from "auth failure" so the fallback detection doesn't mislabel terminations.
• In process_registry.py, confirm unregister + reap runs on every finally: path, not just the success path.
• Add a hard wall-clock guard in the reader loop (e.g. if no bytes read for N seconds AND child has exited, break) so a stuck pipe read can't starve the event loop indefinitely.

Workaround

docker restart agent-<name>

After restart the process tree is clean (CPU back to baseline, endpoints return 200) and the backend logs Circuit CLOSED for agent <agent> (recovered).

Environment

• Previous working commit: 8e469cb (fix(messages): remove duplicate Depends() …)
• Reproducing commit: 13a0018 (feat: branch ownership enforcement S7 Branch ownership enforcement (S7) #382 feat: branch ownership enforcement (S7, #382) #396)
• Base image: trinity-agent-base:latest rebuilt 2026-04-19
• Base image files changed in the reproducing deploy:
  • docker/base-image/agent_server/routers/files.py
  • docker/base-image/agent_server/routers/git.py
  • docker/base-image/agent_server/utils/git_conflict.py
  • docker/base-image/startup.sh

Ops impact

• Single agent becomes unreachable until container restart.
• Backend circuit breaker correctly opens, but keeps retrying in a ~35 s loop, generating log noise (~4 warnings/min per affected agent).
• Scheduled executions against the hung agent are watchdog-terminated and recorded as failed with a misleading "Auth failure" message.
• No cross-agent impact observed; container memory limits contain the spin.

[vybe]

Starting narrow fix on branch feature/407-subprocess-zombie-reaper.

Scope: start_new_session=True + process-group kill (os.killpg(SIGTERM) → wait with timeout → SIGKILL), bounded wait(timeout=) on all exit paths, watchdog after stderr thread join, and a regression test that holds the stderr pipe open via a grandchild while Claude exits.

Broader architectural question (migrate to Claude Agent SDK) tracked separately in #409.