bug: SIGKILL/timeout terminations of claude subprocess misclassified as authentication failure

[vybe]

Summary

When the agent-server's claude subprocess is killed by an external signal (schedule timeout SIGKILL, OOM-kill, parent SIGTERM, etc.), agent-server falls through into the auth-failure heuristics and surfaces the error as Authentication failure: Subscription token may be expired or revoked. Generate a new one with 'claude setup-token'. In the operator UI this looks identical to a real expired-token incident, sending operators down the wrong diagnostic path while the actual cause (skill exceeded its schedule timeout) goes unnoticed and keeps repeating every cron tick.

This is the same shape as #361 (max-turns misclassified as auth) — same file, same heuristic block — but a different exit path. #361 only special-cased the max-turns exit; the signal-kill exit still falls through.

Component

Agent Runtime (agent_server.services.claude_code)

Priority

P2 — feature impaired with workaround (raise the schedule timeout); diagnostic UX is broken so the workaround is hard to discover.

Error

Symptom in agent-server log on every signal-kill:

[Subprocess] Reader thread(s) stuck after process exit (pid=NNNNN, stuck_count=1) — killing process group and closing pipes to unwind
[Subprocess] 1 reader thread(s) leaked for pid=NNNNN after close+killpg; continuing anyway
[Headless Task] Error reading stdout: I/O operation on closed file.
[Headless Task] Auth failure (fallback detection): Subscription token may be expired or revoked. Generate a new one with 'claude setup-token'.
POST /api/task HTTP/1.1 503 Service Unavailable

The first three lines are the SIGKILL fingerprint (process gone, but stdout pipe is still held → reader thread stuck → forced unwind). The fourth line is the misclassification.

What surfaces in the platform schedule_executions.error column:

Authentication failure: Subscription token may be expired or revoked. Generate a new one with 'claude setup-token'.. Check subscription token or API key configuration.

…even though the token is fine, sister agents on the same subscription are running successfully at the same moment, and the actual cause is the schedule timeout SIGKILL.

Location

• File: docker/base-image/agent_server/services/claude_code.py
• Lines: ~1255–1280 (the if return_code != 0: block, after the max-turns special-case from bug: Max-turns termination misclassified as authentication failure #361)
• Two heuristic paths both wrongly fire for signal-kills:
  • _is_auth_failure_message(error_preview) or _is_auth_failure_message(verbose_transcript) — line ~1263
  • metadata.input_tokens == 0 and metadata.output_tokens == 0 — line ~1273

Root Cause

In the --print headless path, all non-zero subprocess exits funnel into one branch and the auth heuristics get first crack at classifying. Two of those heuristics fire by default when we have no usable transcript:

• The fallback string-match runs against verbose_transcript, but on a SIGKILL there's typically no auth-related text — the heuristic still has to make a call and easily catches generic "token" / "auth" substrings from the system prompt or partial transcript.
• The "zero tokens processed" path is the more direct culprit: a SIGKILL'd subprocess never emits the final result message, so metadata.input_tokens == 0 and metadata.output_tokens == 0, and the code declares it "likely auth failure".

Neither heuristic checks the signed return code first. POSIX exit codes for signal terminations are negative in Python's subprocess (-9, -15) or shell-encoded as 128 + signum (137, 143). Distinguishing those before running the auth heuristics would eliminate the false positive entirely.

The same bug affects:

• Schedule timeout enforcement (the most common trigger — every cron whose skill outgrows its timeout_seconds produces a stream of fake "auth" errors).
• Cgroup OOM kills (rare today but a real risk on large skills).
• Operator-initiated terminate via POST /api/executions/{id}/terminate (internally already handled, but the same code path can fire for parent-side cancel).

Reproduction Steps

1. Create or use any agent that runs claude --print headless via the agent-server (i.e., any normal scheduled task).
2. Set the agent's schedule timeout_seconds to something deliberately shorter than the skill's real runtime — e.g. 60 for a skill that needs ~5 min.
3. Trigger the schedule once (manually or via cron).
4. Observe in agent-server logs:
   • Reader thread(s) stuck after process exit … killing process group
   • Error reading stdout: I/O operation on closed file.
   • Auth failure (fallback detection): Subscription token may be expired or revoked.
5. Observe in schedule_executions.error for that execution: the misleading "Subscription token may be expired or revoked" string. Token is fine; subprocess was killed.

Suggested Fix

Special-case signal terminations before the auth heuristics, mirroring how #361 special-cased max-turns.

# In claude_code.py, inside the `if return_code != 0:` block,
# AFTER the max-turns 422 check, BEFORE the auth-fallback heuristics:

# Signal terminations (SIGKILL/SIGTERM from timeout, OOM, parent cancel)
# Python subprocess exposes these as negative return codes; some shells
# expose them as 128 + signum.
if return_code < 0 or return_code in (137, 143):
    signum = -return_code if return_code < 0 else return_code - 128
    sig_name = {9: "SIGKILL", 15: "SIGTERM"}.get(signum, f"signal {signum}")
    logger.warning(
        f"[Headless Task] Subprocess terminated by {sig_name} "
        f"(exit={return_code}, tools={metadata.tool_use_count}, "
        f"raw_messages={metadata.raw_message_count})"
    )
    raise HTTPException(
        status_code=504,  # Gateway Timeout — reflects external kill, not auth
        detail=(
            f"Execution terminated by {sig_name} after "
            f"{metadata.tool_use_count} tool calls / "
            f"{metadata.raw_message_count} messages. "
            f"Likely cause: schedule/agent timeout exceeded, OOM kill, or operator cancel. "
            f"Increase the schedule's timeout_seconds, raise agent memory, "
            f"or split the skill into smaller steps."
        ),
    )

Also worth tightening the two heuristics that follow:

• The _is_auth_failure_message(verbose_transcript) call should ignore the system prompt portion — it's currently happy to match token/auth/etc. anywhere in the captured transcript, which guarantees false positives on long skills that mention those words.
• The "zero tokens" branch should additionally require return_code > 0 (true exit, not signal) — once the signal special-case lands, this becomes correct by construction.

A test fixture could simulate this by spawning a sleep subprocess via the same code path and SIGKILL'ing it; the response should be 504, not 503-with-auth-message.

Environment

• Trinity version: observed on c16fafc (3 days behind origin/main at time of report)
• Affected since: bug: expired subscription token causes hour-long zombie executions instead of fast failure #285 introduced the auth-fallback heuristics; bug: Max-turns termination misclassified as authentication failure #361 fixed only the max-turns slice of the same shape.
• Operator-side cost: every misclassified exec triggers PR feat(subscription): auto-switch on first failure + auth-class triggers (#441) #508's broadened auto-switch (when merged), which would unnecessarily rotate the subscription on what is actually a timeout.

Related

• bug: Max-turns termination misclassified as authentication failure #361 — Max-turns termination misclassified as authentication failure (CLOSED, same file, same shape, partial fix)
• bug: expired subscription token causes hour-long zombie executions instead of fast failure #285 — Original expired-token fast-fail work that introduced these heuristics
• bug: agent-server.py spins at ~83% CPU after claude CLI subprocess becomes defunct #407 — Subprocess zombie reaper (related plumbing in the same code path)
• PR feat(subscription): auto-switch on first failure + auth-class triggers (#441) #508 — Auto-switch on first auth-class failure (broadens the blast radius of any auth false-positive; this bug becomes more impactful once that lands)
• File: docker/base-image/agent_server/services/claude_code.py