bug: scheduled runs ignore per-agent execution_timeout_seconds (canary S-03 + E-01 surfaced)

[obasilakis]

Summary

The canary invariant harness (CANARY-001, #884) consistently fires S-03 (slot TTL below floor) and E-01 (terminal-state closure) on scheduled executions whenever agent_ownership.execution_timeout_seconds is set to anything other than the schedule default. Both invariants are surfacing the same underlying drift: scheduled runs never use the per-agent timeout — they always use agent_schedules.timeout_seconds, default 900s.

Component

Backend / Scheduler / Slot Service / Cleanup Service

Priority

P2 — PUT /api/agents/{name}/timeout is documented as the per-agent execution-timeout control (/api/agents/{name}/timeout endpoint, TIMEOUT-001 migration) but is silently ineffective for scheduled runs. Manual /chat, /task, inter-agent (#418), backlog drain, and session endpoints all honor it correctly. Only the scheduler path is broken. Same class of bug as #418, related to #226 (closed) and #869 (open, status-in-dev).

What the canary surfaces

On any agent with execution_timeout_seconds != 900:

• S-03 below_floor — slot HASH stores timeout_seconds=900 (the schedule value), so slot TTL becomes 1200s. Canary computes floor from per-agent value (e.g., 3600+300=3900s). HASH TTL of ~1200s < floor of 3900s → S-03 fires.
• E-01 terminal-state closure — a row that runs past the per-agent threshold (exec_timeout + 300s) but is still within the per-schedule threshold (900 + 300s = 1200s) is alive at canary cycle time. Canary uses per-agent; cleanup uses per-schedule. E-01 fires; cleanup leaves the row alone.

Both invariants resolve to one root cause.

Root Cause

src/scheduler/service.py:805 passes schedule.timeout_seconds (loaded from DB) to backend's /api/internal/execute-task:

result = await self._call_backend_execute_task(
    ...
    timeout_seconds=schedule.timeout_seconds,
)

The TIMEOUT-001 migration docstring (src/backend/db/migrations.py:801-806) states the design intent:

"All execution paths (task API, chat, scheduler, MCP, paid endpoints) read from this setting when no explicit timeout is provided."

src/backend/services/task_execution_service.py:281-282 has the fallback:

if timeout_seconds is None:
    timeout_seconds = db.get_execution_timeout(agent_name)

But the scheduler always passes a concrete integer, never None. The fallback is dead code on the scheduler path.

The reason the scheduler can't pass None:

• ScheduleCreate.timeout_seconds: Optional[int] = None # None = use agent's config (src/backend/models.py:90)
• agent_schedules.timeout_seconds INTEGER DEFAULT 900 (running schema)
• src/scheduler/database.py:76 coerces any falsy DB value back to 900:

  timeout_seconds=row["timeout_seconds"] if "timeout_seconds" in row_keys and row["timeout_seconds"] else 900

So "user didn't set per-schedule timeout" is indistinguishable from "user explicitly set 900" by the time the scheduler reads the row.

src/backend/services/cleanup_service.py:647 uses ex.timeout_seconds or 900 (per-schedule) for its terminate decision — same drift on the termination side.

Reproduction

1. Pick an agent with agent_ownership.execution_timeout_seconds != 900 (set via PUT /api/agents/{name}/timeout or any agent created after the feat: increase default chat execution timeout from 15m to 60m #665 default bump to 3600).
2. Create a schedule without specifying timeout_seconds.
3. Trigger the schedule, snapshot agent:slot:{name}:{execution_id} HASH. Field timeout_seconds = 900 (matches schedule default, not per-agent value).
4. Run POST /api/canary/run-cycle while the task is running — S-03 fires with kind=below_floor, redis_ttl_seconds=~1200, floor_seconds=execution_timeout+300.
5. Verified locally on canary-fleet-burst (per-agent 3600s, schedule 900s) and canary-fleet-slow (per-agent 180s, schedule 900s) — S-03 fires every cycle for both, E-01 fires on rows that cross per-agent floor while still under per-schedule floor.

Suggested Fix

Make the per-agent value the actual default for scheduled runs. Pick one of:

Option A — fix at the scheduler boundary (minimal blast radius):

• src/scheduler/database.py:76 — return None (not 900) when the DB cell is NULL.
• src/scheduler/service.py:805 — if schedule.timeout_seconds is None, pass None to the backend so task_execution_service.py:281 fallback fires.
• src/backend/services/cleanup_service.py:647 — if ex.timeout_seconds is None, call db.get_execution_timeout(agent_name).
• Make agent_schedules.timeout_seconds nullable in schema; remove the DEFAULT 900. Add a migration to null out rows where the value equals the legacy default and was never explicitly set (or accept the small loss of fidelity for legacy rows).

Option B — propagate per-agent value at schedule creation:

• src/backend/db/schedules.py:create_schedule — when schedule_data.timeout_seconds is None, look up db.get_execution_timeout(agent_name) and store that value. Loses the "stay in sync with agent" property (per-agent updates won't retroactively affect existing schedules), but is operationally simpler and visible in the schedule row.

Option A preserves the design intent in the TIMEOUT-001 docstring. Option B is what most users probably expect the slider to do (apply now, stay applied), and is simpler to ship.

Related

• bug: Stale slot cleanup uses fixed 20-min TTL regardless of agent timeout #226 (closed) — same drift, cleanup-side, fixed.
• bug: inter-agent task timeout hardcoded at 600s ignores per-agent execution_timeout_seconds #418 (closed) — same class of bug, inter-agent path, fixed.
• bug: Cleanup watchdog falsely kills long-running executions as stale before timeout expires #869 (open, status-in-dev) — cleanup watchdog respecting per-slot timeout. PR fix(cleanup): per-slot TTL prevents false-kill of long-running executions (#869) #871 fixed the ping-and-kill path but not the per-agent-vs-per-schedule reconciliation.
• feat: increase default chat execution timeout from 15m to 60m #665 — bumped default per-agent timeout 15m → 60m, surfacing this drift more often.

Detection

This issue is detected automatically by the canary harness (#411, #882) on any staging/dev instance running the canary fleet. S-03 + E-01 will fire on every 5-min cycle until the per-agent and per-schedule timeouts are reconciled.