Summary
When the circuit breaker fires probes against an agent, each probe creates a schedule_executions record. If the probe is rejected (CB open), the record is never closed — it stays in a non-terminal state until the next backend restart, which runs startup cleanup and marks it failed. The result is a large red block on the execution timeline spanning the entire window from probe creation to backend restart, giving a misleading picture of agent activity.
Component
Backend / Circuit Breaker / Execution Timeline (Frontend)
Priority
P3
Observed Behavior
- Circuit breaker opens for an agent
- CB fires probe executions — each creates a
schedule_executions row
- Probes are rejected (CB open), but their DB records are never closed
- Records sit in non-terminal state (
running, pending_retry, etc.)
- On next backend restart, startup cleanup closes them with
error = "Execution orphaned — recovered on backend restart"
completed_at is set to the restart time, not the actual probe end time- Timeline displays a large red failure block spanning the full gap (e.g. 40 minutes) instead of a brief probe event
Root Cause
CB probe executions are not closed at the point of rejection. The backend creates the execution record before dispatching to the agent, but when the CB rejects the call, no corresponding completed_at / failed update is written back to schedule_executions. The record is only resolved as a side-effect of backend restart orphan cleanup, which stamps the current wall-clock time as completed_at — not the actual failure time.
Reproduction Steps
- Trigger enough consecutive agent failures to open the circuit breaker
- Observe CB firing probe executions in scheduler logs
- Let the CB remain open for several minutes without a backend restart
- Check
schedule_executions — probe records will be in non-terminal state
- Restart the backend
- View the agent's execution timeline — probe records now show as long-duration failures
Suggested Fix
When the circuit breaker rejects a call, immediately close the associated execution record with status = 'failed', completed_at = now(), and a descriptive error such as "Rejected by circuit breaker (open state)". This ensures the timeline accurately reflects a brief probe attempt rather than a multi-minute failure span.
Alternatively, CB probe executions could be marked with a distinct status (e.g. cb_rejected) so the frontend can render them differently (e.g. as a small marker rather than a full red block).
Related
Summary
When the circuit breaker fires probes against an agent, each probe creates a
schedule_executionsrecord. If the probe is rejected (CB open), the record is never closed — it stays in a non-terminal state until the next backend restart, which runs startup cleanup and marks itfailed. The result is a large red block on the execution timeline spanning the entire window from probe creation to backend restart, giving a misleading picture of agent activity.Component
Backend / Circuit Breaker / Execution Timeline (Frontend)
Priority
P3
Observed Behavior
schedule_executionsrowrunning,pending_retry, etc.)error = "Execution orphaned — recovered on backend restart"completed_atis set to the restart time, not the actual probe end timeRoot Cause
CB probe executions are not closed at the point of rejection. The backend creates the execution record before dispatching to the agent, but when the CB rejects the call, no corresponding
completed_at/failedupdate is written back toschedule_executions. The record is only resolved as a side-effect of backend restart orphan cleanup, which stamps the current wall-clock time ascompleted_at— not the actual failure time.Reproduction Steps
schedule_executions— probe records will be in non-terminal stateSuggested Fix
When the circuit breaker rejects a call, immediately close the associated execution record with
status = 'failed',completed_at = now(), and a descriptive error such as"Rejected by circuit breaker (open state)". This ensures the timeline accurately reflects a brief probe attempt rather than a multi-minute failure span.Alternatively, CB probe executions could be marked with a distinct status (e.g.
cb_rejected) so the frontend can render them differently (e.g. as a small marker rather than a full red block).Related
src/backend/services/agent_client.pysrc/backend/services/cleanup_service.py(or similar)