feat(agents): ephemeral ghost agents — budgeted hard-discard lifecycle + spawn provenance (trinity-enterprise#69)

[vybe]

Summary

• Ephemeral "ghost" agents (abilityai/trinity-enterprise#69): agents created with a hard budget (max_executions and/or TTL — expiry ALWAYS stamped, 24h default ceiling), volume-less (container writable layer; ghosts never recreate), and hard-discarded at budget — container removed, DB rows cascade-purged, Redis state cleared; no soft-delete, no 180-day name reservation. Creation is entitlement-gated (ephemeral_agents, fail-closed — the feature is inert until the enterprise module registers it); every lifecycle mechanic is an edition-agnostic OSS primitive (the suspended_at pattern).
• Part 2 — spawn provenance + parent control: any agent-spawned creation persists spawned_by_agent/spawned_by_key_id and auto-grants the agent_permissions parent→child edge, so an orchestrator can immediately chat/list/control the child it spawned.
• Scoped to heterogeneous-workspace jobs (a different repo/config per ghost); same-agent burst parallelism stays with fan_out / replica groups (ghost-clones is the target architecture's named anti-pattern).

⚠️ Behavior changes (release-notes callout)

1. Agent-scoped keys may start/stop/delete ONLY agents they spawned (spawned_by name AND key-id match; interim until feat: workflow-scoped capability tokens for agent-to-agent calls #948 capability tokens). A durable agent key that previously start/stopped sibling agents via owner-equivalence now gets 403 on those three routes.
2. Sharing, permission grants, rename, and credential inject/export/import are now human-only — agent-scoped keys get 403 (they previously resolved to the owner with full power).
3. A ghost's own key is confined at the auth entry point to heartbeat / result-callback / reports / notifications / self-info.

Fleet-wide narrowing of the remaining agent-key breadth on other mutating routes is a recorded accepted-risk follow-up (per the issue's own AC).

Key design points

• Budget gate at the TOP of CapacityManager.acquire (no-enqueue invariant, beside the dispatch-breaker gate) — covers every admission surface; routers map to 410 Gone; pull-mode (feat: pull / work-stealing coordination (push→pull dispatch) #1081) must re-check the predicate at the claim endpoint (noted in the feature flow).
• Crash-convergent discard: durable intent marker first (expires_at=now), CAS-fail all non-terminal rows (canary L-03/E-01 stay green; defuses late-writer breaker resurrection), Redis clear BEFORE the purge frees the name (bug(lifecycle): stale transport circuit-breaker Redis key survives agent delete/recreate — a fresh healthy agent fast-fails as "unhealthy" #1560 ordering), schedule_executions KEEP (ages out via 90d retention; admin-only visible post-purge — documented deviation).
• GC: cleanup-service sweep (DB pass + Docker-as-truth orphan pass with a 15-min newborn grace window), capped per cycle; folds into the refactor: collapse 9-path cleanup pyramid once agent is authoritative #429 consolidated reaper later.
• Abuse controls: atomic per-owner quota (Redis INCR-with-cap, NX reseed, DB fallback), per-parent spawn rate limit, ephemeral-cannot-spawn-ephemeral, max_parallel_tasks=1 default, no credential auto-injection.

Changes

Backend: models.py, dependencies.py (fence + 2 guards), services/agent_service/{crud,lifecycle,ephemeral(new)}.py, services/{capacity_manager,task_execution_service,cleanup_service,docker_service,heartbeat_service,monitoring_service,settings_service}.py, routers/{agents,chat,schedules,sharing,agent_rename,credentials,agent_files,monitoring}.py, db/{schema,tables,migrations,agents,schedules}.py, db/agent_settings/ephemeral.py (new mixin), Alembic 0016, database.py facade. MCP: agents.ts (ephemeral param, delete description), types.ts. Frontend: GHOST badge (Agents list + AgentHeader). Docs: requirements (lifecycle-observability), architecture.md subsystem block, feature-flows/ephemeral-agents.md, learnings.md.

Test Plan

• tests/unit/test_69_ephemeral_agents.py — 40 tests on the real-engine db_harness: mixin accessors round-trips, facade delegations, acquire-gate deny matrix (fires BEFORE slot work), key-fence allow/deny matrix, Part 2 guard matrix (name+key-id), budget hook fail-open + trigger, discard full-path/idempotent-rerun/half-discarded-resume/lock-contention, atomic quota + Redis-down fallback
• Guard suites green: schema-parity, Alembic id-length, models-centralized, bug(lifecycle): stale transport circuit-breaker Redis key survives agent delete/recreate — a fresh healthy agent fast-fails as "unhealthy" #1560 keyspace parity, AGENT_REFS cascade parity, status-write inventory (new writer classified)
• Full unit suite: residual 3-test order-dependent flake reproduces on clean dev (2 of 3 baseline worktree runs) — pre-existing, documented in learnings.md
• Live smoke once the enterprise ephemeral_agents registration lands (feature is 403/inert until then — fail-closed by design)

Follow-ups (filed separately)

• Public bug: durable-agent volume leak — volume_remove has zero callers; purge never reclaims volumes (ghosts sidestep it by being volume-less)
• Fleet-wide agent-key narrowing on remaining mutating routes (accepted risk)
• refactor: collapse 9-path cleanup pyramid once agent is authoritative #429 note: ephemeral orphan-reclaim folds into the consolidated lease reaper
• Enterprise repo: register the ephemeral_agents module (activates the feature)

Refs abilityai/trinity-enterprise#69 — private-tracker issue; closed manually at release (cross-repo keywords don't auto-close). Phase 0 design record: https://github.com/Abilityai/trinity-enterprise/issues/69#issuecomment-4952092419

🤖 Generated with Claude Code

[dolho]

Review — #1580 feat(agents): ephemeral ghost agents

Verdict: Approve with one correctness finding to resolve (here or as a documented fast-follow). This is a large, careful, defense-in-depth PR. The security fences, crash-convergent discard ordering, and fail-open/fail-closed choices are all deliberate and well-reasoned. One real budget-counting edge case, plus minor notes.

What's solid

• Auth fences (dependencies.py) are all default-deny / correct direction:
  • _enforce_ephemeral_key_fence — allowlist of (method, path) for a ghost's own key; a path typo fails closed (over-restrict), never open. {name} group pinned to the key's own agent. Keyed off is_ephemeral so it dies with the ghost; fail-open only on DB read error.
  • reject_agent_principal (share/permissions/rename/credentials) and enforce_agent_spawn_scope (start/stop/delete gated on spawned_by_agent AND spawned_by_key_id, not name alone — name reuse can't forge control). Correct.
• Budget admission gate at the top of CapacityManager.acquire, beside the breaker (no-enqueue invariant). ISO-Z lexicographic compare is valid per Invariant #16. Fail-open on DB error, isinstance-dict guarded against mock-truthiness.
• Hard discard ordering is genuinely crash-convergent: intent marker → CAS-fail non-terminal rows (late apply_result loses its CAS → no breaker resurrection) → container rm → clear Redis state before the purge frees the name (bug(lifecycle): stale transport circuit-breaker Redis key survives agent delete/recreate — a fresh healthy agent fast-fails as "unhealthy" #1560) → cascade purge (refuses non-ephemeral rows) → audit. SETNX lock with compare-and-delete release, idempotent-so-fail-open-when-Redis-down. purge_ephemeral_agent_ownership refusing is_ephemeral=0 is a good guardrail against ever hard-deleting a durable agent.
• Quota is atomic (Redis INCR-with-cap, NX reseed from DB count to survive a Redis restart, floor-clamped release), with DB-count fail-open. Both creation-failure paths (docker-available except-rollback and the docker-unavailable else, review M2) release the reservation.
• Creation gate ordering matches the doc (entitlement 403 → recursion 403 → spawn rate-limit → TTL ceiling → unique name → atomic quota just before the docker block). Entitlement is fail-closed — inert in OSS until the enterprise module registers ephemeral_agents. Good.
• Fleet hygiene (heartbeat/monitoring exclude ghosts via getattr(a,'ephemeral',False) is not True, keeps old shapes covered), schedules blocked (400), GHOST badge, MCP ephemeral param + delete description. All coherent.

Finding (medium) — a ghost can count its own in-flight row against its budget on the execute_task-internal path

The gate does count_ephemeral_budget_usage(agent_name) and denies when terminal + active >= max_executions, where active = queued/running/pending_retry. It does not exclude the current execution_id — even though acquire receives it.

• On the /chat path this is fine: chat_execution_id is a bare UUID at acquire time; the create_task_execution row is written after admission. So chat_with_agent → /chat (the primary orchestrator path) counts only other concurrent rows. Correct.
• On the execute_task-internal path it is not: execute_task calls db.create_task_execution(...) with status=RUNNING before capacity.acquire (task_execution_service.py §1). loop_service invokes execute_task with execution_id=None, so a loop on a ghost creates the RUNNING row first → the ghost's own row is counted as active:
  • max_executions=1 loop → run 1: terminal=0, active=1 (its own row) → 1 >= 1 → EphemeralBudgetExhausted → the ghost never executes.
  • max_executions=3 loop → yields only 2 runs (off-by-one).

Schedules are blocked on ghosts (400) but loops and fan-out are not, so this is reachable, not purely theoretical. /task also pre-creates the row early (per the execute_task docstring) and may hit the same skew depending on whether it pre-acquires in the router — worth confirming.

Fix options (either is small): exclude the passed execution_id from the active count in count_ephemeral_budget_usage (the gate already has it), or block loop/fan_out on ephemeral agents the same way schedules are (400), matching the "chat/heterogeneous-jobs only" positioning. The first is more robust and keeps the gate honest for any future internal caller.

Minor

• _enforce_ephemeral_key_fence adds one PK-indexed DB read to every agent-scoped-key request (not just ghosts). Mirrors the connector fence, so acceptable — just noting the per-request cost on agent-to-agent traffic.
• count_ephemeral_budget_usage runs a COUNT over schedule_executions WHERE agent_name per acquire for ghosts. Fine at ghost volume (max_parallel=1), but confirm schedule_executions.agent_name is indexed if a high-max_executions ghost is ever expected.
• Post-discard KEEP schedule_executions become admin-only visible (owner visibility derives from the purged ownership row) — already documented as an intentional deviation; fine, just make sure billing/stats consumers expect it.

Behavior changes (correctly called out in the PR body)

The narrowing of durable agent-scoped keys (can no longer start/stop/delete siblings via owner-equivalence; share/permissions/rename/credentials now human-only) is intentional per the issue AC and release-noted. No objection — just flagging that any existing multi-agent setup relying on agent-key sibling control will get 403s after this lands.

Tests (40, real-engine harness) and the guard-suite coverage (schema-parity, Alembic id-length, #1560 keyspace parity, AGENT_REFS cascade parity) are strong. Resolve the loop/fan-out budget skew and this is good to merge.