Abilityai · vybe · Jul 31, 2026 · Jul 30, 2026 · Jul 30, 2026 · Jul 30, 2026
diff --git a/.env.example b/.env.example
@@ -229,6 +229,11 @@ VOIP_INTENT_TTL_SECONDS=180       # staged Gemini-session intent TTL
 # Agent structured reports (#918)
 REPORT_RATE_LIMIT=30              # max reports an agent may create per 60s window
 
+# Skills library lifecycle automation (ent#236) — auto-sync/re-inject toggles live in
+# Settings (system_settings), not here; these two are the tuning knobs.
+SKILLS_RECONCILE_MAX_REMOVALS=10  # start-path reconcile refuses above this many removals per agent
+SKILLS_FLEET_INJECT_CONCURRENCY=5 # agents re-injected in parallel during a fleet sweep
+
 # Public webhook triggers (#1023, #1424) — 60s windows fixed in code
 WEBHOOK_RATE_LIMIT=10             # triggers per token per 60s
 WEBHOOK_IP_RATE_LIMIT=60          # pre-auth requests per IP per 60s (unknown-token flood guard)

diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml
@@ -110,6 +110,10 @@ services:
       # Agent structured reports (#918) — per-agent create-rate cap (60s window fixed in code).
       # Prod compose launches standalone (no base merge / env_file), so wire it here too.
       - REPORT_RATE_LIMIT=${REPORT_RATE_LIMIT:-30}               # reports per agent per 60s
+      # Skills library lifecycle automation (ent#236).
+      # Prod compose launches standalone (no base merge / env_file), so wire it here too.
+      - SKILLS_RECONCILE_MAX_REMOVALS=${SKILLS_RECONCILE_MAX_REMOVALS:-10}   # per-agent blast-radius cap
+      - SKILLS_FLEET_INJECT_CONCURRENCY=${SKILLS_FLEET_INJECT_CONCURRENCY:-5}  # parallel agents per sweep
       # Public webhook trigger hardening (#1023, #1424) — 60s windows fixed in code.
       # Prod compose launches standalone (no base merge / env_file), so wire it here too.
       - WEBHOOK_RATE_LIMIT=${WEBHOOK_RATE_LIMIT:-10}             # triggers per token per 60s

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -73,6 +73,11 @@ services:
       # Agent structured reports (#918) — per-agent create-rate cap (60s window fixed in code).
       # Must reach the container or the .env lever is inert in prod (the #1039 packaging class).
       - REPORT_RATE_LIMIT=${REPORT_RATE_LIMIT:-30}               # reports per agent per 60s
+      # Skills library lifecycle automation (ent#236). The reconcile cap is named in the
+      # refusal alarm's own remediation text, so it must be tunable from .env or the
+      # operator is told to raise a lever that never reaches the container.
+      - SKILLS_RECONCILE_MAX_REMOVALS=${SKILLS_RECONCILE_MAX_REMOVALS:-10}   # per-agent blast-radius cap
+      - SKILLS_FLEET_INJECT_CONCURRENCY=${SKILLS_FLEET_INJECT_CONCURRENCY:-5}  # parallel agents per sweep
       # Public webhook trigger hardening (#1023, #1424) — 60s windows fixed in code.
       # Must reach the container or the .env lever is inert (the #1056/#1039 packaging class).
       - WEBHOOK_RATE_LIMIT=${WEBHOOK_RATE_LIMIT:-10}             # triggers per token per 60s

diff --git a/docs/memory/architecture.md b/docs/memory/architecture.md
@@ -213,8 +213,9 @@
 - `image_generation_service.py` / `image_generation_prompts.py` - Platform image generation via Gemini (IMG-001)
 
 *Skills & System:*
-- `skill_service.py` - Skills library sync + full-directory package injection (ent#183): `git archive`-sourced tars via the existing agent-server restore primitive, tree-SHA versioning, manifest prune, declaration-only dep check — see [skill-injection.md](feature-flows/skill-injection.md)
-- `skill_packaging.py` - Pure skill-package primitives (ent#183): hardened frontmatter contract parse, archive member vetting, injection-tar assembly, prune diff
+- `skill_service.py` - Skills library sync + full-directory package injection (ent#183): `git archive`-sourced tars via the existing agent-server restore primitive, tree-SHA versioning, manifest prune, declaration-only dep check — see [skill-injection.md](feature-flows/skill-injection.md). **ent#236** adds the removal half — `remove_skills` (manifest-driven, takes the same per-agent inject lock) and `reconcile_agent_skills` (start-path diff of the agent's platform-managed skill dirs against the assignment set, so a removal reaches an agent that was stopped when it happened), plus durable sync status in `system_settings` (the in-process `_last_sync` is invisible to the other uvicorn worker)
+- `skill_packaging.py` - Pure skill-package primitives (ent#183): hardened frontmatter contract parse, archive member vetting, injection-tar assembly, prune diff; `compute_removal` (ent#236) is `compute_prune` against an empty new manifest, so confinement + cap live in one place
+- `skills_sync_service.py` - Scheduled skills-library auto-sync + fleet-wide re-inject (ent#236). Leader-locked (`skills:sync:leader`), both flags default OFF; sweeps only when the library commit actually changed, over **running** non-ghost agents at bounded concurrency with skip-and-report on inject-lock contention. Backend-hosted rather than in the standalone scheduler because the sweep must reach agent containers and the scheduler is platform-network-only
 - `system_agent_service.py` - System agent lifecycle. **#1816:** `ensure_deployed` is read-only when the container is running (3-state `check_base_image_state` → `base_image_state`, WARNING + an edge-triggered `base-image-stale-` operator alarm on `stale` only, **never** on `unknown`) and delegates to `start_agent_internal` when it is stopped, so the cold boundary adopts a rebuilt base image through the shared lifecycle instead of a bare `container_start`. Creation converges on the recreate path's contract (`TRINITY_AGENT_AUTH_TOKEN`, `trinity.full-capabilities`) so no predicate is permanently false. The generic `recreate_missing_container` rebuild **refuses** `trinity-system` (409, ADOPT-006) — it reconstructs a *regular* agent and would irreversibly downgrade the orchestrator (system-scoped MCP key deactivated for an agent-scoped one); `ensure_deployed`'s create branch is the only supported rebuild
 - `cornelius_agent_service.py` - First-run auto-seed of the default "Cornelius" second-brain agent (public `github:Abilityai/cornelius`, cloned anonymously on the ent#123 tokenless path, Brain Orb enabled) — see [Brain Orb](#brain-orb--self-rendering-mind-page-58-trinity-enterprise) (trinity-enterprise#107, source-seeded #1656). Invoked via the ent#124 first-run orchestrator; accepts an optional precomputed `fresh` verdict
 - `system_seed_service.py` - First-run seed of the default system manifest (trinity-enterprise#124): `ensure_first_run_seeded()` (both call sites: setup-completion bg task + lifespan safety-net) resolves a **persisted first-run verdict** (`first_run_fresh` — computed once BEFORE Cornelius provisions, so sibling-seeded agents can't poison later passes), runs the Cornelius seeder, then deploys the bundled `config/manifests/default-system.yaml` (env override/disable via `TRINITY_DEFAULT_SYSTEM_MANIFEST`) through `system_service.deploy_manifest` — durable `default_system_seeded` flag + `system_seed:provision` SETNX lock + reserved-name existence backstop (the deploy path suffixes collisions instead of 409ing); partial/failed seeds raise an operator-queue alert; fail-open, never blocks boot. See requirements §16.5.1 (roadmap.md)
@@ -347,6 +348,7 @@ Services that run continuously in the backend process:
 | **Cleanup Service** | `cleanup_service.py` | Every 5 min: active watchdog reconciliation against agent process registries (orphan recovery, auto-terminate timeouts) + passive stale recovery (CLEANUP-001, #129). Also runs retention + soft-delete purge sweeps, the **expired-SSH sweep** (`_sweep_expired_ssh_credentials` → `SshService.cleanup_expired_credentials` — removes an expired ephemeral key's line from the container `authorized_keys` sshd reads; TTL was previously enforced only on Redis metadata, #1616), and the #740 startup orphan-loop hook, and the **agent_reminders retention sweep** (`_sweep_agent_reminders_retention` — DELETEs terminal `fired`/`cancelled`/`failed` reminders past `agent_reminders_retention_days`, #1296) — see [Soft Delete & Retention](#soft-delete-retention--recovery-834-772). Runs the additive **lease-reaper** (`lease_reaper_service`) each cycle — re-queues (preserving `execution_id`) or poison-parks expired pull leases (#1081 Phase 3, #429/#1402; inert until an agent is piloted). #1804: every recovery path also closes its execution's dispatch activity (`_close_bulk_swept_activities` for the bulk sweeps, the shared helper elsewhere), counted in `activities_closed_on_recovery`; the 120-minute activity backstop now runs **last** in the cycle |
 | **Operator Queue Sync** | `operator_queue_service.py` | Polls running agents every 5s, reads `~/.trinity/operator-queue.json`, syncs to DB, writes responses back (OPS-001). The item `id` is a platform-minted uuid; the agent's correlation string is `request_id` with `(agent_name, request_id)` uniqueness, so all sync reads/writes (exists, acknowledge, response write-back) are agent-scoped and two agents can't collide (#1631). **Leader-locked (#1632):** only the holder of `opqueue:leader` (SET NX, TTL `max(3×interval, 30s)` floor so a slow-write cycle can't flap leadership, own-lease refresh, fail-open — mirror monitoring #1464) runs a cycle, so `--workers 2` doesn't double-charge the ingestion rate limiter or double-broadcast the flood alert. Ingestion is capped per agent (depth + rate + fleet + field hygiene, #1632) — see [Operator Queue](#operator-queue-ops-001) |
 | **Sync Health Service** | `sync_health_service.py` | Polls git-enabled agents every 60s — see [Git Sync Health](#git-sync-health-389390) |
+| **Skills Library Sync** | `skills_sync_service.py` | Scheduled skills-library `git pull` + optional fleet-wide skill re-inject (ent#236). Runs in every worker but only the `skills:sync:leader` lease-holder performs a cycle (fail-open, mirrors #1464); self-gates on the default-OFF `skills_library_auto_sync_enabled` setting, re-read each cycle so an interval change needs no restart. A sweep fires only on a changed library commit, targets running non-ghost agents at `SKILLS_FLEET_INJECT_CONCURRENCY` (5), and persists an honest per-agent report + raises an operator alarm on any failure |
 | **Monitoring Service** | `monitoring_service.py` | Fleet-wide health checks on configurable interval (30s default); authoritative for aggregate status. **Lifespan-resumed (#1121):** boot reads the persisted `monitoring_config` (staggered +12s) and starts the loop only when `enabled` — the flag is the single source of truth, **defaults OFF**, persisted by `enable`/`disable`/`PUT /config` (which also reconcile the running loop) so the choice survives restarts; `*_check_interval` rejects non-positive values (422), loop clamps sleep ≥1s (MON-001). **Cross-worker leader lock (#1464):** the loop runs in every uvicorn worker but only the holder of the Redis `monitoring:leader` lease (SET NX, TTL 3×interval, own-lease-only refresh; fail-open to leader when Redis is down) performs each probe cycle, so `--workers 2` no longer double-probes the fleet or double-feeds the circuit breaker; leadership fails over automatically when the holder dies |
 | **Heartbeat Watch Loop** | `heartbeat_service.py` | 5s loop acting on missed agent heartbeats — see [Heartbeat Liveness](#heartbeat-liveness-reliability-004-307) |
 | **Scheduler Service** | `scheduler_service.py` | APScheduler cron execution; async fire-and-forget with DB polling for status. On each cron fire, optionally invokes the agent's `~/.trinity/pre-check` (see Agent Containers). Also owns one-shot `DateTrigger`s for RETRY-001 retries and **agent self-reminders** (#1296): `_reconcile_reminders` arms pending reminders + reclaims stale `firing` rows at boot, in the 60s sync loop (own try/except), and on full reload — see [Agent Self-Reminders](#agent-self-reminders-1296) |
@@ -1085,6 +1087,7 @@ Coverage: agent lifecycle, auth, sharing, credentials, settings, rename; request
 | GET/PUT | `/api/settings/agent-defaults/resources` | Fleet-wide default CPU/memory for new containers (admin-only; CPU 1/2/4/8/16, memory 1g–32g) (RES-001) |
 | GET/PUT | `/api/settings/agent-defaults/access-policy` | Fleet-wide default `require_email` for new agents (admin-only, #1129). Stored in `system_settings`, **secure-by-default ON** (code fallback when unset — no migration); seeds `agent_ownership.require_email` at creation (`register_agent_owner`) for **new** agents only, never rewrites existing rows; owners still override per agent via `PUT /api/agents/{name}/access-policy` |
 | GET/PUT | `/api/settings/max-parallel-tasks-ceiling` | Fleet-wide ceiling on per-agent `max_parallel_tasks` (admin-only, #506). Returns `{value, default, min, max}`; PUT range-validated 1–32 (400 otherwise), audit-logged. Stored in `system_settings` (no migration). The generic catch-all `PUT /{key}` is blocked for this key (422 → dedicated route). Clamp is runtime/clamp-on-use — see [Capacity & Backlog](#capacity--backlog-428) |
+| GET/PUT | `/api/settings/skills-library` | Skills-library lifecycle automation (admin-only, ent#236). GET: `auto_sync_enabled` / `auto_sync_interval_seconds` / `auto_reinject_enabled` + interval bounds, **plus** the durable sync status (`last_sync`, `last_sync_status`, `last_sync_error`) and the last fleet-re-inject report — the panel must be able to show a *failing* auto-sync. PUT: partial update (an omitted field is untouched), interval range-validated 300–86400 with a descriptive 400 rather than a silent clamp; audit-logged. The three keys are blocked on the generic `PUT /{key}` (unvalidated `Dict[str,str]`; `"10"` would be accepted verbatim and fetch GitHub six times a minute — #1644 class). Registered before `/{key}` (Invariant #4) |
 | GET/PUT | `/api/settings/brain-orb` | Brain Orb platform flags (admin-only, trinity-enterprise#85). GET: per-flag `{value, source: override\|env\|default}` + `gemini_key_configured` (boolean only — never the key). PUT: partial booleans (`enabled`/`voice_enabled`/`write_enabled`) and/or `clear: [flag,…]` reverting a flag to its env/default (400 on unknown name or set+clear conflict); audit-logged with per-flag old→new. Stored in `system_settings` (no migration); route gates resolve at request time — no restart. Registered before `/{key}` (Invariant #4) — see [Brain Orb](#brain-orb--self-rendering-mind-page-58-trinity-enterprise) |
 | GET/PUT | `/api/settings/elevenlabs` | ElevenLabs / voice platform settings (admin-only, ent#117). GET: `{key_configured, key_source: override\|env\|none, default_voice_id}` — the key value is never echoed. PUT: partial `{api_key?, default_voice_id?, clear: ["api_key"\|"default_voice_id"]}`; key stored AES-256-GCM encrypted (Invariant #12) in `system_settings`; runtime-resolved (no restart); audit-logged masked. Registered before `/{key}` (Invariant #4) |
 
@@ -1924,6 +1927,7 @@ CREATE INDEX idx_agent_reports_created ON agent_reports(created_at);  -- retenti
 - **Capacity/breaker keys**: `agent:slots:{name}` (ZSET) + `agent:slot:{name}:{eid}` (HASH), `agent:circuit:{name}`, `agent:dispatch:{name}`, `canary:drain_tick_at` — see the respective subsystem blocks. Every **name-keyed** per-agent keyspace is enumerated once in `services/agent_runtime_state.py` (`CLEARED_KEYSPACES` / `EXEMPT_KEYSPACES`) and cleared across the agent lifecycle, so a recycled name never inherits its predecessor's state (#1560); a parity test fails CI on an unregistered `agent:*` key.
 - **Session tab keys**: `session_lock:*`, `session_inflight:*` — see [Session Tab](#session-tab).
 - **Compatibility fix lock**: `compat_fix:{name}` (SET NX, 30s TTL) serialises the per-agent gitignore auto-fix read-modify-write (#668).
+- **Skill injection / removal lock**: `skill_inject:{name}` (SET NX + TTL, fail-open) serialises injection against removal — both mutate `~/.claude/skills/` and read-modify-write CLAUDE.md (ent#183 / ent#236). Deliberately outside `agent:*` (the `compat_fix` precedent), so the #1560 name-keyed registry doesn't apply. **Skills auto-sync leader**: `skills:sync:leader` (SET NX, TTL 3× interval, own-lease refresh, fail-open) — one worker per cycle (ent#236).
 - **Operator-queue sync keys** (#1632): `opqueue:leader` (SET NX, TTL `max(3×poll-interval, 30s)` floor — the single-syncing-worker lease, mirror `monitoring:leader`) and the create rate-limit windows `ratelimit:operator_queue_create:{agent}` + `ratelimit:operator_queue_create:_fleet` (ZSET, via `rate_limiter.check`, fail-open). Not `agent:*`-named, so the #1560 name-keyed registry doesn't apply; both fail open.
 - **Ephemeral-agent keys** (trinity-enterprise#69): `ephemeral:quota:{owner_id}` (owner-keyed atomic ghost-quota counter — deliberately not `agent:*`, so the #1560 name-keyed registry doesn't apply) and `ephemeral:discard:{name}` (SETNX+TTL discard lock). Both fail-open.