diff --git a/.claude/agents/relavium-reviewer.md b/.claude/agents/relavium-reviewer.md index d79d73a7..2f3e3f09 100644 --- a/.claude/agents/relavium-reviewer.md +++ b/.claude/agents/relavium-reviewer.md @@ -41,33 +41,45 @@ finding, and you list what you verified as clean. in Node, the Tauri WebView, and the extension host. - Signal: `grep -rn "node:\|from 'fs'\|from 'path'\|@tauri-apps\|\bwindow\.\|\bdocument\." packages/core/src` -4. **No new dependency without an ADR.** A new runtime dependency — especially in - `packages/core`/`packages/llm` or a new provider SDK — needs an +4. **`packages/mcp`'s own fences hold.** The inbound MCP client is the one package besides + `apps/cli` that imports `@relavium/core`, and the only one importing the MCP SDK. The SDK + stays behind `packages/mcp`'s adapters; a server-supplied tool description or schema is + untrusted input (bounded, and never admitted verbatim into a prompt); every network + transport keeps its SSRF pre-connect floor and its connect timeout. + - Signal: `grep -rn "@modelcontextprotocol/sdk" packages apps | grep -v 'packages/mcp/src/'` — any hit outside is a boundary break. + - Source: [ADR-0052](../../docs/decisions/0052-inbound-mcp-client-package-lifecycle-registration.md), [ADR-0053](../../docs/decisions/0053-mcp-network-transport-egress-security.md) + +5. **No new dependency without an ADR.** A new runtime dependency — especially in + `packages/core`/`packages/llm`/`packages/mcp` or a new provider SDK — needs an [ADR](../../docs/decisions/README.md). Reject a casual install that adds or re-introduces a banned framework (Vercel AI SDK, LangChain, a Python sidecar). - Signal: inspect the `package.json` / `pnpm-lock.yaml` diff. - Source: [code-review.md](../../docs/standards/code-review.md) -5. **Secrets never in plaintext, logs, or the frontend.** Keys live only in the OS keychain; +6. **Secrets never in plaintext, logs, or the frontend.** Keys live only in the OS keychain; none in an IPC payload to the WebView, a Zustand store, a React prop, localStorage, a log, - an unencrypted DB column, or an error/`node:failed`/`run:failed` event. On the **desktop** - the WebView adapter holds only a key *reference*; the raw key is read and attached inside - the Rust `llm_stream` command and never crosses into the WebView (ADR-0018). + an unencrypted DB column, or an error/`node:failed`/`run:failed` event. On the **CLI** — the + surface that actually ships today — a key is read from stdin, never argv; `history.db` and + `config.toml` stay `0600`; a tool-approval preview, a persisted run summary, and any `--json` + payload go through the secret-shaped redaction helper before they are written. On the + **desktop** (not yet built) the WebView adapter holds only a key *reference*; the raw key is + read and attached inside the Rust `llm_stream` command and never crosses into the WebView + (ADR-0018). - Signal: `grep -rni "apikey\|api_key\|secret\|process.env.*KEY" $changed_files` then trace each hit. - Source: [security-review.md](../../docs/standards/security-review.md), [ADR-0006](../../docs/decisions/0006-os-keychain-for-api-keys.md), [ADR-0018](../../docs/decisions/0018-desktop-execution-and-rust-egress.md) -6. **One canonical home for specs.** A change to a schema (workflow/agent YAML, SSE/run +7. **One canonical home for specs.** A change to a schema (workflow/agent YAML, SSE/run events, IPC, DB DDL, node types, tools, routes) updates its one [docs/reference/](../../docs/reference/) file — no pasted copy elsewhere. Run-event names are the canonical colon-namespaced form with `sequenceNumber`. - Signal: `grep -rn "node\.\(started\|completed\)\|agent\.token\|\bseqNo\b\|node:error\|run:error\|human_gate:pending" $changed_files` — legacy dotted names, `seqNo`, and the non-canonical `node:error`/`run:error`/`human_gate:pending` are all wrong (canonical: `node:failed`/`run:failed`/`human_gate:paused`, field `sequenceNumber`). -7. **Desktop is an agent-management center, not an IDE.** A change under `apps/desktop` +8. **Desktop is an agent-management center, not an IDE.** A change under `apps/desktop` adding a code editor, file browser, or terminal is out of scope; code-adjacent work belongs to the VS Code extension. - Source: ADR-0007, [architectural-principles.md](../../docs/standards/architectural-principles.md) §4 -8. **Conventional Commits.** `(): `, imperative, scope per package +9. **Conventional Commits.** `(): `, imperative, scope per package (`llm`, `core`, `shared`, `db`, `ui`, `cli`, `desktop`, `vscode`, `api`, `portal`, `docs`, `repo`), `Refs: ADR-XXXX` when implementing a decision. - Source: [commit-style.md](../../docs/standards/commit-style.md) diff --git a/.claude/skills/add-package/SKILL.md b/.claude/skills/add-package/SKILL.md index 246e3486..555f667e 100644 --- a/.claude/skills/add-package/SKILL.md +++ b/.claude/skills/add-package/SKILL.md @@ -35,7 +35,8 @@ Create a new `packages/` or `apps/` workspace that is born consistent with 3. **Create the directory and the source/test skeleton.** ```bash PKG=run-history # your slug - mkdir -p /Users/dev/Documents/Projects/Agent-Organizer/packages/$PKG/src + R=$(git rev-parse --show-toplevel) + mkdir -p "$R/packages/$PKG/src" ``` Resulting tree for a shared package: ```text diff --git a/.claude/skills/security-review/SKILL.md b/.claude/skills/security-review/SKILL.md index 03eeffbd..72a8c875 100644 --- a/.claude/skills/security-review/SKILL.md +++ b/.claude/skills/security-review/SKILL.md @@ -76,6 +76,11 @@ encryption (SQLCipher) path, or a new third-party dependency. When in doubt, run user explicitly opted into a local endpoint. An agent-config URL must never make the engine call an internal address with a real key attached. Confirm TLS verification is not disabled and every outbound call carries an `AbortSignal` + timeout. + **Confirm the change REUSES the shared guard rather than hand-rolling a second one** — a + parallel range check is how the two drift and one of them silently stops covering a range + (CLAUDE.md rule 3: never reinvent a security-critical primitive). + - Signal: `grep -rn "isPrivateOrLocalHost\|safe-egress" $changed_files` — a new URL path with + no hit is either reusing nothing or re-deriving the check locally. 5. **The `run_command` sandbox.** `run_command` spawns model-driven shell execution, so it runs sandboxed: only commands on the workflow's `allowedCommands` allowlist execute (never diff --git a/.claude/skills/start-task/SKILL.md b/.claude/skills/start-task/SKILL.md index 519e11f4..2178a599 100644 --- a/.claude/skills/start-task/SKILL.md +++ b/.claude/skills/start-task/SKILL.md @@ -28,7 +28,8 @@ Convert a roadmap phase workstream into a tightly scoped plan an engineer (human ## Workflow 1. **Locate the workstream.** Read `docs/roadmap/current.md` for what is active, then open the phase file and find the workstream by id. Copy its Tasks + Acceptance as the starting point. ```bash - grep -rn "FallbackChain\|1\.K" /Users/dev/Documents/Projects/Agent-Organizer/docs/roadmap/phases/ + R=$(git rev-parse --show-toplevel) + grep -rn "FallbackChain\|1\.K" "$R/docs/roadmap/phases/" ``` 2. **Check dependencies and build order.** Confirm the workstreams it depends on (per the phase's Mermaid graph) are done, and that you are not building a surface before its engine (architectural-principles §1, engine-first: `shared → llm → core → cli → desktop → vscode`). 3. **Define scope explicitly — in and out.** Write a short scope block: diff --git a/.claude/skills/write-architecture-doc/SKILL.md b/.claude/skills/write-architecture-doc/SKILL.md index e2d27398..0086c9c7 100644 --- a/.claude/skills/write-architecture-doc/SKILL.md +++ b/.claude/skills/write-architecture-doc/SKILL.md @@ -40,7 +40,8 @@ Produce or update a `docs/architecture/*.md` doc that answers **"how is this bui 5. **Explain the diagram in prose**, then the flow and the boundaries. Reinforce the invariants the topology depends on: engine zero-platform-imports, no vendor type across the `@relavium/llm` seam, secrets staying engine-side, the canonical `RunEvent` union. 6. **Cite specs — never restate them.** Every concrete shape is a relative link to its `reference/` home (e.g. `[run-event schema](../reference/contracts/sse-event-schema.md)`, `[node types](../reference/shared-core/node-types.md)`, `[LLM-provider seam](../reference/shared-core/llm-provider-seam.md)`). If you catch yourself pasting a schema or event body, replace it with a link. ```bash - ls /Users/dev/Documents/Projects/Agent-Organizer/docs/reference/contracts/ /Users/dev/Documents/Projects/Agent-Organizer/docs/reference/shared-core/ + R=$(git rev-parse --show-toplevel) + ls "$R/docs/reference/contracts/" "$R/docs/reference/shared-core/" ``` 7. **Link decisions and mark phases.** Cite the relevant ADRs for *why*; mark any Phase-2 (cloud) behavior explicitly with a bold marker or blockquote so it is never mistaken for shipped Phase-1 (documentation-style §9). 8. **Checkpoint — run ../standards-check/SKILL.md docs checks.** Confirm: one H1, no front-matter, diagram-first, relative links resolve, **no duplicated spec body**, ISO dates, Phase-2 marked. Then commit with ../commit-and-pr/SKILL.md (`docs(architecture): …`). diff --git a/CLAUDE.md b/CLAUDE.md index 4471a8d3..6e386480 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,6 +26,7 @@ It is a **Turborepo + pnpm monorepo**: | `packages/llm` (`@relavium/llm`) | Relavium's **own** multi-LLM abstraction: the `LLMProvider` seam + thin hand-rolled adapters over the official provider SDKs. No Vercel AI SDK, no LangChain. | | `packages/core` (`@relavium/core`) | **The engine** — YAML→DAG parse, runner, checkpoint/resume, retry. **Zero platform-specific imports.** The most important package. | | `packages/db` (`@relavium/db`) | Drizzle schema + migrations — same schema for SQLite (local) and Postgres (cloud). | +| `packages/mcp` (`@relavium/mcp`) | The inbound MCP client — the SDK-fenced package, the dependency-free JSON-Schema→Zod compiler, and the `http`/`sse`/`websocket` transports behind the SSRF floor. | | `packages/ui` (`@relavium/ui`) | Shared React components: ReactFlow node types + shadcn/ui. | | `apps/desktop` | Tauri v2 desktop app — the agent-management center (canvas, run monitoring). | | `apps/cli` | Terminal CLI (`commander.js` + `ink`). The engine's first real consumer + regression harness. | diff --git a/apps/cli/src/chat/chat-mode-host.test.ts b/apps/cli/src/chat/chat-mode-host.test.ts index 5682027c..9da8c648 100644 --- a/apps/cli/src/chat/chat-mode-host.test.ts +++ b/apps/cli/src/chat/chat-mode-host.test.ts @@ -2,7 +2,12 @@ import { mkdtemp, realpath, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { BUILTIN_TOOLS, type SessionTurnPolicy } from '@relavium/core'; +import { + BUILTIN_TOOLS, + type SessionTurnPolicy, + type ToolActionPreview, + type ToolApprovalRequest, +} from '@relavium/core'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { applyChatMode, makeChatModeEnv } from './chat-mode-host.js'; @@ -59,14 +64,41 @@ describe('makeChatModeEnv + applyChatMode', () => { expect(env.cache.isAlways('write_file')).toBe(true); // the cache lives on the env, not per-policy }); - it('isProtectedTarget resolves a preview path against the workspace and matches a protected target', () => { + /** A minimal approval request: the classifier reads `target`, never `preview` (#91). */ + const req = ( + unredactedPreview: ToolActionPreview, + preview: ToolActionPreview = {}, + ): ToolApprovalRequest => ({ + toolId: 'write_file', + action: 'fs_write', + preview, + unredactedPreview, + }); + + it('isProtectedTarget resolves the TARGET path against the workspace and matches a protected target', () => { + const { session } = fakeSession(); + const env = makeChatModeEnv({ session, tools: BUILTIN_TOOLS, workspaceDir: workspace, prompt }); + expect(env.isProtectedTarget(req({ path: '.git/config' }))).toBe(true); + expect(env.isProtectedTarget(req({ path: '.ssh/authorized_keys' }))).toBe(true); + expect(env.isProtectedTarget(req({ path: 'notes.md' }))).toBe(false); + expect(env.isProtectedTarget(req({}))).toBe(false); // no path (egress/process) ⇒ never protected + expect(env.isProtectedTarget(req({ command: 'ls' }))).toBe(false); + }); + + it('classifies from the TARGET even when the redacted preview no longer looks protected (#91)', () => { const { session } = fakeSession(); const env = makeChatModeEnv({ session, tools: BUILTIN_TOOLS, workspaceDir: workspace, prompt }); - expect(env.isProtectedTarget({ path: '.git/config' })).toBe(true); - expect(env.isProtectedTarget({ path: '.ssh/authorized_keys' })).toBe(true); - expect(env.isProtectedTarget({ path: 'notes.md' })).toBe(false); - expect(env.isProtectedTarget({})).toBe(false); // no path (egress/process preview) ⇒ never protected - expect(env.isProtectedTarget({ command: 'ls' })).toBe(false); + // Exactly the collapse a whole-string scrub produces on this path: the display copy has lost its `.ssh` + // segment entirely. Reading `preview` here would return false and silently auto-approve; reading `target` + // — which is what the classifier does — still sees the real path. + expect( + env.isProtectedTarget( + req( + { path: 'Access Token Backup/.ssh/authorized_keys' }, + { path: 'Access Token [redacted]' }, + ), + ), + ).toBe(true); }); it('auto uses isProtectedTarget: a protected write prompts, a normal write auto-approves', async () => { @@ -82,14 +114,12 @@ describe('makeChatModeEnv + applyChatMode', () => { const confirm = policies[0]?.confirm; expect(confirm).toBeDefined(); // A normal write auto-approves without prompting… - expect( - await confirm!({ toolId: 'write_file', action: 'fs_write', preview: { path: 'ok.md' } }), - ).toEqual({ + expect(await confirm!(req({ path: 'ok.md' }, { path: 'ok.md' }))).toEqual({ outcome: 'approve', }); expect(promptSpy).not.toHaveBeenCalled(); // …a protected write falls back to the prompt. - await confirm!({ toolId: 'write_file', action: 'fs_write', preview: { path: '.git/config' } }); + await confirm!(req({ path: '.git/config' }, { path: '.git/config' })); expect(promptSpy).toHaveBeenCalledTimes(1); }); }); diff --git a/apps/cli/src/chat/chat-mode-host.ts b/apps/cli/src/chat/chat-mode-host.ts index 1ddf446c..285e8a54 100644 --- a/apps/cli/src/chat/chat-mode-host.ts +++ b/apps/cli/src/chat/chat-mode-host.ts @@ -1,6 +1,6 @@ import { resolve } from 'node:path'; -import type { AgentSession, ToolActionPreview, ToolDef } from '@relavium/core'; +import type { AgentSession, ToolApprovalRequest, ToolDef } from '@relavium/core'; import { isProtectedPath } from '../engine/tool-host/fs.js'; import { @@ -34,8 +34,8 @@ export interface ChatModeEnv { /** The REPL's interactive `[y] yes / [a] always / [n] no / [c] reason / [esc] abort` prompt (accept-edits, and * auto's protected-path fallback). `[c]` opens the typed-reason capture (Step 14 — a reject carrying WHY). */ readonly prompt: ApprovalPrompt; - /** Whether an approval preview targets a protected path — `auto` then falls back to a prompt (ADR-0057). */ - readonly isProtectedTarget: (preview: ToolActionPreview) => boolean; + /** Whether an approval REQUEST targets a protected path — `auto` then falls back to a prompt (ADR-0057). */ + readonly isProtectedTarget: (request: ToolApprovalRequest) => boolean; } export interface MakeChatModeEnvOptions { @@ -55,11 +55,22 @@ export function makeChatModeEnv(opts: MakeChatModeEnvOptions): ChatModeEnv { governed: governedToolIds(opts.tools), cache: new ApprovalCache(), prompt: opts.prompt, - // fs_write is the only preview class with a path; egress/process have none, so they are never "protected" + // fs_write is the only action class with a path; egress/process have none, so they are never "protected" // (auto approves them directly). A relative path resolves against the session workspace — the SAME anchor // the fs jail uses — so the classification matches what the fs layer would enforce. - isProtectedTarget: (preview) => - preview.path !== undefined && isProtectedPath(resolve(opts.workspaceDir, preview.path)), + // + // Reads `request.unredactedPreview`, NOT `request.preview` (#91): the preview is redacted for display, + // and a scrub + // that collapses `./x/.ssh/authorized_keys` would silently reclassify a protected path as unprotected. + // Precedence: the REAL target first, the display preview only as a fallback for a hand-built request + // that carries none (a test fixture / surface stub — the engine always sets `target`). Falling back is + // safe-by-default here because `previewFor` only ever LOSES information: a scrub can turn a protected + // path into an unprotected-looking one, never the reverse, so the fallback can under-classify but the + // fs floor still hard-denies the write — while the primary path cannot under-classify at all. + isProtectedTarget: (request) => { + const path = request.unredactedPreview?.path ?? request.preview.path; + return path !== undefined && isProtectedPath(resolve(opts.workspaceDir, path)); + }, }; } diff --git a/apps/cli/src/chat/chat-mode.test.ts b/apps/cli/src/chat/chat-mode.test.ts index 6a26d3b3..0700e709 100644 --- a/apps/cli/src/chat/chat-mode.test.ts +++ b/apps/cli/src/chat/chat-mode.test.ts @@ -25,6 +25,8 @@ const req = (over: Partial = {}): ToolApprovalRequest => ({ toolId: 'write_file', action: 'fs_write', preview: { path: 'notes.md' }, + // The unredacted classification copy (#91) — the classifier's input, distinct from the display preview. + unredactedPreview: { path: 'notes.md' }, ...over, }); @@ -316,17 +318,20 @@ describe('buildTurnPolicy — the mode → { advertise, confirm } mapping', () = ); const policy = buildTurnPolicy('auto', { ...deps({ prompt }), - isProtectedTarget: (preview) => preview.path === '.git/config', + isProtectedTarget: (request) => request.unredactedPreview?.path === '.git/config', }); // A normal write auto-approves… - expect(await policy.confirm!(req({ preview: { path: 'ok.md' } }))).toEqual({ + expect(await policy.confirm!(req({ unredactedPreview: { path: 'ok.md' } }))).toEqual({ outcome: 'approve', }); expect(prompt).not.toHaveBeenCalled(); // …but a protected-path write prompts, forwarding the request + cacheable:false (the REPL greys out // "always" here) + the cancel signal. const signal = new AbortController().signal; - const protectedReq = req({ preview: { path: '.git/config' } }); + const protectedReq = req({ + preview: { path: '.git/config' }, + unredactedPreview: { path: '.git/config' }, // the classifier reads THIS copy (#91), not the display one + }); const decision = await policy.confirm!(protectedReq, signal); expect(prompt).toHaveBeenCalledTimes(1); expect(prompt).toHaveBeenCalledWith(protectedReq, false, signal); diff --git a/apps/cli/src/chat/chat-mode.ts b/apps/cli/src/chat/chat-mode.ts index cc9e4129..fe9481a4 100644 --- a/apps/cli/src/chat/chat-mode.ts +++ b/apps/cli/src/chat/chat-mode.ts @@ -159,11 +159,15 @@ export interface TurnPolicyDeps { /** The session once/always memory. */ readonly cache: ApprovalCache; /** - * Whether an approval preview targets a protected path, so `auto` falls back to a prompt rather than + * Whether the approval REQUEST targets a protected path, so `auto` falls back to a prompt rather than * auto-approving (ADR-0057). Absent ⇒ `auto` auto-approves every governed action (the fs-layer * protected-paths refusal is still the hard floor either way). + * + * Takes the whole request, not the preview: `request.preview` is a redacted DISPLAY projection (#91) whose + * scrub can turn a protected path into an unprotected-looking one, so a security classifier must read + * `request.unredactedPreview` — see {@link ToolApprovalRequest.unredactedPreview}. */ - readonly isProtectedTarget?: (preview: ToolActionPreview) => boolean; + readonly isProtectedTarget?: (request: ToolApprovalRequest) => boolean; } /** @@ -232,7 +236,7 @@ function confirmFor(mode: ChatMode, deps: TurnPolicyDeps): ConfirmActionHook { // answer is NOT cacheable: a protected-path prompt must re-ask every time, and — since the session // cache is shared across modes — an "always" here must not silently blanket-approve that tool id in a // later accept-edits turn (a cross-mode consent escalation). So the auto fallback never remembers. - if (deps.isProtectedTarget?.(request.preview) === true) { + if (deps.isProtectedTarget?.(request) === true) { const cacheable = false; const answer = await deps.prompt(request, cacheable, signal); return toDecision(answer, request.toolId, deps.cache, cacheable); diff --git a/apps/cli/src/chat/persister.test.ts b/apps/cli/src/chat/persister.test.ts index 3755dc27..95538026 100644 --- a/apps/cli/src/chat/persister.test.ts +++ b/apps/cli/src/chat/persister.test.ts @@ -10,7 +10,7 @@ import { type SessionStore, } from '@relavium/db'; import type { AgentSessionRecord, DurableContentPart, SessionMessage } from '@relavium/shared'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { ResolvedChatConfig } from '../config/resolve.js'; import { buildDefaultChatAgent } from './default-agent.js'; @@ -50,6 +50,18 @@ describe('createSessionPersister', () => { client.sqlite.close(); }); + /** + * Notes the bus's `onListenerError` sink reported during a test. Wiring the sink is not cosmetic: a persister + * write that throws travels out of a `RunEventBus` subscriber, and with NO sink the bus deliberately + * re-throws out-of-band (session-host.ts) — which vitest counts as an unhandled rejection and turns into a + * non-zero exit even while every summary line says "passed". So the tests that provoke a write failure MUST + * wire it, and asserting on what it reported also pins the #228 fail-loud path the atomicity fix leans on. + */ + let listenerNotes: string[]; + beforeEach(() => { + listenerNotes = []; + }); + async function setup(providers: ProviderResolver, initialSequenceNumber?: number) { let tick = Date.parse('2026-06-25T00:00:00.000Z'); const now = () => tick++; @@ -62,6 +74,7 @@ describe('createSessionPersister', () => { now, uuid: () => 'sess-1', providers, + onListenerError: (note: string) => listenerNotes.push(note), }); const persister = createSessionPersister({ store, @@ -76,6 +89,135 @@ describe('createSessionPersister', () => { return { built, persister, store }; } + /** {@link setup} against a caller-supplied store — for a scenario that must not share a session row. */ + async function setupWith(target: SessionStore, providers: ProviderResolver) { + let tick = Date.parse('2026-06-25T00:00:00.000Z'); + const now = () => tick++; + let msgId = 0; + const built = await buildChatSession({ + chat: EMPTY_CHAT, + agentRef: undefined, + cwd: '/workspace', + projectConfigDir: undefined, + now, + uuid: () => 'sess-1', + providers, + onListenerError: (note: string) => listenerNotes.push(note), + }); + const persister = createSessionPersister({ + store: target, + handle: built.handle, + sessionId: built.sessionId, + agent: built.agent, + context: built.context, + now, + uuid: () => `msg-${msgId++}`, + }); + return { built, persister }; + } + + /* --- #228: a failed turn write must leave NOTHING, not half a turn --- */ + + it('writes a turn ATOMICALLY — a mid-turn failure persists neither message', async () => { + const { built, persister } = await setup(scriptedResolver([textTurn('hi'), textTurn('again')])); + built.session.start(); + persister.start(); + // Fail the FIRST turn's write outright. Before the atomic write this was three auto-committed + // statements, so the `user` row landed and the `assistant` row did not. + const boom = Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + const writeTurn = vi.spyOn(store, 'writeTurn').mockImplementationOnce(() => { + throw boom; + }); + persister.beginUserTurn('first'); + await built.session.sendMessage('first'); + expect(store.loadMessages('sess-1')).toHaveLength(0); // all of it, or none of it + // …and the user was TOLD. A silently dropped turn is the other half of #228. + expect(listenerNotes.join('\n')).toMatch(/database is locked/); + + // The session survives (that is the point of #228's other layers) and the NEXT turn must be clean — + // this is where the old behaviour buried an orphan `user` row mid-transcript, which resume does not + // roll back and which replays as two consecutive `user` messages a provider rejects. + writeTurn.mockRestore(); + persister.beginUserTurn('second'); + await built.session.sendMessage('second'); + const roles = store.loadMessages('sess-1').map((m) => m.role); + expect(roles).toEqual(['user', 'assistant']); + // No gap: the surviving rows are contiguous from 0, because the failed turn advanced nothing. + expect(store.loadMessages('sess-1').map((m) => m.sequenceNumber)).toEqual([0, 1]); + }); + + it('a FAILED turn contributes NO tokens to the session totals', async () => { + // The bug this pins was real and mine: the totals were assigned BEFORE `writeTurn`, so a failed write left + // them advanced and the next turn's row claimed two turns' tokens for one visible exchange. No magic + // numbers — a control run of ONE clean turn establishes what a turn is worth, and failed-then-clean must + // match it exactly. `stop()`'s usage is fixed, so a leak shows up as double. + const control = await setup(scriptedResolver([textTurn('hi')])); + control.built.session.start(); + control.persister.start(); + control.persister.beginUserTurn('only'); + await control.built.session.sendMessage('only'); + const oneTurn = store.loadSession('sess-1'); + + // A separate DB so the two scenarios cannot share a session row. + const other = createClient(':memory:'); + runMigrations(other.db); + const store2 = createSessionStore(other.db); + try { + const { built, persister } = await setupWith( + store2, + scriptedResolver([textTurn('hi'), textTurn('again')]), + ); + built.session.start(); + persister.start(); + vi.spyOn(store2, 'writeTurn').mockImplementationOnce(() => { + throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + }); + persister.beginUserTurn('first'); + await built.session.sendMessage('first'); + vi.restoreAllMocks(); + persister.beginUserTurn('second'); + await built.session.sendMessage('second'); + + const after = store2.loadSession('sess-1'); + expect(after?.totalInputTokens).toBe(oneTurn?.totalInputTokens); + expect(after?.totalOutputTokens).toBe(oneTurn?.totalOutputTokens); + } finally { + other.sqlite.close(); + } + }); + + it('leaves the in-memory sequence untouched when a turn write fails (no phantom seq)', async () => { + // Three scripted turns: the failed one, the landed one, and the summariser's own turn for the /compact + // that this test uses to observe `realMessageSeqs`. + const { built, persister } = await setup( + scriptedResolver([textTurn('hi'), textTurn('again'), textTurn('the summary')]), + ); + built.session.start(); + persister.start(); + vi.spyOn(store, 'writeTurn').mockImplementationOnce(() => { + throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + }); + persister.beginUserTurn('first'); + await built.session.sendMessage('first'); + vi.restoreAllMocks(); + persister.beginUserTurn('second'); + await built.session.sendMessage('second'); + // The turn totals must also not have advanced for the turn that never landed. + const session = store.loadSession('sess-1'); + expect(session?.title).toBe('second'); // the FAILED turn never got to title the session either + + // The load-bearing half, and the one nothing else observes: `realMessageSeqs`. It is the ADR-0062 + // boundary seed, so a phantom entry from the failed turn shifts + // `realMessageSeqs[length - keptMessageCount - 1]` on EVERY later /compact and /trim — a silent + // kept-message loss, exactly the "step-3 review data-loss trap" this file's own comment warns about. + // Driving a real compaction is the only way to see it: after ONE landed turn (rows 0,1), keeping the + // last exchange must drop nothing and therefore write NO marker. A leaked phantom seq makes the array + // look two entries longer and a spurious marker appears. + const result = await built.session.compact('manual'); + expect(result.kind).toBe('compacted'); + expect(store.loadMessages('sess-1').filter((m) => m.role === 'system')).toHaveLength(0); + }); + it('persists the session row eagerly on start (auto-persisted from the moment it starts)', async () => { const { built, persister } = await setup(scriptedResolver([textTurn('hi')])); persister.start(); diff --git a/apps/cli/src/chat/persister.ts b/apps/cli/src/chat/persister.ts index 5f442303..d04eb36d 100644 --- a/apps/cli/src/chat/persister.ts +++ b/apps/cli/src/chat/persister.ts @@ -5,7 +5,12 @@ import { type SessionStreamHandleEvent, } from '@relavium/core'; import { createModelCatalogStore, type Db, type SessionStore } from '@relavium/db'; -import type { AgentSessionRecord, SessionContext, SessionStatus } from '@relavium/shared'; +import type { + AgentSessionRecord, + SessionContext, + SessionMessage, + SessionStatus, +} from '@relavium/shared'; import { deriveSessionTitle } from './session-title.js'; @@ -112,23 +117,27 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi // prior marker is interleaved — the step-1-review trap). Seeded from the durable transcript on resume. const realMessageSeqs: number[] = []; - /** Append a REAL transcript row + record its sequence for the boundary mapping. `modelCatalogId` (assistant rows - * only) is the already-resolved `model_catalog.id` FK target attributing the row to the model that produced it - * (ADR-0059) — omitted (a NULL column) when unknown/uncataloged; a user row never carries one. */ - const appendText = (role: 'user' | 'assistant', text: string, modelCatalogId?: string): void => { - const seq = sequenceNumber++; - realMessageSeqs.push(seq); - deps.store.appendMessage({ - id: deps.uuid(), - sessionId: deps.sessionId, - sequenceNumber: seq, - role, - content: [{ type: 'text', text }], - // Conditional spread ⇒ no explicit `undefined` under exactOptionalPropertyTypes; a user row never carries one. - ...(modelCatalogId === undefined ? {} : { modelId: modelCatalogId }), - timestamp: iso(), - }); - }; + /** BUILD a REAL transcript row at `seq` — pure, no write and no in-memory mutation. `modelCatalogId` + * (assistant rows only) is the already-resolved `model_catalog.id` FK target attributing the row to the model + * that produced it (ADR-0059) — omitted (a NULL column) when unknown/uncataloged; a user row never carries one. + * + * Staging is what makes the turn atomic (#228): the sequence counter and `realMessageSeqs` advance ONLY after + * the whole turn has been durably written, so a failed write leaves no phantom sequence behind. */ + const stageText = ( + seq: number, + role: 'user' | 'assistant', + text: string, + modelCatalogId?: string, + ): SessionMessage => ({ + id: deps.uuid(), + sessionId: deps.sessionId, + sequenceNumber: seq, + role, + content: [{ type: 'text', text }], + // Conditional spread ⇒ no explicit `undefined` under exactOptionalPropertyTypes; a user row never carries one. + ...(modelCatalogId === undefined ? {} : { modelId: modelCatalogId }), + timestamp: iso(), + }); /** * Append an append-only compaction/trim boundary MARKER row (ADR-0062): `role:'system'`, the summary text @@ -136,43 +145,82 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi * ROLE-FILTERED real-message sequences (never the raw row count). Returns `false` when there is nothing to * drop (fewer real rows than kept — no marker written). The marker's own seq is NOT a real-message seq. */ - const appendMarker = (summary: string, keptMessageCount: number): boolean => { - if (keptMessageCount >= realMessageSeqs.length) return false; // nothing older to supersede + const stageMarker = (summary: string, keptMessageCount: number): SessionMessage | undefined => { + if (keptMessageCount >= realMessageSeqs.length) return undefined; // nothing older to supersede const droppedThroughSequence = realMessageSeqs[realMessageSeqs.length - keptMessageCount - 1]; - if (droppedThroughSequence === undefined) return false; - deps.store.appendMessage({ + if (droppedThroughSequence === undefined) return undefined; + return { id: deps.uuid(), sessionId: deps.sessionId, - sequenceNumber: sequenceNumber++, + sequenceNumber, role: 'system', content: summary.length > 0 ? [{ type: 'text', text: summary }] : [], compaction: { droppedThroughSequence }, timestamp: iso(), - }); - return true; + }; }; // Derived from the FIRST user message so the Home list shows a readable label (2.5.B). Set once; a resumed // session hydrates the existing title in start() so a later message never overwrites it. let title: string | undefined; - const record = (status: SessionStatus): AgentSessionRecord => ({ + /** Build the session row. `staged` supplies the values a not-yet-committed turn is ABOUT to adopt, so the + * row written inside `writeTurn`'s transaction matches exactly what this process commits on success. */ + const record = ( + status: SessionStatus, + staged: { title?: string | undefined; input?: number; output?: number } = {}, + ): AgentSessionRecord => ({ id: deps.sessionId, agentSlug: deps.agent.id, agentSnapshot: deps.agent, context: deps.context, status, - totalInputTokens, - totalOutputTokens, + totalInputTokens: staged.input ?? totalInputTokens, + totalOutputTokens: staged.output ?? totalOutputTokens, totalCostMicrocents, createdAt, updatedAt: iso(), - ...(title === undefined ? {} : { title }), + ...((staged.title ?? title) === undefined ? {} : { title: staged.title ?? title }), // The session's coarse primary model (ADR-0059) — the bound model's `model_catalog.id` (the FK target), so a // reseat's new persister records the switched model. Omitted (NULL) when the model is not cataloged. Per-turn // attribution rides each assistant `SessionMessage.modelId`; this is the row-level label. ...(sessionModelCatalogId === undefined ? {} : { modelId: sessionModelCatalogId }), }); + /** + * Persist one COMPLETED exchange: stage its rows, write them and the session row in one transaction, then — + * and only then — adopt the staged state in memory. Extracted from `onEvent`'s `session:turn_completed` arm + * so that arm reads as one decision (persist or just flush) rather than carrying the whole staging protocol + * inline; the ordering inside it is load-bearing and is documented where it happens. + */ + const commitTurn = (userText: string, tokensUsed: { input: number; output: number }): void => { + const nextTitle = title ?? deriveSessionTitle(userText); + // STAGE the whole turn against provisional sequence numbers, then write it in ONE transaction (#228). + // Nothing in-memory moves until that write succeeds, so a failure leaves neither a half-written turn + // in the transcript nor a phantom sequence in this process. + const staged: SessionMessage[] = [stageText(sequenceNumber, 'user', userText)]; + // The assistant row carries the (resolved) catalog id of the model that produced it (ADR-0059): + // `turnModelCatalogId` from this turn's last `cost:updated`, or `undefined` (NULL) when uncataloged. + if (assistantText.length > 0) { + staged.push(stageText(sequenceNumber + 1, 'assistant', assistantText, turnModelCatalogId)); + } + const nextInput = totalInputTokens + tokensUsed.input; + const nextOutput = totalOutputTokens + tokensUsed.output; + deps.store.writeTurn({ + messages: staged.map((message) => ({ message })), + session: record('active', { title: nextTitle, input: nextInput, output: nextOutput }), + }); + // Committed. EVERY in-memory mutation for this turn happens here, AFTER the durable write returned — + // never before it. The token totals in particular: the columns are deliberately not accumulated for + // a turn whose messages did not persist (the same rule the comment above states for an errored or + // aborted turn), so advancing them ahead of the write made a failed turn's tokens leak into the next + // turn's row — the session then claimed two turns' tokens for one visible exchange. + title = nextTitle; + sequenceNumber += staged.length; + realMessageSeqs.push(...staged.map((m) => m.sequenceNumber)); + totalInputTokens = nextInput; + totalOutputTokens = nextOutput; + }; + const onEvent = (event: SessionStreamHandleEvent): void => { switch (event.type) { case 'agent:token': @@ -235,15 +283,12 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi // Derive the title HERE (not in beginUserTurn) — from the FIRST user message of a COMPLETED exchange of // a titleless session, so an aborted/errored earlier turn never labels the row. A blank message yields // undefined, so the next non-blank completed message becomes the title; a resumed session keeps its own. - title ??= deriveSessionTitle(pendingUserText); - appendText('user', pendingUserText); - // The assistant row carries the (resolved) catalog id of the model that produced it (ADR-0059): - // `turnModelCatalogId` from this turn's last `cost:updated`, or `undefined` (NULL) when uncataloged. - if (assistantText.length > 0) appendText('assistant', assistantText, turnModelCatalogId); - totalInputTokens += event.tokensUsed.input; - totalOutputTokens += event.tokensUsed.output; + commitTurn(pendingUserText, event.tokensUsed); + } else { + // An errored or aborted turn persists no messages and accumulates no TOKENS, but the row is still + // flushed: `updatedAt` moves, and the session COST (already folded per `cost:updated`) is real. + deps.store.updateSession(record('active')); } - deps.store.updateSession(record('active')); pendingUserText = undefined; assistantText = ''; turnModelCatalogId = undefined; @@ -252,15 +297,29 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi // ADR-0062: write the append-only boundary marker (summary + role-filtered droppedThroughSequence) and // add the summariser's REAL token usage to the totals (the cost microcents already flowed via // cost:updated → totalCostMicrocents; flush the row to persist both). Nothing durable is deleted. - appendMarker(event.summary, event.keptMessageCount); - totalInputTokens += event.tokensUsed.input; - totalOutputTokens += event.tokensUsed.output; - deps.store.updateSession(record('active')); + { + const marker = stageMarker(event.summary, event.keptMessageCount); + const nextInput = totalInputTokens + event.tokensUsed.input; + const nextOutput = totalOutputTokens + event.tokensUsed.output; + deps.store.writeTurn({ + messages: marker === undefined ? [] : [{ message: marker }], + session: record('active', { input: nextInput, output: nextOutput }), + }); + if (marker !== undefined) sequenceNumber += 1; // the marker's seq is NOT a real-message seq + totalInputTokens = nextInput; + totalOutputTokens = nextOutput; + } return; case 'session:trimmed': // A deterministic /trim — a summary-less boundary marker, no cost. Flush the row (updatedAt) after. - appendMarker('', event.keptMessageCount); - deps.store.updateSession(record('active')); + { + const marker = stageMarker('', event.keptMessageCount); + deps.store.writeTurn({ + messages: marker === undefined ? [] : [{ message: marker }], + session: record('active'), + }); + if (marker !== undefined) sequenceNumber += 1; + } return; case 'session:cancelled': // The session's sole terminal — mark it ended (still resumable from the persisted transcript), then diff --git a/apps/cli/src/chat/session-host.test.ts b/apps/cli/src/chat/session-host.test.ts index d6649666..c00ddf5a 100644 --- a/apps/cli/src/chat/session-host.test.ts +++ b/apps/cli/src/chat/session-host.test.ts @@ -283,6 +283,76 @@ describe('buildChatSession', () => { }); }); +/* --- #228: a throwing passive subscriber must be REPORTED, never left to kill the process --- */ + +describe('buildChatSession — the onListenerError sink (#228)', () => { + /** + * The failure chain this pins: the chat persister writes `history.db` from inside a `RunEventBus` + * subscriber. `deliver()` isolates the throw so the turn survives — but with no sink the bus re-throws it + * out-of-band, and Node kills the process on the resulting unhandled rejection, asynchronously, AFTER the + * user already saw the reply. A billed turn is lost and the session dies with it. + */ + it('routes a throwing subscriber to onListenerError instead of an out-of-band rejection', async () => { + const notes: string[] = []; + const built = await build({ onListenerError: (note: string) => notes.push(note) }); + built.handle.subscribe(() => { + throw new Error('database is locked'); + }); + + // A real turn, so the throw happens on the genuine delivery path rather than a hand-rolled emit. + built.session.start(); + await built.session.sendMessage('hi'); + + expect(notes.length).toBeGreaterThan(0); + // The note names WHICH event failed and why. Deliberately NEUTRAL about the cause: this sink is bus-wide + // (the renderer, the Home store and the NDJSON printer subscribe here too), so it must not blame the + // database for what may be a render fault. + expect(notes[0]).toMatch( + /a background handler failed while processing the \S+ event: database is locked/, + ); + }); + + it('does NOT swallow when no sink is supplied — the failure still surfaces out-of-band', async () => { + // The alternative implementation (an absent sink degrading to a no-op) would be a silent catch: the turn + // would vanish from history with nothing anywhere saying so. `index.ts`'s process-level net is the floor + // beneath this path, which is what makes surfacing survivable rather than fatal. + const built = await build(); // no onListenerError + const rejections: unknown[] = []; + const onRejection = (reason: unknown): void => { + rejections.push(reason); + }; + process.on('unhandledRejection', onRejection); + try { + built.handle.subscribe(() => { + throw new Error('database is locked'); + }); + built.session.start(); + await built.session.sendMessage('hi'); + // The out-of-band rethrow is a microtask; give it a macrotask turn to become an unhandled rejection. + await new Promise((resolve) => setTimeout(resolve, 10)); + } finally { + process.off('unhandledRejection', onRejection); + } + expect(rejections.some((r) => r instanceof Error && r.message === 'database is locked')).toBe( + true, + ); + }); + + it('keeps the turn and its SIBLING subscribers intact — a failing observer never breaks the producer', async () => { + const built = await build({ onListenerError: () => undefined }); + const seen: string[] = []; + built.handle.subscribe(() => { + throw new Error('database is locked'); + }); + // Registered AFTER the thrower, so it only ever runs if `deliver()` really isolates each subscriber — + // persistence is an observer, never a gate on the turn or on the surface's own rendering. + built.handle.subscribe((event) => seen.push(event.type)); + built.session.start(); + await built.session.sendMessage('hi'); + expect(seen).toContain('session:turn_completed'); + }); +}); + describe('buildChatSession + MCP host wiring (2.R)', () => { /** Write a throwaway `.agent.yaml` declaring one inline stdio MCP server (optional extra server lines). */ function writeMcpAgent(extraServerLines: readonly string[] = []): string { @@ -905,9 +975,11 @@ describe('buildGovernorWiring', () => { (warning) => warnings.push(warning), ); wiring?.updateCost(999_999); - await expect(wiring?.preEgress(OVER_CAP)).resolves.toBeUndefined(); // warn never blocks - await expect(wiring?.preEgress(OVER_CAP)).resolves.toBeUndefined(); // still non-blocking the 2nd time - // The governor emits the warning ONCE (#warningEmitted) — the 2nd over-cap call must not re-notify. + const first = await wiring?.preEgress(OVER_CAP); // warn never blocks + const second = await wiring?.preEgress(OVER_CAP); // still non-blocking the 2nd time + first?.release(); + second?.release(); + // No realized cost landed between these checks, so the standing condition remains deduped. expect(warnings).toHaveLength(1); expect(warnings[0]?.limitMicrocents).toBe(1); }); @@ -917,7 +989,11 @@ describe('buildGovernorWiring', () => { // never a rejection that would surface as an `internal` turn error. const wiring = buildGovernorWiring({ ...EMPTY_CHAT, maxCostMicrocents: 1, onExceed: 'warn' }); wiring?.updateCost(999_999); - await expect(wiring?.preEgress(OVER_CAP)).resolves.toBeUndefined(); + // Assert the OUTCOME, not merely the absence of a throw: `warn` must still ADMIT the egress. Without this + // the test would pass just as happily if admission stopped being granted altogether. + const admission = await wiring?.preEgress(OVER_CAP); + expect(admission).toBeDefined(); + expect(() => admission?.release()).not.toThrow(); }); it('on_exceed:warn — a throwing onWarning surface never rejects preEgress (warn stays non-blocking)', async () => { @@ -928,8 +1004,11 @@ describe('buildGovernorWiring', () => { }, ); wiring?.updateCost(999_999); - // A misbehaving warn surface must NOT surface as an `internal` turn error — the throw is swallowed. - await expect(wiring?.preEgress(OVER_CAP)).resolves.toBeUndefined(); + // A misbehaving warn surface must NOT surface as an `internal` turn error — the throw is swallowed, AND + // the egress is still admitted. Asserting both is what makes this fail for the right reason. + const admission = await wiring?.preEgress(OVER_CAP); + expect(admission).toBeDefined(); + expect(() => admission?.release()).not.toThrow(); }); }); diff --git a/apps/cli/src/chat/session-host.ts b/apps/cli/src/chat/session-host.ts index 0b52bf88..c047d687 100644 --- a/apps/cli/src/chat/session-host.ts +++ b/apps/cli/src/chat/session-host.ts @@ -128,6 +128,18 @@ export interface BuildChatSessionOptions { * Absent ⇒ a no-op, and the tier is still withheld (never guessed at). */ readonly onEffortWithheld?: (note: string) => void; + /** + * Sink for a **passive subscriber** that threw while handling a session event (#228) — in practice the chat + * persister failing a durable `history.db` write after its bounded retry was exhausted. Same channel shape as + * {@link BuildChatSessionOptions.onBudgetWarning}: the surface puts the sentence in the transcript's notice + * channel so the user learns the turn was not saved, while the session itself continues (the reply is already + * on screen and the in-memory transcript is intact). + * + * Absent ⇒ **not** a no-op, unlike the sinks above. A dropped durable write must never be silent + * (`docs/standards/error-handling.md`), so the runtime still surfaces it out-of-band, where the process-level + * `unhandledRejection` net in `index.ts` reports it and keeps the process alive. + */ + readonly onListenerError?: (note: string) => void; /** * The ADR-0065 §2 user-pricing overlay (2.5.G S10) — a `ReadonlyMap` the command projects * from the `model_catalog` `source='user'` rows (via `buildUserPricing`). It flows into BOTH the pre-egress @@ -196,6 +208,7 @@ type SessionRuntimeOptions = Pick< | 'onBudgetWarning' | 'onUnpriced' | 'onEffortWithheld' + | 'onListenerError' | 'resolvePrice' >; @@ -217,7 +230,28 @@ function buildSessionRuntime( mcp: McpClient | undefined, context: SessionContext, ): { bus: RunEventBus; deps: SessionDeps; emit: SessionEventSink; host: ToolHost } { - const bus = new RunEventBus({ now: () => new Date(opts.now()).toISOString() }); + // #228: a passive subscriber that throws — the chat persister failing a durable `history.db` write — is + // isolated by `deliver()` either way, so the turn itself is never broken. What was missing is where the + // failure GOES: with no sink the bus re-throws it out-of-band, and Node kills the process on the resulting + // unhandled rejection, asynchronously, after the reply was already on screen. Wiring the sink turns that into + // a reported, survivable event. When the surface supplied no `onListenerError` the sink deliberately re-throws + // rather than swallowing (no silent catch) — the process-level net in `index.ts` is the floor beneath it. + const bus = new RunEventBus({ + now: () => new Date(opts.now()).toISOString(), + onListenerError: (error, event) => { + const report = opts.onListenerError; + if (report === undefined) { + // Rethrowing from the sink is not a shrug — `RunEventBus.#reportListenerError` catches a throwing sink + // and routes it to `#surfaceOutOfBand`, i.e. exactly the no-sink path, which is the fallback we want. + // Stated explicitly because "throw to fall through" reads as a bug otherwise. + throw error; + } + // Bus-WIDE, not persister-specific: the renderer, the Home store and the NDJSON printer all subscribe + // here too, so a render fault must not be reported as a database problem. + const reason = error instanceof Error ? error.message : String(error); + report(`a background handler failed while processing the ${event.type} event: ${reason}`); + }, + }); const providers = opts.providers ?? createProviderResolver(); const tools = mcp === undefined ? BUILTIN_TOOLS : [...BUILTIN_TOOLS, ...mcp.toolDefs]; // 2.5.E (ADR-0057): the shared factory wires the FULL-CAPABILITY chat host (fs read+WRITE, process, egress, @@ -474,6 +508,12 @@ export interface BuildResumedChatSessionOptions { readonly onEffortWithheld?: (note: string) => void; /** Unpriced-model notice (ADR-0071 §K7) — see {@link BuildChatSessionOptions.onUnpriced}. */ readonly onUnpriced?: (note: string) => void; + /** + * Sink for a throwing passive subscriber (#228) — see {@link BuildChatSessionOptions.onListenerError}. + * A RESUMED session is the HIGHEST-contention surface for it: two `chat-resume` processes on one session + * write the same `history.db` rows concurrently (#227). + */ + readonly onListenerError?: (note: string) => void; /** The loaded session record (its frozen `agentSnapshot` + `context` rebind the session). */ readonly record: AgentSessionRecord; /** The session's persisted transcript, in any order ({@link reconstructSessionState} sorts it). */ diff --git a/apps/cli/src/commands/agent-run.ts b/apps/cli/src/commands/agent-run.ts index 8e2b50ba..8bf8bfee 100644 --- a/apps/cli/src/commands/agent-run.ts +++ b/apps/cli/src/commands/agent-run.ts @@ -87,6 +87,8 @@ export async function agentRunCommand( // completely — the turn runs, the authored knob does nothing, and the bill arrives at the provider's default. // STDERR, never stdout: `--json` owns stdout, and a warning line mid-stream is a parse error downstream. onEffortWithheld: onceEffortNotice((note) => deps.io.writeErr(`warning: ${note}\n`)), + // stderr, never stdout: `--json` owns stdout and a notice must not corrupt the NDJSON stream. + onListenerError: (note) => deps.io.writeErr(`warning: ${note}\n`), onUnpriced: (note) => deps.io.writeErr(`warning: ${note}\n`), agentRef: args.agent, cwd: deps.global.cwd, diff --git a/apps/cli/src/commands/chat-alt-hoist.test.ts b/apps/cli/src/commands/chat-alt-hoist.test.ts index 38a66834..748f312c 100644 --- a/apps/cli/src/commands/chat-alt-hoist.test.ts +++ b/apps/cli/src/commands/chat-alt-hoist.test.ts @@ -150,6 +150,25 @@ describe('withHoistedAltScreen (2.6.F Step 4b-3, ADR-0068 §c)', () => { expect(h.exit).toHaveBeenCalledWith(143); // 128 + 15 (SIGTERM) }); + it('disarms a reversible job-control handoff BEFORE TERM performs the final alt restore', async () => { + const h = harness(); + await withHoistedAltScreen( + { + ...opts(h, true), + beforeTerminalRestore: () => h.events.push('job-control:dispose'), + }, + () => { + h.fireSignal(15); + return Promise.resolve({}); + }, + ); + + const dispose = h.events.indexOf('job-control:dispose'); + const exit = h.events.indexOf(`write:${EXIT_SEQ}`); + expect(dispose).toBeGreaterThanOrEqual(0); + expect(exit).toBeGreaterThan(dispose); + }); + it('a SIGHUP is 128+1 = 129, a SIGQUIT is 128+3 = 131 (the external kills that skip Node’s exit event)', async () => { const hup = harness(); await withHoistedAltScreen(opts(hup, true), () => { diff --git a/apps/cli/src/commands/chat.test.ts b/apps/cli/src/commands/chat.test.ts index 8bc4ec71..26f5e195 100644 --- a/apps/cli/src/commands/chat.test.ts +++ b/apps/cli/src/commands/chat.test.ts @@ -15,7 +15,7 @@ import { type DbClient, } from '@relavium/db'; import { startMcpClient as realStartMcpClient, type McpConnection } from '@relavium/mcp'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { DoctorProbes } from '../chat/doctor.js'; import { buildChatSession, buildResumedChatSession } from '../chat/session-host.js'; @@ -153,13 +153,23 @@ const INERT_HOIST = { setRawMode: (): void => undefined, exit: (): void => undefined, }, + jobControlLifecycle: { + supported: false, + onSuspend: (): (() => void) => () => undefined, + onContinue: (): (() => void) => () => undefined, + suspendSelf: (): void => undefined, + }, } as const; /** Like {@link INERT_HOIST} but RECORDS the control-sequence writes (still never touching the real fd) so a test can * assert WHETHER the hoist entered the alt buffer — pinning `runReplLoop`'s `altActive` derivation, the single wiring * the whole flicker fix depends on, which the fully-inert `INERT_HOIST` cannot observe (Step-4b-3 Sonnet review). */ function recordingHoist(): { - hoist: { writeControl: (s: string) => void; lifecycle: typeof INERT_HOIST.lifecycle }; + hoist: { + writeControl: (s: string) => void; + lifecycle: typeof INERT_HOIST.lifecycle; + jobControlLifecycle: typeof INERT_HOIST.jobControlLifecycle; + }; writes: string[]; } { const writes: string[] = []; @@ -170,6 +180,7 @@ function recordingHoist(): { writes.push(s); }, lifecycle: INERT_HOIST.lifecycle, + jobControlLifecycle: INERT_HOIST.jobControlLifecycle, }, }; } @@ -237,6 +248,59 @@ describe('chatCommand', () => { expect(full?.messages[1]?.content[0]).toEqual({ type: 'text', text: 'hi there' }); }); + it('routes chat SIGTSTP/SIGCONT through reversible job control, never the termination lifecycle (G0)', async () => { + const { d } = deps([], [textTurn('unused')]); + const writes: string[] = []; + const exit = vi.fn(); + const setRawMode = vi.fn(); + const suspendSelf = vi.fn(); + let onSuspendSignal: () => void = () => undefined; + let onContinueSignal: () => void = () => undefined; + const drive: ChatDriver = async (ctx) => { + ctx.startSession(); + onContinueSignal(); // stray SIGCONT: no terminal teardown and no redraw before a stop + onSuspendSignal(); + await ctx.processLine('/exit'); + return { kind: ctx.stopReason() }; + }; + + await chatCommand( + { agent: undefined }, + { + ...d, + io: { ...d.io, stdoutIsTty: true }, + writeControl: (sequence) => writes.push(sequence), + lifecycle: { + onProcessExit: () => () => undefined, + onTerminationSignal: () => () => undefined, + onInterrupt: () => () => undefined, + setRawMode, + exit, + }, + jobControlLifecycle: { + supported: true, + onSuspend: (listener) => { + onSuspendSignal = listener; + return () => undefined; + }, + onContinue: (listener) => { + onContinueSignal = listener; + return () => undefined; + }, + suspendSelf, + }, + drive, + }, + ); + + expect(suspendSelf).toHaveBeenCalledTimes(1); + expect(exit).not.toHaveBeenCalled(); + expect(setRawMode).not.toHaveBeenCalledWith(false); + // Hoisted 1049 is TEMPORARILY released/reclaimed around the stop, then finally restored at normal REPL exit. + expect(writes.filter((sequence) => sequence.includes(EXIT_ALT_SCREEN))).toHaveLength(2); + expect(writes.filter((sequence) => sequence.includes(ENTER_ALT_SCREEN))).toHaveLength(2); + }); + it('streams a multi-turn conversation that includes a tool call, persisting each turn', async () => { const { d, store, sessionId } = deps( ['use a tool', '/exit'], diff --git a/apps/cli/src/commands/chat.ts b/apps/cli/src/commands/chat.ts index c330bd7c..65724649 100644 --- a/apps/cli/src/commands/chat.ts +++ b/apps/cli/src/commands/chat.ts @@ -91,7 +91,14 @@ import { type TranscriptBound, } from '../render/tui/session-view-model.js'; import { copyToClipboard, type ClipboardOutcome } from '../render/clipboard.js'; -import { createSuspendPort, type SuspendPort } from '../render/suspend.js'; +import { + createSuspendPort, + defaultJobControlLifecycle, + suspendFullScreen, + wireJobControl, + type JobControlLifecycle, + type SuspendPort, +} from '../render/suspend.js'; import { DISABLE_BRACKETED_PASTE } from '../render/tui/home-input.js'; import { errorRecoveryHint, @@ -291,6 +298,8 @@ export interface ChatDriveContext { * a plain / `--json` driver — the hatches then surface an honest "needs an interactive terminal" notice. */ readonly suspendPort?: SuspendPort | undefined; + /** Raw-mode Ctrl-Z / external SIGTSTP converge here. Absent for a plain driver, where no Ink terminal is owned. */ + readonly onSuspend?: (() => void) | undefined; /** * Write the mouse selection to the system clipboard over OSC 52 (2.6.F Step 6). Only an INK driver has a terminal * to write to; a plain / `--json` driver never mounts the viewport, so nothing can be selected there. @@ -336,6 +345,8 @@ export interface ChatCommandDeps { * `writeControl` + `lifecycle` so the hoist never touches the REAL terminal / process signals; default = real. */ readonly writeControl?: (sequence: string) => void; readonly lifecycle?: ReplLifecycle; + /** Injectable POSIX SIGTSTP/SIGCONT seam; kept separate from irreversible termination handling. */ + readonly jobControlLifecycle?: JobControlLifecycle; /** Wall-clock (ms) + id sources (injectable for tests). */ readonly now?: () => number; readonly uuid?: () => string; @@ -367,6 +378,8 @@ export interface ChatResumeCommandDeps { * never touches the real terminal / process signals (default = real). */ readonly writeControl?: (sequence: string) => void; readonly lifecycle?: ReplLifecycle; + /** Injectable POSIX SIGTSTP/SIGCONT seam; kept separate from irreversible termination handling. */ + readonly jobControlLifecycle?: JobControlLifecycle; /** Wall-clock (ms) + id sources (injectable for tests). */ readonly now?: () => number; readonly uuid?: () => string; @@ -383,6 +396,10 @@ interface ChatReplDeps { /** The process lifecycle seam for the alt-buffer exit-safety net (Step 4b-3) — `process.on('exit')`, SIGTERM/SIGHUP, * raw-mode, and exit. Default {@link defaultReplLifecycle}; a test injects fakes to drive the exit paths. */ readonly lifecycle?: ReplLifecycle; + /** Injectable POSIX SIGTSTP/SIGCONT seam; active only for an interactive Ink chat. */ + readonly jobControlLifecycle?: JobControlLifecycle; + /** The current loop's raw Ctrl-Z request. Constructed beside the hoisted terminal state, not by a plain driver. */ + readonly onSuspend?: (() => void) | undefined; /** * `[preferences].copy_on_select`, ALREADY resolved against the mouse decision (`resolveCopyOnSelect`). `false` (or * absent, on a non-interactive path) means the ink tree gets no `clipboard` prop: a released drag still highlights, @@ -530,6 +547,7 @@ export async function chatCommand(args: ChatCommandArgs, deps: ChatCommandDeps): // turn runs, the field is gone, and the user is billed at the provider's default tier with nothing to explain // why the knob they set did nothing. onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(deps.io, note)), + onListenerError: (note: string) => emitLiveNotice(deps.io, note), onUnpriced: (note) => emitLiveNotice(deps.io, note), }); // The session now OWNS the live MCP connections (built.closeMcp). `runReplLoop`'s finally is the steady-state @@ -694,6 +712,7 @@ export async function chatResumeCommand( // turn runs, the field is gone, and the user is billed at the provider's default tier with nothing to explain // why the knob they set did nothing. onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(deps.io, note)), + onListenerError: (note: string) => emitLiveNotice(deps.io, note), onUnpriced: (note) => emitLiveNotice(deps.io, note), }); closeMcp = resumed.closeMcp; @@ -1429,6 +1448,7 @@ interface FreshChatWiringDeps { /** Withheld-tier sink (ADR-0071 §6) — threaded exactly like {@link FreshChatWiringDeps.onBudgetWarning}, because a * `/clear` rebuild binds a NEW session and a session with no sink withholds a tier in silence. */ readonly onEffortWithheld: NonNullable; + readonly onListenerError: NonNullable; readonly onUnpriced: NonNullable; /** `[preferences].alt_screen` (2.6.F, ADR-0068 §e) — carried into the rebuilt `ReplWiring` so a `/clear` re-drive * keeps the full-screen render mode (else the mode reverts to the phase default mid-conversation). */ @@ -1455,6 +1475,7 @@ async function buildFreshChatWiring(deps: FreshChatWiringDeps, intro: string): P ...(resolvePrice.size === 0 ? {} : { resolvePrice }), onBudgetWarning: deps.onBudgetWarning, onEffortWithheld: deps.onEffortWithheld, + onListenerError: deps.onListenerError, onUnpriced: deps.onUnpriced, }); // The SAME signals `runReplLoop` used for the hoist (`deps.altScreen` is `[preferences].alt_screen`, carried here @@ -1564,6 +1585,7 @@ function createClearRebuild(params: { altScreen: params.altScreen, onBudgetWarning: (warning) => emitLiveNotice(params.io, budgetWarningText(warning)), onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(params.io, note)), + onListenerError: (note: string) => emitLiveNotice(params.io, note), onUnpriced: (note) => emitLiveNotice(params.io, note), }; return (oldSessionId) => buildFreshChatWiring(wiringDeps, clearedNotice(oldSessionId)); @@ -1640,6 +1662,7 @@ interface ReseatWiringDeps { /** Withheld-tier sink (ADR-0071 §6) — a `/models` reseat binds a DIFFERENT model, which is precisely when a tier * that was fine a moment ago stops being accepted. Threaded like {@link ReseatWiringDeps.onBudgetWarning}. */ readonly onEffortWithheld: NonNullable; + readonly onListenerError: NonNullable; readonly onUnpriced: NonNullable; /** `[preferences].alt_screen` (2.6.F, ADR-0068 §e) — carried into the rebuilt `ReplWiring` so a `/models` reseat * keeps the full-screen render mode (else it reverts to the phase default after a mid-session model switch). */ @@ -1699,6 +1722,7 @@ async function buildReseatWiring( ...(resolvePrice.size === 0 ? {} : { resolvePrice }), onBudgetWarning: deps.onBudgetWarning, onEffortWithheld: deps.onEffortWithheld, + onListenerError: deps.onListenerError, onUnpriced: deps.onUnpriced, }); let seeded: { store: ChatStoreController; persister: SessionPersister }; @@ -1777,6 +1801,7 @@ function createReseatRebuild(params: { ) => Promise { const wiringDeps: ReseatWiringDeps = { chat: params.chat, + onListenerError: (note: string) => emitLiveNotice(params.io, note), now: params.now, uuid: params.uuid, providers: params.providers, @@ -1919,6 +1944,7 @@ async function driveOneSession(wiring: ReplWiring, deps: ChatReplDeps): Promise< ...(deps.hatchPorts?.suspendPort === undefined ? {} : { suspendPort: deps.hatchPorts.suspendPort }), + ...(deps.onSuspend === undefined ? {} : { onSuspend: deps.onSuspend }), // The clipboard rides the SAME control-write sink as the alt-buffer toggles (Step 6). OSC 52 prints nothing and // moves no cursor, so writing it mid-frame cannot corrupt ink's line accounting. // ABSENT when `[preferences].copy_on_select = false` (or `--no-mouse`, which resolves it false): the selection @@ -2017,6 +2043,12 @@ export async function withHoistedAltScreen( readonly lifecycle: ReplLifecycle; readonly writeOut: (text: string) => void; readonly writeErr: (text: string) => void; + /** + * Permanently disarm a reversible terminal handoff before this wrapper restores its final terminal state. The + * job-control coordinator intentionally leaves a pending SIGCONT waiter unresolved on an external termination: + * allowing it to reclaim afterward could re-enter 1049 / re-enable mouse reporting behind this final restore. + */ + readonly beforeTerminalRestore?: () => void; /** The ONE suspend port for the loop. `current() === undefined` means NO ink tree is mounted — the only window in * which SIGINT is unowned. Absent ⇒ no SIGINT net (a caller with no ink, e.g. a unit test). */ readonly suspendPort?: SuspendPort; @@ -2039,7 +2071,12 @@ export async function withHoistedAltScreen( let result: HoistedLoopResult = {}; try { alt.enter(); - removeExitNet = opts.active ? opts.lifecycle.onProcessExit(() => alt.restore()) : noop; + removeExitNet = opts.active + ? opts.lifecycle.onProcessExit(() => { + opts.beforeTerminalRestore?.(); + alt.restore(); + }) + : noop; // SIGINT belongs to ink: while a tree is mounted, `driveInk`'s `onSigintGated` runs the cooperative `/cancel`, and // during a `/scrollback` or `/edit` hatch it deliberately DROPS the signal so the suspension can reclaim. But a // `/clear` or `/models` rebuild unmounts ink and mounts a fresh tree, and in that window NOTHING listens for @@ -2050,6 +2087,7 @@ export async function withHoistedAltScreen( opts.active && suspendPortForSigint !== undefined ? opts.lifecycle.onInterrupt(() => { if (suspendPortForSigint.current() !== undefined) return; // ink owns it (mounted, or suspended) + opts.beforeTerminalRestore?.(); alt.restore(); opts.write(DISABLE_BRACKETED_PASTE); try { @@ -2062,6 +2100,7 @@ export async function withHoistedAltScreen( : noop; removeSignalNet = opts.active ? opts.lifecycle.onTerminationSignal((signo) => { + opts.beforeTerminalRestore?.(); alt.restore(); opts.write(DISABLE_BRACKETED_PASTE); try { @@ -2079,6 +2118,7 @@ export async function withHoistedAltScreen( // DECRST-1049 restore (the Step-4b-3 Opus-review regression: the rebuild-failure `chat-resume ` hint vanished // once alt became the default). `restore()` is idempotent — the `onProcessExit` net may also have fired. Remove // the nets so they cannot outlive the loop. + opts.beforeTerminalRestore?.(); alt.restore(); removeExitNet(); removeSignalNet(); @@ -2182,6 +2222,10 @@ export async function runReplLoop( // The hoisted controller, captured so `hatchPorts.terminal()` can read the LIVE buffer state (it is created inside // `withHoistedAltScreen` below, after `hatchPorts` is built — hence the mutable binding read lazily). let altScreenController: AltScreenController | undefined; + // The hoist owns the irreversible restore nets, while the job-control binding is created only after its live + // controller exists. Keep a tiny indirection so those nets can disarm a pending reversible reclaim BEFORE they + // restore 1049/mouse permanently (a SIGCONT must never re-enter the terminal after a TERM/HUP/QUIT path). + let disposeJobControl = (): void => undefined; // Copy-on-select (2.6.F Step 6e): a durable preference, resolved from the ALREADY-RESOLVED mouse decision, so // `--no-mouse` structurally turns it off too. `/copy` is unaffected: it binds the clipboard through `hatchPorts`. const copyOnSelect = resolveCopyOnSelect({ @@ -2199,49 +2243,82 @@ export async function runReplLoop( lifecycle: deps.lifecycle ?? defaultReplLifecycle, writeOut: (text) => deps.io.writeOut(text), writeErr: (text) => deps.io.writeErr(text), + beforeTerminalRestore: () => disposeJobControl(), suspendPort, }, async (alt): Promise => { altScreenController = alt; // so `hatchPorts.terminal()` reads the LIVE buffer state, not the startup decision - let current = wiring; - let summaryText: string | undefined; // the final `/exit` summary, printed after the single alt-exit - for (;;) { - const outcome = await driveOneSession(current, replDeps); - // The end-of-session summary is LIFTED out of driveInk (ADR-0068 §c): the wrapper prints it after the single - // alt-exit, on the PRIMARY buffer. Only a final `'exit'` carries one; a `/clear` / reseat swap carries none. - if (outcome.kind === 'exit') summaryText = outcome.summaryText; - // The old session is ALREADY torn down (driveOneSession's finally fired its terminal → the row is 'ended' + - // resumable). Resolve the rebuild for this swap kind, or `undefined` to END the REPL (see resolveSwapRebuild). - const oldSessionId = current.built.sessionId; - // The old session is torn down but its view STORE is still live (teardown closes the persister + MCP, never - // the store), so this is the rendered conversation the reseated session must open with (2.6.C / F1). - const next = resolveSwapRebuild( - outcome, - oldSessionId, - rebuild, - reseatRebuild, - current.store.getSnapshot().state.transcript, - ); - if (next === undefined) return { summaryText }; - // Clear the persistent alt buffer between the just-unmounted session and the next mount, so a `/clear` swap - // never briefly shows two stacked transcripts (ink's non-fullscreen unmount does not erase). No-op inactive. - alt.clearBetween(); - // Build the swap session over the same db and re-drive; a build failure is surfaced actionably (the prior - // conversation is still resumable) and ends the REPL rather than looping on a broken build. The message is - // RETURNED (not written here) so the wrapper emits it AFTER the alt-exit — writing it inside the still-entered - // alt buffer would discard it (the recovery hint with the resumable session id — Step-4b-3 Opus review). - try { - current = await next(); - } catch (err) { - const errorText = - // Sanitize the error text too (not just the id beside it) — a rebuild fault can rethrow an unclassified - // message verbatim (session-host.ts), which could carry an ANSI/OSC escape from a spawned MCP server's - // error text; strip it exactly as the id + every other display string on this surface is stripped. - `could not start a new session after ${outcome.kind === 'reseat' ? 'a model switch' : '/clear'}: ` + - `${sanitizeInline(err instanceof Error ? err.message : String(err))}. ` + - `Your previous conversation is saved — resume it with \`relavium chat-resume ${sanitizeInline(oldSessionId)}\`.\n`; - return { summaryText, errorText }; + // The raw-mode Ink surface needs job control even in inline render mode; only plain/non-TTY drivers omit + // these listeners. During a `/clear`/reseat gap there is no Ink tree, so the hoisted controller supplies its + // reversible 1049/mouse release instead of calling its FINAL `restore()`. + const jobControl = chatIsInteractive(deps.io, deps.global) + ? wireJobControl({ + suspendPort, + lifecycle: deps.jobControlLifecycle ?? defaultJobControlLifecycle, + run: (suspendTerminal, waitForContinue) => + suspendFullScreen( + { + suspendTerminal, + writeControl, + inkOwnsAltScreen: false, + altActive: alt.isEntered(), + mouseActive: alt.isMouseEnabled(), + }, + waitForContinue, + ), + releaseWithoutInk: () => alt.releaseForSuspend(), + reclaimWithoutInk: () => alt.reclaimAfterSuspend(), + }) + : undefined; + disposeJobControl = jobControl?.dispose ?? (() => undefined); + const sessionDeps = + jobControl === undefined + ? replDeps + : { ...replDeps, onSuspend: jobControl.requestSuspend }; + try { + let current = wiring; + let summaryText: string | undefined; // the final `/exit` summary, printed after the single alt-exit + for (;;) { + const outcome = await driveOneSession(current, sessionDeps); + // The end-of-session summary is LIFTED out of driveInk (ADR-0068 §c): the wrapper prints it after the single + // alt-exit, on the PRIMARY buffer. Only a final `'exit'` carries one; a `/clear` / reseat swap carries none. + if (outcome.kind === 'exit') summaryText = outcome.summaryText; + // The old session is ALREADY torn down (driveOneSession's finally fired its terminal → the row is 'ended' + + // resumable). Resolve the rebuild for this swap kind, or `undefined` to END the REPL (see resolveSwapRebuild). + const oldSessionId = current.built.sessionId; + // The old session is torn down but its view STORE is still live (teardown closes the persister + MCP, never + // the store), so this is the rendered conversation the reseated session must open with (2.6.C / F1). + const next = resolveSwapRebuild( + outcome, + oldSessionId, + rebuild, + reseatRebuild, + current.store.getSnapshot().state.transcript, + ); + if (next === undefined) return { summaryText }; + // Clear the persistent alt buffer between the just-unmounted session and the next mount, so a `/clear` swap + // never briefly shows two stacked transcripts (ink's non-fullscreen unmount does not erase). No-op inactive. + alt.clearBetween(); + // Build the swap session over the same db and re-drive; a build failure is surfaced actionably (the prior + // conversation is still resumable) and ends the REPL rather than looping on a broken build. The message is + // RETURNED (not written here) so the wrapper emits it AFTER the alt-exit — writing it inside the still-entered + // alt buffer would discard it (the recovery hint with the resumable session id — Step-4b-3 Opus review). + try { + current = await next(); + } catch (err) { + const errorText = + // Sanitize the error text too (not just the id beside it) — a rebuild fault can rethrow an unclassified + // message verbatim (session-host.ts), which could carry an ANSI/OSC escape from a spawned MCP server's + // error text; strip it exactly as the id + every other display string on this surface is stripped. + `could not start a new session after ${outcome.kind === 'reseat' ? 'a model switch' : '/clear'}: ` + + `${sanitizeInline(err instanceof Error ? err.message : String(err))}. ` + + `Your previous conversation is saved — resume it with \`relavium chat-resume ${sanitizeInline(oldSessionId)}\`.\n`; + return { summaryText, errorText }; + } } + } finally { + jobControl?.dispose(); + disposeJobControl = (): void => undefined; } }, ); diff --git a/apps/cli/src/commands/drive.ts b/apps/cli/src/commands/drive.ts index 1b07f425..dec54a4d 100644 --- a/apps/cli/src/commands/drive.ts +++ b/apps/cli/src/commands/drive.ts @@ -14,6 +14,7 @@ import { CliError } from '../process/errors.js'; import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { RunRenderer } from '../render/renderer.js'; +import { wireRunJobControl } from '../render/run-job-control.js'; /** * The D15 catalog load-check, shared by `run` (a fresh load) and `gate` (a resume — re-validated against the @@ -90,6 +91,11 @@ export async function driveRun(deps: DriveRunDeps): Promise io.writeOut(text) }); let renderer: RunRenderer | undefined; try { renderer = makeRenderer(); @@ -118,6 +124,7 @@ export async function driveRun(deps: DriveRunDeps): Promise { expect(text).not.toContain('r3'); }); + it('SANITIZES the untrusted gate message (lane-c security review)', async () => { + // `human_gate:paused.message` is `resolveTemplate(node.message_template, …)` — interpolated upstream node + // output, i.e. the most model-controlled string in the product. The interactive prompt sanitized it + // (`clack-prompter.ts`); this command leaf did not, so an OSC 52 clipboard write, an OSC 8 hyperlink, a + // CSI erase-display and a bidi override all reached the terminal intact. Confirmed by executing the real + // command against a real DB during the review, not by reading. + const { io, out } = captureIo(); + const hostile = + 'hi\u001b]52;c;ZXZpbA==\u0007\u001b]8;;file:///etc/passwd\u0007click\u001b]8;;\u0007\u001b[2J\u202Egnp.txt'; + await seedRun(db, { + slug: 'x', + runId: 'r9', + state: 'paused', + gate: { gateId: 'g-x', gateType: 'approval', message: hostile }, + }); + + expect(gateListCommand({}, deps(io))).toBe(EXIT_CODES.success); + const text = out(); + expect(text).not.toContain('\u001b'); // no CSI/OSC introducer survives + expect(text).not.toContain('\u0007'); // nor the OSC terminator + expect(text).not.toContain('\u202E'); // nor a Trojan-Source bidi override + expect(text).toContain('hi'); // …while the readable text still reaches the user + }); + it('scopes to one run when given a runId', async () => { const { io, out } = captureIo(); await seedRun(db, { diff --git a/apps/cli/src/commands/gate-list.ts b/apps/cli/src/commands/gate-list.ts index 5ffd1fc3..6730456a 100644 --- a/apps/cli/src/commands/gate-list.ts +++ b/apps/cli/src/commands/gate-list.ts @@ -8,6 +8,7 @@ import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; import { writeRecordLines } from '../render/records.js'; import { openHistoryReader } from '../history/reader.js'; +import { sanitizeInline } from '../render/sanitize.js'; /** A pending gate row tagged with its run id — the listing unit (`gate list` JSON record + human line). */ type GateRow = PendingGate & { readonly runId: string }; @@ -85,7 +86,8 @@ function renderGateRows(io: CliIo, rows: readonly GateRow[], runId: string | und } io.writeOut(`Pending human gates (${rows.length}):\n`); for (const row of rows) { - const message = row.message === '' ? '' : ` "${row.message}"`; + // Same untrusted `human_gate:paused.message` as `status.ts` — see the note there. + const message = row.message === '' ? '' : ` "${sanitizeInline(row.message)}"`; io.writeOut(` ${row.runId} ${row.gateId} ${row.gateType} node=${row.nodeId}${message}\n`); } } diff --git a/apps/cli/src/commands/status.ts b/apps/cli/src/commands/status.ts index c3cc6464..8fe14a1a 100644 --- a/apps/cli/src/commands/status.ts +++ b/apps/cli/src/commands/status.ts @@ -7,6 +7,7 @@ import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; import { writeRecordLines } from '../render/records.js'; import { openHistoryReader } from '../history/reader.js'; +import { sanitizeInline } from '../render/sanitize.js'; export interface StatusCommandDeps { readonly io: CliIo; @@ -99,7 +100,11 @@ function renderRun(io: CliIo, status: ActiveRunStatus): void { io.writeOut(` ${step.status.padEnd(9)} ${step.nodeId} [${step.nodeType}]${attempt}\n`); } for (const gate of status.pendingGates) { - const message = gate.message === '' ? '' : ` — "${gate.message}"`; + // The SAME `human_gate:paused.message` the interactive prompt sanitizes (`clack-prompter.ts`), and it is + // `resolveTemplate(node.message_template, …)` — i.e. interpolated upstream NODE OUTPUT, the most + // model-controlled string in the product. The TUI leaf was hardened; this command leaf was not, so a + // pasted OSC 52 / OSC 8 / CSI payload reached the terminal intact here (G34/#57's class). + const message = gate.message === '' ? '' : ` — "${sanitizeInline(gate.message)}"`; io.writeOut(` ⏸ pending gate ${gate.gateId} (${gate.gateType}) at ${gate.nodeId}${message}\n`); } } diff --git a/apps/cli/src/config/load.test.ts b/apps/cli/src/config/load.test.ts index d28756e2..79520e2e 100644 --- a/apps/cli/src/config/load.test.ts +++ b/apps/cli/src/config/load.test.ts @@ -1,4 +1,4 @@ -import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { chmodSync, mkdirSync, mkdtempSync, rmSync, statSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -104,6 +104,64 @@ describe('loadConfigFile', () => { }); }); +/** + * #33 — `config/write.ts` applies `0600` at WRITE time only, so a layer that predates that guard, or one a + * user, an editor, a `cp` or a restored backup left permissive, stayed world-readable forever. `history.db` + * already self-heals on open (ADR-0050); this is the config twin. POSIX-only — `chmod` is a documented no-op + * on Windows. + */ +describe('loadConfigFile — the 0600 re-assert on read (#33)', () => { + const POSIX = process.platform !== 'win32'; + let dir: string; + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'relavium-load-mode-')); + }); + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + const modeOf = (path: string): number => statSync(path).mode & 0o777; + + it.skipIf(!POSIX)('tightens a world-readable config layer on load', () => { + const file = join(dir, 'config.toml'); + writeFileSync(file, 'update_channel = "beta"\n'); + chmodSync(file, 0o644); + // …and still returns the parsed config: the self-heal is a side effect, never a gate. + expect(loadConfigFile(file, GlobalConfigSchema)).toEqual({ update_channel: 'beta' }); + expect(modeOf(file)).toBe(0o600); + }); + + it.skipIf(!POSIX)( + 'leaves an already-0600 layer alone (no needless syscall on the common path)', + () => { + const file = join(dir, 'config.toml'); + writeFileSync(file, 'update_channel = "beta"\n'); + chmodSync(file, 0o600); + expect(loadConfigFile(file, GlobalConfigSchema)).toEqual({ update_channel: 'beta' }); + expect(modeOf(file)).toBe(0o600); + }, + ); + + it.skipIf(!POSIX)( + 'NEVER throws when the mode cannot be changed — a config layer holds no secrets', + () => { + // A read-only parent directory blocks the chmod. Unlike `history.db`, the at-rest guarantee does not rest + // on this, so a permissive-but-readable config must still load rather than failing startup. + const readOnlyDir = join(dir, 'locked'); + mkdirSync(readOnlyDir); + const file = join(readOnlyDir, 'config.toml'); + writeFileSync(file, 'update_channel = "beta"\n'); + chmodSync(file, 0o444); + chmodSync(readOnlyDir, 0o500); + try { + expect(loadConfigFile(file, GlobalConfigSchema)).toEqual({ update_channel: 'beta' }); + } finally { + chmodSync(readOnlyDir, 0o700); // so the temp dir can be swept + } + }, + ); +}); + describe('loadResolvedConfig', () => { let home: string; let project: string; diff --git a/apps/cli/src/config/load.ts b/apps/cli/src/config/load.ts index 4da5bae1..562fbb3d 100644 --- a/apps/cli/src/config/load.ts +++ b/apps/cli/src/config/load.ts @@ -1,4 +1,4 @@ -import { readFileSync, statSync, type Stats } from 'node:fs'; +import { chmodSync, readFileSync, statSync, type Stats } from 'node:fs'; import { homedir } from 'node:os'; import { join } from 'node:path'; @@ -22,6 +22,11 @@ const MAX_CONFIG_BYTES = 256 * 1024; * contents (config files hold no secrets — config-spec.md). */ export function loadConfigFile(filePath: string, schema: ZodType): T | undefined { + // #33: re-assert `0600` on READ, mirroring `history.db`'s self-heal in `db/open.ts`. `config/write.ts` + // applies the mode at WRITE time only, so a file that predates that guard — or one a user, an editor, a + // `cp`, or a restored backup left at the umask default — stays world-readable forever. Cheap: one `stat` + // we already did, and a `chmod` only when the mode is actually wrong. + let stats: Stats; try { stats = statSync(filePath); @@ -34,6 +39,7 @@ export function loadConfigFile(filePath: string, schema: ZodType): T | und if (!stats.isFile()) { throw new ConfigError(filePath, 'is not a regular file.'); } + reassertOwnerOnly(filePath, stats); // Enforce the size cap BEFORE reading into memory (ADR-0048 hardened loader) — an oversize // file is rejected without ever being loaded. if (stats.size > MAX_CONFIG_BYTES) { @@ -119,6 +125,29 @@ export function loadResolvedConfig(options: LoadConfigOptions): LoadedConfig { return { config: resolveConfig({ global, workspace, project }), projectConfigDir, homeDir: home }; } +/** + * Best-effort `0600` self-heal for a config layer (#33) — the read-side twin of `config/write.ts`'s write-time + * mode and of `db/open.ts`'s db self-heal (ADR-0050). + * + * Deliberately **never throws**: unlike `history.db`, a config file holds no secrets (config-spec.md — secret + * VALUES live in the keychain and are referenced by name), so this is defence in depth, not the guarantee. A + * read-only file, a foreign owner, or Windows — where POSIX mode bits do not apply and `chmod` is a documented + * no-op — must not turn a perfectly readable config into a startup failure. + * + * Skips the `chmod` entirely when the mode is already right, so the common path adds no syscall at all. + */ +function reassertOwnerOnly(filePath: string, stats: Stats): void { + // The permission bits of `st_mode` — masking is the only way to read them. + if ((stats.mode & 0o777) === 0o600) { + return; + } + try { + chmodSync(filePath, 0o600); + } catch { + // Best-effort by design — see the doc comment. A config layer is readable either way. + } +} + function errnoCode(err: unknown): string | undefined { if (typeof err === 'object' && err !== null && 'code' in err) { const code: unknown = err.code; diff --git a/apps/cli/src/db/open.test.ts b/apps/cli/src/db/open.test.ts new file mode 100644 index 00000000..cec3515d --- /dev/null +++ b/apps/cli/src/db/open.test.ts @@ -0,0 +1,85 @@ +import { chmodSync, mkdtempSync, rmSync, statSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { globalConfigDir } from '../config/paths.js'; + +/** + * #28 — the ADR-0050 at-rest `0600` on `history.db` must NOT be conditional on the migrations succeeding. + * + * `createClient` creates the file at the process umask (typically 644). The chmod used to run only AFTER + * `runMigrations`, so a migration that threw — disk full, an interrupted first run — left the file + * world-readable for the lifetime of the install, with nothing ever revisiting it. The guarantee has to hold + * on the failure path too, which is the whole point of an at-rest guarantee. + * + * POSIX-only: `chmod` is a documented no-op on Windows (ADR-0050), so the mode assertions are gated off it + * exactly as the 2.5.I concurrency lane does. + */ +const POSIX = process.platform !== 'win32'; + +let home: string; + +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), 'relavium-open-db-')); +}); + +afterEach(() => { + vi.resetModules(); + vi.doUnmock('@relavium/db'); + rmSync(home, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); +}); + +const modeOf = (path: string): number => statSync(path).mode & 0o777; + +describe('openLocalDb — the at-rest 0600 (#28, ADR-0050)', () => { + it.skipIf(!POSIX)('applies 0600 on a successful open', async () => { + const { openLocalDb } = await import('./open.js'); + const opened = openLocalDb(home); + try { + expect(modeOf(join(globalConfigDir(home), 'history.db'))).toBe(0o600); + } finally { + opened.close(); + } + }); + + it.skipIf(!POSIX)( + 'applies 0600 even when the MIGRATIONS throw — the guarantee is unconditional', + async () => { + // The real driver still opens (and therefore creates) the file; only the migration batch fails. That is + // exactly the disk-full / interrupted-first-run shape, and the shape that used to leave the file at 644. + const actual = await import('@relavium/db'); + vi.doMock('@relavium/db', () => ({ + ...actual, + runMigrations: () => { + throw new Error('disk full'); + }, + })); + vi.resetModules(); + const { openLocalDb } = await import('./open.js'); + + expect(() => openLocalDb(home)).toThrow('disk full'); + // The file exists (createClient made it) and is owner-only despite the failure. + expect(modeOf(join(globalConfigDir(home), 'history.db'))).toBe(0o600); + }, + ); + + it.skipIf(!POSIX)( + 're-asserts 0600 on a SUBSEQUENT open of a file left world-readable', + async () => { + const { openLocalDb } = await import('./open.js'); + const dbPath = join(globalConfigDir(home), 'history.db'); + const first = openLocalDb(home); + first.close(); + // Simulate a file that predates the guard, or one a `cp`/restore/editor left permissive. + chmodSync(dbPath, 0o644); + const second = openLocalDb(home); + try { + expect(modeOf(dbPath)).toBe(0o600); + } finally { + second.close(); + } + }, + ); +}); diff --git a/apps/cli/src/db/open.ts b/apps/cli/src/db/open.ts index 90d5a572..68dcb390 100644 --- a/apps/cli/src/db/open.ts +++ b/apps/cli/src/db/open.ts @@ -24,19 +24,16 @@ export function openLocalDb(homeDir: string): OpenedDb { // If setup (migrations / the at-rest chmod) throws, close the just-opened handle before propagating — // otherwise the SQLite connection leaks for the lifetime of the failing process. try { - runMigrations(client.db); - // The db file is guaranteed to exist here — a chmod failure on IT must be LOUD (ADR-0050's at-rest - // guarantee rests on this 0600). Its WAL/SHM sidecars may not exist yet (no checkpoint) — best-effort. - chmodSync(path, 0o600); - for (const suffix of ['-wal', '-shm']) { - try { - chmodSync(`${path}${suffix}`, 0o600); - } catch (err) { - if (errnoCode(err) !== 'ENOENT') { - throw err; - } - } - } + // #28: the 0600 runs BEFORE the migrations, and again after. Before, because `createClient` has already + // CREATED the file at the process umask — typically 644 — and a migration that throws (disk full, an + // interrupted first run) used to leave it there, world-readable, for the lifetime of the install. The + // ADR-0050 at-rest guarantee cannot be conditional on the migration succeeding. After, because the WAL + // and SHM sidecars only come into existence once SQLite actually writes. + hardenAtRest(path); + // The batch is serialized across processes (ADR-0073): two `relavium` invocations racing a fresh + // `history.db` wait for each other instead of one dying on a duplicate `CREATE TABLE` (#99). + runMigrations(client.db, { dbPath: client.path }); + hardenAtRest(path); } catch (err) { client.sqlite.close(); throw err; @@ -53,6 +50,27 @@ export function openLocalDb(homeDir: string): OpenedDb { }; } +/** + * Apply the ADR-0050 at-rest mode to `history.db` and its sidecars. The db file itself is guaranteed to exist + * (`createClient` opened it), so a chmod failure on IT is LOUD — the whole at-rest guarantee rests on this + * `0600`. The `-wal`/`-shm` sidecars may legitimately not exist yet (no checkpoint), so `ENOENT` on them is + * expected and ignored; any other error still propagates. + * + * Idempotent, so it is safe to call on both sides of the migration batch. + */ +function hardenAtRest(path: string): void { + chmodSync(path, 0o600); + for (const suffix of ['-wal', '-shm']) { + try { + chmodSync(`${path}${suffix}`, 0o600); + } catch (err) { + if (errnoCode(err) !== 'ENOENT') { + throw err; + } + } + } +} + /** The `errno` code of a Node fs error (`ENOENT`, `EPERM`, …), or `undefined` if it is not one. */ function errnoCode(err: unknown): string | undefined { if (typeof err === 'object' && err !== null && 'code' in err) { diff --git a/apps/cli/src/engine/model-catalog-view.test.ts b/apps/cli/src/engine/model-catalog-view.test.ts index 439c7016..52380d5d 100644 --- a/apps/cli/src/engine/model-catalog-view.test.ts +++ b/apps/cli/src/engine/model-catalog-view.test.ts @@ -198,6 +198,51 @@ describe('buildMergedCatalog', () => { }); }); +describe('a corrupt deprecation_date degrades one row, never the whole catalog (G1)', () => { + // `new Date(x).toISOString()` throws RangeError on NaN or |x| > 8.64e15, and the conversion runs inside a + // `.map()` over EVERY row — so before the guard, one bad value aborted the entire `/models` projection. + // SQLite is dynamically typed, so an INTEGER column really can hold these. + it.each([ + ['NaN', Number.NaN], + ['out of Date range', 8.64e15 + 1], + ['negative out of range', -(8.64e15 + 1)], + ['Infinity', Number.POSITIVE_INFINITY], + ])('survives a %s deprecation_date and still returns the catalog', (_label, value) => { + const rows = [ + row({ + modelId: MODEL_PRESENT, + providerId: 'p-anthropic', + source: 'live', + deprecationDate: 0, + }), + row({ + modelId: MODEL_ABSENT, + providerId: 'p-anthropic', + source: 'live', + deprecationDate: value, + }), + ]; + let view: ReturnType | undefined; + expect(() => { + view = buildMergedCatalog({ + rows, + providerSlug: slugResolver({ 'p-anthropic': 'anthropic' }), + now: 0, + }); + }).not.toThrow(); + // The whole projection survived — every static model plus both live rows. + expect(view?.entries.length ?? 0).toBeGreaterThan(1); + // The corrupt row is still PRESENT; only its decorative date is dropped. + const bad = view?.entries.find((entry) => entry.modelId === MODEL_ABSENT); + expect(bad).toBeDefined(); + expect(bad?.deprecatedAt).toBeUndefined(); + // …and a valid neighbour keeps its date, so the guard is not blanket-dropping the field. + expect(view?.entries.find((entry) => entry.modelId === MODEL_PRESENT)?.deprecatedAt).toBe( + new Date(0).toISOString(), + ); + }); +}); + describe('buildUserPricing (2.5.G S10, ADR-0065 §2)', () => { it('projects a source="user" row into a ModelPricing keyed by model id', () => { const overlay = buildUserPricing({ diff --git a/apps/cli/src/engine/model-catalog-view.ts b/apps/cli/src/engine/model-catalog-view.ts index 6475d278..3ae99f97 100644 --- a/apps/cli/src/engine/model-catalog-view.ts +++ b/apps/cli/src/engine/model-catalog-view.ts @@ -58,6 +58,29 @@ function isProviderId(slug: string): slug is ProviderId { return (LLM_PROVIDERS as readonly string[]).includes(slug); } +/** + * A stored epoch-ms → an ISO string, or `undefined` when the value cannot be a date (G1). + * + * `new Date(x).toISOString()` throws `RangeError: Invalid time value` on `NaN` or any value outside + * ±8.64e15 — and SQLite is dynamically typed, so an `INTEGER` column can hold whatever a bad write or an + * external tool put there. Both conversions below run inside a `.map()` over EVERY catalog row, so one + * corrupt `deprecation_date` would abort the entire `/models` projection instead of degrading a single row. + * A deprecation date is decoration; the catalog is not. Same guarded-parse discipline the limit columns get + * from {@link statedLimit} a few lines down. + */ +function spreadIfSet( + key: K, + value: string | undefined, +): Record | Record { + return value === undefined ? {} : ({ [key]: value } as Record); +} + +function isoDateOrUndefined(epochMs: number | undefined): string | undefined { + if (epochMs === undefined) return undefined; + const date = new Date(epochMs); + return Number.isNaN(date.getTime()) ? undefined : date.toISOString(); +} + /** Map a stored catalog row → a seam {@link ModelListing} (the live-discovery half): id + the optional limits + * the live deprecation date as ISO (the store carries epoch-ms; the merge unions ISO dates). Pricing is NOT * carried — the merge's pricing authority is the user tier / the catalog, never a live row (ADR-0064 §6). */ @@ -69,9 +92,7 @@ function rowToListing(row: ModelCatalogListing): ModelListing { ? { contextWindowTokens: row.contextWindowTokens } : {}), ...(row.maxOutputTokens !== undefined ? { maxOutputTokens: row.maxOutputTokens } : {}), - ...(row.deprecationDate !== undefined - ? { deprecatedAt: new Date(row.deprecationDate).toISOString() } - : {}), + ...spreadIfSet('deprecatedAt', isoDateOrUndefined(row.deprecationDate)), }; } @@ -185,9 +206,7 @@ function rowToUserPricing(row: ModelCatalogListing, provider: ProviderId): Model ? {} : { cacheWritePerMtokMicrocents: base.cacheWritePerMtokMicrocents }), ...(tiers === undefined ? {} : { contextTiers: tiers }), - ...(row.deprecationDate !== undefined - ? { deprecatedAt: new Date(row.deprecationDate).toISOString() } - : {}), + ...spreadIfSet('deprecatedAt', isoDateOrUndefined(row.deprecationDate)), }; } diff --git a/apps/cli/src/gate/clack-prompter.test.ts b/apps/cli/src/gate/clack-prompter.test.ts index 76191531..8e84cb0b 100644 --- a/apps/cli/src/gate/clack-prompter.test.ts +++ b/apps/cli/src/gate/clack-prompter.test.ts @@ -40,6 +40,62 @@ describe('createClackGatePrompter', () => { expect(body).toContain('Ship it?'); }); + it('sanitizes the card body and title while retaining readable multiline gate text (G44)', async () => { + const attack = '\x1b[2J\x1b]52;c;Zm9v\x07\x1bPtmux;\x1b\\\x9b2J\r\b\u202E'; + const note = vi.fn<(message: string, title: string) => void>(); + await createClackGatePrompter(deps({ note })).prompt( + gate({ + nodeId: `node visible${attack}\nforged title row`, + message: `message visible${attack}\nsecond message line`, + }), + ); + + const [body, title] = note.mock.calls[0] ?? []; + for (const forbidden of ['\x1b', '\x9b', '\r', '\b', '\u202E']) { + expect(body).not.toContain(forbidden); + expect(title).not.toContain(forbidden); + } + expect(body).toContain('message visible'); + expect(body).toContain('second message line'); + expect(title).toContain('node visible'); + expect(title).not.toContain('\n'); // a title cannot forge another terminal row + }); + + it('uses the no-message fallback after controls are stripped (not before)', async () => { + const note = vi.fn<(message: string, title: string) => void>(); + await createClackGatePrompter(deps({ note })).prompt( + gate({ message: '\x1b]52;c;Zm9v\x07\u202E' }), + ); + expect(note.mock.calls[0]?.[0]).toBe('(no message)'); + }); + + it('sanitizes a forged runtime timeout action before it reaches the deadline card', async () => { + const note = vi.fn<(message: string, title: string) => void>(); + const event = gate({ + expiresAt: '2026-06-24T11:00:00.000Z', + timeoutAction: 'reject', + }); + // Events are schema-validated at the bus boundary, but the renderer itself must remain safe when a host or + // test double violates that contract. Reflect simulates that runtime condition without weakening TypeScript types. + expect( + Reflect.set( + event, + 'timeoutAction', + 'reject\x1b]52;c;Zm9v\x07\x9b2J\u202E\nforged timeout row', + ), + ).toBe(true); + + await createClackGatePrompter(deps({ note })).prompt(event); + + const body = note.mock.calls[0]?.[0] ?? ''; + for (const forbidden of ['\x1b', '\x9b', '\r', '\b', '\u202E']) { + expect(body).not.toContain(forbidden); + } + expect(body).toContain('auto-reject'); + expect(body).toContain('forged timeout row'); + expect(body.split('\n')).toHaveLength(3); // message + blank separator + one deadline row + }); + it('approval gate → approve yields an approved decision', async () => { const d = await createClackGatePrompter(deps({ confirm: () => Promise.resolve(true) })).prompt( gate(), diff --git a/apps/cli/src/gate/clack-prompter.ts b/apps/cli/src/gate/clack-prompter.ts index 84e63323..368e1735 100644 --- a/apps/cli/src/gate/clack-prompter.ts +++ b/apps/cli/src/gate/clack-prompter.ts @@ -3,6 +3,7 @@ import type { HumanGatePausedEvent } from '@relavium/shared'; import { approvalDecision, inputDecision, rejectionDecision } from './decision.js'; import type { GatePrompter } from './prompter.js'; +import { sanitizeInline, stripTerminalControls } from '../render/sanitize.js'; /** * The `@clack/prompts`-backed {@link GatePrompter} (2.G, [ADR-0047](../../../../docs/decisions/0047-cli-framework-commander-ink-clack.md)) @@ -42,11 +43,12 @@ const GATE_TITLE: Record = { /** The boxed card body shown above the prompt — the gate message, plus the deadline when the gate has one. */ function cardBody(event: HumanGatePausedEvent): string { - const lines = [event.message.trim() === '' ? '(no message)' : event.message]; + const message = stripTerminalControls(event.message); + const lines = [message.trim() === '' ? '(no message)' : message]; if (event.expiresAt !== undefined) { lines.push( '', - `Expires at ${event.expiresAt} — auto-${event.timeoutAction ?? 'reject'} on timeout`, + `Expires at ${sanitizeInline(event.expiresAt)} — auto-${sanitizeInline(event.timeoutAction ?? 'reject')} on timeout`, ); } return lines.join('\n'); @@ -55,7 +57,10 @@ function cardBody(event: HumanGatePausedEvent): string { export function createClackGatePrompter(deps: ClackPromptDeps = defaultDeps): GatePrompter { return { prompt: async (event) => { - deps.note(cardBody(event), `⏸ ${GATE_TITLE[event.gateType]} · ${event.nodeId}`); + deps.note( + cardBody(event), + `⏸ ${GATE_TITLE[event.gateType]} · ${sanitizeInline(event.nodeId)}`, + ); if (event.gateType === 'input') { // A human types a value: kept as the raw string (predictable). The structured/JSON path is the diff --git a/apps/cli/src/home/drive-home.test.ts b/apps/cli/src/home/drive-home.test.ts index ccd073d4..f440dd37 100644 --- a/apps/cli/src/home/drive-home.test.ts +++ b/apps/cli/src/home/drive-home.test.ts @@ -16,14 +16,15 @@ import { EXIT_CODES } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; import type { RootAppProps } from '../render/tui/home-app.js'; -import { DISABLE_MOUSE, ENABLE_MOUSE } from '../render/alt-screen.js'; -import type { SuspendPort } from '../render/suspend.js'; +import { DISABLE_MOUSE, ENABLE_MOUSE, HIDE_CURSOR, SHOW_CURSOR } from '../render/alt-screen.js'; +import type { JobControlLifecycle, SuspendPort } from '../render/suspend.js'; import { DISABLE_BRACKETED_PASTE } from '../render/tui/home-input.js'; import { defaultSubscribeProcessExit, defaultSubscribeSignals, driveHome, type HomeDeps, + type OnboardingTerminalLifecycle, } from './drive-home.js'; // Regression for the `provider_auth` bug: the Home built an ENV-ONLY key resolver, so a key stored in the OS @@ -75,6 +76,17 @@ const CANCEL_ONBOARDING: ClackOnboardingDeps = { }; const ENTER = { return: true } as const; const CTRL_C = { ctrl: true } as const; +const INERT_JOB_CONTROL: JobControlLifecycle = { + supported: false, + onSuspend: () => () => undefined, + onContinue: () => () => undefined, + suspendSelf: () => undefined, +}; +const INERT_ONBOARDING_TERMINAL: OnboardingTerminalLifecycle = { + onInput: () => () => undefined, + isRawMode: () => false, + setRawMode: () => undefined, +}; describe('driveHome (2.5.B / ADR-0054)', () => { let client: DbClient; @@ -151,6 +163,10 @@ describe('driveHome (2.5.B / ADR-0054)', () => { exitHandlers.push(onExit); return () => undefined; }, + // Signal-oriented tests intentionally leave some drives pending after a simulated process exit. Keep their + // default hermetic: dedicated G0 tests inject a recording POSIX lifecycle instead of leaking real listeners. + jobControlLifecycle: INERT_JOB_CONTROL, + onboardingTerminalLifecycle: INERT_ONBOARDING_TERMINAL, writeControl, exit: () => undefined, // A cancel-immediately onboarding prompter by DEFAULT, so a key-less resolver (e.g. the real keychain-backed @@ -599,6 +615,417 @@ describe('driveHome (2.5.B / ADR-0054)', () => { expect(await drivePromise).toBe(EXIT_CODES.success); }); + it('registers the exit-safety nets BEFORE a key-less onboarding prompt can await the network (G0, #50)', async () => { + // Hold the first clack prompt open. The old ordering subscribed only AFTER the wizard settled, so a Ctrl-C or + // Ctrl-Z during the wizard's later key-validation request had no Home-owned safety net at all. Capture the + // ordering before settling the prompt, then complete the normal clean-exit path so this regression test leaves + // no open db handle or pending Home promise behind. + const order: string[] = []; + let resolveSelect: ((value: string | symbol) => void) | undefined; + const select = new Promise((resolve) => { + resolveSelect = resolve; + }); + const onboardingPrompter: ClackOnboardingDeps = { + ...CANCEL_ONBOARDING, + select: () => { + order.push('wizard-select'); + return select; + }, + }; + const keylessResolver: ProviderResolver = { + resolveProvider: () => undefined, + keyFor: () => { + throw new Error('no key'); + }, + }; + let captured: RootAppProps | undefined; + const { deps } = makeDeps((props) => (captured = props), { + providers: keylessResolver, + onboardingPrompter, + subscribeSignals: () => { + order.push('signals'); + return () => undefined; + }, + subscribeProcessExit: () => { + order.push('exit-net'); + return () => undefined; + }, + jobControlLifecycle: { + supported: true, + onSuspend: () => { + order.push('job-suspend'); + return () => undefined; + }, + onContinue: () => { + order.push('job-continue'); + return () => undefined; + }, + suspendSelf: () => undefined, + }, + }); + + const drivePromise = driveHome(deps); + await flush(); + const beforeWizardSettles = [...order]; + + if (resolveSelect === undefined) throw new Error('the onboarding select did not start'); + resolveSelect(Symbol('cancel')); + await flush(); + const props = captured; + if (props === undefined) throw new Error('the Home did not mount after onboarding'); + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + + expect(beforeWizardSettles).toEqual([ + 'signals', + 'exit-net', + 'job-continue', + 'job-suspend', + 'wizard-select', + ]); + }); + + it('hands Clack raw input + cursor back around BOTH raw Ctrl-Z and external SIGTSTP during onboarding (G0)', async () => { + // Hold Clack's first raw-mode prompt open. Unlike the earlier order-only regression, this models the terminal + // custody that exists before Ink mounts: Ctrl-Z is a SUB byte here, and an external SIGTSTP has no SuspendPort. + let resolveSelect: ((value: string | symbol) => void) | undefined; + const select = new Promise((resolve) => { + resolveSelect = resolve; + }); + const onboardingPrompter: ClackOnboardingDeps = { + ...CANCEL_ONBOARDING, + select: () => select, + }; + const keylessResolver: ProviderResolver = { + resolveProvider: () => undefined, + keyFor: () => { + throw new Error('no key'); + }, + }; + const terminalEvents: string[] = []; + let clackOwnsRawMode = true; + let rawInput: ((input: string) => void) | undefined; + let suspend: (() => void) | undefined; + let activeSuspend: (() => void) | undefined; + let captured: RootAppProps | undefined; + const { deps } = makeDeps((props) => (captured = props), { + providers: keylessResolver, + onboardingPrompter, + writeControl: (sequence) => terminalEvents.push(`control:${sequence}`), + onboardingTerminalLifecycle: { + onInput: (listener) => { + rawInput = listener; + return () => { + rawInput = undefined; + }; + }, + isRawMode: () => clackOwnsRawMode, + setRawMode: (enabled) => { + clackOwnsRawMode = enabled; + terminalEvents.push(`raw:${String(enabled)}`); + }, + }, + jobControlLifecycle: { + supported: true, + onSuspend: (listener) => { + activeSuspend = listener; + suspend = listener; + return () => { + if (activeSuspend === listener) activeSuspend = undefined; + }; + }, + onContinue: () => () => undefined, + // In production this returns only after `fg`/SIGCONT. The synchronous fake therefore observes the exact + // release → stop → reclaim order without suspending Vitest itself. + suspendSelf: () => terminalEvents.push('self-stop'), + }, + }); + + const drivePromise = driveHome(deps); + await flush(); + if (rawInput === undefined || suspend === undefined) + throw new Error('onboarding did not install its raw-input/job-control handoff'); + + rawInput('\x1a'); + expect(terminalEvents).toEqual([ + 'raw:false', + `control:${SHOW_CURSOR}`, + 'self-stop', + 'raw:true', + `control:${HIDE_CURSOR}`, + ]); + + terminalEvents.length = 0; + activeSuspend?.(); // the external `kill -TSTP` path uses the SAME no-Ink custody pair + expect(terminalEvents).toEqual([ + 'raw:false', + `control:${SHOW_CURSOR}`, + 'self-stop', + 'raw:true', + `control:${HIDE_CURSOR}`, + ]); + + // After Clack closes its spinner/prompt it has returned cooked input + a visible cursor. The wizard may still be + // awaiting keychain/config work, but that interval owns no terminal modes and must not synthesize raw+hidden + // state merely because a SIGTSTP arrives. + terminalEvents.length = 0; + clackOwnsRawMode = false; + activeSuspend?.(); + expect(terminalEvents).toEqual(['self-stop']); + + if (resolveSelect === undefined) throw new Error('the onboarding select did not start'); + resolveSelect(Symbol('cancel')); + await flush(); + const props = captured; + if (props === undefined) throw new Error('the Home did not mount after onboarding'); + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + }); + + it('reclaims Clack cursor ownership when the release write faults after raw mode is handed off (G0)', async () => { + // stdout can fail after the terminal accepts part of an escape sequence. In particular, a failed SHOW_CURSOR + // must be treated as a possible visible cursor: rolling raw mode back alone would return Clack to a live prompt + // with its cursor exposed. + let resolveSelect: ((value: string | symbol) => void) | undefined; + const select = new Promise((resolve) => { + resolveSelect = resolve; + }); + const keylessResolver: ProviderResolver = { + resolveProvider: () => undefined, + keyFor: () => { + throw new Error('no key'); + }, + }; + const terminalEvents: string[] = []; + let clackOwnsRawMode = true; + let rawInput: ((input: string) => void) | undefined; + let captured: RootAppProps | undefined; + const { deps } = makeDeps((props) => (captured = props), { + providers: keylessResolver, + onboardingPrompter: { ...CANCEL_ONBOARDING, select: () => select }, + writeControl: (sequence) => { + terminalEvents.push(`control:${sequence}`); + if (sequence === SHOW_CURSOR) throw new Error('partial cursor write'); + }, + onboardingTerminalLifecycle: { + onInput: (listener) => { + rawInput = listener; + return () => { + rawInput = undefined; + }; + }, + isRawMode: () => clackOwnsRawMode, + setRawMode: (enabled) => { + clackOwnsRawMode = enabled; + terminalEvents.push(`raw:${String(enabled)}`); + }, + }, + jobControlLifecycle: { + supported: true, + onSuspend: () => () => undefined, + onContinue: () => () => undefined, + // A failed terminal release must never forward SIGTSTP: there is no safe stopped state to return from. + suspendSelf: () => terminalEvents.push('self-stop'), + }, + }); + + const drivePromise = driveHome(deps); + await flush(); + if (rawInput === undefined) throw new Error('onboarding did not install its raw-input handoff'); + + rawInput('\x1a'); + expect(terminalEvents).toEqual([ + 'raw:false', + `control:${SHOW_CURSOR}`, + 'raw:true', + `control:${HIDE_CURSOR}`, + ]); + expect(clackOwnsRawMode).toBe(true); + + if (resolveSelect === undefined) throw new Error('the onboarding select did not start'); + resolveSelect(Symbol('cancel')); + await flush(); + const props = captured; + if (props === undefined) throw new Error('the Home did not mount after onboarding'); + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + }); + + it('retries a failed Clack cursor reclaim before a later stop request (G0)', async () => { + let resolveSelect: ((value: string | symbol) => void) | undefined; + const select = new Promise((resolve) => { + resolveSelect = resolve; + }); + const keylessResolver: ProviderResolver = { + resolveProvider: () => undefined, + keyFor: () => { + throw new Error('no key'); + }, + }; + const terminalEvents: string[] = []; + let clackOwnsRawMode = true; + let rawInput: ((input: string) => void) | undefined; + let captured: RootAppProps | undefined; + let failShowOnce = true; + let failHideOnce = true; + const { deps } = makeDeps((props) => (captured = props), { + providers: keylessResolver, + onboardingPrompter: { ...CANCEL_ONBOARDING, select: () => select }, + writeControl: (sequence) => { + terminalEvents.push(`control:${sequence}`); + if (sequence === SHOW_CURSOR && failShowOnce) { + failShowOnce = false; + throw new Error('partial cursor release'); + } + if (sequence === HIDE_CURSOR && failHideOnce) { + failHideOnce = false; + throw new Error('partial cursor reclaim'); + } + }, + onboardingTerminalLifecycle: { + onInput: (listener) => { + rawInput = listener; + return () => { + rawInput = undefined; + }; + }, + isRawMode: () => clackOwnsRawMode, + setRawMode: (enabled) => { + clackOwnsRawMode = enabled; + terminalEvents.push(`raw:${String(enabled)}`); + }, + }, + jobControlLifecycle: { + supported: true, + onSuspend: () => () => undefined, + onContinue: () => () => undefined, + suspendSelf: () => terminalEvents.push('self-stop'), + }, + }); + + const drivePromise = driveHome(deps); + await flush(); + if (rawInput === undefined) throw new Error('onboarding did not install its raw-input handoff'); + + rawInput('\x1a'); + expect(terminalEvents).toEqual([ + 'raw:false', + `control:${SHOW_CURSOR}`, + 'raw:true', + `control:${HIDE_CURSOR}`, + ]); + + // The second request must first retry the cursor reclaim left latched above, then safely release/stop/reclaim. + rawInput('\x1a'); + expect(terminalEvents).toEqual([ + 'raw:false', + `control:${SHOW_CURSOR}`, + 'raw:true', + `control:${HIDE_CURSOR}`, + `control:${HIDE_CURSOR}`, + 'raw:false', + `control:${SHOW_CURSOR}`, + 'self-stop', + 'raw:true', + `control:${HIDE_CURSOR}`, + ]); + expect(clackOwnsRawMode).toBe(true); + + if (resolveSelect === undefined) throw new Error('the onboarding select did not start'); + resolveSelect(Symbol('cancel')); + await flush(); + const props = captured; + if (props === undefined) throw new Error('the Home did not mount after onboarding'); + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + }); + + it('SIGTSTP/SIGCONT never take the Home termination path (G0)', async () => { + let onSuspend: () => void = () => undefined; + let onContinue: () => void = () => undefined; + const suspendSelf = vi.fn(); + const exit = vi.fn(); + let captured: RootAppProps | undefined; + const { deps, unmount, writeControl } = makeDeps((props) => (captured = props), { + exit: exit as (code: number) => void, + jobControlLifecycle: { + supported: true, + onSuspend: (listener) => { + onSuspend = listener; + return () => undefined; + }, + onContinue: (listener) => { + onContinue = listener; + return () => undefined; + }, + suspendSelf, + }, + }); + const drivePromise = driveHome(deps); + const props = captured; + if (props === undefined) throw new Error('the Home did not mount'); + + onContinue(); // unexpected continuation: a no-op, not a teardown + onSuspend(); + expect(suspendSelf).toHaveBeenCalledTimes(1); + expect(exit).not.toHaveBeenCalled(); + expect(unmount).not.toHaveBeenCalled(); + expect(writeControl).not.toHaveBeenCalled(); + expect(closeSpy).not.toHaveBeenCalled(); + + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + }); + + it('disarms a pending job-control continuation BEFORE TERM restores the Home terminal (G0)', async () => { + // Regression for the adversarial order TERM/HUP/QUIT → SIGCONT: permanent teardown owns the final terminal + // state, so an old reversible waiter must never subsequently emit ENABLE_MOUSE behind its one-shot restore latch. + let onSuspend: (() => void) | undefined; + let onContinue: (() => void) | undefined; + let captured: RootAppProps | undefined; + const exit = vi.fn(); + const { deps, signalHandlers, writeControl } = makeDeps((props) => (captured = props), { + exit: exit as (code: number) => void, + jobControlLifecycle: { + supported: true, + onSuspend: (listener) => { + onSuspend = listener; + return () => { + if (onSuspend === listener) onSuspend = undefined; + }; + }, + onContinue: (listener) => { + onContinue = listener; + return () => { + if (onContinue === listener) onContinue = undefined; + }; + }, + suspendSelf: () => undefined, + }, + }); + void driveHome(deps); // a real TERM calls process.exit; the injected exit intentionally keeps this harness alive + const props = captured; + if (props?.suspendPort === undefined) + throw new Error('the Home did not expose its suspend port'); + props.setMouseCapture?.(true); + props.suspendPort.attach(async (callback) => callback()); + + onSuspend?.(); + await flush(); + expect(writeControl).toHaveBeenCalledWith(DISABLE_MOUSE); // mounted Ink suspension released our live mouse mode + const writesBeforeTerminalTeardown = writeControl.mock.calls.length; + + signalHandlers[0]?.(15); // permanent teardown first disposes the reversible continuation listener + onContinue?.(); // removed listener: cannot trigger `suspendFullScreen`'s ENABLE_MOUSE reclaim + await flush(); + + const controlsAfterTerminalTeardown = writeControl.mock.calls + .slice(writesBeforeTerminalTeardown) + .map((call) => String(call[0])); + expect(controlsAfterTerminalTeardown).not.toContain(ENABLE_MOUSE); + expect(exit).toHaveBeenCalledWith(143); + }); + it('a run WITH a resolvable key SKIPS the onboarding wizard entirely', async () => { // The default scriptedResolver's keyFor returns 'test-key' ⇒ not key-less ⇒ the wizard must never run. const select = vi.fn(() => Promise.resolve('anthropic')); diff --git a/apps/cli/src/home/drive-home.tsx b/apps/cli/src/home/drive-home.tsx index 619466cf..0c6ec454 100644 --- a/apps/cli/src/home/drive-home.tsx +++ b/apps/cli/src/home/drive-home.tsx @@ -1,4 +1,5 @@ import { randomUUID } from 'node:crypto'; +import { Buffer } from 'node:buffer'; import { createProviderStore, createRunHistoryReader } from '@relavium/db'; import type { AgentSessionRecord, ReasoningEffort } from '@relavium/shared'; @@ -55,12 +56,18 @@ import { type HomeController, type HomeModelsPort, } from '../render/tui/home-controller.js'; -import { DISABLE_MOUSE, ENABLE_MOUSE } from '../render/alt-screen.js'; +import { DISABLE_MOUSE, ENABLE_MOUSE, HIDE_CURSOR, SHOW_CURSOR } from '../render/alt-screen.js'; import { nodeCreateTempDocument, nodeSpawnEditor } from '../render/editor.js'; import { inkOwnedTerminal, type HatchDeps } from '../render/hatches.js'; import { nodeWaitForContinue, nodeWriteOut } from '../render/scrollback.js'; import { copyToClipboard, type ClipboardOutcome } from '../render/clipboard.js'; -import { createSuspendPort } from '../render/suspend.js'; +import { + createSuspendPort, + defaultJobControlLifecycle, + suspendFullScreen, + wireJobControl, + type JobControlLifecycle, +} from '../render/suspend.js'; import { DISABLE_BRACKETED_PASTE } from '../render/tui/home-input.js'; import { RootApp, type RootAppProps } from '../render/tui/home-app.js'; import { FORCE_TEARDOWN_MS, FRAME_MS } from '../render/tui/tui-constants.js'; @@ -111,6 +118,14 @@ export interface HomeDeps { readonly subscribeResize?: (onResize: () => void) => () => void; /** Subscribe to SIGINT(2)/SIGTERM(15)/SIGHUP(1)/SIGQUIT(3); returns an unsubscribe. Default registers on `process`. */ readonly subscribeSignals?: (onSignal: (signo: number) => void) => () => void; + /** POSIX SIGTSTP/SIGCONT seam. Separate from termination signals: a job stop is reversible and MUST NOT exit. */ + readonly jobControlLifecycle?: JobControlLifecycle; + /** + * The short pre-Ink terminal custody window owned by `@clack/prompts` during first-run onboarding. Clack enables + * raw input and hides the cursor, so it needs the same reversible release/reclaim discipline as Ink before a + * SIGTSTP can stop the process. Injectable to keep the signal path testable without touching the real TTY. + */ + readonly onboardingTerminalLifecycle?: OnboardingTerminalLifecycle; /** Register a synchronous `process.on('exit')` net; returns a remover. Default registers on `process`. The LAST * chance to restore the terminal when something calls `process.exit()` past the `finally` (2.6.F Step 6f). */ readonly subscribeProcessExit?: (onExit: () => void) => () => void; @@ -153,6 +168,33 @@ export function defaultSubscribeProcessExit(onExit: () => void): () => void { return () => process.removeListener('exit', onExit); } +/** The narrow `@clack/prompts` terminal custody seam needed only before Ink mounts. */ +export interface OnboardingTerminalLifecycle { + /** Observe raw input bytes while Clack owns stdin; callers remove the exact listener after the wizard settles. */ + readonly onInput: (listener: (input: string) => void) => () => void; + /** Whether Clack currently owns raw input (and, by its contract, its hidden cursor). */ + readonly isRawMode: () => boolean; + /** Transfer raw-mode ownership between Clack and the user's shell. */ + readonly setRawMode: (enabled: boolean) => void; +} + +/** Production terminal bridge for the real Clack prompt, intentionally kept out of the generic `CliIo` seam. */ +export const defaultOnboardingTerminalLifecycle: OnboardingTerminalLifecycle = { + onInput: (listener) => { + const onData = (chunk: Buffer | string): void => { + listener(typeof chunk === 'string' ? chunk : chunk.toString('utf8')); + }; + process.stdin.on('data', onData); + return () => process.stdin.removeListener('data', onData); + }, + isRawMode: () => process.stdin.isRaw === true, + setRawMode: (enabled) => { + // The bare Home is TTY-gated, but retain this guard so a future direct caller never invokes an unsupported + // `setRawMode` on a redirected stream. + if (process.stdin.isTTY) process.stdin.setRawMode(enabled); + }, +}; + export async function driveHome(deps: HomeDeps): Promise { const now = deps.now ?? Date.now; const uuid = deps.uuid ?? randomUUID; @@ -182,6 +224,8 @@ export async function driveHome(deps: HomeDeps): Promise { let controller: HomeController | undefined; let unsubscribeSignals: (() => void) | undefined; let unsubscribeProcessExit: (() => void) | undefined; + let unsubscribeOnboardingInput: (() => void) | undefined; + let disposeJobControl: (() => void) | undefined; let dbClosed = false; const closeDb = (): void => { if (dbClosed) return; @@ -194,6 +238,88 @@ export async function driveHome(deps: HomeDeps): Promise { process.stdout.write(sequence); }); + // Before Ink mounts, the first-run wizard owns raw input and cursor visibility through `@clack/prompts`. It is a + // real full-screen-adjacent terminal owner: a raw Ctrl-Z becomes a SUB byte (not a kernel SIGTSTP), while an + // external SIGTSTP would otherwise stop the process with Clack's raw mode and hidden cursor still live. Keep its + // state separate from Ink's SuspendPort, then hand it off reversibly through the same job-control coordinator. + const onboardingTerminal = deps.onboardingTerminalLifecycle ?? defaultOnboardingTerminalLifecycle; + let onboardingActive = false; + let onboardingFinalized = false; + let onboardingRawReleased = false; + let onboardingCursorReleased = false; + + const reclaimOnboardingTerminal = (): void => { + if (!onboardingActive || onboardingFinalized) return; + // Restore raw input before Clack redraws/reads again. Each latch is independent: a transient cursor-write fault + // must remain retryable and cannot make us incorrectly claim that the terminal is wholly reclaimed. + if (onboardingRawReleased) { + try { + onboardingTerminal.setRawMode(true); + onboardingRawReleased = false; + } catch { + // A future recovery opportunity gets another exact retry. + } + } + if (onboardingCursorReleased) { + try { + writeControl(HIDE_CURSOR); + onboardingCursorReleased = false; + } catch { + // A future recovery opportunity gets another exact retry. + } + } + }; + + const releaseOnboardingTerminal = (): boolean => { + if (!onboardingActive) return true; // no Ink and no Clack ownership (the tiny pre-mount handoff window) + if (onboardingFinalized) return false; + // A prior partial release may have restored raw input but failed while hiding the cursor. Retry that exact + // reclaim before refusing a later stop request: terminal writes are fallible and retaining the latches without a + // recovery attempt would turn one transient stdout fault into a permanently unsuspendable Clack prompt. + if (onboardingRawReleased || onboardingCursorReleased) { + reclaimOnboardingTerminal(); + if (onboardingRawReleased || onboardingCursorReleased) return false; + } + // `runOnboardingWizard` also awaits keychain/config work after Clack has stopped its spinner. Do not manufacture + // raw mode + a hidden cursor when a stop lands in that ordinary cooked-input interval: Clack has nothing to + // hand over there, so a default SIGTSTP is already terminal-safe. + if (!onboardingTerminal.isRawMode()) return true; + try { + onboardingTerminal.setRawMode(false); + onboardingRawReleased = true; + // Mark before the write: a TTY write can throw after emitting a partial escape sequence. The rollback must + // conservatively re-hide the cursor in that case; a harmless extra HIDE is far safer than returning Clack to a + // live prompt with its cursor visible. + onboardingCursorReleased = true; + writeControl(SHOW_CURSOR); + return true; + } catch { + // We did not stop the process, so put any partial release back immediately rather than leaving Clack half-owned. + reclaimOnboardingTerminal(); + return false; + } + }; + + const restoreOnboardingTerminal = (): void => { + // On a normal wizard handoff Clack restores itself before `onboardingActive` is cleared. This path is only for a + // signal/process-exit net while the wizard still owns the terminal; it deliberately leaves cooked input + a + // visible cursor, never a reversible Clack reclaim behind a terminal teardown. + if (!onboardingActive && !onboardingRawReleased && !onboardingCursorReleased) return; + onboardingFinalized = true; + try { + onboardingTerminal.setRawMode(false); + onboardingRawReleased = false; + } catch { + // The overlapping exit net/finally retries the exact operation. + } + try { + writeControl(SHOW_CURSOR); + onboardingCursorReleased = false; + } catch { + // The overlapping exit net/finally retries the exact operation. + } + }; + /** * Undo every terminal mode this command turned on, in reverse order: unmount ink FIRST (leaving raw mode and the * alternate buffer), then disable bracketed paste (DECSET 2004, enabled by ink 7's `usePaste`) and mouse reporting @@ -214,6 +340,7 @@ export async function driveHome(deps: HomeDeps): Promise { let pasteDisabled = false; let mouseDisabled = false; const restoreTerminalControls = (): void => { + restoreOnboardingTerminal(); if (!unmounted) { try { instance?.unmount(); // restore the terminal from raw mode BEFORE anything else @@ -529,6 +656,7 @@ export async function driveHome(deps: HomeDeps): Promise { // saying so on raw stderr would land on the alt buffer for one frame. `onceEffortNotice` keeps a standing // condition — a stale `off` on a model that cannot disable thinking — from repeating every single turn. onEffortWithheld: onceEffortNotice((note) => store.notice(note)), + onListenerError: (note) => store.notice(note), onUnpriced: (note) => store.notice(note), }); return wireHomeChatSession(built, store, { open: true }); @@ -586,6 +714,7 @@ export async function driveHome(deps: HomeDeps): Promise { // A RESEAT binds a different model — precisely when a tier that was fine a moment ago stops being accepted. // Through `noteToStore`, like every sink here, so none can TDZ on the store declared below. onEffortWithheld: onceEffortNotice(noteToStore), + onListenerError: noteToStore, onUnpriced: noteToStore, }); // Seed the view store with the carried model + cost/turns — a resumed session never re-emits session:started, @@ -629,33 +758,11 @@ export async function driveHome(deps: HomeDeps): Promise { }); const exitProcess = deps.exit ?? ((code: number) => process.exit(code)); - // First-run onboarding (2.5.G S8): a truly KEY-LESS bare Home offers a `@clack` wizard to connect a provider - // BEFORE mounting ink — clack + ink both take the terminal's raw mode, so the wizard must fully settle (and - // clack restore the terminal) before `render()`. Already behind `shouldOpenHome`'s TTY/CI gate; a cancel or a - // keychain-write failure ends the wizard cleanly and the Home mounts key-less (retry, or add a key manually). - if (isProviderKeyless(providers)) { - await runOnboardingWizard({ - store: providerStore, - keychain, - resolver: providers, - io: deps.io, - // Reuse the SAME config-write target as the `/models` port (honors `--config`) so the wizard's starter - // model + a later `/models` pick + the started session all agree on one file (2.5.G S7/S8). - writeDefaultModel: (modelId, provider) => - writeGlobalPreferences( - { defaultModel: modelId, defaultProvider: provider }, - homeDir, - deps.global.configPath, - ), - ...(deps.onboardingPrompter === undefined ? {} : { prompter: deps.onboardingPrompter }), - }); - } - - // Bracketed paste (DECSET 2004) is enabled by ink 7's `usePaste` on mount (home-app.tsx). The defensive - // `DISABLE_BRACKETED_PASTE` writes in `restoreTerminalControls` are belt-and-suspenders (usePaste also - // disables on unmount) so an external signal can never leave the terminal in bracketed-paste mode. - // One external-signal lifecycle covering the Home, the in-Home chat, and MCP teardown. + // + // This is deliberately established BEFORE first-run onboarding. Its key-validation probe is network-bound; a + // signal arriving while that promise is pending must still restore the terminal / close the db, rather than + // falling through a listener-free window (#50). `controller` is optional below, so the pre-mount path is safe. let signaled = false; const onSignal = (signo: number): void => { // A KEYBOARD Ctrl-C during a `/scrollback` or `/edit` hatch arrives here as a REAL SIGINT: the suspension turns @@ -671,6 +778,10 @@ export async function driveHome(deps: HomeDeps): Promise { return; } signaled = true; + // Disarm the reversible handoff BEFORE the permanent teardown writes terminal state. In particular, a late + // SIGCONT must not resolve an old suspension and re-enable mouse/raw/1049 after this path has restored shell + // custody (TERM/HUP/QUIT while stopped is the adversarial case). + disposeJobControl?.(); // Best-effort terminal restore — it swallows its own throw, so it can NOT skip scheduling the bounded // teardown + exit below (else an external signal could neither close the db nor exit). restoreTerminalControls(); @@ -697,9 +808,77 @@ export async function driveHome(deps: HomeDeps): Promise { // The LAST net. `onSignal` covers the catchable kills; this covers everything that reaches Node's `'exit'` without // unwinding our `finally` — a `process.exit()` from a nested command, an uncaught throw, an unhandled rejection. // It must be synchronous, which `restoreTerminalControls` is; the latch makes the overlap with the other nets free. - unsubscribeProcessExit = (deps.subscribeProcessExit ?? defaultSubscribeProcessExit)( - restoreTerminalControls, - ); + unsubscribeProcessExit = (deps.subscribeProcessExit ?? defaultSubscribeProcessExit)(() => { + disposeJobControl?.(); + restoreTerminalControls(); + }); + + // SIGTSTP/SIGCONT are a SEPARATE, reversible lifecycle. `suspendFullScreen` gives Ink ownership of raw mode and + // Home's alt buffer while it temporarily releases our live mouse capture; on SIGCONT Ink restores both and forces + // a redraw. It never reaches the termination handler above, so it cannot close the db or end a live chat. + const jobControl = wireJobControl({ + suspendPort, + lifecycle: deps.jobControlLifecycle ?? defaultJobControlLifecycle, + run: (suspendTerminal, waitForContinue) => + suspendFullScreen( + { + suspendTerminal, + writeControl, + // Home mounts Ink with its OWN alternate-screen render option. `suspendFullScreen` must never double + // toggle 1049 here, even on an inline Home where `altActive` is false. + inkOwnsAltScreen: true, + altActive: altScreenActive, + mouseActive: mouseCaptured, + }, + waitForContinue, + ), + releaseWithoutInk: releaseOnboardingTerminal, + reclaimWithoutInk: reclaimOnboardingTerminal, + }); + disposeJobControl = jobControl.dispose; + + // First-run onboarding (2.5.G S8): a truly KEY-LESS bare Home offers a `@clack` wizard to connect a provider + // BEFORE mounting ink — clack + ink both take the terminal's raw mode, so the wizard must fully settle (and + // clack restore the terminal) before `render()`. The signal and job-control nets above are already live before + // the wizard can reach its network-bound key-validation request. + if (isProviderKeyless(providers)) { + onboardingActive = true; + onboardingFinalized = false; + // Clack's raw-mode listener is registered after this one, so this sees SUB first. Clack itself treats Ctrl-Z as + // neither submit nor cancel, but consuming the byte here makes the intended POSIX handoff explicit and keeps it + // out of every wizard prompt/secret field. + unsubscribeOnboardingInput = onboardingTerminal.onInput((input) => { + if (input.includes('\x1a')) jobControl.requestSuspend(); + }); + try { + await runOnboardingWizard({ + store: providerStore, + keychain, + resolver: providers, + io: deps.io, + // Reuse the SAME config-write target as the `/models` port (honors `--config`) so the wizard's starter + // model + a later `/models` pick + the started session all agree on one file (2.5.G S7/S8). + writeDefaultModel: (modelId, provider) => + writeGlobalPreferences( + { defaultModel: modelId, defaultProvider: provider }, + homeDir, + deps.global.configPath, + ), + ...(deps.onboardingPrompter === undefined ? {} : { prompter: deps.onboardingPrompter }), + }); + } finally { + unsubscribeOnboardingInput?.(); + unsubscribeOnboardingInput = undefined; + // A real process cannot execute this while stopped. It makes an injected/failing lifecycle safe, and is a + // no-op after a normal continue where the reversible coordinator already reclaimed Clack's state. + reclaimOnboardingTerminal(); + onboardingActive = false; + } + } + + // Bracketed paste (DECSET 2004) is enabled by ink 7's `usePaste` on mount (home-app.tsx). The defensive + // `DISABLE_BRACKETED_PASTE` writes in `restoreTerminalControls` are belt-and-suspenders (usePaste also + // disables on unmount) so an external signal can never leave the terminal in bracketed-paste mode. // Resolve the effective render mode (2.6.F, ADR-0068 §e). driveHome only runs on a TTY interactive path // (shouldOpenHome-gated), so the output mode is 'tui'; the resolver still short-circuits a 'plain' path to @@ -751,6 +930,9 @@ export async function driveHome(deps: HomeDeps): Promise { alternateScreen, // `RootApp` attaches ink's `suspendTerminal` here while mounted (2.6.F Step 5d, ADR-0068 §e). suspendPort, + // Raw Ink mode delivers Ctrl-Z as a key event, not a process SIGTSTP. Route it into the same coordinator as + // an external `kill -TSTP`, so both paths release/reclaim terminal state identically. + onSuspend: jobControl.requestSuspend, // The branded banner's durable switch (Step 5g); `HomeView` owns the empty-Home rule when it is absent. showBanner: config.showBanner, // Armed only while the in-Home chat owns the screen (Step 6g). `mouseActive` is the RESOLVED mode @@ -791,6 +973,8 @@ export async function driveHome(deps: HomeDeps): Promise { // terminal can never leak the session or the db handle. unsubscribeSignals?.(); unsubscribeProcessExit?.(); + unsubscribeOnboardingInput?.(); + disposeJobControl?.(); restoreTerminalControls(); await controller?.teardownActive().catch(() => undefined); // always reclaim a live session closeDb(); // always close the shared db diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.ts index dd74dbb0..8aa7cd77 100644 --- a/apps/cli/src/index.ts +++ b/apps/cli/src/index.ts @@ -1,9 +1,16 @@ +import { escalateExitCode, installBackgroundFailureNet } from './process/background-failure.js'; import { EXIT_CODES } from './process/exit-codes.js'; import { processIo } from './process/io.js'; import { run } from './run.js'; -// The `bin` entry: wire the real-process IO seam, run, and set the deterministic exit code. +// The `bin` entry: wire the real-process IO seam, install the background-failure net, run, and set the +// deterministic exit code. const io = processIo(); + +// #228 — installed BEFORE `run()`, so a rejection during startup is caught too. The module doc explains why +// the process survives one rather than dying on it, and why the net is bounded and sanitized. +const net = installBackgroundFailureNet(io); + try { process.exitCode = await run(process.argv, io); } catch (err) { @@ -15,3 +22,9 @@ try { } process.exitCode = EXIT_CODES.workflowFailed; } + +// The mirror of the in-handler upgrade, for a rejection that fired BEFORE `run()` assigned the code and would +// otherwise have been overwritten by it. Same rule: only a clean `0` is escalated. +if (net.occurred()) { + escalateExitCode(); +} diff --git a/apps/cli/src/process/background-failure.test.ts b/apps/cli/src/process/background-failure.test.ts new file mode 100644 index 00000000..fdfd8920 --- /dev/null +++ b/apps/cli/src/process/background-failure.test.ts @@ -0,0 +1,148 @@ +import { afterEach, describe, expect, it } from 'vitest'; + +import { captureIo } from '../test-support.js'; +import { + describeReason, + escalateExitCode, + installBackgroundFailureNet, +} from './background-failure.js'; +import { EXIT_CODES } from './exit-codes.js'; + +/** CSI + OSC introducers, built from escapes so the bytes survive every editor and diff tool. */ +const CSI = '['; +const OSC = ']'; +const BEL = ''; + +/** + * The process-level net (#228). Its whole job is to keep the CLI alive after an unhandled rejection instead of + * letting Node kill it mid-session — so it is exactly the kind of code that silently rots without tests: the + * listener is attached to the real `process`, and nothing else in the CLI suite would notice its absence. + */ +describe('installBackgroundFailureNet (#228)', () => { + const originalExitCode = process.exitCode; + let dispose: (() => void) | undefined; + + afterEach(() => { + dispose?.(); + dispose = undefined; + process.exitCode = originalExitCode; + }); + + /** + * Drive one real unhandled rejection through Node and wait for the listener to see it. Typed `Error` + * because every case here rejects with one; the non-Error reasons `describeReason` must survive are + * covered by calling it directly, which is cheaper and does not need a real rejection. + */ + async function rejectOnce(reason: Error): Promise { + void Promise.reject(reason); + await new Promise((resolve) => setTimeout(resolve, 10)); + } + + it('reports a rejection and does NOT let it kill the process', async () => { + const { io, err } = captureIo(); + const net = installBackgroundFailureNet(io); + dispose = net.dispose; + process.exitCode = EXIT_CODES.success; + + await rejectOnce(new Error('database is locked')); + + expect(net.occurred()).toBe(true); + expect(err()).toContain('a background operation failed: database is locked'); + // Still running — reaching this assertion at all is itself the proof. + expect(process.exitCode).toBe(EXIT_CODES.workflowFailed); + }); + + it('upgrades a clean 0 but NEVER overwrites a more specific outcome', () => { + const { io } = captureIo(); + const net = installBackgroundFailureNet(io); + dispose = net.dispose; + + // chatEnded / gatePaused / invalidInvocation each carry a truth CI and scripts key on. + for (const code of [ + EXIT_CODES.chatEnded, + EXIT_CODES.gatePaused, + EXIT_CODES.invalidInvocation, + ]) { + process.exitCode = code; + escalateExitCode(); + expect(process.exitCode).toBe(code); + } + process.exitCode = EXIT_CODES.success; + escalateExitCode(); + expect(process.exitCode).toBe(EXIT_CODES.workflowFailed); + }); + + it('SANITIZES the reported reason — a rejection reason is untrusted text', async () => { + const { io, err } = captureIo(); + const net = installBackgroundFailureNet(io); + dispose = net.dispose; + // An MCP server's error text, a provider response body and a tool result can all land here, so the same + // ANSI/OSC guard every other terminal boundary applies has to apply here too (G34's class). + await rejectOnce(new Error(`lost ${CSI}31mred${OSC}0;pwned${BEL} write`)); + expect(err()).not.toContain(''); // no CSI/OSC introducer survives + expect(err()).not.toContain(BEL); // nor the OSC terminator + expect(err()).toContain('lost'); // …while the readable text still reaches the user + }); + + it('bounds repeated reports so a repeating fault cannot scroll a session away', async () => { + const { io, err } = captureIo(); + const net = installBackgroundFailureNet(io); + dispose = net.dispose; + for (let i = 0; i < 8; i += 1) { + await rejectOnce(new Error(`boom-${i}`)); + } + expect(err()).toContain('boom-4'); // the 5th, still reported in full + expect(err()).not.toContain('boom-5'); // the 6th, suppressed + expect(err()).toContain('further background failures suppressed'); + }); + + it('RELAVIUM_DEBUG lifts the bound — the suppression line must not promise what it does not deliver', async () => { + const { io, err } = captureIo({ RELAVIUM_DEBUG: '1' }); + const net = installBackgroundFailureNet(io); + dispose = net.dispose; + for (let i = 0; i < 8; i += 1) { + await rejectOnce(new Error(`boom-${i}`)); + } + // Every one reported, and no suppression notice claiming a flag would reveal them. + expect(err()).toContain('boom-7'); + expect(err()).not.toContain('further background failures suppressed'); + }); + + it('stops listening once disposed', async () => { + const { io, err } = captureIo(); + const net = installBackgroundFailureNet(io); + net.dispose(); + // A throwaway absorber, because this is the ONE case with no net installed: without it the rejection is + // genuinely unhandled, vitest counts it as a file-level error, and the run exits NON-ZERO while every + // summary line still says "passed" — the exact failure mode that hid this from me the first time. + const absorb = (): void => undefined; + process.on('unhandledRejection', absorb); + try { + await rejectOnce(new Error('after dispose')); + } finally { + process.off('unhandledRejection', absorb); + } + expect(net.occurred()).toBe(false); + expect(err()).toBe(''); + }); +}); + +describe('describeReason', () => { + it('carries the discriminant, not just the message — a bare message cannot identify a wiring gap', () => { + expect(describeReason(Object.assign(new Error('locked'), { code: 'SQLITE_BUSY' }))).toBe( + 'locked [SQLITE_BUSY]', + ); + }); + + it('handles the realistic non-Error shapes instead of collapsing them to "unknown"', () => { + // A driver-shaped plain object is exactly what `retry.ts` matches structurally, so it is a real case. + expect(describeReason({ code: 'SQLITE_BUSY', message: 'database is locked' })).toBe( + 'database is locked', + ); + expect(describeReason('a bare string')).toBe('a bare string'); + expect(describeReason(new AggregateError([new Error('a'), new Error('b')]))).toContain( + '2 background errors', + ); + expect(describeReason(42)).toBe('unknown reason'); + }); +}); diff --git a/apps/cli/src/process/background-failure.ts b/apps/cli/src/process/background-failure.ts new file mode 100644 index 00000000..c5360c33 --- /dev/null +++ b/apps/cli/src/process/background-failure.ts @@ -0,0 +1,104 @@ +import { sanitizeInline, stripTerminalControls } from '../render/sanitize.js'; +import { EXIT_CODES } from './exit-codes.js'; +import type { CliIo } from './io.js'; + +/** + * The process-level last-resort net for an unhandled promise rejection (#228). + * + * Node's default for one is to **kill the process**. That default is load-bearing in the wrong direction here: + * the engine's `RunEventBus` deliberately re-throws a passive subscriber's fault out-of-band, on a microtask, + * when no `onListenerError` sink is wired — precisely so it is never silently swallowed. The consequence was + * that a transient `history.db` write failure inside the chat persister took the whole CLI down + * **asynchronously, after the user had already seen the reply**, losing a turn they had been billed for. + * + * So: report it loudly and keep the process alive. This is a BACKSTOP, not a replacement for the sinks + * upstream of it — the store's bounded retry absorbs the transient case and the bus's `onListenerError` sink + * reports the genuine one; anything reaching here is a wiring gap worth seeing. Deliberately scoped to + * rejections: an `uncaughtException` is a different class of fault and continuing past one would be unsafe. + * + * Lives in its own module, rather than inline in the `bin` entry, so it is unit-testable — `index.ts` has a + * top-level `await run(...)` and importing it from a test would run the whole CLI. + */ +export interface BackgroundFailureNet { + /** Whether a background failure was reported during this process's lifetime. */ + readonly occurred: () => boolean; + /** Detach the listener (tests; the real process never needs it). */ + readonly dispose: () => void; +} + +/** How many distinct rejections are reported in full before the net starts counting instead of printing. */ +const MAX_REPORTED = 5; + +export function installBackgroundFailureNet(io: CliIo): BackgroundFailureNet { + let count = 0; + // With `RELAVIUM_DEBUG` set the bound is lifted entirely — otherwise the suppression line below would be + // making a promise ("set RELAVIUM_DEBUG to see them") the code does not keep, and a diagnostic flag that + // does not actually reveal the diagnostics is worse than no flag. + const verbose = io.env['RELAVIUM_DEBUG'] !== undefined; + const onRejection = (reason: unknown): void => { + count += 1; + if (verbose || count <= MAX_REPORTED) { + // Sanitized like every other terminal boundary (`process/render-error.ts`): a rejection reason is + // arbitrary `unknown` — realistically an MCP server's error text, a provider response body, or a tool + // result — i.e. exactly the untrusted sources the ANSI/OSC/Trojan-Source guard exists for. + io.writeErr( + `relavium: a background operation failed: ${sanitizeInline(describeReason(reason))}\n`, + ); + if (verbose && reason instanceof Error && reason.stack !== undefined) { + io.writeErr(`${stripTerminalControls(reason.stack)}\n`); + } + } else if (count === MAX_REPORTED + 1) { + // Bounded: a repeating fault in a long interactive session must not scroll the transcript away. + io.writeErr( + 'relavium: further background failures suppressed (set RELAVIUM_DEBUG to see them).\n', + ); + } + // Fires after `run()` already set the code: only ever upgrade a clean `0`, never overwrite a more + // specific outcome (a `chatEnded` 4 or a `gatePaused` 3 still means exactly what it says). + escalateExitCode(); + }; + process.on('unhandledRejection', onRejection); + return { + occurred: () => count > 0, + dispose: () => { + process.off('unhandledRejection', onRejection); + }, + }; +} + +/** + * Upgrade a clean success to a failure, and nothing else. A run that lost a durable write must not report + * `0` — but a `chatEnded` (4), `gatePaused` (3) or `invalidInvocation` (2) already carries a more specific + * truth that CI and scripts key on, so those survive untouched. + */ +export function escalateExitCode(): void { + if (process.exitCode === undefined || process.exitCode === EXIT_CODES.success) { + process.exitCode = EXIT_CODES.workflowFailed; + } +} + +/** A one-line, user-safe description of a rejection reason — never a stack as primary output. */ +export function describeReason(reason: unknown): string { + if (reason instanceof AggregateError) { + return `${reason.errors.length} background errors: ${reason.errors + .slice(0, 3) + .map((e: unknown) => describeReason(e)) + .join('; ')}`; + } + if (reason instanceof Error) { + // Include the discriminant when there is one — a bare message often cannot identify the wiring gap. + const code = 'code' in reason && typeof reason.code === 'string' ? ` [${reason.code}]` : ''; + return `${reason.message}${code}`; + } + if (typeof reason === 'string') { + return reason; + } + // A driver-shaped plain object (`{ code, message }`) is the realistic non-Error case on this path. + if (typeof reason === 'object' && reason !== null && 'message' in reason) { + const message: unknown = reason.message; + if (typeof message === 'string') { + return message; + } + } + return 'unknown reason'; +} diff --git a/apps/cli/src/process/render-error.test.ts b/apps/cli/src/process/render-error.test.ts new file mode 100644 index 00000000..a337b38e --- /dev/null +++ b/apps/cli/src/process/render-error.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from 'vitest'; + +import { CliError } from './errors.js'; +import type { CliIo } from './io.js'; +import { renderError } from './render-error.js'; + +const ATTACK = '\x1b[2J\x1b]52;c;Zm9v\x07\x1bPtmux;\x1b\\\x9b2J\r\b\u202E'; + +function captureIo(): { readonly io: CliIo; readonly stderr: () => string } { + const writes: string[] = []; + return { + io: { + writeOut: () => {}, + writeErr: (text) => { + writes.push(text); + }, + env: {}, + stdoutIsTty: false, + stdinIsTty: false, + stdin: process.stdin, + }, + stderr: () => writes.join(''), + }; +} + +function messageFromJson(value: unknown): string { + if ( + typeof value !== 'object' || + value === null || + !('message' in value) || + typeof value.message !== 'string' + ) { + throw new Error('expected an error JSON envelope with a string message'); + } + return value.message; +} + +describe('renderError terminal projection', () => { + it('sanitizes the human diagnostic and its verbose stack without suppressing readable content (G34)', () => { + const error = new CliError('invalid_invocation', `message visible${ATTACK}\nforged message`); + error.stack = `stack visible${ATTACK}\nsecond stack line`; + const capture = captureIo(); + + renderError(error, { json: false, verbose: true }, capture.io); + + const output = capture.stderr(); + for (const forbidden of ['\x1b', '\x9b', '\r', '\b', '\u202E']) { + expect(output).not.toContain(forbidden); + } + expect(output).toContain('message visible'); + expect(output).toContain('stack visible'); + expect(output).toContain('second stack line'); // a stack remains readable multiline prose + }); + + it('sanitizes JSON diagnostic values and never appends a raw verbose stack', () => { + const error = new CliError('invalid_invocation', `message visible${ATTACK}\nforged message`); + error.stack = `stack visible${ATTACK}`; + const capture = captureIo(); + + renderError(error, { json: true, verbose: true }, capture.io); + + const output = capture.stderr(); + expect(output).not.toContain('\u202E'); + expect(output).not.toContain('stack visible'); + const message = messageFromJson(JSON.parse(output)); + for (const forbidden of ['\x1b', '\x9b', '\r', '\b', '\u202E']) { + expect(message).not.toContain(forbidden); + } + expect(message).toContain('message visible'); + }); +}); diff --git a/apps/cli/src/process/render-error.ts b/apps/cli/src/process/render-error.ts index 039be0b2..30bda994 100644 --- a/apps/cli/src/process/render-error.ts +++ b/apps/cli/src/process/render-error.ts @@ -1,5 +1,6 @@ import { toUserFacing } from './errors.js'; import type { CliIo } from './io.js'; +import { sanitizeInline, stripTerminalControls } from '../render/sanitize.js'; /** * Render a fatal error once, at the top-level boundary (error-handling.md: the internal → @@ -17,17 +18,19 @@ export function renderError( io: CliIo, ): void { const userFacing = toUserFacing(value); + // The error boundary is shared by every command, including JSON diagnostics and verbose stacks. Keep the + // structured envelope valid while also removing terminal/bidi controls before it reaches stderr. + const code = sanitizeInline(userFacing.code); + const message = sanitizeInline(userFacing.message); if (opts.json) { - io.writeErr( - JSON.stringify({ type: 'error', code: userFacing.code, message: userFacing.message }) + '\n', - ); + io.writeErr(JSON.stringify({ type: 'error', code, message }) + '\n'); } else { - io.writeErr(`relavium: ${userFacing.message}\n`); + io.writeErr(`relavium: ${message}\n`); } // The raw stack is a human-mode debugging affordance only — never under `--json`, where it would // mix non-JSON text into the structured stderr diagnostic (error-handling.md: a stack is not machine // output). Under `--json` the `{ type: 'error', … }` envelope is the whole stderr diagnostic. if (opts.verbose && !opts.json && value instanceof Error && value.stack !== undefined) { - io.writeErr(value.stack + '\n'); + io.writeErr(stripTerminalControls(value.stack) + '\n'); } } diff --git a/apps/cli/src/render/alt-screen.test.ts b/apps/cli/src/render/alt-screen.test.ts index a0c1188f..474031d5 100644 --- a/apps/cli/src/render/alt-screen.test.ts +++ b/apps/cli/src/render/alt-screen.test.ts @@ -219,3 +219,69 @@ describe('createAltScreenController — a failed restore must not disarm the lat expect(alt.isEntered()).toBe(false); }); }); + +/** The chat REPL has a tiny no-Ink interval between `/clear`/reseat sessions while its hoisted 1049 controller stays + * alive. G0 uses this reversible pair there: job control must hand the shell a normal primary buffer, then reclaim + * the exact state on SIGCONT — never call the FINAL `restore()` just because the process was briefly stopped. */ +describe('createAltScreenController — reversible job-control release', () => { + it('temporarily releases and reclaims the live buffer without tripping the final restore latch', () => { + const out: string[] = []; + const alt = createAltScreenController({ + write: (sequence) => out.push(sequence), + active: true, + }); + alt.enter(); + + expect(alt.releaseForSuspend()).toBe(true); + expect(out).toEqual([ENTER_SEQ, EXIT_SEQ]); + expect(alt.isEntered()).toBe(false); + expect(alt.isMouseEnabled()).toBe(false); + + alt.reclaimAfterSuspend(); + expect(out).toEqual([ENTER_SEQ, EXIT_SEQ, ENTER_SEQ]); + expect(alt.isEntered()).toBe(true); + expect(alt.isMouseEnabled()).toBe(true); + + alt.restore(); // still a live controller — final teardown remains available after a suspend/resume cycle + expect(out).toEqual([ENTER_SEQ, EXIT_SEQ, ENTER_SEQ, EXIT_SEQ]); + }); + + it('does not double-release, and a failed temporary release never invents a reclaim', () => { + const writes: string[] = []; + let failRelease = true; + const alt = createAltScreenController({ + write: (sequence) => { + if (sequence === EXIT_SEQ && failRelease) { + failRelease = false; + throw new Error('EIO'); + } + writes.push(sequence); + }, + active: true, + }); + alt.enter(); + + expect(alt.releaseForSuspend()).toBe(false); + alt.reclaimAfterSuspend(); + expect(writes).toEqual([ENTER_SEQ]); + + expect(alt.releaseForSuspend()).toBe(true); + expect(alt.releaseForSuspend()).toBe(false); + expect(writes).toEqual([ENTER_SEQ, EXIT_SEQ]); + }); + + it('a final restore while temporarily released keeps the primary buffer visible', () => { + const out: string[] = []; + const alt = createAltScreenController({ + write: (sequence) => out.push(sequence), + active: true, + }); + alt.enter(); + alt.releaseForSuspend(); + alt.restore(); + alt.reclaimAfterSuspend(); // final state wins: a process that exits while stopped must not re-enter 1049 + + expect(out).toEqual([ENTER_SEQ, EXIT_SEQ]); + expect(alt.isEntered()).toBe(false); + }); +}); diff --git a/apps/cli/src/render/alt-screen.ts b/apps/cli/src/render/alt-screen.ts index 8204c266..fb4c57f5 100644 --- a/apps/cli/src/render/alt-screen.ts +++ b/apps/cli/src/render/alt-screen.ts @@ -56,6 +56,14 @@ export interface AltScreenController { restore(): void; /** Clear the persistent alt buffer between sessions (see {@link CLEAR_ALT_SCREEN}). No-op once restored / inactive. */ clearBetween(): void; + /** + * Temporarily give the terminal back for POSIX job control while a chat re-drive has no mounted Ink tree. Unlike + * {@link restore}, this is reversible: SIGCONT calls {@link reclaimAfterSuspend} and the REPL remains alive. + * Returns `false` only when the release write itself failed or a second release is already in flight. + */ + releaseForSuspend(): boolean; + /** Re-enter the buffer after {@link releaseForSuspend}; no-op unless this controller released it successfully. */ + reclaimAfterSuspend(): void; /** Whether the alt buffer is currently entered (and not yet restored) — for a caller that must know the live state. */ readonly isEntered: () => boolean; } @@ -77,6 +85,7 @@ export function createAltScreenController(opts: { const mouse = opts.mouse ?? true; let entered = false; let restored = false; + let temporarilyReleased = false; return { enter: (): void => { if (!active || entered) return; @@ -87,6 +96,13 @@ export function createAltScreenController(opts: { }, restore: (): void => { if (!entered || restored) return; // never exit a buffer we never entered; never exit twice + // A job-control stop already wrote the exact exit sequence and left the terminal on its primary buffer. Mark + // the controller final without writing it again: a SIGTERM delivered while the process is stopped must not + // double-toggle 1049 back INTO the alt buffer on its way out. + if (temporarilyReleased) { + restored = true; + return; + } // Disable mouse reporting FIRST (restore native selection), then exit the alt buffer + show the cursor. The // DISABLE is UNCONDITIONAL even when `mouse` is off: a disable of a mode that was never enabled is a no-op, and // an unconditional teardown can never strand DECSET-1002 if the option is ever mis-threaded. @@ -108,10 +124,30 @@ export function createAltScreenController(opts: { restored = true; }, clearBetween: (): void => { - if (!entered || restored) return; + if (!entered || restored || temporarilyReleased) return; write(CLEAR_ALT_SCREEN); }, - isEntered: (): boolean => entered && !restored, - isMouseEnabled: (): boolean => mouse && entered && !restored, + releaseForSuspend: (): boolean => { + if (!entered || restored) return true; // inline / inactive: no terminal state belongs to this controller + if (temporarilyReleased) return false; // the coordinator gates this; a duplicate must not write 1049 twice + try { + write(DISABLE_MOUSE + EXIT_ALT_SCREEN + SHOW_CURSOR); + } catch { + return false; // never claim a release we could not complete — no phantom reclaim on SIGCONT + } + temporarilyReleased = true; + return true; + }, + reclaimAfterSuspend: (): void => { + if (!temporarilyReleased || restored) return; + try { + write(ENTER_ALT_SCREEN + HIDE_CURSOR + (mouse ? ENABLE_MOUSE : '')); + } catch { + return; // leave the latch up so a later recovery opportunity can retry the exact missing reclaim + } + temporarilyReleased = false; + }, + isEntered: (): boolean => entered && !restored && !temporarilyReleased, + isMouseEnabled: (): boolean => mouse && entered && !restored && !temporarilyReleased, }; } diff --git a/apps/cli/src/render/renderer.test.ts b/apps/cli/src/render/renderer.test.ts index 8fab26ea..9851d52e 100644 --- a/apps/cli/src/render/renderer.test.ts +++ b/apps/cli/src/render/renderer.test.ts @@ -49,6 +49,42 @@ describe('createPlainRenderer', () => { expect(text).toContain('run completed'); }); + it('SANITIZES at the single write point, so a later arm cannot reopen the hole (G34/#57 class)', () => { + // The third leaf of the same "one seam, three renderers" split lane (c) hardened. The guard sits at the + // write, not on today's fields, so an arm added later inherits it. A CI log is a terminal too. + const { io, out } = captureIo(); + createPlainRenderer(io).onEvent( + ev({ + type: 'node:started', + nodeType: 'agent', + nodeId: 'n\u001b]52;c;ZXZpbA==\u0007\u001b[2J\u202Egnp', + }), + ); + const text = out(); + expect(text).not.toContain('\u001b'); // no CSI/OSC introducer + expect(text).not.toContain('\u0007'); // no OSC terminator + expect(text).not.toContain('\u202E'); // no Trojan-Source bidi override + expect(text).toContain('n'); // …the readable part still renders + }); + + it('keeps a legitimately multi-line line as multiple rows while neutralizing each', () => { + // `node:completed` emits the ok line plus one row per produced media handle; sanitizing the whole string + // must not collapse those rows into one. + const { io, out } = captureIo(); + createPlainRenderer(io).onEvent( + ev({ + type: 'node:started', + nodeType: 'agent', + nodeId: 'plain', + }), + ); + expect( + out() + .split('\n') + .filter((l) => l.length > 0), + ).toHaveLength(1); + }); + it('surfaces a node failure with its error code', () => { const { io, out } = captureIo(); createPlainRenderer(io).onEvent( diff --git a/apps/cli/src/render/renderer.ts b/apps/cli/src/render/renderer.ts index 26e5b31e..1fd23b0c 100644 --- a/apps/cli/src/render/renderer.ts +++ b/apps/cli/src/render/renderer.ts @@ -2,6 +2,7 @@ import { collectDurableMediaHandles, type RunEvent } from '@relavium/shared'; import type { CliIo } from '../process/io.js'; import { formatProducedMedia } from './tui/format.js'; +import { sanitizeInline } from './sanitize.js'; /** * A renderer consumes the run's canonical {@link RunEvent} stream. The renderers below sit behind this one @@ -45,13 +46,27 @@ export function createJsonRenderer(io: CliIo): RunRenderer { }; } -/** Minimal human renderer — a terse line per lifecycle event; the no-TTY / CI fallback beside the ink TUI (2.E). */ +/** + * Minimal human renderer — a terse line per lifecycle event; the no-TTY / CI fallback beside the ink TUI (2.E). + * + * Sanitized at the SINGLE write, not per field (G34/#57's class). Lane (c) hardened the TUI leaf and four + * specific call sites; this is the third leaf of the same "one seam, three renderers" split, and putting the + * guard at the boundary rather than on today's fields means an arm added later cannot reopen the hole by + * forgetting it. The fields that matter today are authored `nodeId`s and a provider-declared media `mimeType` + * (never verified against the bytes — finding #107), so the exposure is narrower than the human-gate message + * that `status.ts`/`gate-list.ts` printed, but it is the same class and the same fix. + * + * `sanitizeInline` rather than `stripTerminalControls`: a lifecycle line is one row, so an embedded newline + * would forge extra rows in a CI log just as it would on a terminal. + */ export function createPlainRenderer(io: CliIo): RunRenderer { return { onEvent: (event) => { const line = describe(event); if (line !== undefined) { - io.writeOut(`${line}\n`); + // Per line, so a legitimate multi-line describe() (node:completed's media handles) keeps its rows + // while each row is individually neutralized. + io.writeOut(`${line.split('\n').map(sanitizeInline).join('\n')}\n`); } }, }; diff --git a/apps/cli/src/render/run-job-control.test.ts b/apps/cli/src/render/run-job-control.test.ts new file mode 100644 index 00000000..230e21fb --- /dev/null +++ b/apps/cli/src/render/run-job-control.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { HIDE_CURSOR, SHOW_CURSOR } from './alt-screen.js'; +import { wireRunJobControl } from './run-job-control.js'; +import type { JobControlLifecycle } from './suspend.js'; + +/** + * G0's residue on the run/gate path. ink hides the cursor while it renders and `signal-exit`'s signal list + * contains no `SIGTSTP`, so a Ctrl-Z during `relavium run` used to return the user to a shell with a + * permanently invisible cursor — for the rest of that terminal's life, not just the run's. + */ +function harness(supported = true) { + let onSuspend: (() => void) | undefined; + let onContinue: (() => void) | undefined; + const removeSuspend = vi.fn(); + const removeContinue = vi.fn(); + const suspendSelf = vi.fn(); + const written: string[] = []; + const lifecycle: JobControlLifecycle = { + supported, + onSuspend: (listener) => { + onSuspend = listener; + return removeSuspend; + }, + onContinue: (listener) => { + onContinue = listener; + return removeContinue; + }, + suspendSelf, + }; + return { + lifecycle, + written, + removeSuspend, + removeContinue, + suspendSelf, + write: (text: string) => written.push(text), + fireSuspend: () => onSuspend?.(), + fireContinue: () => onContinue?.(), + out: () => written.join(''), + }; +} + +describe('wireRunJobControl (G0, the run/gate path)', () => { + it('SHOWS the cursor before stopping, then re-raises so the process actually suspends', () => { + const h = harness(); + const jc = wireRunJobControl({ write: h.write, lifecycle: h.lifecycle }); + try { + h.fireSuspend(); + // Order matters: once stopped we run no code, so the restore has to precede the stop. + expect(h.out()).toBe(SHOW_CURSOR); + // Re-raised — swallowing Ctrl-Z would be worse than the bug: the user pressed a key that did nothing. + expect(h.suspendSelf).toHaveBeenCalledTimes(1); + } finally { + jc.dispose(); + } + }); + + it('HIDES it again on SIGCONT, because ink resumes drawing', () => { + const h = harness(); + const jc = wireRunJobControl({ write: h.write, lifecycle: h.lifecycle }); + try { + h.fireSuspend(); + h.fireContinue(); + expect(h.out()).toBe(SHOW_CURSOR + HIDE_CURSOR); + } finally { + jc.dispose(); + } + }); + + it('dispose removes BOTH listeners — the leak shape suspend.ts already had', () => { + const h = harness(); + const jc = wireRunJobControl({ write: h.write, lifecycle: h.lifecycle }); + jc.dispose(); + expect(h.removeSuspend).toHaveBeenCalledTimes(1); + expect(h.removeContinue).toHaveBeenCalledTimes(1); + // Idempotent: the caller's `finally` also runs on the throw path. + jc.dispose(); + expect(h.removeSuspend).toHaveBeenCalledTimes(1); + }); + + it('is a no-op where job control does not exist, rather than half-wired', () => { + const h = harness(false); + const jc = wireRunJobControl({ write: h.write, lifecycle: h.lifecycle }); + h.fireSuspend(); + expect(h.out()).toBe(''); + expect(h.suspendSelf).not.toHaveBeenCalled(); + expect(() => jc.dispose()).not.toThrow(); + }); +}); diff --git a/apps/cli/src/render/run-job-control.ts b/apps/cli/src/render/run-job-control.ts new file mode 100644 index 00000000..274065f0 --- /dev/null +++ b/apps/cli/src/render/run-job-control.ts @@ -0,0 +1,65 @@ +import { HIDE_CURSOR, SHOW_CURSOR } from './alt-screen.js'; +import { defaultJobControlLifecycle, type JobControlLifecycle } from './suspend.js'; + +/** + * Minimal `SIGTSTP`/`SIGCONT` handling for the **run/gate** path (G0's residue). + * + * Deliberately NOT `wireJobControl`. That primitive serves the full-screen chat/Home surfaces: it releases and + * reclaims the alternate screen and mouse reporting, coalesces concurrent requests, and owns a suspend hatch. + * `driveRun` mounts none of that — `RunApp` uses no `useInput`, so the terminal stays in cooked mode and Ctrl-Z + * is a real `SIGTSTP` rather than a keystroke the app reads. Reusing the heavy primitive here would add a + * release/reclaim of state that was never entered. + * + * What `driveRun` DOES leave behind is narrower and was genuinely broken: ink hides the cursor while it renders, + * and `signal-exit`'s signal list contains no `SIGTSTP`, so nothing restored it. A Ctrl-Z during + * `relavium run` therefore returned the user to a shell with a **permanently invisible cursor** — for the rest + * of that terminal's life, not just the run's. + * + * So: show the cursor before we stop, hide it again when we are foregrounded and ink resumes drawing. Nothing + * else, because nothing else was entered. + */ +export interface RunJobControl { + /** Detach the listeners. Idempotent; safe to call from a `finally` that also runs on the throw path. */ + readonly dispose: () => void; +} + +export interface WireRunJobControlOptions { + /** Where the cursor codes go — the same seam the renderers write through, so tests capture them. */ + readonly write: (text: string) => void; + /** Injectable signal registration (`defaultJobControlLifecycle` in production; a fake in tests). */ + readonly lifecycle?: JobControlLifecycle; +} + +export function wireRunJobControl(opts: WireRunJobControlOptions): RunJobControl { + const lifecycle = opts.lifecycle ?? defaultJobControlLifecycle; + // Windows has no job control: `chmod`-style no-op rather than a half-wired handler (ADR-0050's precedent for + // a documented per-platform divergence). + if (!lifecycle.supported) { + return { dispose: () => undefined }; + } + + let disposed = false; + const removeSuspend = lifecycle.onSuspend(() => { + // Restore the cursor BEFORE the process stops. Once stopped we run no code, so there is no later chance — + // and the shell the user lands in inherits whatever state we left. + opts.write(SHOW_CURSOR); + // Re-raise so the process actually stops. Without this the handler swallows Ctrl-Z and the run keeps + // going, which is worse than the bug being fixed: the user pressed a key that did nothing. + lifecycle.suspendSelf(); + }); + const removeContinue = lifecycle.onContinue(() => { + // Foregrounded: ink is drawing again, so hide the cursor as it expects. If the run already finished while + // we were stopped, the extra hide is harmless — the renderer's own teardown shows it again. + opts.write(HIDE_CURSOR); + }); + + return { + dispose: () => { + if (disposed) return; + disposed = true; + // BOTH, not just one — the leak `suspend.ts`'s own `dispose` had (see its comment) is exactly this shape. + removeSuspend(); + removeContinue(); + }, + }; +} diff --git a/apps/cli/src/render/suspend.test.ts b/apps/cli/src/render/suspend.test.ts index 009e2787..024bb0ac 100644 --- a/apps/cli/src/render/suspend.test.ts +++ b/apps/cli/src/render/suspend.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; import { DISABLE_MOUSE, @@ -8,7 +8,14 @@ import { HIDE_CURSOR, SHOW_CURSOR, } from './alt-screen.js'; -import { createSuspendPort, suspendFullScreen, type SuspendFullScreenOptions } from './suspend.js'; +import { + createSuspendPort, + defaultJobControlLifecycle, + suspendFullScreen, + wireJobControl, + type JobControlLifecycle, + type SuspendFullScreenOptions, +} from './suspend.js'; /** * The suspend-full-screen primitive (2.6.F Step 5d, ADR-0068 §e). These pin the contract the `/scrollback` and @@ -296,3 +303,281 @@ describe('createSuspendPort — the suspension window', () => { expect(port.current()).toBeUndefined(); }); }); + +/** + * G0 — POSIX job control is intentionally built on the SAME suspension primitive as `/scrollback` and `/edit`, not + * on an irreversible termination handler. These use a named signal seam: emitting a real SIGTSTP in a test would + * stop the Vitest worker, so the harness proves the release → self-stop → CONT → reclaim ordering without a shell. + */ +describe('wireJobControl (G0)', () => { + interface Harness { + readonly trace: string[]; + readonly port: ReturnType; + readonly lifecycle: JobControlLifecycle; + readonly suspendSelf: ReturnType; + readonly fireSuspend: () => void; + readonly fireContinue: () => void; + readonly removeSuspend: ReturnType; + readonly removeContinue: ReturnType; + } + + const flush = (): Promise => new Promise((resolve) => setTimeout(resolve, 0)); + + const harness = (): Harness => { + const trace: string[] = []; + const port = createSuspendPort(); + let onSuspend: (() => void) | undefined; + let onContinue: (() => void) | undefined; + const removeSuspend = vi.fn(); + const removeContinue = vi.fn(); + const suspendSelf = vi.fn(() => trace.push('self-stop')); + const lifecycle: JobControlLifecycle = { + supported: true, + onSuspend: (listener) => { + onSuspend = listener; + return () => { + removeSuspend(); + if (onSuspend === listener) onSuspend = undefined; + }; + }, + onContinue: (listener) => { + onContinue = listener; + return () => { + removeContinue(); + if (onContinue === listener) onContinue = undefined; + }; + }, + suspendSelf, + }; + return { + trace, + port, + lifecycle, + suspendSelf, + fireSuspend: () => onSuspend?.(), + fireContinue: () => onContinue?.(), + removeSuspend, + removeContinue, + }; + }; + + const bind = (h: Harness) => + wireJobControl({ + suspendPort: h.port, + lifecycle: h.lifecycle, + run: (suspendTerminal, waitForContinue) => + suspendFullScreen( + { + suspendTerminal, + writeControl: (sequence) => h.trace.push(sequence), + inkOwnsAltScreen: false, + altActive: true, + mouseActive: true, + }, + waitForContinue, + ), + }); + + it('releases terminal state BEFORE self-stop, then reclaims and redraws only after SIGCONT', async () => { + const h = harness(); + h.port.attach(async (callback) => { + h.trace.push('ink:begin'); + try { + await callback(); + } finally { + h.trace.push('ink:redraw'); + } + }); + const binding = bind(h); + + h.fireSuspend(); + await flush(); + expect(h.trace).toEqual([ + 'ink:begin', + DISABLE_MOUSE, + EXIT_ALT_SCREEN + SHOW_CURSOR, + 'self-stop', + ]); + expect(h.port.isSuspended()).toBe(true); + + h.fireContinue(); + await flush(); + expect(h.trace).toEqual([ + 'ink:begin', + DISABLE_MOUSE, + EXIT_ALT_SCREEN + SHOW_CURSOR, + 'self-stop', + ENTER_ALT_SCREEN + HIDE_CURSOR, + ENABLE_MOUSE, + 'ink:redraw', + ]); + expect(h.port.isSuspended()).toBe(false); + binding.dispose(); + }); + + it('coalesces a fast second stop request until the first reclaim/redraw is complete', async () => { + const h = harness(); + h.port.attach(async (callback) => callback()); + const binding = bind(h); + + h.fireContinue(); // no pending wait: must not redraw, reclaim, or throw + h.fireSuspend(); + await flush(); + expect(h.suspendSelf).toHaveBeenCalledTimes(1); + expect(h.trace.filter((entry) => entry === DISABLE_MOUSE)).toHaveLength(1); + + // A second request lands after the self-stop returns but before CONT has let Ink reclaim. It is retained rather + // than consumed by the re-armed listener and silently discarded. + h.fireSuspend(); + expect(h.suspendSelf).toHaveBeenCalledTimes(1); + h.fireContinue(); + await flush(); + expect(h.suspendSelf).toHaveBeenCalledTimes(2); + h.fireContinue(); + await flush(); + binding.dispose(); + }); + + it('replays a captured TSTP only AFTER an explicitly held Ink redraw settles', async () => { + const h = harness(); + let releaseRedraw: (() => void) | undefined; + let holdNextRedraw = true; + h.port.attach(async (callback) => { + h.trace.push('ink:begin'); + await callback(); + if (holdNextRedraw) { + holdNextRedraw = false; + await new Promise((resolve) => { + releaseRedraw = resolve; + }); + } + h.trace.push('ink:redraw'); + }); + const binding = bind(h); + + h.fireSuspend(); + await flush(); + h.fireContinue(); + await flush(); + // Reclaim has started, but the fake Ink outer promise (its force-redraw boundary) has not returned yet. + h.fireSuspend(); + expect(h.suspendSelf).toHaveBeenCalledTimes(1); + + if (releaseRedraw === undefined) throw new Error('the first redraw was not held'); + releaseRedraw(); + await flush(); + expect(h.suspendSelf).toHaveBeenCalledTimes(2); + + h.fireContinue(); + await flush(); + binding.dispose(); + }); + + it('keeps no-Ink/onboarding stops re-armed after each continuation', () => { + const h = harness(); + const binding = bind(h); // no `port.attach` ⇒ the onboarding/no-Ink branch + + h.fireSuspend(); + h.fireSuspend(); + + expect(h.suspendSelf).toHaveBeenCalledTimes(2); + binding.dispose(); + }); + + it('dispose removes the SIGTSTP listener too, not just SIGCONT (no listener leak)', () => { + // `wireJobControl` installs a SIGTSTP listener at construction; `dispose` used to remove only the SIGCONT + // one. Every rebuild of the session — a `/clear`, a `/model` reseat, an onboarding→Home handoff — left + // another SIGTSTP listener behind, so one Ctrl-Z eventually ran several handlers and Node would warn about + // a leak. The harness's `removeSuspend` spy is the direct observation. + const h = harness(); + const binding = bind(h); + expect(h.removeSuspend).not.toHaveBeenCalled(); + binding.dispose(); + expect(h.removeSuspend).toHaveBeenCalledTimes(1); + // …and a Ctrl-Z after disposal reaches nothing. + h.fireSuspend(); + expect(h.suspendSelf).not.toHaveBeenCalled(); + }); + + it('permanent disposal suppresses a late SIGCONT reclaim behind terminal teardown', async () => { + const h = harness(); + h.port.attach(async (callback) => callback()); + const binding = bind(h); + + h.fireSuspend(); + await flush(); + binding.dispose(); + h.fireContinue(); // the listener was removed; a late foregrounding event cannot re-enter 1049/mouse + await flush(); + + expect(h.trace).toEqual([DISABLE_MOUSE, EXIT_ALT_SCREEN + SHOW_CURSOR, 'self-stop']); + expect(h.removeContinue).toHaveBeenCalledTimes(1); + }); + + it('does not nest Ink suspension while a hatch already owns it', async () => { + const h = harness(); + let releaseHatch: (() => void) | undefined; + h.port.attach( + (callback) => + new Promise((resolve) => { + void callback().then(resolve); + }), + ); + const binding = bind(h); + const hatch = h.port.current()?.( + () => + new Promise((resolve) => { + releaseHatch = resolve; + }), + ); + await flush(); + expect(h.port.isSuspended()).toBe(true); + + h.fireSuspend(); + expect(h.suspendSelf).toHaveBeenCalledTimes(1); + expect(h.trace).toEqual(['self-stop']); // no second mouse/alt release behind the hatch owner's back + + if (releaseHatch === undefined) + throw new Error('the hatch did not enter its suspension window'); + releaseHatch(); + await hatch; + binding.dispose(); + }); + + it('unsupported platforms register nothing and make raw Ctrl-Z a no-op', () => { + const h = harness(); + const binding = wireJobControl({ + suspendPort: h.port, + lifecycle: { ...h.lifecycle, supported: false }, + run: () => Promise.resolve(), + }); + binding.requestSuspend(); + expect(h.suspendSelf).not.toHaveBeenCalled(); + expect(h.removeSuspend).not.toHaveBeenCalled(); + expect(h.removeContinue).not.toHaveBeenCalled(); + }); + + it('the production lifecycle adds and removes SIGTSTP/SIGCONT only on POSIX', (ctx) => { + // `ctx.skip()` rather than an early `return`: a bare return reports a PASS on Windows, so the suite would + // claim to have verified signal registration on a platform where it never ran. + if (process.platform === 'win32') { + ctx.skip(); + return; + } + const before = { + suspend: process.listenerCount('SIGTSTP'), + continue: process.listenerCount('SIGCONT'), + }; + const removeSuspend = defaultJobControlLifecycle.onSuspend(() => undefined); + const removeContinue = defaultJobControlLifecycle.onContinue(() => undefined); + try { + expect(process.listenerCount('SIGTSTP')).toBe(before.suspend + 1); + expect(process.listenerCount('SIGCONT')).toBe(before.continue + 1); + } finally { + removeSuspend(); + removeContinue(); + } + expect(process.listenerCount('SIGTSTP')).toBe(before.suspend); + expect(process.listenerCount('SIGCONT')).toBe(before.continue); + }); +}); diff --git a/apps/cli/src/render/suspend.ts b/apps/cli/src/render/suspend.ts index 524e6492..c732fc92 100644 --- a/apps/cli/src/render/suspend.ts +++ b/apps/cli/src/render/suspend.ts @@ -183,3 +183,200 @@ export function createSuspendPort(): SuspendPort { isSuspended: () => suspended, }; } + +/** + * The named, POSIX-only job-control signal seam. `SIGTSTP`/`SIGCONT` are deliberately NOT folded into the normal + * termination callbacks: stopping a foreground job is reversible, whereas SIGINT/TERM/HUP/QUIT tear the command + * down, close its database, and exit. Keeping the names here also avoids treating platform-specific signal numbers + * as a public contract. + */ +export interface JobControlLifecycle { + /** `false` on platforms without POSIX job control (notably Windows): no listeners are installed and Ctrl-Z is inert. */ + readonly supported: boolean; + /** Register the catchable stop request; returns its exact remover. */ + readonly onSuspend: (listener: () => void) => () => void; + /** Register the continuation notification; it remains installed while the process is stopped. */ + readonly onContinue: (listener: () => void) => () => void; + /** Re-deliver SIGTSTP to this process after our listener has released terminal state and removed itself. */ + readonly suspendSelf: () => void; +} + +const noop = (): void => undefined; + +/** Production process binding for {@link JobControlLifecycle}. No unsupported Windows signal is ever registered. */ +export const defaultJobControlLifecycle: JobControlLifecycle = { + supported: process.platform !== 'win32', + onSuspend: (listener) => { + if (process.platform === 'win32') return noop; + process.on('SIGTSTP', listener); + return () => process.removeListener('SIGTSTP', listener); + }, + onContinue: (listener) => { + if (process.platform === 'win32') return noop; + process.on('SIGCONT', listener); + return () => process.removeListener('SIGCONT', listener); + }, + suspendSelf: () => { + if (process.platform !== 'win32') process.kill(process.pid, 'SIGTSTP'); + }, +}; + +/** + * Perform an Ink-aware terminal release and wait for a matching SIGCONT. `run` must use {@link suspendFullScreen} + * (rather than calling Ink directly), preserving each surface's proven alt-screen and mouse ownership rules. + */ +export type JobControlRun = ( + suspendTerminal: SuspendTerminal, + waitForContinue: () => Promise, +) => Promise; + +/** The reversible job-control binding returned by {@link wireJobControl}. */ +export interface JobControlBinding { + /** Request a stop from either a process SIGTSTP or the raw-mode Ctrl-Z input byte. */ + readonly requestSuspend: () => void; + /** Remove the process listeners; safe to call more than once. */ + readonly dispose: () => void; +} + +/** + * Bind POSIX job control to one mounted Ink surface. + * + * The critical ordering is enforced in one place: + * + * 1. `run` enters Ink's suspension window and releases mouse/alt state. + * 2. only inside that released window do we remove our SIGTSTP listener and re-deliver SIGTSTP to the process; + * 3. SIGCONT re-arms SIGTSTP before resolving the wait, after which Ink restores raw input and redraws. A second + * stop received while that reclaim is in flight is remembered and replayed only after Ink has finished its redraw; + * it is never silently swallowed. + * + * A hatch may already own `SuspendPort` (`/scrollback`/`/edit`), and a chat rebuild has a brief interval with no Ink + * tree at all. Those paths deliberately do not nest Ink suspension: they release/reclaim only the optional hoisted + * terminal state around the real stop, then let the existing owner resume normally. + */ +export function wireJobControl(opts: { + readonly suspendPort: SuspendPort; + readonly lifecycle: JobControlLifecycle; + readonly run: JobControlRun; + /** Optional reversible release for an interval with no Ink tree (the chat hoist's rebuild window). */ + readonly releaseWithoutInk?: () => boolean; + /** Mirrors {@link releaseWithoutInk} after the stopped process is foregrounded again. */ + readonly reclaimWithoutInk?: () => void; +}): JobControlBinding { + if (!opts.lifecycle.supported) return { requestSuspend: noop, dispose: noop }; + + let disposed = false; + let suspending = false; + let pendingSuspend = false; + let resolveContinue: (() => void) | undefined; + let removeSuspend: (() => void) | undefined; + let removeContinue: (() => void) | undefined; + + const removeSuspendListener = (): void => { + removeSuspend?.(); + removeSuspend = undefined; + }; + + const installSuspendListener = (): void => { + if (disposed || removeSuspend !== undefined) return; + removeSuspend = opts.lifecycle.onSuspend(requestSuspend); + }; + + const continueSuspension = (): void => { + const resolve = resolveContinue; + if (resolve === undefined) return; // a stray SIGCONT must not perturb an active surface + // Re-arm BEFORE Ink resumes input/redraws. A fast second Ctrl-Z is then captured by our handler rather than + // falling through to a default stop while the terminal is still being reclaimed. + installSuspendListener(); + resolveContinue = undefined; + resolve(); + }; + + /** Remove our handler just long enough for the OS default SIGTSTP action to stop the process. */ + const forwardSuspend = (): void => { + removeSuspendListener(); + try { + opts.lifecycle.suspendSelf(); + } catch { + // A failed self-signal means there was no stop/continuation. Resolve a pending release so Ink can reclaim; + // the finally below restores our SIGTSTP listener too. + continueSuspension(); + } finally { + // `suspendSelf()` returns only after SIGCONT. This is the re-arm for the no-Ink and hatch-owned paths; the + // Ink-owned path already re-arms in `continueSuspension` before it resolves the deferred, so this is a no-op. + installSuspendListener(); + } + }; + + const settle = (): void => { + resolveContinue = undefined; + suspending = false; + installSuspendListener(); + // `SIGCONT` deliberately re-arms before Ink starts reclaiming terminal state: an external SIGTSTP can otherwise + // default-stop the process in the tiny interval after 1049/raw input has been restored but before the mouse/ + // redraw have completed. The coordinator is still busy, so retain ONE request and replay it after the terminal is + // coherent again. `queueMicrotask` also keeps the signal callback itself synchronous and side-effect bounded. + queueMicrotask(() => { + if (disposed || suspending || !pendingSuspend) return; + pendingSuspend = false; + requestSuspend(); + }); + }; + + function requestSuspend(): void { + if (disposed) return; + if (suspending) { + // Coalesce a burst into one real follow-up stop. Dropping it would make a fast Ctrl-Z / SIGTSTP after `fg` + // appear to succeed (the listener consumed it) while leaving the process running. + pendingSuspend = true; + return; + } + + // A hatch has already released Ink's input and its terminal modes. Nesting `suspendTerminal` would throw + // "already suspended" and could reclaim the hatch's terminal behind its back, so just perform genuine POSIX job + // control and let the still-pending hatch resume after foregrounding. + if (opts.suspendPort.isSuspended()) { + forwardSuspend(); + return; + } + + const suspendTerminal = opts.suspendPort.current(); + if (suspendTerminal === undefined) { + // Before Home mounts (onboarding) there is no Ink state to release. In the chat's rebuild window there can be + // a live hoisted alt buffer, supplied by the reversible hooks below. + if (opts.releaseWithoutInk !== undefined && !opts.releaseWithoutInk()) return; + forwardSuspend(); + opts.reclaimWithoutInk?.(); + return; + } + + suspending = true; + const waitForContinue = (): Promise => + new Promise((resolve) => { + resolveContinue = resolve; + forwardSuspend(); + }); + // Signals cannot await a promise. Absorb a terminal-write/Ink failure here: `suspendFullScreen` has already + // performed its own partial-state reclaim, and an unhandled rejection from a signal handler is strictly worse. + // We intentionally do not re-raise after a failed release because terminal release never reached the safe point. + void opts.run(suspendTerminal, waitForContinue).catch(noop).finally(settle); + } + + removeContinue = opts.lifecycle.onContinue(continueSuspension); + installSuspendListener(); + + return { + requestSuspend, + dispose: (): void => { + if (disposed) return; + disposed = true; + pendingSuspend = false; + // BOTH listeners, not just SIGCONT: `wireJobControl` installs the SIGTSTP one at construction, so + // removing only the continue handler leaked a suspend handler per binding. Every session rebuild — a + // `/clear`, a `/model` reseat, the onboarding→Home handoff — added another, so one Ctrl-Z eventually ran + // several handlers and Node warned about a listener leak. + removeSuspendListener(); + removeContinue?.(); + removeContinue = undefined; + }, + }; +} diff --git a/apps/cli/src/render/tui/RunApp.test.tsx b/apps/cli/src/render/tui/RunApp.test.tsx new file mode 100644 index 00000000..c6768a7b --- /dev/null +++ b/apps/cli/src/render/tui/RunApp.test.tsx @@ -0,0 +1,72 @@ +import { cleanup, render } from 'ink-testing-library'; +import { afterEach, describe, expect, it } from 'vitest'; + +import { waitFor } from './harness-util.js'; +import { RunApp } from './RunApp.js'; +import type { RunStore, RunStoreSnapshot } from './run-store.js'; +import type { RunViewState } from './run-view-model.js'; + +afterEach(cleanup); + +/** + * A deliberately mixed terminal-control payload: CSI cursor/erase, OSC-52 clipboard, DCS passthrough, + * C1 CSI, C0 carriage-return/backspace, and a bidi override. Each value below also carries a readable + * label, so this regression proves the display retains content while removing the control channel. + */ +const ATTACK = '\x1b[2J\x1b]52;c;Zm9v\x07\x1bPtmux;\x1b\\\x9b2J\r\b\u202E'; + +function staticStore(state: RunViewState): RunStore { + const snapshot: RunStoreSnapshot = { state, tick: 0, color: false }; + return { + subscribe: () => () => {}, + getSnapshot: () => snapshot, + }; +} + +describe('RunApp terminal projection', () => { + it('removes terminal and bidi controls from every untrusted display field in a real Ink frame (#56)', async () => { + const nodeId = `node visible${ATTACK}\nforged row`; + const state: RunViewState = { + nodeOrder: [nodeId], + nodes: { + [nodeId]: { + nodeId, + status: 'failed', + errorCode: `error visible${ATTACK}\nforged suffix`, + }, + }, + activeNodeId: nodeId, + activeModel: `model visible${ATTACK}\nforged model`, + activeTokens: `token visible${ATTACK}\nsecond token visible`, + toolLines: [`tool visible${ATTACK}\nforged tool row`], + cumulativeCostMicrocents: 0, + gapDetected: false, + warnings: [`warning visible${ATTACK}\nforged warning row`], + producedMedia: [ + { + nodeId: `media node visible${ATTACK}\nforged media node`, + mimeType: `image/png visible${ATTACK}\nforged mime`, + handle: `media://visible${ATTACK}\nforged handle`, + }, + ], + }; + + const { lastFrame } = render(); + await waitFor(() => (lastFrame() ?? '').includes('token visible')); + const frame = lastFrame() ?? ''; + + // Ink is explicitly color-disabled above. Any remaining ESC/C1/C0/bidi byte therefore came from the + // untrusted payload, rather than the renderer's own control protocol. + for (const forbidden of ['\x1b', '\x9b', '\r', '\b', '\u202E']) { + expect(frame).not.toContain(forbidden); + } + expect(frame).toContain('node visible'); + expect(frame).toContain('error visible'); + expect(frame).toContain('model visible'); + expect(frame).toContain('token visible'); + expect(frame).toContain('tool visible'); + expect(frame).toContain('warning visible'); + expect(frame).toContain('image/png visible'); + expect(frame).toContain('media://visible'); + }); +}); diff --git a/apps/cli/src/render/tui/RunApp.tsx b/apps/cli/src/render/tui/RunApp.tsx index e9916458..4f63be18 100644 --- a/apps/cli/src/render/tui/RunApp.tsx +++ b/apps/cli/src/render/tui/RunApp.tsx @@ -12,6 +12,7 @@ import { import { colorProps, dimProps, nodeSuffix } from './projection.js'; import type { RunStore } from './run-store.js'; import { MAX_ACTIVE_TOKEN_LINES, type NodeView } from './run-view-model.js'; +import { sanitizeInline, stripTerminalControls } from '../sanitize.js'; /** * The thin `ink` projection of the {@link RunStore}'s snapshot (workstream **2.E**). It holds NO logic of @@ -28,8 +29,8 @@ function NodeLine( const glyph = node.status === 'running' ? spinnerFrame(tick) : statusGlyph(node.status); return ( - {glyph} {node.nodeId} - {nodeSuffix(node)} + {glyph} {sanitizeInline(node.nodeId)} + {sanitizeInline(nodeSuffix(node))} ); } @@ -60,14 +61,14 @@ export function RunApp(props: Readonly<{ store: RunStore }>): ReactElement { {activeNode !== undefined && activeLines.length > 0 ? ( - ▌ {activeNode.nodeId} - {state.activeModel === undefined ? '' : ` · ${state.activeModel}`} + ▌ {sanitizeInline(activeNode.nodeId)} + {state.activeModel === undefined ? '' : ` · ${sanitizeInline(state.activeModel)}`} {/* `truncate-end` bounds each logical line to one terminal row — a newline-free token blast or a narrow terminal can't blow the live region up to dozens of wrapped rows (§2.E narrow-terminal). */} {activeLines.map((line, i) => ( - {line} + {stripTerminalControls(line)} ))} @@ -78,7 +79,7 @@ export function RunApp(props: Readonly<{ store: RunStore }>): ReactElement { {state.toolLines.map((line, i) => ( - {line} + {sanitizeInline(line)} ))} @@ -90,7 +91,7 @@ export function RunApp(props: Readonly<{ store: RunStore }>): ReactElement { {state.producedMedia.map((media) => ( // The content-addressed handle is unique (the view-model dedups by it) + stable — a sound React key. - {formatProducedMedia(media)} + {sanitizeInline(formatProducedMedia(media))} ))} @@ -101,7 +102,7 @@ export function RunApp(props: Readonly<{ store: RunStore }>): ReactElement { {state.warnings.map((w, i) => ( - ⚠ {w} + ⚠ {sanitizeInline(w)} ))} diff --git a/apps/cli/src/render/tui/chat-app.test.tsx b/apps/cli/src/render/tui/chat-app.test.tsx index 460f8ea5..adc81420 100644 --- a/apps/cli/src/render/tui/chat-app.test.tsx +++ b/apps/cli/src/render/tui/chat-app.test.tsx @@ -43,7 +43,10 @@ afterEach(cleanup); /** Mount `ChatApp` with the minimal REQUIRED props (no optional ports) — the surface under test is the ink * lifecycle + the raw-mode input/paste handlers, so the driver callbacks are inert stubs. */ -function mountChat(store: ChatStoreController): ReturnType { +function mountChat( + store: ChatStoreController, + opts: { onSuspend?: () => void } = {}, +): ReturnType { // Self-policing fixture (2.6.C): this helper mounts INLINE, so its store must carry the INLINE bound — the pairing // production builds. A divergent fixture would otherwise pass on a DEAD TREE: the tripwire throws inside the // component and ink swallows it (no-op error callbacks), leaving an empty frame and a green test. @@ -56,6 +59,7 @@ function mountChat(store: ChatStoreController): ReturnType { onExit={() => {}} onError={() => {}} onModeChange={() => {}} + {...(opts.onSuspend === undefined ? {} : { onSuspend: opts.onSuspend })} />, ); } @@ -64,6 +68,7 @@ const approvalReq: ToolApprovalRequest = { toolId: 'write_file', action: 'fs_write', preview: { path: 'notes.md' }, + unredactedPreview: { path: 'notes.md' }, }; const turnStarted = (timestamp: string): SessionStreamHandleEvent => ({ @@ -73,6 +78,19 @@ const turnStarted = (timestamp: string): SessionStreamHandleEvent => ({ timestamp, }); +describe('ChatApp — raw Ctrl-Z job-control routing (G0)', () => { + it('routes the raw control byte to onSuspend exactly once and never submits prompt text', async () => { + const onSuspend = vi.fn(); + const h = mountChat(createChatStore(false, undefined, INLINE_TRANSCRIPT_BOUND), { onSuspend }); + await waitFor(() => (h.lastFrame() ?? '').length > 0); + + h.stdin.write('\x1a'); + await waitFor(() => onSuspend.mock.calls.length === 1); + expect(onSuspend).toHaveBeenCalledTimes(1); + expect(h.lastFrame() ?? '').not.toContain('^Z'); + }); +}); + describe('ChatApp bracketed paste — ink-7 usePaste channel (ADR-0068)', () => { it('inserts an idle paste into the compose buffer', async () => { const store = createChatStore(false, undefined, INLINE_TRANSCRIPT_BOUND); diff --git a/apps/cli/src/render/tui/chat-ink.tsx b/apps/cli/src/render/tui/chat-ink.tsx index d83e28b3..f730ae9e 100644 --- a/apps/cli/src/render/tui/chat-ink.tsx +++ b/apps/cli/src/render/tui/chat-ink.tsx @@ -235,6 +235,8 @@ interface ChatAppProps { * mounted, which is the ONLY way the non-React slash dispatch can reach `/scrollback` and `/edit`. Absent ⇒ the * hatches surface an honest "needs an interactive terminal" notice. */ readonly suspendPort?: SuspendPort | undefined; + /** Request the shared POSIX job-control flow. Ink raw mode delivers Ctrl-Z as input, not SIGTSTP. */ + readonly onSuspend?: (() => void) | undefined; /** Write the mouse selection to the system clipboard (OSC 52, 2.6.F Step 6). Absent ⇒ selection still highlights * but copy-on-select is inert (a driver/test that wires no terminal). */ readonly clipboard?: ((text: string) => ClipboardOutcome) | undefined; @@ -1149,6 +1151,12 @@ export function ChatApp(props: Readonly): ReactElement { }; useInput((char, key) => { + // Raw Ink input represents Ctrl-Z as Ctrl+`z` (and a few test streams retain the literal SUB byte). It must win + // over mouse parsing, overlays, and approval routing: job control is never prompt text and never consent. + if (char === '\x1a' || (key.ctrl === true && key.meta !== true && char.toLowerCase() === 'z')) { + props.onSuspend?.(); + return; + } // Mouse reports (Step 5): the alt screen enables mouse reporting, so a wheel/click arrives in EVERY state — // including while an overlay owns the keyboard. CONSUME every report HERE, ahead of the overlay routing below, // so its raw bytes can never type into the prompt, the `/` palette filter, or the `[c]` reason capture. The wheel @@ -1740,6 +1748,7 @@ export function driveInk(ctx: ChatDriveContext): Promise { // `!`-shell runner (2.5.D, ADR-0061) — interactive-only; absent ⇒ a leading `!` is a literal message. runShellCommand: ctx.runShellCommand, suspendPort: ctx.suspendPort, + onSuspend: ctx.onSuspend, clipboard: ctx.clipboard, }), { diff --git a/apps/cli/src/render/tui/final-summary.test.ts b/apps/cli/src/render/tui/final-summary.test.ts index 9805d4ca..7476649d 100644 --- a/apps/cli/src/render/tui/final-summary.test.ts +++ b/apps/cli/src/render/tui/final-summary.test.ts @@ -156,4 +156,69 @@ describe('renderFinalSummary', () => { ]); expect(renderFinalSummary(state)).not.toContain('produced media'); }); + + it('projects every untrusted terminal value safely while retaining readable final-summary content (#57)', () => { + const attack = '\x1b[2J\x1b]52;c;Zm9v\x07\x1bPtmux;\x1b\\\x9b2J\r\b\u202E'; + const nodeId = `node visible${attack}\nforged node row`; + const state: RunViewState = { + ...initialRunViewState(), + nodeOrder: [nodeId], + nodes: { + [nodeId]: { + nodeId, + status: 'failed', + errorCode: `node error visible${attack}\nforged node suffix`, + }, + }, + producedMedia: [ + { + nodeId: `media node visible${attack}\nforged media node`, + mimeType: `image/png visible${attack}\nforged mime`, + handle: `media://visible${attack}\nforged handle`, + }, + ], + summary: { + outcome: 'failed', + errorCode: `run error visible${attack}\nforged run suffix`, + errorMessage: `failure visible${attack}\nsecond failure line`, + }, + }; + const paused: RunViewState = { + ...state, + summary: { + outcome: 'paused', + pausedGateIds: [`gate visible${attack}\nforged gate`], + }, + }; + + const output = `${renderFinalSummary(state)}${renderFinalSummary(paused)}`; + for (const forbidden of ['\x1b', '\x9b', '\r', '\b', '\u202E']) { + expect(output).not.toContain(forbidden); + } + expect(output).toContain('run error visible'); + expect(output).toContain('failure visible'); + expect(output).toContain('second failure line'); // summary errors stay readable after a newline collapses + expect(output).toContain('node visible'); + expect(output).toContain('node error visible'); + expect(output).toContain('image/png visible'); + expect(output).toContain('media://visible'); + expect(output).toContain('media node visible'); + expect(output).toContain('gate visible'); + }); + + it('keeps an untrusted failure message on its one summary row (it cannot forge a node status)', () => { + const state: RunViewState = { + ...initialRunViewState(), + summary: { + outcome: 'failed', + errorMessage: 'provider failed\n ✓ deploy completed', + }, + }; + + const lines = renderFinalSummary(state).trimEnd().split('\n'); + expect(lines).toHaveLength(2); // headline + one labeled error row; no forged third status row + expect(lines[1]).toContain('provider failed'); + expect(lines[1]).toContain('✓ deploy completed'); + expect(lines).not.toContain(' ✓ deploy completed'); + }); }); diff --git a/apps/cli/src/render/tui/final-summary.ts b/apps/cli/src/render/tui/final-summary.ts index 61d35501..a5167057 100644 --- a/apps/cli/src/render/tui/final-summary.ts +++ b/apps/cli/src/render/tui/final-summary.ts @@ -7,6 +7,7 @@ import { } from './format.js'; import { nodeSuffix } from './projection.js'; import type { RunViewState } from './run-view-model.js'; +import { sanitizeInline } from '../sanitize.js'; /** * Render the **persistent** final summary the `ink` renderer writes after it unmounts (workstream **2.E**). @@ -23,7 +24,8 @@ export function renderFinalSummary(state: RunViewState): string { case 'completed': return 'run completed'; case 'failed': { - const code = summary.errorCode === undefined ? '' : ` (${summary.errorCode})`; + const code = + summary.errorCode === undefined ? '' : ` (${sanitizeInline(summary.errorCode)})`; return `run failed${code}`; } case 'cancelled': @@ -32,7 +34,7 @@ export function renderFinalSummary(state: RunViewState): string { const gates = summary.pausedGateIds === undefined || summary.pausedGateIds.length === 0 ? '' - : ` at gate ${summary.pausedGateIds.join(', ')}`; + : ` at gate ${summary.pausedGateIds.map(sanitizeInline).join(', ')}`; return `run paused${gates}`; } default: @@ -53,7 +55,9 @@ export function renderFinalSummary(state: RunViewState): string { lines.push(meta.join(' · ')); if (summary?.errorMessage !== undefined) { - lines.push(` ${summary.errorMessage}`); + // A summary error occupies ONE structural row. Collapse line/column separators as well as terminal + // controls, so provider/model text cannot forge a believable node-status row beneath the headline. + lines.push(` ${sanitizeInline(summary.errorMessage)}`); } for (const id of state.nodeOrder) { @@ -62,7 +66,9 @@ export function renderFinalSummary(state: RunViewState): string { continue; } // Reuse the live view's suffix logic (one source of truth) — completed→duration, failed→error code, etc. - lines.push(` ${statusGlyph(node.status)} ${id}${nodeSuffix(node)}`); + lines.push( + ` ${statusGlyph(node.status)} ${sanitizeInline(id)}${sanitizeInline(nodeSuffix(node))}`, + ); } // The run's media deliverables — the durable handle per produced artifact (never bytes), attributed to its @@ -70,7 +76,9 @@ export function renderFinalSummary(state: RunViewState): string { if (state.producedMedia.length > 0) { lines.push(' produced media:'); for (const media of state.producedMedia) { - lines.push(` ${formatProducedMedia(media)} (${media.nodeId})`); + lines.push( + ` ${sanitizeInline(formatProducedMedia(media))} (${sanitizeInline(media.nodeId)})`, + ); } } diff --git a/apps/cli/src/render/tui/home-app.test.tsx b/apps/cli/src/render/tui/home-app.test.tsx index 496bd6f5..825165a8 100644 --- a/apps/cli/src/render/tui/home-app.test.tsx +++ b/apps/cli/src/render/tui/home-app.test.tsx @@ -150,6 +150,8 @@ function mountHome( color?: boolean; /** Record the mouse-capture toggles `RootApp` requests (2.6.F Step 6g). */ setMouseCapture?: (enabled: boolean) => void; + /** Capture the raw Ctrl-Z job-control request. */ + onSuspend?: () => void; } = {}, ): MountedHome { let onResize: () => void = () => {}; @@ -183,6 +185,7 @@ function mountHome( {...(opts.clipboard === undefined ? {} : { clipboard: opts.clipboard })} {...(opts.showBanner === undefined ? {} : { showBanner: opts.showBanner })} {...(opts.setMouseCapture === undefined ? {} : { setMouseCapture: opts.setMouseCapture })} + {...(opts.onSuspend === undefined ? {} : { onSuspend: opts.onSuspend })} />, ); return { @@ -204,6 +207,20 @@ async function enterChat(c: HomeController): Promise { await waitFor(() => c.getSnapshot().mode === 'chat'); // startChat resolves on a microtask } +describe('RootApp — raw Ctrl-Z job-control routing (G0)', () => { + it('routes the raw control byte to onSuspend without changing the Home controller state', async () => { + const onSuspend = vi.fn(); + const m = mountHome(createChatStore(false, undefined, INLINE_TRANSCRIPT_BOUND), { onSuspend }); + await settleFrames(); + + m.harness.stdin.write('\x1a'); + await waitFor(() => onSuspend.mock.calls.length === 1); + expect(onSuspend).toHaveBeenCalledTimes(1); + expect(m.c.getSnapshot().mode).toBe('home'); + expect(m.c.getSnapshot().input.text).toBe(''); + }); +}); + describe('RootApp (Home) bracketed paste — usePaste → controller.handlePaste wiring (ADR-0068)', () => { it('inserts an idle paste into the in-Home chat buffer', async () => { const store = createChatStore(false, undefined, INLINE_TRANSCRIPT_BOUND); diff --git a/apps/cli/src/render/tui/home-app.tsx b/apps/cli/src/render/tui/home-app.tsx index 617b289c..06014696 100644 --- a/apps/cli/src/render/tui/home-app.tsx +++ b/apps/cli/src/render/tui/home-app.tsx @@ -67,6 +67,9 @@ export interface RootAppProps { * mounted — the ONLY way the non-React slash dispatch (`createHomeController` is built before this tree exists) * can reach `/scrollback` and `/edit`. Absent (a test) ⇒ the hatches notice "needs an interactive terminal". */ readonly suspendPort?: SuspendPort | undefined; + /** Request POSIX job control. In Ink raw mode Ctrl-Z is input, not a kernel SIGTSTP, so this routes both forms + * through the shared release → stop → reclaim coordinator. Absent on a focused component test ⇒ a harmless no-op. */ + readonly onSuspend?: (() => void) | undefined; /** Write the in-Home chat's mouse selection to the system clipboard over OSC 52 (2.6.F Step 6). Absent ⇒ the * selection still highlights but copy-on-select is inert (a test that wires no terminal). */ readonly clipboard?: ((text: string) => ClipboardOutcome) | undefined; @@ -380,6 +383,15 @@ export function RootApp(props: Readonly): ReactElement { }; useInput((input, key) => { + // Ink raw mode parses Ctrl-Z as Ctrl+`z` (some harnesses surface the literal SUB byte). Consume it BEFORE mouse, + // overlays, and the Home controller so it can neither type into a prompt nor answer a pending approval. + if ( + input === '\x1a' || + (key.ctrl === true && key.meta !== true && input.toLowerCase() === 'z') + ) { + props.onSuspend?.(); + return; + } if (consumeMouseReport(input)) return; if (altChat) { // Esc DISMISSES a live selection. Gated on an IDLE chat: while a turn streams, Esc is the mid-turn ABORT that diff --git a/apps/cli/src/test-support.ts b/apps/cli/src/test-support.ts index b8cf2def..deaf7bb4 100644 --- a/apps/cli/src/test-support.ts +++ b/apps/cli/src/test-support.ts @@ -51,7 +51,11 @@ export const GENERATIVE_IMAGE_CAPABILITY_FLAGS: CapabilityFlags = CapabilityFlag * can assert on the exact stdout (NDJSON / human lines) and stderr (diagnostics) a command produced. * Shared by the command tests and the 2.K regression harness so the capture shape never diverges. */ -export function captureIo(): { io: CliIo; out: () => string; err: () => string } { +export function captureIo(env: Readonly> = {}): { + io: CliIo; + out: () => string; + err: () => string; +} { const outChunks: string[] = []; const errChunks: string[] = []; const io: CliIo = { @@ -61,7 +65,7 @@ export function captureIo(): { io: CliIo; out: () => string; err: () => string } writeErr: (text) => { errChunks.push(text); }, - env: {}, + env, stdoutIsTty: false, stdinIsTty: false, // An empty, already-ended stub: a chat test that exercises the plain loop overrides it with its own stream; diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 9bd0031d..391376d7 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -23,6 +23,7 @@ flowchart TB LLM["packages/llm
LLMProvider seam · adapters over official SDKs
fallback chains · cost · tool normalize"] Shared["packages/shared
Zod schemas + types"] DB["packages/db
Drizzle schema (SQLite + Postgres)"] + MCP["packages/mcp
inbound MCP client · SDK fence
JSON-Schema→Zod compiler · SSRF-floored transports"] UI["packages/ui
shadcn/ui components"] end @@ -127,6 +128,9 @@ canonical definitions and per-package purpose are in Gemini / DeepSeek plus fallback chains, tool-schema normalization, and cost accounting — no 3rd-party framework, and no vendor type crosses the seam. See [multi-llm-providers.md](multi-llm-providers.md). +- **`packages/mcp`** — the inbound MCP client: the SDK-fenced package, the dependency-free + JSON-Schema→Zod compiler, and the `http`/`sse`/`websocket` transports behind the SSRF + pre-connect floor ([ADR-0052](../decisions/0052-inbound-mcp-client-package-lifecycle-registration.md), [ADR-0053](../decisions/0053-mcp-network-transport-egress-security.md)). - **`packages/shared`** — the Zod schemas and TypeScript types every package imports (workflow, agent, run, node, edge, run-event, cost-event). The schema shapes are documented under [../reference/contracts/](../reference/contracts/). diff --git a/docs/decisions/0028-workflow-resource-governance.md b/docs/decisions/0028-workflow-resource-governance.md index e5192873..68cb68cd 100644 --- a/docs/decisions/0028-workflow-resource-governance.md +++ b/docs/decisions/0028-workflow-resource-governance.md @@ -4,6 +4,8 @@ - **Date**: 2026-06-05 - **Related**: [0008-local-first-phase-1-cloud-phase-2.md](0008-local-first-phase-1-cloud-phase-2.md), [0014-managed-metering-quota-and-billing.md](0014-managed-metering-quota-and-billing.md), [../reference/contracts/workflow-yaml-spec.md](../reference/contracts/workflow-yaml-spec.md), [../reference/contracts/sse-event-schema.md](../reference/contracts/sse-event-schema.md), [../reference/contracts/ipc-contract.md](../reference/contracts/ipc-contract.md), [../architecture/execution-model.md](../architecture/execution-model.md) +> **Amended 2026-07-30 by [ADR-0074](0074-durable-conservative-budget-commitments.md) — a refinement, not a reversal.** ADR-0074 keeps this ADR's estimate-and-block pre-egress cap and its `on_exceed` branches exactly as decided, and adds a **third** cap-consuming quantity beside realized cost and live admissions: a *durable conservative commitment*, retained when a provider may already have been billed but returned no trustworthy usage. The projection becomes `realized + conservativeCommitted + liveAdmissions + nextWorstCaseEstimate`, so the cap survives a partial stream, a crash and a resume. ADR-0074 is **Accepted** as of 2026-07-30; its implementation is staged, so the behaviour described below is what ships until §2–§5 land. + > **Amended 2026-06-18 by [ADR-0044](0044-media-access-governance-read-media-save-to-cost.md).** A refinement, not a reversal: ADR-0044 adds a **disjoint per-modality media cost class** to this ADR's pre-egress governor — it widens the pre-egress hook to carry `outputModalities`/a media-unit estimate and folds the media estimate into the **existing** `max_cost_microcents` cap (no new cap dimension, no new event/error class). This ADR's budget / timeout / concurrency decisions are unchanged. ## Context diff --git a/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md b/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md index 11b6159a..857e800b 100644 --- a/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md +++ b/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md @@ -4,6 +4,8 @@ - **Date**: 2026-06-13 - **Related**: [ADR-0003](0003-pure-ts-engine-not-langgraph-python.md), [ADR-0011](0011-internal-llm-abstraction.md), [ADR-0018](0018-desktop-execution-and-rust-egress.md), [ADR-0022](0022-run-references-workflow-by-uuid.md), [ADR-0024](0024-agent-first-entry-point-agentsession.md), [ADR-0027](0027-expression-sandbox.md), [ADR-0028](0028-workflow-resource-governance.md), [ADR-0029](0029-tool-policy-hardening.md), [ADR-0035](0035-yaml-parser-dependency.md), [sse-event-schema.md](../reference/contracts/sse-event-schema.md), [execution-model.md](../architecture/execution-model.md), [shared-core-engine.md](../architecture/shared-core-engine.md), [error-handling.md](../standards/error-handling.md), [architectural-principles.md](../standards/architectural-principles.md) +> **Amended 2026-07-30 by [ADR-0074](0074-durable-conservative-budget-commitments.md) — a refinement, not a reversal.** ADR-0074 adds one additive, secret-free dual-envelope event to the canonical stream — `budget:estimate_committed` — and makes its durability a precondition for the next provider attempt. It changes nothing about the single producer-side translation point, the monotonic `sequenceNumber`, or the persistence-before-delivery rule. ADR-0074 is **Accepted** as of 2026-07-30; its implementation is staged, so the behaviour described below is what ships until §2–§5 land. + > **Amended 2026-06-18 by [ADR-0042](0042-engine-media-storage-substrate-mediastore-deinline-retention.md).** A refinement, not a reversal: ADR-0042 adds an optional `mediaStore?` port to the `ExecutionHost` seam (1.AF) and pins where the async `deInlineMedia` pass sits relative to the single producer-side translation point this ADR defines (the gap-free `sequenceNumber` + persist-before-deliver chokepoint is unchanged). This ADR's substrate decisions stand. > **Amended 2026-06-20 by [ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md).** A refinement, not a reversal: 1.AG adds a new non-terminal node **suspension state** (a parked async media job) + a durable `media_job:submitted` event, both settling through the single `#emitDurable` choke point this ADR defines (the gap-free `sequenceNumber` + persist-before-deliver are unchanged). This ADR's substrate decisions stand. diff --git a/docs/decisions/0040-node-retry-budget-above-the-chain.md b/docs/decisions/0040-node-retry-budget-above-the-chain.md index 70089ee4..b50b1c14 100644 --- a/docs/decisions/0040-node-retry-budget-above-the-chain.md +++ b/docs/decisions/0040-node-retry-budget-above-the-chain.md @@ -4,6 +4,26 @@ - **Date**: 2026-06-15 - **Related**: [ADR-0038](0038-agentrunner-llm-call-boundary.md) (amended — see Decision A.2), [ADR-0011](0011-internal-llm-abstraction.md) (the `FallbackChain` owns within-chain policy), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) (run loop + the injected one-shot timer), [ADR-0027](0027-expression-sandbox.md) / [ADR-0029](0029-tool-policy-hardening.md) (failure classification), [error-handling.md](../standards/error-handling.md), [run-plan.md](../reference/shared-core/run-plan.md) and [node-types.md](../reference/shared-core/node-types.md) (which already pre-describe this layer), [expression-sandbox-spec.md](../reference/shared-core/expression-sandbox-spec.md) (the `runId + nodeId + retryCount` idempotency key) +> **Amended 2026-07-30 (append-only — the primary's default budget on the SESSION path only).** This ADR puts +> the retry budget ABOVE the chain and lets the primary `FallbackPlanEntry` default to `maxAttempts: 1`. That +> holds on the **workflow** path, where `WorkflowEngine` really does retry above the chain. It does **not** hold +> on the **session** path (`relavium chat`, one-shot `agent run`): an `AgentSession` has no above-chain loop, so +> the chain is the only retry — and at a budget of 1 the chain's own guard (`attempt < budget`) makes even its +> backoff unreachable, so nothing retried at all. That was invisible until **#276**, because the vendor SDKs' +> internal retry was silently absorbing transient 429/5xx inside the adapter; with that correctly disabled, a +> bare 429 would have ended the user's turn with no wait. The session path's primary therefore defaults to +> **`maxAttempts: 2`** (`SESSION_PRIMARY_MAX_ATTEMPTS`, `agent-session.ts`). +> +> **Extended the same day, after review:** the workflow path needed the same treatment for a narrower case, and +> the first reading of this note was wrong to leave it wholly at 1. The above-chain budget only exists when a +> node AUTHORS `retry:` — `#shouldRetry` returns false for `retry === undefined` and `RetrySchema.max` has no +> default — so an unauthored node had no retry above the chain either, and the #276 regression was live there +> too. The workflow primary is therefore **2 when `node.retry` is absent and 1 when it is present** +> (`UNAUTHORED_PRIMARY_MAX_ATTEMPTS`, `agent-runner.ts`): never both, because that would multiply the author's +> budget (an authored `retry.max: 3` becoming up to 6 real calls) — which is exactly what this ADR exists to +> prevent. Nothing else here changes: the above-chain budget, the no-jitter/deterministic backoff convention, +> and A.2's still-deferred authored primary `max_attempts` field all stand. + > **Amended 2026-06-20 by [ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md).** A scoped exception, not a reversal: for the async-media node only, ADR-0045 **re-attaches** to a persisted provider job on crash-resume instead of re-running from pending (the default A.6 quotes), and **carves the wall-clock media-poll cadence out of the A.3 deterministic-replay invariant** (an external job result is non-deterministic state already). The node-retry budget + classification rules are otherwise unchanged. ## Context diff --git a/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md b/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md index 768e3267..fa600199 100644 --- a/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md +++ b/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md @@ -4,6 +4,8 @@ - **Date**: 2026-06-20 - **Related**: [0031-llm-seam-shape-amendment-multimodal-io.md](0031-llm-seam-shape-amendment-multimodal-io.md) (**this ADR amends it** — append-only, ADR-0031 is unchanged in history; it wires the reserved `generateMedia`/`pollMediaJob` methods + additively adds a `signal` param to `pollMediaJob`, the only shape change to the 1.AD-frozen seam, made while the methods are still un-implemented), [0036-run-loop-substrate-event-bus-and-execution-host.md](0036-run-loop-substrate-event-bus-and-execution-host.md) (**amends** — adds a new non-terminal node *suspension* state + the durable `media_job:submitted` event through the existing `#emitDurable` choke point; the gap-free monotone `sequenceNumber` + persist-before-deliver are unchanged), [0040-node-retry-budget-above-the-chain.md](0040-node-retry-budget-above-the-chain.md) (**amends** — overrides, for the async-media node, the checkpoint model's "running-at-crash vertex re-runs from pending" default A.6 quotes (re-attach, not restart) and carves the wall-clock poll cadence out of the A.3 deterministic-replay invariant; reuses the `setTimer`/abort pattern), [0003-pure-ts-engine-not-langgraph-python.md](0003-pure-ts-engine-not-langgraph-python.md) (the derived-from-`run_events` checkpoint rule this stays inside, and the source of the "omitted running vertex re-runs from pending" default — a derived `pendingMediaJobs` slot, **no `media_jobs` table**), [0042-engine-media-storage-substrate-mediastore-deinline-retention.md](0042-engine-media-storage-substrate-mediastore-deinline-retention.md) (the §4 terminal-sweep boundary it explicitly left open for this ADR; the cancel-sweep composes with it), [0043-media-egress-failover-rematerialization-ssrf.md](0043-media-egress-failover-rematerialization-ssrf.md) (the one SSRF-validated media-egress this reuses to re-host a provider output URL — never a second fetch site), [0044-media-access-governance-read-media-save-to-cost.md](0044-media-access-governance-read-media-save-to-cost.md) (the per-modality media cost class + `PreEgressHook` this prices the generate path through), [0011-internal-llm-abstraction.md](0011-internal-llm-abstraction.md) (the seam-purity rule — the opaque jobId, no vendor type across the seam), [0014-managed-metering-quota-and-billing.md](0014-managed-metering-quota-and-billing.md) (the cost-integrity / orphaned-provider-cost caveat on a local-only cancel), [0015-managed-mode-data-handling-and-compliance.md](0015-managed-mode-data-handling-and-compliance.md) (the gateway poll-through / counts-not-content the loop must not foreclose), [../reference/shared-core/llm-provider-seam.md](../reference/shared-core/llm-provider-seam.md), [../reference/contracts/sse-event-schema.md](../reference/contracts/sse-event-schema.md), [../reference/contracts/config-spec.md](../reference/contracts/config-spec.md), [../reference/shared-core/database-schema.md](../reference/shared-core/database-schema.md), [../reference/contracts/workflow-yaml-spec.md](../reference/contracts/workflow-yaml-spec.md), [../analysis/multimodal-io-design-2026-06-07.md](../analysis/multimodal-io-design-2026-06-07.md) (decision A5). +> **Amended 2026-07-30 by [ADR-0074](0074-durable-conservative-budget-commitments.md) — a refinement, not a reversal.** ADR-0074 freezes an async media job's money basis at submission: `media_job:submitted` gains additive `units` and `acceptedCostMicrocents`, so re-attach after a crash reconstructs the admission from what was accepted rather than re-deriving it from the current pricing overlay and mutable workflow content. The poll/checkpoint/resume/cancel loop is unchanged. ADR-0074 is **Accepted** as of 2026-07-30; its implementation is staged, so the behaviour described below is what ships until §2–§5 land. + ## Context Workstream **1.AG (Output generation — Phase D of the 1.m6 multimodal sub-spine)** lights up media *output*. [ADR-0031](0031-llm-seam-shape-amendment-multimodal-io.md) froze the seam shape for it at 1.AD: `generateMedia?(req, key): Promise` and `pollMediaJob?(jobId, key): Promise` are **reserved, optional** methods on the one `LlmProvider` seam ([`types.ts`](../../packages/llm/src/types.ts) — `MediaGenRequest`/`MediaGenResult`/`MediaJobStatus`), and the disclaimer reads "**RESERVED at 1.AD; wired at Phase D (1.AG) with its own ADR for the engine-owned poll/checkpoint/resume/cancel loop (A5)**." This is that ADR. diff --git a/docs/decisions/0070-durable-per-model-session-cost-attribution.md b/docs/decisions/0070-durable-per-model-session-cost-attribution.md index d9576551..a6e04971 100644 --- a/docs/decisions/0070-durable-per-model-session-cost-attribution.md +++ b/docs/decisions/0070-durable-per-model-session-cost-attribution.md @@ -8,6 +8,8 @@ FK target and the `BEGIN IMMEDIATE`/`SQLITE_BUSY` write convention this obeys · [ADR-0028](0028-workflow-resource-governance.md) — the budget governor that *reads* the session total this ADR gives a single writer. - **Related (secondary)**: [ADR-0005](0005-sqlite-drizzle-local-postgres-cloud.md) (one schema, two dialects) · + +> **Amended 2026-07-30 by [ADR-0074](0074-durable-conservative-budget-commitments.md) — a refinement, not a reversal.** ADR-0074 keeps this ADR's realized-cost invariant intact — `SUM(session_costs) == agent_sessions.total_cost_microcents` stays realized-only — and adds a **separate** conservative aggregate beside it rather than mixing an estimate into the actual total. `recordSessionCost` remains the single writer of the realized total. ADR-0074 is **Accepted** as of 2026-07-30; its implementation is staged, so the behaviour described below is what ships until §2–§5 land. [ADR-0024](0024-agent-first-entry-point-agentsession.md) (one model per `AgentSession` *instance*, not per billed egress) · [ADR-0065](0065-provider-economics-and-extensibility.md) · [ADR-0067](0067-node-supported-floor-22-reaffirm-better-sqlite3.md) · [ADR-0050](0050-cli-history-db-at-rest-posture.md) · [ADR-0056](0056-cli-in-app-slash-command-system-and-manifest.md) diff --git a/docs/decisions/0073-history-db-migration-lock.md b/docs/decisions/0073-history-db-migration-lock.md new file mode 100644 index 00000000..db6ea4b6 --- /dev/null +++ b/docs/decisions/0073-history-db-migration-lock.md @@ -0,0 +1,213 @@ +# ADR-0073: `runMigrations` is serialized across processes by an OS lock file, not by a transaction + +- **Status**: Accepted +- **Date**: 2026-07-29 +- **Related**: [ADR-0064](0064-live-model-catalog.md) (its 2026-07-07 "2.5.I — DB write-path concurrency" amendment note establishes the `BEGIN IMMEDIATE` + bounded-retry convention this extends), [ADR-0021](0021-node-sqlite-driver-better-sqlite3.md) + [ADR-0067](0067-node-supported-floor-22-reaffirm-better-sqlite3.md) (the synchronous driver), [ADR-0050](0050-cli-history-db-at-rest-posture.md) (the `0700`/`0600` guard the lock file inherits), [ADR-0005](0005-sqlite-drizzle-local-postgres-cloud.md) (one schema, two dialects — the Postgres port must not inherit this mechanism). Canonical home for the mechanism: the "Concurrency & transaction behavior" section of [database-schema.md](../reference/shared-core/database-schema.md). + +> **Amended 2026-07-30 (append-only, mechanism replaced — the decision stands).** The Decision below chose an +> **OS lock file** with a staleness threshold and an atomic-`rename` takeover. Review found that protocol +> **unsound, and it was replaced before merge.** Two failures, both reproducible: the takeover's `rename` fixes +> only the file's final contents, so two takers can each `rename` and then each read back their own nonce and +> both enter the critical section; and `release()` unlinked unconditionally, so a superseded holder could delete +> the new owner's lock and admit a third process. Breaking a stale lock with plain file operations cannot be +> made mutually exclusive — which is exactly why this ADR named `flock`/`LockFileEx` the mechanically superior +> option and rejected it only for lacking a dependency-free Node binding. +> +> **The implementation now takes that advisory lock through SQLite instead**: `BEGIN EXCLUSIVE` on a dedicated +> `.migrate.lock` database. That is a real `fcntl` lock held for the life of the transaction, and the +> kernel releases it when the process dies — so the 30 s threshold, the 50 ms poll, the pid field and the whole +> takeover protocol are **deleted rather than tuned**, and the negative this ADR recorded about `finally` not +> surviving `SIGKILL` no longer applies. It needs no new dependency (`better-sqlite3` is already the driver) and +> keeps a separate file from `history.db`, because taking `BEGIN EXCLUSIVE` on the database drizzle is about to +> migrate would deadlock it against our own connection. +> +> Unchanged: the decision to serialize the batch at all, why `BEGIN IMMEDIATE` cannot do it, the `:memory:` +> no-op, the run-then-reconcile fallback for a filesystem that refuses the lock, and the Node/CLI scoping (the +> Phase-6 Postgres port still gets a real advisory lock of its own, not this). + +## Decision — current (2026-07-30) + +**`runMigrations`'s whole batch runs under an OS advisory lock taken through SQLite: `BEGIN EXCLUSIVE` on a +dedicated `.migrate.lock` database, held for the duration of the batch and committed at the end.** + +- **Why it is sound where the lock file was not.** `BEGIN EXCLUSIVE` takes a real `fcntl` write lock, and the + kernel releases it when the holder dies — however it dies. There is therefore **no staleness to detect and no + lock to break**, which is what made the original protocol unfixable: breaking a stale lock with plain file + operations cannot be mutually exclusive. +- **A separate file, never `history.db` itself.** Taking `BEGIN EXCLUSIVE` on the database drizzle is about to + migrate would deadlock it against our own connection. +- **No new dependency.** `better-sqlite3` is already the driver ([ADR-0021](0021-node-sqlite-driver-better-sqlite3.md)). +- **Bounded, then reconcile.** A waiter blocks in SQLite's busy handler for up to 10 s; past that — or if the + lock file cannot be opened, or the filesystem's locking does not work — it falls through to + run-then-reconcile (run `migrate()`, and on failure run it once more, rethrowing the FIRST error) rather than + hang or crash first-run startup. +- **`:memory:` is a no-op.** A private in-memory database cannot be contended by construction. + +Everything the amendment note says is unchanged still holds: that the batch must be serialized at all, why +`BEGIN IMMEDIATE` cannot do it, and the Node/CLI scoping. The mechanism is the only thing that changed. + +## Context + +`~/.relavium/history.db` is a **single shared file** that two `relavium` processes may open at +once — the case [ADR-0064](0064-live-model-catalog.md) §5 already accepted and its 2.5.I +amendment note gave a concrete mechanism for. That note covers steady-state **writes**. It does +not cover **first-run migration**, which is a different race with a different shape. + +`runMigrations` ([client.ts](../../packages/db/src/client.ts)) delegates to drizzle's +`migrate()`. Its synchronous SQLite implementation does three things in this order: + +1. `CREATE TABLE IF NOT EXISTS __drizzle_migrations` — auto-committed, outside any transaction. +2. `SELECT id, hash, created_at FROM __drizzle_migrations ORDER BY created_at DESC LIMIT 1` — + **also outside any transaction**. This single read decides which migrations are pending. +3. `BEGIN` (drizzle's **DEFERRED** default) → apply each pending migration's statements → `COMMIT`. + +The deciding read in step 2 is therefore not covered by the write in step 3. Two processes +launched together against a fresh file — two shells, or a CLI and a VS Code extension host — +both observe an empty table, both decide the full migration set is pending, and both proceed. +One commits; the other then runs `CREATE TABLE \`runs\`` against a database that already has it +and dies with a raw `DrizzleError`. Reproduced live; recorded as finding `#99`. + +The stakes are a **first-run** crash, which is the worst place to have one: the user has no +history to lose, but also no reason to believe the tool works. And the failure is timing- +dependent, so it survives casual testing. + +Two constraints bound the solution space. The engine-purity rule (CLAUDE.md rule 5) does not +apply — `packages/db` is a Node-side package that already imports `node:fs`. But +[architectural-principles.md](../standards/architectural-principles.md) §9 does: **no new +runtime dependency without an ADR**, which rules out reaching for `proper-lockfile` and is part +of why this decision is being recorded rather than quietly implemented. + +## Decision — original (2026-07-29, SUPERSEDED) + +> **Historical.** Everything in this section is the lock-file protocol the amendment above replaced: the +> `'wx'` acquisition, the constants table, the pid/staleness handling and the `rename` takeover. It is kept +> because the corpus is append-only and because the rejected-alternatives reasoning below is still the record +> of why a lock was chosen at all — but **do not implement from it.** The `flock`/`LockFileEx` option it +> rejected for lacking a Node binding is, in effect, what the current decision adopts by routing the same +> advisory lock through SQLite. + +**We will serialize the whole `runMigrations` batch across processes with an OS lock file at +`.migrate.lock`, created with `openSync(path, 'wx')`, and fall back to a +run-then-reconcile retry when the lock cannot be created at all.** + +**The constants, pinned here rather than left to the implementation** — they are this decision's +whole tuning surface: + +| Constant | Value | Why | +|---|---|---| +| Stale threshold | **30 s** | Two orders of magnitude above a real batch (milliseconds on a local file), so a live holder is never mistaken for a dead one even on a loaded CI box; and short enough that a crashed holder costs one pause, not a support ticket. | +| Max wait | **10 s** | A waiter that has not been let in by then is not waiting on a slow migration; falling through to reconcile beats hanging first-run startup. | +| Poll interval | **50 ms** | Bounded busy-wait; the whole batch is shorter than this on the happy path. | + +**Staleness is decided by the recorded start time, never by the pid.** The holder writes both, +but pids recycle — a "is that pid alive?" check can be confidently wrong in either direction +(reporting a stranger's process as our live holder, or a recycled pid as dead). The pid is +therefore **diagnostic only**, for a human reading a stuck lock file; the timestamp is the only +input to the takeover decision. + +**Takeover is an atomic swap, not delete-then-create.** Naïve "unlink the stale lock, then +create ours" recreates the original race one level up: two waiters can both observe the same +stale lock, both unlink, and both create. Instead the taker writes its claim to a +uniquely-named temp file in the same directory and `rename`s it over the lock path — `rename` +is atomic and last-writer-wins, so concurrent takers converge on exactly one owner rather than +two believing they hold it. A taker then re-reads the lock and proceeds only if the claim it +reads back is its own. + +`'wx'` (`O_CREAT|O_EXCL`) is the atomic-create idiom this repo already uses in +[config/write.ts](../../apps/cli/src/config/write.ts), +[media-write.ts](../../packages/db/src/media-write.ts) and the tool-host fs arm, so this +introduces a new *use*, not a new *technique*. + +**This decision is scoped to migration.** It changes nothing about steady-state writes, which +keep ADR-0064's `BEGIN IMMEDIATE` + `withBusyRetry` convention unchanged, and it is a **Node/CLI** +mechanism: the Phase-6 Postgres port replaces it with a real advisory lock rather than inheriting +a lock file, and the desktop's Rust path is untouched. + +**It is also a no-op for an in-memory database.** `createClient(':memory:')` is private to its +own connection, so cross-process contention is impossible by construction and a lock file named +`:memory:.migrate.lock` would be nonsense. The lock is taken only for a real filesystem path — +which also keeps the entire test suite (hundreds of `:memory:` migrations) off the lock path. + +Considered: + +- **`BEGIN IMMEDIATE` around the migration batch** — *rejected: not implementable.* This was the + originally-proposed fix and it cannot work against the migrator described above. Taking the + write lock up front closes the read→write **upgrade** race, but the read that decides what to + apply happens *before* the transaction, so both processes still decide identically; and we + cannot hoist drizzle's `migrate()` into our own transaction, because its raw `BEGIN` would + throw inside one. Recorded here explicitly so the option is not re-proposed. +- **Run-then-reconcile only, with no lock file** — *rejected as the primary, kept as the + fallback.* On failure, re-run `migrate()` once: the loser's second pass observes the winner's + committed row and applies nothing. It is roughly fifteen lines, keeps no state on disk, and has + no stale-lock failure mode. But it **recovers from** the collision instead of preventing it, + and its correctness rests entirely on every migration being replay-safe after a partial + concurrent apply — a property that holds today only because drizzle wraps the batch in a + transaction, and that no test would catch us losing. It is retained as the fallback for the + case where the lock file genuinely cannot be created (a read-only directory, an exotic mount + where `'wx'` is not atomic), where crashing would be strictly worse. +- **Owning the migration runner** — *rejected: highest risk, no proportionate gain.* Reading + `_journal.json` and the `.sql` files ourselves and applying them inside one `BEGIN IMMEDIATE` + is the only option that makes the read and the write genuinely atomic. But it obliges us to + reproduce drizzle-kit's `hash` and `folderMillis` semantics exactly and forever: get it wrong + and a released build re-applies every migration against a populated database. That is a much + worse failure than the one being fixed, traded for elegance. +- **An OS advisory lock (`flock` / `LockFileEx`)** — *rejected, reluctantly: no dependency-free + Node primitive exists.* This is the mechanically superior answer and deserves naming as such, + because it has **no stale-lock problem at all**: the kernel releases an advisory lock when the + holding process dies, however it dies, so the 30 s threshold and the takeover protocol above + would both be unnecessary. Node's `fs` does not expose `flock(2)` (`fs.open`'s `'wx'` is + `O_CREAT|O_EXCL`, an atomic *create*, which is a different primitive), and `LockFileEx` is + unreachable too — so taking this route means a native addon or an npm dependency, which + [architectural-principles.md](../standards/architectural-principles.md) §9 gates behind exactly + this kind of ADR and which we are not willing to add for a first-run edge case. Recorded as the + upgrade path: if Node ever exposes advisory locking, or if `packages/db` acquires a native + dependency for another reason, this mechanism should be replaced by it rather than tuned. +- **Doing nothing / documenting the race** — *rejected.* It is a live, reproduced first-run + crash, and Wave 1 exists to stop exactly this class of bleeding. + +## Consequences + +### Positive + +- Two processes racing a fresh `history.db` produce a clean **wait-and-continue** rather than a + raw `DrizzleError` — the behaviour the finding asked for, not merely the absence of a crash. +- The guarantee is **independent of drizzle's internals**. If a future drizzle release changes + where its deciding read sits, or makes a migration non-replay-safe, the lock still holds; the + reconcile fallback alone would not. +- It generalizes to the migration we cannot see yet: a data-backfilling migration that is *not* + idempotent on replay is safe under a lock and unsafe under reconcile-only. +- No new runtime dependency, and `'wx'` atomic-create is an idiom already established in three + places in this repo. + +### Negative + +> **Partly historical.** The entries about a stale lock file, the tuning of its threshold, and `finally` not +> surviving `SIGKILL` describe the superseded protocol and **no longer apply** — the kernel releases the +> advisory lock on process death. What still applies: two mechanisms rather than one (the lock plus the +> reconcile fallback), and that this is a local-only answer the Phase-6 Postgres port must not inherit. + + +- **A new on-disk artifact with its own failure mode, and `finally` is not a guarantee.** The + holder releases in a `finally`, which covers a thrown migration and a normal exit — but + **`SIGKILL`, a power loss, or an OOM kill skips it**, so a crashed holder WILL leave a lock file + behind. That is a design assumption, not an oversight: the stale threshold is the only thing + that recovers it, which is precisely why an advisory lock (rejected above) would have been + better. A wrong threshold either wedges startup or lets two processes in; 30 s against a + millisecond-scale batch leaves three orders of magnitude of headroom, and the reconcile + fallback still catches the double-entry case. +- **Windows deserves naming, not lumping in with "exotic".** `'wx'` maps to `CREATE_NEW`, which + is atomic on NTFS, so the primary path holds — but a virus scanner or an indexer holding a + transient handle can make the *unlink*/`rename` fail in ways POSIX does not, and network + redirectors (SMB/DFS) do not guarantee `O_EXCL` semantics at all. This joins the corpus's + existing Windows caveat (ADR-0050's `0600`/`0700` no-op) rather than hiding behind it: on + Windows the reconcile fallback is load-bearing, not decorative. +- **Two mechanisms instead of one** (lock, then reconcile), so there are two paths to test rather + than one. Accepted deliberately: the fallback exists precisely for the environments where the + primary cannot run, and both **will be** covered by the two-process regression test the + implementation lands with. +- **A local-only answer to a general problem.** This does not serialize anything for the Phase-2 + cloud/Postgres path, which will need its own advisory lock. Named here so the Postgres port + does not inherit a lock file by accident. +- The lock file sits beside `history.db` inside `~/.relavium/`, so it inherits ADR-0050's `0700` + directory guard — but it is one more file a user may see and wonder about. It is named for what + it is. diff --git a/docs/decisions/0074-durable-conservative-budget-commitments.md b/docs/decisions/0074-durable-conservative-budget-commitments.md new file mode 100644 index 00000000..b8fe57e5 --- /dev/null +++ b/docs/decisions/0074-durable-conservative-budget-commitments.md @@ -0,0 +1,109 @@ +# ADR-0074: Durable conservative budget commitments across run and session resume (amends ADR-0028, ADR-0036, ADR-0045, and ADR-0070) + +- **Status**: Accepted +- **Date**: 2026-07-29 +- **Related**: [ADR-0028](0028-workflow-resource-governance.md) (the pre-egress cap), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) (durable run events), [ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) (async media re-attach), [ADR-0070](0070-durable-per-model-session-cost-attribution.md) (session-cost accounting), [ADR-0071](0071-models-dev-as-the-model-metadata-source.md) (the strict-cost-cap posture), and [sse-event-schema.md](../reference/contracts/sse-event-schema.md) (the canonical event contract). + +## Context + +The pre-egress governor must make two deliberately different statements about money: + +1. **Realized cost** is a provider-usage-derived amount that belongs in the user-visible actual-cost total. +2. **Conservative commitment** is a bounded estimate retained when a provider may already have accepted or billed a request but the response supplies no trustworthy usage. It must consume cap capacity, but it is not evidence of an actual invoice amount. + +Keeping only the first amount reopens a hard cap after a clean EOF, a partial-stream failure, a crash, or a resume. Folding the second into `cost:updated` or `totalCostMicrocents` prevents that bypass, but falsely presents an estimate as realized spend and breaks the realized-cost contract that the second 2.6.Q money PR is intended to complete. + +Async media has a parallel replay hazard. A submitted provider job is already irrevocably billable, but the existing durable `media_job:submitted` descriptor does not contain its authored unit volume or the price accepted at submission. A new process consequently re-derives both from mutable workflow content and the current pricing overlay. A lower later overlay can under-reserve the already-submitted job, admit a sibling, and silently exceed the authored cap; it can also rewrite the job's eventual reported cost. + +The same governor backs resumable chat. Session persistence currently stores only realized `total_cost_microcents`, so an uncertain post-egress charge is also lost across `chat-resume`. A workflow-only repair would leave the shared safety control materially inconsistent by surface. + +## Decision + +### 1. Keep actual and conservative money separate + +Every governed execution keeps three independent quantities: + +- durable **realized** cost; +- durable **conservative committed** cost; and +- live, process-local **admissions** for prospective egress. + +The pre-egress projection is: + +`realized + conservativeCommitted + liveAdmissions + nextWorstCaseEstimate`. + +Actual totals, per-model actual-cost attribution, and `cost:updated` remain realized-only. A conservative commitment is never silently downgraded by a later actual-cost snapshot; it remains cap-consuming for the lifetime of the owning run or session. + +**It is not, however, unbounded in time.** "Until a future provider protocol can reconcile that exact attempt" would mean an indefinite block, which is a worse failure than the overspend it prevents: a single usage-less response would permanently shrink a long-lived chat's cap with no way out. Two bounds apply. The commitment dies with its owner — a completed run and an ended session discard it with the rest of their governor state, so it can never leak into unrelated work. And within a live owner the user must be able to clear it deliberately: the surfaces that render the committed amount as *estimated, possibly billed* also expose releasing it, which is an explicit user decision about their own money, exactly like raising the cap under `on_exceed`. What is forbidden is the system silently deciding the estimate was wrong. + +### 2. Persist an uncertain provider charge before later egress can proceed + +Add an additive, secret-free dual-envelope event, `budget:estimate_committed`, carrying the node/agent identity, model id, bounded `estimateMicrocents`, and the owner-local cumulative conservative amount. It is a **new event `type`**, not an optional field on `cost:updated` — the rejected alternatives below say why. Its exact Zod shape, which discriminated-union arms it joins, and its envelope rules have one canonical home in [sse-event-schema.md](../reference/contracts/sse-event-schema.md) and [run-event.ts](../../packages/shared/src/run-event.ts); this ADR decides that it exists and what it means, and deliberately does not restate the spec. For workflows it is a durable `run_events` entry; for sessions it is a durable session-budget write and a streamed session event. + +When an admission becomes uncertain, its ledger mutation is synchronous, but the next provider attempt and the enclosing turn completion wait for the commitment's durability acknowledgement. This closes the crash window between a potentially billable provider call and a later node/session boundary. A persistence failure never releases capacity; it fails the active owner loudly while preserving the conservative reservation in memory for the terminal path. + +Checkpoint reconstruction folds conservative commitments separately from actual-cost snapshots and restores both before it schedules resumed work. The historical `totalCostMicrocents` / run terminal actual-cost field remains actual-only; surfaces render an explicit *estimated, possibly billed* amount rather than claiming that it was realized usage. + +### 3. Freeze the accepted money basis of an async media job + +`media_job:submitted` gains additive, backwards-compatible submit-time fields: + +- `units` — the exact authored billed volume; and +- `acceptedCostMicrocents` — the priced amount used for the admission at submission. + +New emitters always populate both. Resume reconstructs the admission from `acceptedCostMicrocents` without a pricing lookup and uses the frozen units/cost for the job's one terminal addend. A user-price/catalog or same-workflow-content change after submission cannot change an already accepted commitment or rewrite its historical cost. + +Legacy submitted-job rows without the fields remain readable. With a configured cap, a resumed legacy pending job is treated fail-closed for new egress until it settles; it is never silently under-reserved from a newer, cheaper price. The compatibility fallback is observable so operators can distinguish legacy uncertainty from a current priced submission. + +### 4. Make resumable sessions preserve the same safety state + +Session persistence receives a separate conservative-cost aggregate and per-model conservative attribution beside ADR-0070's realized `session_costs` data. The persister owns the additive, transactional write of a `budget:estimate_committed`; `AgentSession.resume` restores the realized and conservative totals together into the same `BudgetGovernor` used by a fresh session. The existing invariant for realized cost stays intact rather than being weakened to mix actual and estimated figures. + +### 5. Settle forward-compatibility first: a tolerant READ, a strict BODY + +The event in §2 cannot be emitted until this is decided, so it is decided here rather than left open. + +**The reader tolerates an unknown event `type` and drops it; a KNOWN type with an invalid body still fails +loud.** That single distinction is the whole rule: an unrecognized discriminator is a newer writer, which is +forward evolution; a recognized discriminator whose payload does not parse is corruption, and +[ADR-0050](0050-cli-history-db-at-rest-posture.md)'s durability-first posture says corruption must never be +swallowed. + +This is **not a new choice** — it is the contract [sse-event-schema.md](../reference/contracts/sse-event-schema.md) +§Forward-compatibility already states: adding a new event `type` is "always v1.0-legal and never a breaking +change, **provided consumers ignore unknown `type`s and unknown fields**". The code is the deviation, and it +predates this ADR: `RunEventSchema` and `SessionEventSchema` are both `z.discriminatedUnion('type', …)` and +`RunOrSessionEventSchema` is a `z.union` of the two, so **all three throw** on an unknown type, and +`loadRunEvents` parses every stored row through one of them. So the work §2 depends on is *fixing a +pre-existing doc↔code contradiction*, not weakening a guarantee to make room for a new event. + +Both paths need it, not just the run path: `budget:estimate_committed` is dual-envelope, so the session +discriminated union and the `z.union` wrapper are equally affected. The correct seam is a `safeParse` at the +**read** boundary that distinguishes the two failure modes, never a `.catch()` at the call site — one rule, one +place, so a third reader cannot get it wrong. + +The rejected alternative was an explicit refusal to downgrade while a run/session carries new events. It needs a +durable version marker the schema deliberately does not have (it is "versioned by additive evolution, not a +version field"), it converts a recoverable read into a hard failure, and it would leave the documented +ignore-unknown promise unfulfilled anyway. + +## Consequences + +### Positive + +- A strict cap stays enforced across concurrent branches, provider responses without usage, async-media re-attach, process crashes, workflow resume, and chat resume. +- Cost reporting stays honest: actual provider-derived cost is not inflated by an upper bound, while the user can still see the committed uncertainty that protects their cap. +- Async media's durable descriptor becomes a true replay record: it preserves the acceptance-time request volume and money basis instead of asking current mutable state to reconstruct history. + +### Negative + +- The event/schema and checkpoint contracts grow, and the session implementation needs a migration plus a transactional persistence path. These are deliberate cross-surface changes, not a local `budget-governor.ts` edit. +- A conservative commitment can block a later request even when the provider ultimately charged less or nothing. This is the intentionally safe direction; its presentation must identify it as an estimate rather than hide the tradeoff. +- **Older binaries cannot replay a log containing the new event discriminant, and this is harder than a first reading suggests.** The premise that "current readers already handle unknown event types" is **false**, verified against the code: `RunEventUnionSchema` is a `z.discriminatedUnion('type', …)` ([run-event.ts](../../packages/shared/src/run-event.ts)) and `loadRunEvents` parses **every** row through it ([run-history-store.ts](../../packages/db/src/run-history-store.ts)), so an unknown `type` **throws rather than being skipped** — one new event in the log makes the whole run unreadable to an older binary, not merely partially understood. **§5 settles this**: the read boundary tolerates an unknown `type` and drops it, while a known type with an invalid body still fails loud — which is the contract `sse-event-schema.md` already documents and the code never implemented. The same fix is required on the session discriminated union and the `z.union` wrapper, because this event is dual-envelope. That work is a **precondition** for emitting the event: until the tolerant read ships, one new discriminant in the log makes the whole run unreadable to an older binary. +- Legacy async-media records cannot recover a price that was never persisted. They therefore choose temporary fail-closed capacity rather than a fabricated historical amount. + +### Rejected alternatives + +- **Add the estimate to `cost:updated` or actual totals.** Rejected: it makes an estimate look like realized spend and corrupts the realized-cost boundary. +- **Keep the commitment process-local until node/session completion.** Rejected: a crash before that boundary reopens the cap. +- **Reprice an async job on resume.** Rejected: current catalog/workflow state is not evidence of the price or volume accepted by a prior provider submission. +- **Fix workflows but leave resumed chat unprotected.** Rejected: both surfaces share the governor; a safety guarantee that disappears after `chat-resume` is not a first-class cost cap. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 9c5efdd0..45183b30 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -116,6 +116,8 @@ flowchart TD | 0070 | [Durable per-model session cost attribution — the `session_costs` aggregate, single-owner cost writes, and the reconciliation invariant](0070-durable-per-model-session-cost-attribution.md) | Accepted | 2026-07-12 | | 0071 | [models.dev as the model-metadata source; the hand-maintained registry is retired (supersedes two ADR-0064 clauses, corrects an ADR-0066 premise)](0071-models-dev-as-the-model-metadata-source.md) | Accepted | 2026-07-13 | | 0072 | [Model metadata in the DB behind the offline snapshot floor (partially supersedes ADR-0064 §4's seed-prohibition; amends ADR-0071 §3/§4/§9)](0072-model-metadata-in-the-db-behind-a-generated-offline-floor.md) | Accepted | 2026-07-14 | +| 0073 | [`runMigrations` is serialized across processes by an OS lock file, not by a transaction](0073-history-db-migration-lock.md) | Accepted | 2026-07-29 | +| 0074 | [Durable conservative budget commitments across run and session resume (amends ADR-0028, ADR-0036, ADR-0045, ADR-0070)](0074-durable-conservative-budget-commitments.md) | Accepted | 2026-07-30 | ## Creating a new ADR diff --git a/docs/project-structure.md b/docs/project-structure.md index d11fa67f..b745323d 100644 --- a/docs/project-structure.md +++ b/docs/project-structure.md @@ -7,7 +7,7 @@ Relavium is a **Turborepo monorepo with pnpm workspaces**. Every package is TypeScript with shared `tsconfig` bases, a single root `package.json` for tooling (ESLint, Prettier, Vitest), and Turborepo remote cache for CI. There are **four product surfaces** (desktop, VS Code, CLI, portal) plus a Phase-2 backend -(`apps/api`, infrastructure rather than a product surface), and five shared packages. +(`apps/api`, infrastructure rather than a product surface), and six shared packages. Every product surface exposes **both entry points of the one shared engine**: a **conversational agent** (chat session) *and* a **workflow interface** (the runner @@ -23,6 +23,7 @@ flowchart TD llm["packages/llm
provider adapters"] core["packages/core
workflow engine"] db["packages/db
Drizzle schema"] + mcp["packages/mcp
inbound MCP client"] ui["packages/ui
ReactFlow nodes + shadcn"] end subgraph apps @@ -66,6 +67,7 @@ flowchart TD | `packages/shared` | **`@relavium/shared`** — Zod schemas, TypeScript types, constants used everywhere (`WorkflowSchema`, `AgentSchema`, `RunSchema`, `NodeSchema`, `EdgeSchema`, `RunEvent`, `CostEvent`, `HumanGateEvent`). No runtime deps except zod. | | `packages/llm` | **`@relavium/llm`** — provider adapters (`AnthropicAdapter`, `GeminiAdapter`, and a shared OpenAI-compatible adapter serving both OpenAI and DeepSeek via a custom `baseURL`) normalizing streaming, tool calls, and usage tokens to the canonical format. Houses `ToolNormalizer`, `CostTracker`, `FallbackChain`. Three adapters, per [tech-stack.md](tech-stack.md) and ADR-0011. | | `packages/db` | **`@relavium/db`** — Drizzle schema + migrations. Same table names and column types for SQLite (local) and Postgres (cloud), different driver. See [reference/shared-core/database-schema.md](reference/shared-core/database-schema.md). | +| `packages/mcp` | **`@relavium/mcp`** — the inbound MCP client: the SDK fence, the dependency-free JSON-Schema→Zod compiler, and the network transports behind the SSRF pre-connect floor ([ADR-0052](decisions/0052-inbound-mcp-client-package-lifecycle-registration.md), [ADR-0053](decisions/0053-mcp-network-transport-egress-security.md)). | | `packages/ui` | **`@relavium/ui`** — shared React component library. All ReactFlow custom node types (Agent, Condition, FanOut, Aggregator, Loop, HumanGate, Input, Output, Tool) and edges, plus shadcn/ui base + Tailwind config. The canvas node types are imported by desktop and VS Code panels; the portal imports only the shadcn/ui base + Tailwind layer for visual consistency (it is a control plane, not a canvas surface). | ## Build Order diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index 9db15a87..b55fa801 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -76,7 +76,7 @@ export type RunEvent = | `agent:reasoning` | A streaming reasoning ("thinking") delta from an agent node (EA6, 2.5.H — a pure host-emit: the `@relavium/llm` seam already carries the reasoning chunks; the turn core emits one per `reasoning_delta`). A **dual-envelope** event like `agent:token` (carried on the session stream too), so a surface renders a live, collapsible "thinking" panel. Never carries the ephemeral same-provider `signature` (ADR-0030). Amends [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md). | `nodeId`, `text`, `model` | | `agent:tool_call` | An agent invoked a tool. | `nodeId`, `model` (the invoking model — so a tool call is attributable across a failover), `toolId`, `toolInput` (sanitized — no secrets), `attemptNumber?` (1-based, matches `cost:updated`) | | `agent:tool_result` | A tool returned. | `nodeId`, `toolId`, `success`, `outputSummary` (truncated for UI), `attemptNumber?` | -| `agent:approval_requested` | A side-effecting tool dispatch is awaiting an **interactive per-tool approval** decision (ADR-0057 EA3/EA5). The engine's `confirmDispatch` emits it — for **every** governed dispatch reaching the gate, whether the host then prompts a human or auto-decides — just before invoking the host's `ConfirmActionHook`; the registry then awaits the verdict (approve ⇒ dispatch, reject ⇒ a fatal `tool_denied`). A **dual-envelope** event (`runId`/`sessionId`), like `agent:tool_call` — in Phase 2.5 emitted only on the chat session path (the approval regime), and **carried on the session stream** (not run-only — it is **not** dropped like `agent:file_patch_proposed`). | `nodeId`, `toolId`, `action: 'fs_write' \| 'process' \| 'egress' \| 'os'` (the governed side-effect class — [tool-registry.md](../shared-core/tool-registry.md)), `preview` (**secret-free, display-only**: `{ path? }` for a write, `{ command? }` for a process, `{ host? }` for egress, `{}` for an `os` action like `read_clipboard`/`notify` — never a full URL/query, never a secret), `attemptNumber?` | +| `agent:approval_requested` | A side-effecting tool dispatch is awaiting an **interactive per-tool approval** decision (ADR-0057 EA3/EA5). The engine's `confirmDispatch` emits it — for **every** governed dispatch reaching the gate, whether the host then prompts a human or auto-decides — just before invoking the host's `ConfirmActionHook`; the registry then awaits the verdict (approve ⇒ dispatch, reject ⇒ a fatal `tool_denied`). A **dual-envelope** event (`runId`/`sessionId`), like `agent:tool_call` — in Phase 2.5 emitted only on the chat session path (the approval regime), and **carried on the session stream** (not run-only — it is **not** dropped like `agent:file_patch_proposed`). | `nodeId`, `toolId`, `action: 'fs_write' \| 'process' \| 'egress' \| 'os'` (the governed side-effect class — [tool-registry.md](../shared-core/tool-registry.md)), `preview` (**secret-free, display-only**: `{ path? }` for a write, `{ command? }` for a process, `{ host? }` for egress, `{}` for an `os` action like `read_clipboard`/`notify` — never a full URL/query, never a secret. Secret-freedom is **enforced** by the registry's redaction, not asserted, so a field **may contain the literal `[redacted]` marker and is not guaranteed to be a resolvable path/command/host — a machine consumer must never parse it**; see [tool-registry.md](../shared-core/tool-registry.md) §Preview redaction), `attemptNumber?` | | `agent:file_patch_proposed` | An agent proposed a file change (**gated — no write until the user accepts**; e.g. the VS Code inline-diff review). | `nodeId`, `patches: [{ uri, unifiedDiff }]` (≥1 — an empty proposal is meaningless), `attemptNumber?` | | `cost:updated` | A node's token cost was tallied (drives the cost waterfall). | `nodeId`, `model`, `inputTokens`, `outputTokens`, `costMicrocents`, `cumulativeCostMicrocents` (integer micro-cents — canonical unit in [llm-provider-seam.md](../shared-core/llm-provider-seam.md#6-usage); **includes realized media spend**, folded as a disjoint addend per [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3 — the per-unit `Usage.mediaUnits` axis is **not yet a field on this event**, deferred, see [deferred-tasks.md](../../roadmap/deferred-tasks.md)), `attemptNumber?` (1-based **within-chain** FallbackChain attempt — resets per node-retry re-dispatch; **distinct** from `node:*.attemptNumber`, see the [two attemptNumber families](#two-attemptnumber-families) note), `priced?` (**ADR-0070** — additive + optional, so an older reader ignores it: `false` when the egress could **not be priced**. An unpriced model still emits this event with its **real tokens** and `costMicrocents: 0`, which makes "cost 0 + tokens > 0" ambiguous between *could not price* and *genuinely free* — an ambiguity nothing else in the event resolves. The durable `session_costs` row records it as an `unpriced_calls` **counter**, not a boolean, because a model can be priced **mid-session**). **Generative-node variant (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5):** a `media_surface: 'generative'` agent node emits **exactly one** `cost:updated` with `inputTokens` / `outputTokens` **= 0** (no token billing — the spend rides entirely in `costMicrocents` as the per-modality media addend) and **no `attemptNumber`** (no FallbackChain on the generative path — one provider, no failover). | | `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — the durable cost source checkpoint/resume restores from, since `cost:updated` is streamed-only; the engine always populates it. `node:failed` mirrors this field, 2.S/D-GC) | diff --git a/docs/reference/shared-core/database-schema.md b/docs/reference/shared-core/database-schema.md index 82f435a0..cbcd6300 100644 --- a/docs/reference/shared-core/database-schema.md +++ b/docs/reference/shared-core/database-schema.md @@ -736,9 +736,10 @@ CREATE INDEX idx_media_references_handle ON media_references (handle); - **Connection PRAGMAs** ([`client.ts`](../../../packages/db/src/client.ts)): `journal_mode = WAL` (readers never block the single writer, and vice-versa), `busy_timeout = 5000` (SQLite's built-in busy handler waits up to 5 s for a contended lock before returning `SQLITE_BUSY`), `synchronous = NORMAL` (the recommended durability/throughput trade-off under WAL), and `foreign_keys = ON`. - **Write transactions use `BEGIN IMMEDIATE`**, never drizzle's `DEFERRED` default. A DEFERRED transaction that reads before it writes takes a read lock first and must *upgrade* to a write lock on the first write — if another connection committed in between, that upgrade fails immediately with `SQLITE_BUSY` (`SQLITE_BUSY_SNAPSHOT`), which `busy_timeout` does **not** cover. `BEGIN IMMEDIATE` takes the write lock up front, so the upgrade race cannot occur. Applied to every multi-statement writer: `persistEvent` (run history), the model-catalog `replaceProviderModels` (bulk live-upsert) and `upsert` (per-model pricing), the provider `upsert` read-then-write, and the media-reference GC writes (`addReference`, `removeRunReferences`, `reclaimExpired`). It applies only to the OUTERMOST `BEGIN` — a store method called inside another transaction is demoted to a `SAVEPOINT` and the IMMEDIATE behavior is ignored, so a future batch-in-one-transaction caller must itself open `BEGIN IMMEDIATE`. -- **The bounded retry** (`packages/db/src/retry.ts`, `withBusyRetry`) wraps those write transactions and retries only `SQLITE_BUSY`/`SQLITE_LOCKED` up to a bounded attempt budget (default 5) with a **deterministic** linear backoff — **no jitter**, never `Math.random`, per the no-jitter/deterministic-replay convention of [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md). It is **fail-loud**: on an exhausted budget (or any non-lock fault) it rethrows the original error and never silently drops a write, preserving [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. A retried transaction rolls back with no partial write and re-runs the whole (idempotent) body. Because each attempt's `BEGIN IMMEDIATE` can itself wait up to `busy_timeout` (5 s) for the lock, the compounded worst case under sustained contention is ~25 s (5 attempts × 5 s + the sub-300 ms backoffs) of a synchronous block before the fail-loud rethrow — a deliberate durability-over-latency trade on a path that only stalls under pathological multi-writer contention. -- **Single-statement writes** (e.g. `appendMessage`, `setKeychainRef`) go straight for the write lock and rely on SQLite's built-in busy handler (`busy_timeout`); they need no explicit transaction. -- **Reads that must be consistent across statements** use a read transaction: `sessionStore.loadFull` reads the session row and its transcript inside one deferred transaction so **both reads observe a single consistent DB snapshot** (never a two-`SELECT` straddle across a concurrent commit). Note this guarantees *snapshot* consistency, not *turn* atomicity: the CLI persister writes a turn's messages and its updated session totals as separate auto-committed statements, so a snapshot can still legitimately observe messages ahead of their totals. A "totals always match the returned messages" guarantee would additionally require the host to persist each turn in one transaction (a tracked follow-up). +- **The bounded retry** (`packages/db/src/retry.ts`, `withBusyRetry` / `withBusyRetryAsync`) wraps those write transactions and retries only `SQLITE_BUSY` / `SQLITE_LOCKED` / `SQLITE_BUSY_SNAPSHOT` / `SQLITE_BUSY_RECOVERY` up to a bounded attempt budget (default 5) with a **deterministic** linear backoff — **no jitter**, never `Math.random`, per the no-jitter/deterministic-replay convention of [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md). `better-sqlite3` reports the **extended** result code, so the stale-DEFERRED upgrade failure arrives as the distinct string `SQLITE_BUSY_SNAPSHOT`, not as a `SQLITE_BUSY` prefix — matched exactly, so an unrelated future `SQLITE_BUSY_*` code is not swept into the set. It is **fail-loud**: on an exhausted budget (or any non-lock fault) it rethrows the original error and never silently drops a write, preserving [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. A retried transaction rolls back with no partial write and re-runs the whole (idempotent) body. Because each attempt's `BEGIN IMMEDIATE` can itself wait up to `busy_timeout` (5 s) for the lock, the compounded worst case under sustained contention is ~25 s (5 attempts × 5 s + the sub-300 ms backoffs) of blocking before the fail-loud rethrow — a deliberate durability-over-latency trade on a path that only stalls under pathological multi-writer contention. +- **Two retry twins, one policy.** `withBusyRetry` is synchronous, because the driver is. `withBusyRetryAsync` is the twin for the one call site whose caller is genuinely async today — `persistEvent` (run history) — where the backoff **yields the event loop** rather than parking the thread on `Atomics.wait`. Both share the retryable-code set, the attempt budget, the linear schedule and the fail-loud exhaustion; only the sleep differs. Read the scope precisely: this removes the **sub-300 ms** term of the ~25 s worst case above, not the dominant one — for `SQLITE_BUSY`/`SQLITE_LOCKED`, the codes that actually fire, each attempt still blocks inside the synchronous driver for up to `busy_timeout` (~5.2 s measured). `SQLITE_BUSY_SNAPSHOT` returns *immediately*, so for that code the backoff is the whole cost; it cannot reach a synchronous call site today because every one of them is `IMMEDIATE`, single-statement, or read-only. The remaining store methods stay synchronous deliberately: converting them would change five port interfaces and ripple through every CLI consumer without making a single write non-blocking. The async twin's `await` between attempts is an **observable yield**, so a sibling writer may commit during the backoff — the outcome the backoff exists to allow — which makes the "`fn` must be one self-contained, re-runnable transaction" requirement load-bearing rather than incidental. +- **Single-statement writes** (e.g. `appendMessage`, `setKeychainRef`) go straight for the write lock and need no explicit transaction — `BEGIN IMMEDIATE` would buy nothing, since a lone INSERT/UPDATE takes the write lock directly and cannot hit a read→write upgrade race. They do still route through `withBusyRetry` where a failure is user-visible data loss: the three `agent_sessions` / `session_messages` writers do (#228), because `busy_timeout` waits 5 s and then *fails*, and the chat persister calls them from inside a `RunEventBus` subscriber. +- **Reads that must be consistent across statements** use a read transaction: `sessionStore.loadFull` reads the session row and its transcript inside one deferred transaction so **both reads observe a single consistent DB snapshot** (never a two-`SELECT` straddle across a concurrent commit). This now composes with **turn atomicity** on the write side: `sessionStore.writeTurn` appends a turn's messages and flushes the session row in ONE `BEGIN IMMEDIATE` transaction (#228), so a snapshot can no longer observe messages ahead of their totals, and a failed write leaves neither a half-written turn nor an unanswered `user` row that resume cannot roll back. This discharges the per-turn-transaction follow-up this section previously tracked. - **Cross-platform:** `BEGIN IMMEDIATE` + the retry behave identically on every OS. The `0600`/`0700` at-rest guard below is a documented Windows no-op, so the concurrency test lane gates POSIX-permission assertions off Windows only. This realizes the concurrent-process write requirement recorded in the [ADR-0064](../../decisions/0064-live-model-catalog.md) §5 amendment note (2.5.I). diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index bbe276b5..f7bc9514 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -704,7 +704,7 @@ modes. The full design — gateway, key vault and pools, metering — is in ## Dependency posture The adapters prefer the official SDKs (`@anthropic-ai/sdk`, `openai`, -`@google/genai`) for typed event parsing and retry plumbing; **DeepSeek reuses +`@google/genai`) for typed event parsing and wire/SSE handling. **Their own retry is deliberately disabled (`maxRetries: 0`) for the chain-governed calls — `FallbackChain` is the sole retry authority there (error-handling.md §6, per ADR-0011; #276). The two surfaces the chain does not sit above — live model discovery and the async media-job poll — keep a small SDK retry, because nothing else can retry them.** **DeepSeek reuses the `openai` SDK with a custom `baseURL` (`api.deepseek.com`)** — no separate dependency. The SDKs live strictly **behind the adapter boundary** so they are swappable without touching any caller. A future implementation may drop the diff --git a/docs/reference/shared-core/tool-registry.md b/docs/reference/shared-core/tool-registry.md index 4ef74db5..4f82215b 100644 --- a/docs/reference/shared-core/tool-registry.md +++ b/docs/reference/shared-core/tool-registry.md @@ -233,12 +233,21 @@ interface ToolApprovalRequest { readonly toolId: ToolId; readonly action: ToolActionClass; // 'fs_write' | 'process' | 'egress' | 'os' (@relavium/shared TOOL_ACTION_CLASSES) readonly preview: ToolActionPreview; + /** + * The same field selection as `preview`, UNSCRUBBED — for IN-PROCESS CLASSIFICATION ONLY (#91). A host that + * classifies the target (the CLI's auto-mode protected-path check) must read this, never `preview`, because + * the scrub can turn a protected path into an unprotected-looking one. Deliberately ABSENT from the + * `agent:approval_requested` event, and MUST NEVER be serialized, logged, persisted, or forwarded across the + * event / IPC / `--json` boundary. Optional so a hand-built request stays constructible; the engine always + * sets it. + */ + readonly unredactedPreview?: ToolActionPreview; } -/** Secret-free, display-only — never a full URL/query, never a secret. */ +/** Secret-free (ENFORCED, not asserted), display-only — never a full URL/query, never a secret. */ interface ToolActionPreview { - readonly path?: string; // fs_write — the resolved target path - readonly command?: string; // process — the resolved command string + readonly path?: string; // fs_write — the resolved target path, scrubbed per SEGMENT + readonly command?: string; // process — the resolved command string, scrubbed whole readonly host?: string; // egress — the target host ONLY } @@ -247,6 +256,28 @@ type ToolApprovalDecision = | { readonly outcome: 'reject'; readonly reason?: string }; // a secret-free, display-safe label ``` +**Preview redaction (#91).** "Secret-free" is enforced by the registry, not asserted: every preview field is +scrubbed with the same `redactSecretShapedText` detector applied to `agent:tool_call.toolInput`, because the +ADR-0029(c) taint gate only covers *declared* `secret`-typed args — a model-placed credential in a +`run_command` arg or a `write_file` path is not one. Two consequences a consumer must know: + +- **A field may contain the literal `[redacted]` marker and is NOT guaranteed to be a resolvable + path/command/host. Never parse it.** It is a display projection; the dispatch always runs against the real, + unscrubbed target. +- **The scrub is whole-string, and a CLASSIFIER must not read it.** Two detector patterns have value classes + containing `/`, which cuts both ways: whole-string catches a credential whose value spans a separator, but + one match can also swallow the rest of a path (`./Access Token Backup/.ssh/authorized_keys` → + `./Access Token [redacted]`). No pattern gives both. So the two *uses* are split: `preview` is the display + copy, and `ToolApprovalRequest.unredactedPreview` carries the same field selection unscrubbed for + in-process classification only. It is deliberately **not** the raw `PolicyTarget` — an egress entry carries + the host and never the URL, whose query string is where a credential lives — and it is absent from the + `agent:approval_requested` event, so it never crosses the event / IPC / `--json` boundary. +- **A `command` preview can be shaped to blind the approver.** The command is fully model-controlled and the + detector's patterns are public, so an injected model can match one deliberately and be displayed as + `[redacted]` — which reads as "we protected you" rather than "you cannot see this". Bounded by + `allowedCommands`/`allowedCommandGlobs` being deny-all by default and by the fs/egress floors, but real; + surfacing "this was redacted" to the prompt is an open follow-up. + A reject ⇒ `ToolDeniedByUserError` (reason `user_rejected`); a hook that throws a non-abort error ⇒ the same fatal `tool_denied` (reason `approval_error`, fail-closed — consent could not be obtained, never the retryable `tool_failed` a *host-capability* throw gets); an absent hook under an active regime ⇒ diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index c97ebf22..fc6238c2 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -110,26 +110,32 @@ checklists before ~30 security-gated PRs are reviewed against them. 1. ✅ **Merge PR #76** (2.6.Q P1–P5, ADR-0071/0072, the three picker bugs, the param self-heal) — **done 2026-07-26**, along with #77 (this plan + the Phase-2.5.5 opening + `release.yml`'s `G26` ancestry gate). -2. **Reserve numbers**, one per number in landing order — migrations end at `0012` and ADRs at `0072` - (verified): `0013` = the Wave-1 approval-preview scrub · `0014` = 2.5.5.C's enum CHECKs (#101) · - `0015` = 2.6.H · `0016` = 2.6.G's pins · `0017` = 2.6.N lineage; **ADR-0073+** for the nine unwritten 2.6 ADRs. -3. **One `ci.yml`/`turbo.json`/`tsup.config.ts` PR** (a five-way collision file — do not split): +2. ✅ **Reserve numbers** — **done**, recorded here: migrations end at `0012` and ADRs at `0072` (verified), + so `0013` = the Wave-1 approval-preview scrub · `0014` = 2.5.5.C's enum CHECKs (#101) · `0015` = 2.6.H · + `0016` = 2.6.G's pins · `0017` = 2.6.N lineage; **ADR-0075+** for the nine unwritten 2.6 ADRs (`0073` = the history.db migration lock, `0074` = durable conservative budget commitments — both landed in Wave 1). +3. ✅ **One `ci.yml`/`turbo.json`/`tsup.config.ts` PR** — **done 2026-07-29 (PR #80)**, all eight items: 2.5.5.H · the required gate never runs the compiled binary + undeclared `drizzle` output (#294, #315) → local `pnpm run ci` vs `ci.yml` divergence (#312 — and `pnpm ci` is shadowed by a pnpm builtin, so the script was unreachable by the name everyone types) → the coverage-floor **ruling and its implementation** (#296, #152) → the `(advisory)` labels (#320) → `THIRD_PARTY_EXTERNAL` (G27, #248) → bundle-closure - single-chunk assert (#314) → `sync:models:check` (#317). *(`release.yml`'s ancestry check, `G26`, already - landed in #77; only its tag-protection half remains, and that is now configured too.)* -4. 2.5.5.F · **three skills' hardcoded foreign-project paths** (#162) — filed as docs, actually a functional - bug: they write outside the repo. -5. **One markdown PR** (shared files): `packages/mcp` missing from five inventory tables (#128, #129, #153, - #163, #254) + the two review-checklist gaps (#164, #167) + the CLAUDE.md/README i18n clause (#74, #260, #134). -6. The **locale-bar amendment** (decision D5) and the **snapshot regen** after the Gemini price ruling (D3). - -> Why first: every wave after this adds a migration, and the required gate still does not execute the -> binary it ships while turbo does not declare `apps/cli/drizzle/**` as a build output — so a cache-hit -> replay can leave `dist/index.js` fresh beside a stale `drizzle/`, crashing on first DB touch. The -> coverage gate must also flip *before* ~120 items land, not mid-flight. + single-chunk assert (#314) → `sync:models:check` (#317). *(`release.yml`'s ancestry check, `G26`, landed in + #77; its tag-protection half is configured.)* The PR also closed the four blockers from the 2026-07-29 + roadmap review, and its own CI caught two real defects in it — a TS4111 that `turbo run typecheck` does not + cover, and `pnpm ci` being unreachable because pnpm reserves `ci` as a builtin. +4. ✅ **2.5.5.F · three skills' hardcoded foreign-project paths** (#162) — **done 2026-07-29.** Filed as docs, + actually a functional bug: they pointed at `/Users/dev/…/Agent-Organizer/`, so `add-package`'s `mkdir -p` + would have created a tree outside the repo. All three now anchor on `$(git rev-parse --show-toplevel)`. +5. ✅ **One markdown PR** — **done 2026-07-29**: `packages/mcp` added to all five inventory sites (#128, #129, + #153, #163, #254), the reviewer agent's secrets item extended to the shipped CLI floor (#164), the + security-review skill's SSRF step now requires reusing the shared guard rather than re-deriving one (#167), + and the i18n clause settled by the D5 ruling (#74, #260, #134). +6. ✅ **The locale-bar amendment** (D5, `en`+`tr`) and the **snapshot regen** (D3) — **done**, propagated to + all seven locale sites and merged as PR #79 (two price rises taken, nine retirements accepted). + +> ✅ **Wave 0 is COMPLETE (2026-07-29).** The required gate now executes the binary it ships, +> `apps/cli/drizzle/**` is a declared turbo output, and `lint · typecheck · test` + +> `engine coverage floor (llm, mcp)` are both required checks under branch protection. In phase terms this +> closes **8 of 2.5.5.H's 14** tasks and **3 of 2.5.5.F's 20**. **Wave 1 is next.** ### Wave 1 — Stop the bleeding diff --git a/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md b/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md index 704e2af5..e7af8c62 100644 --- a/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md +++ b/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md @@ -351,11 +351,11 @@ The project's own documentation conventions — cite-not-restate, status banners - **Sync five roadmap/status docs that fell behind a merge event that updated a sibling doc in the same commit.** `current.md` still says 2.6.Q is blocked on seven decisions the maintainer must answer, but ADR-0071 (Accepted 2026-07-13) and ADR-0072 (Accepted 2026-07-14) already resolved the six — not seven — open ADR questions the phase file itself cites, and the "seven" figure contradicts the phase file's own "six open ADR questions" count in the same paragraph. `AGENTS.md` fell behind `CLAUDE.md`'s mirror contract. `docs/roadmap/phases/phase-2.6-conversational-authoring.md`'s own banner never picked up 2.6.C's completion. `docs/roadmap/phases/phase-3-desktop.md`'s status line predates the 2026-07-08 rescope. Fix all five in one pass; add a note recommending banner updates land in the same commit as any merge/rescope going forward. *(M · docs/roadmap/current.md, AGENTS.md, docs/roadmap/phases/phase-2.6-conversational-authoring.md, docs/roadmap/phases/phase-3-desktop.md · #127, #132, #133, #138, #139)* - **Refresh the flagship chat tutorial and reroute the onboarding path off 0%-built surfaces.** The chat tutorial is still marked "planned" though `chat.ts` is 2300+ lines and feature-complete; the recommended onboarding path routes a new user through desktop and VS Code, both 0%-built; the tutorials sub-index in the same folder omits the chat tutorial entirely. Rewrite the tutorial as shipped, reorder the recommended path around chat and a CI run, and fix the sub-index — one pass. *(M · docs/tutorials/ · #160, #161, #166)* - **Fix overview.md's system-map diagram: a Core→DB edge that doesn't exist, and a misdrawn LLM→Keychain injection boundary.** The diagram draws a `Core`→`DB` edge that has no counterpart in `packages/core/package.json` and contradicts the host-injected-`Checkpointer` framing used elsewhere in the same doc. The same diagram's `LLM`→`Keychain` edge misrepresents the injection boundary — this half is contested and needs a maintainer to confirm the corrected routing (`LLM`→`Transport`(injected)→host→`Keychain`) before the diagram is redrawn. Bundle both fixes into one PR once the routing is confirmed. *(S · docs/architecture/overview.md · #148, #149 — blocked: maintainer must confirm the corrected LLM→Keychain edge routing before redraw)* -- **Add `packages/mcp` to the five package-inventory tables it is missing from.** `packages/mcp` is a real, shipped, tested package absent from `CLAUDE.md`'s package table, `docs/project-structure.md`'s table/diagram/intro text, `docs/architecture/overview.md`'s package list/diagram, and the `.claude/agents/relavium-reviewer.md` reviewer agent's own inventory — the last of which risks an ADR-0052 boundary violation slipping past review undetected because the reviewer's checklist never looks at the package. Add it to all five sites in one pass. *(S · CLAUDE.md, docs/project-structure.md, docs/architecture/overview.md, .claude/agents/relavium-reviewer.md · #128, #129, #153, #163, #254)* +- ✅ **DONE (2026-07-29).** **Add `packages/mcp` to the five package-inventory tables it is missing from.** `packages/mcp` is a real, shipped, tested package absent from `CLAUDE.md`'s package table, `docs/project-structure.md`'s table/diagram/intro text, `docs/architecture/overview.md`'s package list/diagram, and the `.claude/agents/relavium-reviewer.md` reviewer agent's own inventory — the last of which risks an ADR-0052 boundary violation slipping past review undetected because the reviewer's checklist never looks at the package. Add it to all five sites in one pass. *(S · CLAUDE.md, docs/project-structure.md, docs/architecture/overview.md, .claude/agents/relavium-reviewer.md · #128, #129, #153, #163, #254)* - **Rewrite four docs that still name the retired hand-typed `pricing.ts` `MODEL_PRICING` table as the pricing authority, two ADR-generations stale.** `docs/reference/cli/commands.md`'s `relavium models` intro still describes the pre-ADR-0071/ADR-0072 architecture — found independently by three review units at the exact same line — and self-contradicts its own `models pricing` subsection two paragraphs later. The same doc's `models pricing --json` section is missing the `overriddenCatalogPrice` field and the `--clear --json` output shape. `docs/architecture/multi-llm-providers.md` separately names the deleted table as canonical. The models.dev analysis doc's provider-precedence table is stale with no forward link to ADR-0071/ADR-0072. Not gated on the blocked 2.6.Q — this is one coordinated pass rewriting all four docs against the now-Accepted ADRs. *(M · docs/reference/cli/commands.md, docs/architecture/multi-llm-providers.md, models.dev analysis doc · #14, #15, G13, G25, #165, #250)* - **Fix two node-types.md spec-accuracy bugs: an overstated `merge_strategy: 'first'` claim and three fabricated `human_in_the_loop_config` fields.** `docs/reference/core/node-types.md` overstates `merge_strategy: 'first'` as cancelling losing branches, contradicting `run-plan.md`'s accurate description of the same cost-relevant behavior. The same doc documents three `human_in_the_loop_config` fields that don't exist in the schema or the engine anywhere — an author following the doc as written hits an immediate parse rejection. Fix both in one pass. *(S · docs/reference/core/node-types.md · G10, G45)* -- **Close two gaps in the review checklists themselves: the reviewer agent's security item covers only the 0%-built desktop threat model, and the security-review skill's SSRF step never requires confirming reuse of the shared primitive.** `.claude/agents/relavium-reviewer.md`'s checklist item 5 covers only the unbuilt desktop threat model and omits the CLI-specific security floor that actually ships today. The `security-review` skill's SSRF step doesn't require confirming that a change reuses the shared SSRF-guard primitive rather than hand-rolling a new check. Add both missing checklist items in one pass. *(S · .claude/agents/relavium-reviewer.md, .claude/skills/security-review/SKILL.md · #164, #167)* -- **Anchor three skills' hardcoded foreign-project paths to the repo root.** Three skills under `.claude/skills/` embed literal hardcoded paths copy-pasted from an unrelated project — a functional bug, not a style nit: the skill either fails outright or, worse, writes files outside this repo when run as-is. Anchor each on `` `$(git rev-parse --show-toplevel)` `` the way sibling skills already correctly do. *(S · .claude/skills/ (3 files) · #162)* +- ✅ **DONE (2026-07-29).** **Close two gaps in the review checklists themselves: the reviewer agent's security item covers only the 0%-built desktop threat model, and the security-review skill's SSRF step never requires confirming reuse of the shared primitive.** `.claude/agents/relavium-reviewer.md`'s checklist item 6 (renumbered from 5 when the `packages/mcp` item landed) covers only the unbuilt desktop threat model and omits the CLI-specific security floor that actually ships today. The `security-review` skill's SSRF step doesn't require confirming that a change reuses the shared SSRF-guard primitive rather than hand-rolling a new check. Add both missing checklist items in one pass. *(S · .claude/agents/relavium-reviewer.md, .claude/skills/security-review/SKILL.md · #164, #167)* +- ✅ **DONE (2026-07-29).** **Anchor three skills' hardcoded foreign-project paths to the repo root.** Three skills under `.claude/skills/` embed literal hardcoded paths copy-pasted from an unrelated project — a functional bug, not a style nit: the skill either fails outright or, worse, writes files outside this repo when run as-is. Anchor each on `` `$(git rev-parse --show-toplevel)` `` the way sibling skills already correctly do. *(S · .claude/skills/ (3 files) · #162)* - **Repo-wide docs link/index hygiene sweep: four broken relative links, three missing ADR cross-reference pointers, and four stale reference-doc index tables.** Superseded ADR-0021 has no in-body amendment pointer forward to the new Node floor decision. ADR-0069's Related line 404s to a renamed file — it links `0047-cli-render-seam-and-framework-free-cores.md` but the real file is `docs/decisions/0047-cli-framework-commander-ink-clack.md`. ADR-0071's `### K7.` header gives no in-header signal that it belongs to a separate numbering track. An independent repo-wide crawl of every relative markdown link under `docs/` found four genuinely broken links — one of which duplicates the ADR-0069→ADR-0047 link above — plus a wrong-subdirectory link in ADR-0062. Separately, `docs/reference/README.md`'s shared-core table lists only 5 of 11 real files; `docs/standards/documentation-style.md` §6's canonical-artifact registry is missing 6 of 11, including the safety-critical `llm-provider-seam.md`; `docs/reference/shared-core/agent-runner.md` is orphaned from both indexes despite being cited by six other docs; the `docs/reference/cli/` index is missing `accessibility.md` and `regression-harness.md`. Regenerate all four index tables from the real file tree, fix the four broken links, and add the three missing ADR pointers in one coordinated sweep; recommend a CI link-checker afterward to prevent recurrence. *(M · docs/decisions/, docs/reference/README.md, docs/standards/documentation-style.md, docs/reference/cli/ · #123, #124, #126, #143, #151, #251, #252, #253)* - **Trim two ADRs and one project-structure.md row that restate content already living verbatim in their own cited canonical doc.** ADR-0031 states "canonical types live in `llm-provider-seam.md`" and then spends roughly 250 lines re-deriving the exact same content that file already carries. ADR-0027's addendum states exact sandbox resource-cap numbers that already live in `expression-sandbox-spec.md`, even after the ADR's own preamble names that spec the single canonical home. `docs/project-structure.md`'s Apps table restates a hand-picked, now-stale CLI subcommand list instead of linking to `docs/reference/cli/commands.md`, which already owns it canonically, and has drifted out of sync as a direct result. All three are the identical CLAUDE.md rule-8 violation — trim each ADR to drivers/tradeoffs plus a citation, and replace the stale restated list with a link. *(M · docs/decisions/0031-*.md, docs/decisions/0027-*.md, docs/project-structure.md · #120, #121, #257)* - **Add three missing rows/notes to `tech-stack.md`, the single source of truth for pinned choices.** `docs/tech-stack.md` omits `smol-toml` and `string-width` as ADR-backed runtime dependencies (both already shipped and in `package.json`), and is missing an MCP SDK shape-tradeoff note that should cross-reference ADR-0034. The Playwright e2e row is the one un-flagged not-yet-applicable row in an otherwise carefully phase-annotated doc. Add the two missing dependency rows, the tradeoff note, and the same Phase-3 flag every other not-yet-applicable row already carries, in one pass. *(S · docs/tech-stack.md · #246, #247, #140)* @@ -395,17 +395,17 @@ This substream is the development-process substrate itself — `.github/workflow **Tasks:** -- **The required PR gate never executes the compiled binary, and turbo's build task doesn't declare the drizzle side-effect that binary needs.** The required `ci` job in `.github/workflows/ci.yml` builds `apps/cli/dist/index.js` and checks bundle-closure metadata but never runs it — only the advisory `windows-concurrency` job and `release.yml`'s tag-gated smoke ever execute the real artifact (#294). Compounding it, the root `turbo.json` build task declares only `"outputs": ["dist/**"]`, but `apps/cli/tsup.config.ts`'s `onSuccess` hook copies `packages/db/drizzle` into `apps/cli/drizzle/` as a required runtime artifact outside `dist/` — so a cache-hit replay of `build` can leave `dist/index.js` fresh while `apps/cli/drizzle/` is missing or stale, crashing on first DB touch, invisible precisely because nothing in the required gate runs the binary (#315). Add a `node apps/cli/dist/index.js --version`/`--help`/fixture-`run --json` smoke to the required `ci` job, and declare `apps/cli/drizzle/**` as an additional turbo output for `apps/cli`'s build task. *(M · `.github/workflows/ci.yml` · #294, #315)* -- **Local `pnpm ci` and the real `ci.yml` gate have diverged in both directions.** The root `ci` script runs `pnpm lint:tools`, which `ci.yml` never invokes anywhere; conversely `ci.yml` runs a DB-migration-sync check (`pnpm --filter @relavium/db db:generate` + a `git status --porcelain` diff) the root script never calls — so a green local `pnpm ci` is not a faithful predictor of real CI in either direction, and none of `tools/engine-deps/check.mjs`, `tools/bundle-closure/check.mjs`, or `tools/sync-models-dev/sync.mjs` is ever linted in real CI. Add the missing `lint:tools` step to `ci.yml`'s main job and add the migration-sync check to the root `ci` script — ideally have `ci.yml` invoke the root script (or a documented subset) instead of re-listing overlapping-but-different commands. *(S · `package.json` · #312)* -- **`release.yml` publishes with an npm provenance attestation but never verifies the tagged commit is reachable from `main`.** The workflow triggers on any `v*` tag push and checks only that the tag's version string matches `apps/cli/package.json` — it never runs a `git merge-base --is-ancestor` check against `origin/main`, and no tag-protection ruleset is visible anywhere in the repo, so the `publish` job's only real gate beyond a green cross-OS smoke is possession of `NPM_TOKEN`. Add the ancestor-reachability assertion to the `pack` job. *(M · `.github/workflows/release.yml` · G26)* Blocking — full closure additionally needs a maintainer decision to configure a GitHub tag-protection ruleset restricting `v*` tag creation, a repo-settings change outside this PR's diff; the in-workflow ancestor check can land independently. +- ✅ **DONE (PR #80, 2026-07-29).** **The required PR gate never executes the compiled binary, and turbo's build task doesn't declare the drizzle side-effect that binary needs.** The required `ci` job in `.github/workflows/ci.yml` builds `apps/cli/dist/index.js` and checks bundle-closure metadata but never runs it — only the advisory `windows-concurrency` job and `release.yml`'s tag-gated smoke ever execute the real artifact (#294). Compounding it, the root `turbo.json` build task declares only `"outputs": ["dist/**"]`, but `apps/cli/tsup.config.ts`'s `onSuccess` hook copies `packages/db/drizzle` into `apps/cli/drizzle/` as a required runtime artifact outside `dist/` — so a cache-hit replay of `build` can leave `dist/index.js` fresh while `apps/cli/drizzle/` is missing or stale, crashing on first DB touch, invisible precisely because nothing in the required gate runs the binary (#315). Add a `node apps/cli/dist/index.js --version`/`--help`/fixture-`run --json` smoke to the required `ci` job, and declare `apps/cli/drizzle/**` as an additional turbo output for `apps/cli`'s build task. *(M · `.github/workflows/ci.yml` · #294, #315)* +- ✅ **DONE (PR #80).** **Local `pnpm run ci` and the real `ci.yml` gate have diverged in both directions.** *(Also found: `pnpm ci` never reached the script at all — pnpm reserves `ci` as a builtin.)* The root `ci` script runs `pnpm lint:tools`, which `ci.yml` never invokes anywhere; conversely `ci.yml` runs a DB-migration-sync check (`pnpm --filter @relavium/db db:generate` + a `git status --porcelain` diff) the root script never calls — so a green local `pnpm ci` is not a faithful predictor of real CI in either direction, and none of `tools/engine-deps/check.mjs`, `tools/bundle-closure/check.mjs`, or `tools/sync-models-dev/sync.mjs` is ever linted in real CI. Add the missing `lint:tools` step to `ci.yml`'s main job and add the migration-sync check to the root `ci` script — ideally have `ci.yml` invoke the root script (or a documented subset) instead of re-listing overlapping-but-different commands. *(S · `package.json` · #312)* +- ✅ **DONE (PR #77 + repo settings).** **`release.yml` publishes with an npm provenance attestation but never verifies the tagged commit is reachable from `main`.** The workflow triggers on any `v*` tag push and checks only that the tag's version string matches `apps/cli/package.json` — it never runs a `git merge-base --is-ancestor` check against `origin/main`, and no tag-protection ruleset is visible anywhere in the repo, so the `publish` job's only real gate beyond a green cross-OS smoke is possession of `NPM_TOKEN`. Add the ancestor-reachability assertion to the `pack` job. *(M · `.github/workflows/release.yml` · G26)* Blocking — full closure additionally needs a maintainer decision to configure a GitHub tag-protection ruleset restricting `v*` tag creation, a repo-settings change outside this PR's diff; the in-workflow ancestor check can land independently. - **`tools/engine-deps/check.mjs`'s allowlist and its purity model both stop short of the packages that need them.** `ENGINE_ALLOWLISTS` covers only `packages/shared`/`packages/llm`/`packages/core` — `packages/db`'s exclusion is deliberate but undocumented for `packages/mcp`, the one other package (besides `apps/cli`) that imports `@relavium/core`, leaving both packages' dependency growth policed only by human PR review instead of the mechanical gate the other three get (the identical missing-coverage observation, found independently in two review rounds, #285 and #318). Separately, the guard only reads each allowed package's own declared dependency *names* — it never resolves what an allowed transitive dependency (`yaml`, `quickjs-emscripten-core`) actually does, so a routine patch bump introducing a `node:fs`/`node:child_process` call inside one would pass every gate silently (#313). Extend `ENGINE_ALLOWLISTS` to cover `packages/db` and `packages/mcp`, and add a periodic browser-target bundle probe (esbuild `--platform=browser`, `node:*` externals set to fail-on-resolve) to catch transitive purity breaks. *(M · `tools/engine-deps/check.mjs` · #313, #318, #285)* -- **The documented ≥90% coverage floor is advisory in CI, and the standard says otherwise.** `docs/standards/testing.md` states unconditionally that `packages/core`/`packages/llm`/`packages/mcp` carry an "enforced floor >= 90%", but `vitest.config.ts`'s own header comment and `ci.yml`'s job name (`engine coverage floor (advisory)`) both say the job stays non-required pending a stability check — so a PR that drops core/llm/mcp coverage below 90% merges cleanly today, and the standard contradicts itself between its "coverage expectations" and "CI gate" sections. Found independently in two review rounds citing the identical contradiction. One maintainer decision resolves both: promote the job to required (a checked-in run already shows 92-97% margin) or soften `testing.md`'s wording to "advisory, tracked toward required." *(S · `vitest.config.ts` · #296, #152)* +- ✅ **DONE (PR #80).** Ruled, not reworded: `llm`+`mcp` are a required check; `core` is measured but not enforced (+0.83 branch margin), with Wave 3's test items as the promotion trigger. **The documented ≥90% coverage floor is advisory in CI, and the standard says otherwise.** `docs/standards/testing.md` states unconditionally that `packages/core`/`packages/llm`/`packages/mcp` carry an "enforced floor >= 90%", but `vitest.config.ts`'s own header comment and `ci.yml`'s job name (`engine coverage floor (advisory)`) both say the job stays non-required pending a stability check — so a PR that drops core/llm/mcp coverage below 90% merges cleanly today, and the standard contradicts itself between its "coverage expectations" and "CI gate" sections. Found independently in two review rounds citing the identical contradiction. One maintainer decision resolves both: promote the job to required (a checked-in run already shows 92-97% margin) or soften `testing.md`'s wording to "advisory, tracked toward required." *(S · `vitest.config.ts` · #296, #152)* - **No automated enforcement exists for CLAUDE.md's own binding documentation rules.** One-canonical-home, relative-only links, single-H1, kebab-case filenames, and no-front-matter are declared non-negotiable, yet `.prettierignore` explicitly excludes `docs/**`/`**/*.md`, and no markdownlint, link checker, or filename check runs anywhere in `.github/` or `tools/`. Write a ~100-line checker in the project's existing `tools/` style — validate every relative Markdown link under `docs/` resolves, assert exactly one top-level `#` H1 and no front-matter per file, flag non-kebab-case new/changed filenames — and wire it into CI. *(M · `docs/standards/documentation-style.md` · #316)* - **The binding logging standard describes an injected-logger architecture that has zero implementation anywhere in the codebase.** `docs/standards/logging-and-observability.md` says the engine "logs through an injected logger interface... so a surface... supplies its own sink"; a repo-wide grep of `packages/*/src` and `apps/cli/src` finds zero `console.*` calls, zero logger type on `ExecutionHost` (`packages/core/src/engine/execution-host.ts`), and no implementation anywhere (#266). Two live consequences: `correlationId` is stamped on every `node:failed`/`run:failed` event (`packages/core/src/engine/engine.ts:1298`) specifically so an operator can "join it to the structured internal log" — but none of the three `RunEvent` renderers nor `relavium logs`'s `detailOf()` ever print it, because there is nothing to join it to (#267); and `media-gc.ts`'s intentional bare `catch { return undefined; }` (lines 161, 194) means a persistent GC failure — a permissions problem, a locked file — has zero trace anywhere for its entire lifetime, not in a log, a `RunEvent`, or the DB (#270). *(L · `docs/standards/logging-and-observability.md` · #266, #267, #270)* Blocking — a maintainer decision gates #267/#270 closing properly: build the minimal injected-logger interface the standard describes and route the swallowed catches through it at `warn`, or formally amend the standard to name the run-event stream + persisted history as the only observability mechanism through Phase 2.6. -- **`sync:models:check`, the documented staleness-detection mode, is implemented and unused.** `tools/sync-models-dev/sync.mjs`'s own header documents `pnpm sync:models --check` as the CI-facing mode that fails if the committed `packages/llm/src/catalog/snapshot.ts` snapshot is stale, and it is fully implemented with a matching root script (`sync:models:check`), but `models-catalog.yml`'s weekly job calls plain `sync:models` — grepping every workflow file for `sync:models:check` returns zero hits. Wire the `:check` variant into the weekly job as a non-blocking, informational lane. *(S · `tools/sync-models-dev/sync.mjs` · #317)* -- **Two advisory CI jobs are missing the `(advisory)` label the other two carry.** `coverage` and `windows-concurrency` both self-label `(advisory)` in their job `name:` field in `.github/workflows/ci.yml`, which is what shows in the GitHub Actions/PR checks UI; `floor-check` and `peer-dep-gate` are equally advisory per the file's top-of-file comment but carry no such marker, so a contributor scanning the checks list can mistake them for required. Append `(advisory)` to both job names. *(S · `.github/workflows/ci.yml` · #320)* -- **The bundle-closure guard trusts there is exactly one esbuild output chunk without asserting it.** `tools/bundle-closure/check.mjs` picks a single `index.js` entry from the esbuild metafile via `.find()` and reads only its `imports` array; correct today (zero dynamic imports in `apps/cli/src`), but if a future dynamic `import()` causes esbuild to split a chunk, that chunk's external imports would never be inspected. Iterate all `.js` entries in `meta.outputs`, or assert `Object.keys(meta.outputs).length === 1` and fail loudly if that ever changes. *(S · `tools/bundle-closure/check.mjs` · #314)* -- **`THIRD_PARTY_EXTERNAL` is stale versus `package.json`, and its own doc-comment overstates what it enforces.** `apps/cli/tsup.config.ts`'s comment claims the array is what `tools/bundle-closure/check.mjs` diffs against to prevent drift, but `string-width` was added to `package.json` `dependencies` (ADR-0069) without being added to `THIRD_PARTY_EXTERNAL`, and `check.mjs` actually diffs the esbuild metafile against `package.json` directly, never against this array — so the comment's enforcement claim is false, harmlessly so today only because tsup auto-externalizes anything already in `dependencies`. Found independently in two review rounds at the identical lines. Add `'string-width'` to the array, or correct the comment to state plainly that the array is documentation-only. *(S · `apps/cli/tsup.config.ts` · G27, #248)* +- ✅ **DONE (PR #80).** Wired ahead of the mutating sync, so it reads the committed snapshot. **`sync:models:check`, the documented staleness-detection mode, is implemented and unused.** `tools/sync-models-dev/sync.mjs`'s own header documents `pnpm sync:models --check` as the CI-facing mode that fails if the committed `packages/llm/src/catalog/snapshot.ts` snapshot is stale, and it is fully implemented with a matching root script (`sync:models:check`), but `models-catalog.yml`'s weekly job calls plain `sync:models` — grepping every workflow file for `sync:models:check` returns zero hits. Wire the `:check` variant into the weekly job as a non-blocking, informational lane. *(S · `tools/sync-models-dev/sync.mjs` · #317)* +- ✅ **DONE (PR #80).** **Two advisory CI jobs are missing the `(advisory)` label the other two carry.** `coverage` and `windows-concurrency` both self-label `(advisory)` in their job `name:` field in `.github/workflows/ci.yml`, which is what shows in the GitHub Actions/PR checks UI; `floor-check` and `peer-dep-gate` are equally advisory per the file's top-of-file comment but carry no such marker, so a contributor scanning the checks list can mistake them for required. Append `(advisory)` to both job names. *(S · `.github/workflows/ci.yml` · #320)* +- ✅ **DONE (PR #80).** **The bundle-closure guard trusts there is exactly one esbuild output chunk without asserting it.** `tools/bundle-closure/check.mjs` picks a single `index.js` entry from the esbuild metafile via `.find()` and reads only its `imports` array; correct today (zero dynamic imports in `apps/cli/src`), but if a future dynamic `import()` causes esbuild to split a chunk, that chunk's external imports would never be inspected. Iterate all `.js` entries in `meta.outputs`, or assert `Object.keys(meta.outputs).length === 1` and fail loudly if that ever changes. *(S · `tools/bundle-closure/check.mjs` · #314)* +- ✅ **DONE (PR #80).** **`THIRD_PARTY_EXTERNAL` is stale versus `package.json`, and its own doc-comment overstates what it enforces.** `apps/cli/tsup.config.ts`'s comment claims the array is what `tools/bundle-closure/check.mjs` diffs against to prevent drift, but `string-width` was added to `package.json` `dependencies` (ADR-0069) without being added to `THIRD_PARTY_EXTERNAL`, and `check.mjs` actually diffs the esbuild metafile against `package.json` directly, never against this array — so the comment's enforcement claim is false, harmlessly so today only because tsup auto-externalizes anything already in `dependencies`. Found independently in two review rounds at the identical lines. Add `'string-width'` to the array, or correct the comment to state plainly that the array is documentation-only. *(S · `apps/cli/tsup.config.ts` · G27, #248)* - **A cluster of tests in the newest code races real wall-clock timeouts instead of fake timers.** `packages/core/src/expression/sandbox.test.ts:183` expects the QuickJS interrupt handler to fire within a real 50ms window; `apps/cli/src/render/tui/synchronized-output.test.tsx:51` and `apps/cli/src/render/scrollback.test.ts:188` wait real 20-100ms windows for an ink render or listener teardown to "settle" — all in the 2.6.F-era TUI renderer and the QuickJS sandbox, both actively developed, both at CI flakiness risk under runner contention. Convert to `vi.useFakeTimers()` + `vi.advanceTimersByTimeAsync()` wherever the code under test doesn't span a real process boundary. *(S · `packages/core/src/expression/sandbox.test.ts` · #298)* - **No property-based or fuzz testing exists anywhere in the repo.** A repo-wide grep for `fast-check` across every `package.json` and the lockfile returns nothing; the existing example-table style is already thorough (the SSRF `isPrivateOrLocalHost` suite hand-enumerates ~15 adversarial IP forms), so this raises the bar rather than closes a live gap, but the SSRF IP-form parser, the QuickJS sandbox's structured-clone boundary (`packages/core/src/expression/sandbox.ts`), and the `{{ }}` interpolation path resolver (`packages/core/src/interpolation/`) are exactly the class of parser/normalizer property-based testing exists for. Pilot `fast-check` (dev-only, no ADR required under the dependency policy's lower bar for dev tooling) scoped to those three files. *(M · `packages/llm/src/conformance` · #299)* - **The non-`Uint8Array` continue-branch in the catalog-refresh stream reader is untested.** `readCapped` (`apps/cli/src/engine/catalog-refresh.ts:167-195`) correctly narrows each chunk with `chunk.value instanceof Uint8Array` before accumulating bytes and `continue`s past any non-byte chunk — the right pattern given the DOM lib's `ReadableStream` typing — but no test exercises that branch to confirm a non-byte chunk is dropped rather than silently truncating a real pricing-overlay payload. Add a unit test feeding `readCapped` a reader that yields a non-`Uint8Array` chunk. *(S · `apps/cli/src/engine/catalog-refresh.ts` · #302)* diff --git a/packages/core/src/engine/agent-runner.e2e.test.ts b/packages/core/src/engine/agent-runner.e2e.test.ts index 8e034515..4477d2e7 100644 --- a/packages/core/src/engine/agent-runner.e2e.test.ts +++ b/packages/core/src/engine/agent-runner.e2e.test.ts @@ -442,6 +442,36 @@ workflow: ); } +/** Two independently-admissible calls whose combined worst-case output cost exceeds the authored cap (G38). */ +function parallelBudgetWorkflow(): ReturnType { + return parseWorkflow( + `schema_version: '1.0' +workflow: + id: e2e-budget-parallel-admission + max_parallel: 2 + budget: + max_cost_microcents: 750000 + on_exceed: fail + agents: + - id: a + model: claude-haiku-4-5 + provider: anthropic + system_prompt: hi + max_tokens: 1000 + nodes: + - id: n1 + type: agent + agent_ref: a + prompt_template: 'one' + - id: n2 + type: agent + agent_ref: a + prompt_template: 'two' + edges: [] +`, + ); +} + function cheapProvider(): LlmProvider { return provider([ { type: 'text_delta', text: 'ok' }, @@ -450,6 +480,99 @@ function cheapProvider(): LlmProvider { } describe('AgentRunner resource governance end-to-end (ADR-0028, 1.AC)', () => { + it('keeps the true in-provider attempt admission while a max_parallel sibling is denied (G38)', async () => { + // Haiku's 1,000-output-token worst case is 500,000µ¢. Each branch fits a 750,000µ¢ cap in isolation, + // but two max_parallel:2 TRUE provider attempts must NOT both reach egress: their combined reservation is + // 1,000,000µ¢. The first stream deliberately remains in-flight while the sibling reaches its own actual + // attempt boundary; this proves the ledger is not merely a transient loop-top probe. Before G38 this goes red + // with two provider calls and run:completed at 1,000,200µ¢ (the 1-input-token charge is included). + let calls = 0; + let signalFirstStarted: (() => void) | undefined; + const firstStarted = new Promise((resolve) => { + signalFirstStarted = () => resolve(); + }); + let releaseFirst: (() => void) | undefined; + const firstMayFinish = new Promise((resolve) => { + releaseFirst = () => resolve(); + }); + let signalSecondProviderCall: (() => void) | undefined; + const secondProviderCall = new Promise((resolve) => { + signalSecondProviderCall = () => resolve(); + }); + const providerWithCount: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('unused'); + }, + stream: () => { + calls += 1; + if (calls === 1) { + return (async function* (): AsyncGenerator { + if (signalFirstStarted === undefined) + throw new Error('first-attempt barrier was not initialized'); + signalFirstStarted(); + await firstMayFinish; + yield { type: 'text_delta', text: 'ok' }; + yield { + type: 'stop', + stopReason: 'stop', + usage: { inputTokens: 1, outputTokens: 1000 }, + }; + })(); + } + signalSecondProviderCall?.(); + return streamOf([ + { type: 'text_delta', text: 'ok' }, + { type: 'stop', stopReason: 'stop', usage: { inputTokens: 1, outputTokens: 1000 } }, + ]); + }, + }; + const engine = new WorkflowEngine({ + host: createInMemoryHost(), + executor: agentExecutor(() => providerWithCount), + }); + + const handle = engine.start({ workflow: parallelBudgetWorkflow() }); + const events: RunEvent[] = []; + let signalSiblingDenied: (() => void) | undefined; + const siblingDenied = new Promise((resolve) => { + signalSiblingDenied = () => resolve(); + }); + const consume = (async (): Promise => { + for await (const event of handle.events) { + events.push(event); + if (event.type === 'node:failed' && event.error.code === 'budget_exceeded') { + signalSiblingDenied?.(); + } + } + })(); + + try { + await firstStarted; + const release = releaseFirst; + if (release === undefined) throw new Error('first-attempt barrier was not initialized'); + // Race the sibling's budget failure against an observable second seam invocation WHILE the first provider is + // still blocked. Before G38, the old released probe reaches `stream()` first; the repaired ledger instead + // emits the sibling's budget failure before a second provider call exists. + const boundary = await Promise.race([ + siblingDenied.then(() => 'denied' as const), + secondProviderCall.then(() => 'second_provider_call' as const), + ]); + expect(boundary).toBe('denied'); + expect(calls).toBe(1); + } finally { + releaseFirst?.(); + } + await consume; + + expect(calls).toBe(1); + expect(events.at(-1)?.type).toBe('run:failed'); + const terminal = events.at(-1); + expect(terminal?.type === 'run:failed' && terminal.error.code).toBe('budget_exceeded'); + assertGapFreeSeq(events); + }); + it('warns and continues when on_exceed is warn', async () => { const engine = new WorkflowEngine({ host: createInMemoryHost(), @@ -461,8 +584,9 @@ describe('AgentRunner resource governance end-to-end (ADR-0028, 1.AC)', () => { expect(events.at(-1)?.type).toBe('run:completed'); const warning = events.find((e) => e.type === 'budget:warning'); expect(warning).toBeDefined(); - // The warning fires before any spend, so the observed spent/limit fraction is 0%. - expect(warning?.type === 'budget:warning' && warning.thresholdPct).toBe(0); + // `spentMicrocents` remains the honest realized 0, but G47 projects the post-call total for the warning + // threshold; the 1µ¢ cap is therefore fully exceeded (clamped to the event schema's 100%). + expect(warning?.type === 'budget:warning' && warning.thresholdPct).toBe(100); }); it('an UNPRICED model reaches the onUnpriced sink THROUGH WorkflowEngine (ADR-0071 §K7)', async () => { diff --git a/packages/core/src/engine/agent-runner.test.ts b/packages/core/src/engine/agent-runner.test.ts index 4062d44f..d904dd52 100644 --- a/packages/core/src/engine/agent-runner.test.ts +++ b/packages/core/src/engine/agent-runner.test.ts @@ -403,6 +403,41 @@ describe('createAgentNodeExecutor — output_schema + grant', () => { expect(streamCalls).toBe(1); // a single primary attempt — node.retry does NOT drive within-chain retry }); + it('an UNAUTHORED node gets 2 within-chain attempts — nothing else retries it (#276)', async () => { + // The companion to the test above. With an authored `retry:` the engine owns the budget above the chain, + // so the primary stays at one attempt. WITHOUT one, `#shouldRetry` returns false (`RetrySchema.max` has no + // default), so at a chain budget of 1 nothing retried at all and even the chain's backoff was unreachable. + // That was invisible until #276 disabled the vendor SDKs' internal retry, which had been absorbing it. + let streamCalls = 0; + const failing: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('unused'); + }, + stream: () => { + streamCalls += 1; + return streamOf([ + { + type: 'error', + error: { kind: 'overloaded', retryable: true, provider: 'anthropic', message: 'busy' }, + }, + ]); + }, + }; + const exec = createAgentNodeExecutor(deps(failing)); + const { ctx } = ctxFor( + // No `retry` block — the case the roadmap's own default workflow shape produces. + vertexFor({ kind: 'agent', node: agentNode({}), resolvedAgent: AGENT }), + ); + + const outcome = await exec.execute(ctx); + + expect(outcome.kind).toBe('failed'); + // TWO attempts: the chain is the only retry here, so it must carry a minimal budget. At 1 this is 1. + expect(streamCalls).toBe(2); + }); + it('resolves {{inputs.x}} into a user message, leaving system as the authored prompt', async () => { let capturedSystem: string | undefined; let capturedUser: string | undefined; @@ -696,6 +731,37 @@ describe('createAgentNodeExecutor — generative media (1.AG Section C, generate expect(info?.mediaUnitsEstimate).toEqual([{ modality: 'image', units: 2 }]); }); + it('settles the direct generative admission with realized cost before its event is emitted', async () => { + let releases = 0; + const settlements: number[] = []; + let settled = false; + const admission = { + settle: (realizedMicrocents: number): void => { + if (settled) return; + settled = true; + settlements.push(realizedMicrocents); + }, + settleAtReservedEstimate: (): void => { + if (settled) return; + settled = true; + }, + release: (): void => { + if (settled) return; + settled = true; + releases += 1; + }, + }; + const exec = createAgentNodeExecutor( + genDeps(generativeProvider(), { + preEgress: () => admission, + }), + ); + + expect((await exec.execute(ctxFor(genVertex()).ctx)).kind).toBe('completed'); + expect(settlements).toHaveLength(1); + expect(releases).toBe(0); + }); + it('maps a generateMedia provider error through the chat taxonomy (content_filter stays content_filter)', async () => { const exec = createAgentNodeExecutor( genDeps( @@ -930,6 +996,62 @@ describe('createAgentNodeExecutor — generative media (1.AG Section C, generate expect(called).toBe(false); }); + it('a cancel during async credential resolution releases before generateMedia egress', async () => { + const signal = { + aborted: false, + addEventListener: () => undefined, + removeEventListener: () => undefined, + }; + let resolveKey: ((key: string) => void) | undefined; + const deferredKey = new Promise((resolve) => { + resolveKey = resolve; + }); + let signalKeyRequested: (() => void) | undefined; + const keyRequested = new Promise((resolve) => { + signalKeyRequested = resolve; + }); + let generateCalls = 0; + const base = generativeProvider(); + const provider: LlmProvider = { + ...base, + generateMedia: (request, key) => { + generateCalls += 1; + return base.generateMedia?.(request, key) ?? Promise.reject(new Error('missing generator')); + }, + }; + let releases = 0; + let conservativeSettlements = 0; + const admission = { + settle: (): void => undefined, + settleAtReservedEstimate: (): void => { + conservativeSettlements += 1; + }, + release: (): void => { + releases += 1; + }, + }; + const { ctx } = ctxFor(genVertex()); + const pending = createAgentNodeExecutor( + genDeps(provider, { + keyFor: () => { + signalKeyRequested?.(); + return deferredKey; + }, + preEgress: () => admission, + }), + ).execute({ ...ctx, signal }); + + await keyRequested; + signal.aborted = true; + if (resolveKey === undefined) throw new Error('credential lookup was not initialized'); + resolveKey('test-key'); + + await expect(pending).resolves.toMatchObject({ kind: 'failed', error: { code: 'cancelled' } }); + expect(generateCalls).toBe(0); + expect(releases).toBe(1); + expect(conservativeSettlements).toBe(0); + }); + it('a BudgetPauseError on the generative pre-egress gate → paused (the human-gate seam, not internal)', async () => { // The generative path must map a budget PAUSE to the gate seam exactly like the chat path — a future // refactor that dropped the pause arm would leave a budget-paused generative run non-resumably internal-failed. diff --git a/packages/core/src/engine/agent-runner.ts b/packages/core/src/engine/agent-runner.ts index 9ab6ddd4..bbcd8b72 100644 --- a/packages/core/src/engine/agent-runner.ts +++ b/packages/core/src/engine/agent-runner.ts @@ -64,7 +64,7 @@ import { type ChainCapabilities, type PreEgressHook, } from './agent-turn.js'; -import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; +import { BudgetExceededError, BudgetPauseError, type BudgetAdmission } from './budget-governor.js'; import { effortToSend, gateReasoningEffort } from './reasoning-effort.js'; import type { EffortGateResult, @@ -80,6 +80,28 @@ import type { type AgentNode = AgentPlanConfig['node']; +/** + * An async media submission crosses from the runner to the engine as the exact `MediaJobSubmission` object. Keep + * its admission out of the persisted seam object: a lease is process-local bookkeeping, never checkpoint data or + * an adapter-visible type. The engine consumes it once when it parks the job; resume reconstructs a fresh lease + * from durable model/modality/units instead. + */ +const mediaJobAdmissions = new WeakMap(); + +function retainMediaJobAdmission( + job: MediaJobSubmission, + admission: BudgetAdmission | undefined, +): void { + if (admission !== undefined) mediaJobAdmissions.set(job, admission); +} + +/** Consume the process-local budget admission attached to a freshly submitted async media job, if any. */ +export function takeMediaJobAdmission(job: MediaJobSubmission): BudgetAdmission | undefined { + const admission = mediaJobAdmissions.get(job); + if (admission !== undefined) mediaJobAdmissions.delete(job); + return admission; +} + /** * The AgentRunner's injected dependencies — **platform capabilities only**. The genuinely-new one is * `resolveProvider`; `keyFor` / `sleep` / `now` / `onAuthError` are forwarded into the per-node @@ -496,14 +518,16 @@ async function executeGenerativeMedia( // must not add a spurious token addend on top of the media estimate. `outputModalities` is the validated // single modality (singleBilledModality), so the budget governor's media addend resolves the same rate. const preEgress = ctx.preEgress ?? deps.preEgress; + let admission: BudgetAdmission | undefined; if (preEgress !== undefined) { try { - await preEgress({ + const nextAdmission = await preEgress({ model: primary.model, maxTokens: 0, outputModalities: [modality.modality], mediaUnitsEstimate: [{ modality: modality.modality, units }], }); + if (nextAdmission !== undefined) admission = nextAdmission; } catch (err) { if (err instanceof BudgetExceededError) { return failed('budget_exceeded', err.message, false); @@ -511,58 +535,100 @@ async function executeGenerativeMedia( return turnOutcomeForError(err); // BudgetPauseError → paused; anything else re-throws → engine internal } } - // A cancel landing inside the (async) budget check costs no egress: re-check before engaging the provider. - if (ctx.signal.aborted) { - return failed( - 'cancelled', - `agent node '${node.id}': run cancelled before media generation`, - false, - ); - } + let egressStarted = false; + try { + // A cancel landing inside the (async) budget check costs no egress: re-check before engaging the provider. + if (ctx.signal.aborted) { + return failed( + 'cancelled', + `agent node '${node.id}': run cancelled before media generation`, + false, + ); + } - const req: MediaGenRequest = { - model: primary.model, - prompt, - modality: modality.modality, - ...(node.count === undefined ? {} : { count: node.count }), - ...(node.duration_seconds === undefined ? {} : { durationSeconds: node.duration_seconds }), - signal: ctx.signal, - }; + const req: MediaGenRequest = { + model: primary.model, + prompt, + modality: modality.modality, + ...(node.count === undefined ? {} : { count: node.count }), + ...(node.duration_seconds === undefined ? {} : { durationSeconds: node.duration_seconds }), + signal: ctx.signal, + }; - let key: string; - try { - key = await deps.keyFor(provider.id); - } catch { - // A host credential-resolution failure must NEVER surface its (possibly secret-bearing) message — replace - // it with a fixed, secret-free `provider_auth` failure, exactly as the FallbackChain's `#resolveKey` does - // on the chat path (rule 6). The original is dropped, not carried as a cause a sink could serialize. - return failed( - 'provider_auth', - `agent node '${node.id}': credential resolution failed for provider ${provider.id}`, - false, - ); - } + let key: string; + try { + key = await deps.keyFor(provider.id); + } catch { + // A host credential-resolution failure must NEVER surface its (possibly secret-bearing) message — replace + // it with a fixed, secret-free `provider_auth` failure, exactly as the FallbackChain's `#resolveKey` does + // on the chat path (rule 6). The original is dropped, not carried as a cause a sink could serialize. + return failed( + 'provider_auth', + `agent node '${node.id}': credential resolution failed for provider ${provider.id}`, + false, + ); + } - let result: MediaGenResult; - try { - result = await provider.generateMedia(req, key); - } catch (err) { - return mapGenerateMediaError(err); - } + // A key lookup can be asynchronous. If the run was cancelled while it was resolving, do not hand the now-stale + // credential to generateMedia: this is still proven pre-egress, so the `finally` below releases its admission. + if (ctx.signal.aborted) { + return failed( + 'cancelled', + `agent node '${node.id}': run cancelled before media generation`, + false, + ); + } - // A cancel that landed WHILE generateMedia was in-flight (a non-cooperative adapter that ignored the signal, - // or one that resolved just as the run cancelled) must win: skip BOTH the async park / sync media outcome AND - // the stray cost:updated below — mirroring the post-turn abort re-check the inline/stream chat paths run. - // (#onOutcome drops a post-cancel completed anyway; returning `cancelled` here also suppresses the cost emit.) - if (ctx.signal.aborted) { - return failed( - 'cancelled', - `agent node '${node.id}': run cancelled during media generation`, - false, + let result: MediaGenResult; + try { + // From this call onward the provider may have accepted/billed the generation even if its SDK throws or omits + // a terminal payload. Preserve the bounded reservation in those uncertain paths; only credential resolution + // above is proven pre-egress and may release it. + egressStarted = true; + result = await provider.generateMedia(req, key); + } catch (err) { + admission?.settleAtReservedEstimate(); + return mapGenerateMediaError(err); + } + + // A cancel that landed WHILE generateMedia was in-flight (a non-cooperative adapter that ignored the signal, + // or one that resolved just as the run cancelled) must win: skip BOTH the async park / sync media outcome AND + // the stray cost:updated below — mirroring the post-turn abort re-check the inline/stream chat paths run. + // (#onOutcome drops a post-cancel completed anyway; returning `cancelled` here also suppresses the cost emit.) + if (ctx.signal.aborted) { + admission?.settleAtReservedEstimate(); + return failed( + 'cancelled', + `agent node '${node.id}': run cancelled during media generation`, + false, + ); + } + + const outcome = buildGenerativeOutcome( + ctx, + node, + primary, + modality.modality, + units, + result, + deps.resolvePrice, + (realizedMicrocents) => admission?.settle(realizedMicrocents), ); + if (outcome.kind === 'media_job') { + retainMediaJobAdmission(outcome.job, admission); + admission = undefined; // ownership moved to the engine-owned parked-job lifecycle + } else if (outcome.kind !== 'completed' && egressStarted) { + // A malformed/mismatched provider result is still evidence that the provider processed a request. No exact + // charge is trustworthy, so retain the bounded estimate instead of treating an internal validation failure as + // a free call. + admission?.settleAtReservedEstimate(); + } + return outcome; + } finally { + // A synchronous completion settles actual cost before its event; a known pre-egress credential/cancel failure + // releases. Async ownership was transferred above. All ambiguous post-egress paths settled conservatively. + admission?.release(); } - - return buildGenerativeOutcome(ctx, node, primary, modality.modality, units, result); } /** @@ -596,6 +662,8 @@ function buildGenerativeOutcome( modality: MediaBilledModality, units: number, result: MediaGenResult, + resolvePrice: PricingOverlay | undefined, + onRealizedCost: (costMicrocents: number) => void, ): NodeOutcome { if (result.jobId !== undefined && result.media !== undefined) { return failed( @@ -638,13 +706,17 @@ function buildGenerativeOutcome( } // Exactly ONE realized cost:updated (ADR-0045 §5) — derived from the request volume × the per-model media // rate (best-effort: an unknown/unrated model degrades to 0, H4). The engine folds it into the run cumulative. + const costMicrocents = realizedMediaCost(primary.model, modality, units, resolvePrice); + // Settle before emitting to the engine: if a synchronous event sink faults after provider success, the admission + // cannot be released as though the charged generation never happened. + onRealizedCost(costMicrocents); ctx.emit({ type: 'cost:updated', nodeId: node.id, model: primary.model, inputTokens: 0, outputTokens: 0, - costMicrocents: realizedMediaCost(primary.model, modality, units), + costMicrocents, cumulativeCostMicrocents: 0, // placeholder — the engine overwrites with the authoritative run-wide total }); // The pure-media node output ({ text:'', media:[part] }) de-inlines at #emitDurable exactly like the inline @@ -704,12 +776,13 @@ export function realizedMediaCost( model: string, modality: MediaBilledModality, units: number, + resolvePrice?: PricingOverlay, ): number { const mediaUnits: MediaUnitsEntry[] = [ { modality, direction: 'output', units, unit: modality === 'image' ? 'count' : 'second' }, ]; try { - return cost(model, { inputTokens: 0, outputTokens: 0, mediaUnits }); + return cost(model, { inputTokens: 0, outputTokens: 0, mediaUnits }, resolvePrice); } catch { // Best-effort (H4): an unknown/unrated model — OR any pricing-layer fault — must NEVER turn a successful, // already-paid generation into a failed node. Degrade to 0, matching the chain's best-effort cost fold @@ -719,6 +792,13 @@ export function realizedMediaCost( } /** Build the ordered fallback plan: primary (node-over-agent model) + each authored fallback entry. */ +/** + * The primary chain entry's attempt budget on a workflow node with NO authored `retry:` block — the only case + * where the chain is the sole retry, because the engine's above-chain loop needs an authored budget to run. + * Mirrors `SESSION_PRIMARY_MAX_ATTEMPTS` in `agent-session.ts`; ADR-0040 carries the dated amendment. + */ +const UNAUTHORED_PRIMARY_MAX_ATTEMPTS = 2; + function buildPlanEntries( agent: Agent, node: AgentNode, @@ -728,16 +808,28 @@ function buildPlanEntries( if (primary === undefined) { return { ok: false, code: 'internal', message: `no provider wired for '${agent.provider}'` }; } - // The primary entry does NOT consume `node.retry` any more: ADR-0040 (amending ADR-0038) makes - // `node.retry` the engine's ABOVE-chain node-retry budget (applied around the whole chain), not the - // primary provider's within-chain same-model retry. The primary defaults to a single attempt + the - // chain's own default backoff; a within-chain primary retry, if ever wanted, is a future primary + // The primary entry does NOT consume `node.retry`: ADR-0040 (amending ADR-0038) makes `node.retry` the + // engine's ABOVE-chain node-retry budget (applied around the whole chain), not the primary provider's + // within-chain same-model retry. A within-chain primary retry, if ever wanted, is a future primary // `max_attempts` field (ADR-0040 A.2), not `retry`. + // + // But ONE attempt only when there IS an above-chain budget. `#shouldRetry` returns false for + // `retry === undefined`, and `RetrySchema.max` has no default — so a node with no authored `retry:` block + // has no retry above the chain at all, and at a chain budget of 1 the chain's own guard (`attempt < budget`) + // is false too, so nothing retries and even the backoff is unreachable. That was invisible until #276, + // because the vendor SDKs' internal retry was quietly absorbing transient 429/5xx inside the adapter. + // So: 2 when unauthored, 1 when authored — never both, which would MULTIPLY the author's budget (an + // authored `retry.max: 3` would become up to 6 real calls), exactly what ADR-0040 exists to prevent. + // + // "Authored" is `node.retry ?? agent.retry`, matching `Engine#retryConfig` EXACTLY (ADR-0040 A.8). Checking + // only `node.retry` looked right and was wrong: an AGENT-level `retry:` also gives the engine a budget, so + // that reading double-counted it. Two e2e tests caught it, which is what they are for. + const aboveChainBudgetExists = (node.retry ?? agent.retry) !== undefined; const entries: FallbackPlanEntry[] = [ { provider: primary, model: node.model ?? agent.model, - maxAttempts: 1, + maxAttempts: aboveChainBudgetExists ? 1 : UNAUTHORED_PRIMARY_MAX_ATTEMPTS, }, ]; for (const entry of agent.fallback_chain ?? []) { diff --git a/packages/core/src/engine/agent-session.test.ts b/packages/core/src/engine/agent-session.test.ts index 9086bb03..cee01442 100644 --- a/packages/core/src/engine/agent-session.test.ts +++ b/packages/core/src/engine/agent-session.test.ts @@ -451,6 +451,35 @@ describe('AgentSession (1.V) — multi-turn entry point over the shared turn cor expect(completed.tokensUsed).toEqual({ input: 4, output: 2 }); }); + it('RETRIES a retryable failure once on the session path — the primary carries a budget of 2 (#276)', async () => { + // A session has no above-chain retry loop, so the chain is the only retry here. Until #276 the vendor + // SDK's internal retry was quietly absorbing a transient 429 inside the adapter; with that correctly + // disabled, a primary budget of 1 would end the user's turn on the first rate limit with no wait at all + // (at budget 1 the chain's own guard `attempt < budget` is false, so even its backoff is unreachable). + // Two scripted turns: the first fails retryable, the second succeeds — one turn must consume BOTH. + const { deps, events } = harness([ + [ + { + type: 'error', + error: { kind: 'rate_limit', retryable: true, provider: 'anthropic', message: 'rl' }, + }, + ], + [ + { type: 'text_delta', text: 'recovered' }, + { type: 'stop', stopReason: 'stop', usage: { inputTokens: 5, outputTokens: 3 } }, + ], + ]); + const s = session(deps); + s.start(); + await s.sendMessage('go'); + + const completed = events.find((e) => e.type === 'session:turn_completed'); + // The turn SUCCEEDS: the retry landed. At maxAttempts 1 this is `stopReason: 'error'`. + expect(completed?.type === 'session:turn_completed' ? completed.stopReason : undefined).toBe( + 'stop', + ); + }); + it('completes a turn with an error (stopReason error) when the provider chain is exhausted', async () => { const { deps, events } = harness([ [ @@ -950,6 +979,9 @@ describe('AgentSession — reseat-less modes + mid-turn abort (ADR-0057 Step 2)' toolId: 'write_file', action: 'fs_write', preview: { path: './out.txt' }, + // The unredacted classification copy (#91). It rides the in-process request only — the assertion + // below is precisely that it does NOT reach the emitted event. + unredactedPreview: { path: './out.txt' }, }); const approvalEvent = events.find((e) => e.type === 'agent:approval_requested'); expect(approvalEvent).toMatchObject({ @@ -959,6 +991,9 @@ describe('AgentSession — reseat-less modes + mid-turn abort (ADR-0057 Step 2)' action: 'fs_write', preview: { path: './out.txt' }, }); + // #91: the UNREDACTED classification target rides the in-process request only. It must never reach the + // event — that body is persisted, `--json`-visible, and its preview object is `.strict()`. + expect(approvalEvent).not.toHaveProperty('unredactedPreview'); // The emitted body, once the sink stamps the session envelope (1.W), is a SCHEMA-VALID run event — the // action-bound preview + dual-envelope refinements accept it (this is what the bus parses against). const validated = RunEventSchema.safeParse({ diff --git a/packages/core/src/engine/agent-session.ts b/packages/core/src/engine/agent-session.ts index 858b2b51..14a229e3 100644 --- a/packages/core/src/engine/agent-session.ts +++ b/packages/core/src/engine/agent-session.ts @@ -370,6 +370,18 @@ function isProcessResult(value: unknown): value is ProcessResult { * `sessionId`, call {@link start} once, then {@link sendMessage} per user turn; {@link cancel} aborts an * in-flight turn. Events flow through the injected {@link SessionEventSink}. */ +/** + * The primary chain entry's attempt budget on the SESSION path (chat / one-shot `agent run`). + * + * **2, not 1**, and the difference is load-bearing: a session has no above-chain retry loop, so the chain is + * the only retry there — and a budget of 1 makes even the chain's own backoff unreachable (`attempt < budget`). + * Until #276 the vendor SDK's internal retry hid that; with the SDK correctly out of the retry business, a + * transient 429 would end the user's turn with no wait at all. Two attempts restores a single polite retry + * without touching the authored node-retry budget, which does not exist on this path. + * ADR-0040 carries the dated amendment. + */ +const SESSION_PRIMARY_MAX_ATTEMPTS = 2; + export class AgentSession { readonly sessionId: string; @@ -1199,10 +1211,17 @@ export class AgentSession { this.#plan = { ok: false, message: `no provider wired for '${agent.provider}'` }; return this.#plan; } - // The primary entry does NOT consume a node/agent retry budget — ADR-0040 makes node retry the - // engine's ABOVE-chain budget (a session has no such loop in 1.V); the primary is a single attempt. + // ADR-0040 makes node retry the engine's ABOVE-chain budget, and the primary entry does not consume it. + // But a SESSION has no such loop (this comment used to end there, with `maxAttempts: 1`), so on the chat + // and one-shot `agent run` surfaces the chain was the ONLY place a retry could happen — and with a budget + // of 1 the guard `attempt < budget` is false, so `#backoff` is unreachable and nothing retried at all. + // That was masked until #276: the vendor SDK's own retry was silently absorbing transient 429/5xx here. + // With the SDK's retry correctly off, a bare 429 would fail the turn outright with no wait, so the primary + // carries a minimal budget of 2 on this path. The WORKFLOW path deliberately stays at 1 (see + // `agent-runner.ts`) because the engine retries above it there — raising both would multiply the authored + // budget. Recorded as a dated amendment on ADR-0040. const entries: FallbackPlanEntry[] = [ - { provider: primary, model: agent.model, maxAttempts: 1 }, + { provider: primary, model: agent.model, maxAttempts: SESSION_PRIMARY_MAX_ATTEMPTS }, ]; for (const entry of agent.fallback_chain ?? []) { const provider = this.#deps.resolveProvider(entry.provider); diff --git a/packages/core/src/engine/agent-turn.test.ts b/packages/core/src/engine/agent-turn.test.ts index 1e0b1ed4..5a6d6971 100644 --- a/packages/core/src/engine/agent-turn.test.ts +++ b/packages/core/src/engine/agent-turn.test.ts @@ -1008,16 +1008,243 @@ describe('runAgentTurn — failover + cancel + reasoning', () => { return streamOf([{ type: 'text_delta', text: 'must not run' }, STOP()]); }, }; + let releases = 0; + let conservativeSettlements = 0; + const admission = { + settle: (): void => undefined, + settleAtReservedEstimate: (): void => { + conservativeSettlements += 1; + }, + release: (): void => { + releases += 1; + }, + }; // The budget hook is awaited, so the signal can fire during that await; simulate it firing there. const params = baseParams(provider, { signal: aborted, preEgress: () => { aborted.aborted = true; - return Promise.resolve(); + return Promise.resolve(admission); }, }); await expect(runAgentTurn(params)).rejects.toMatchObject({ code: 'cancelled' }); expect(streamed).toBe(false); // the re-check fired before the provider was engaged + expect(releases).toBe(1); + expect(conservativeSettlements).toBe(0); + }); + + it('releases an admission when credential resolution fails before provider egress', async () => { + let streamed = false; + const provider: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('generate not used in these tests'); + }, + stream: (): AsyncIterable => { + streamed = true; + return streamOf([{ type: 'text_delta', text: 'must not run' }, STOP()]); + }, + }; + let releases = 0; + let conservativeSettlements = 0; + const admission = { + settle: (): void => undefined, + settleAtReservedEstimate: (): void => { + conservativeSettlements += 1; + }, + release: (): void => { + releases += 1; + }, + }; + + await expect( + runAgentTurn( + baseParams(provider, { + chainCapabilities: { + ...CAPABILITIES, + keyFor: () => Promise.reject(new Error('secret-bearing credential lookup failure')), + }, + preEgress: () => admission, + }), + ), + ).rejects.toMatchObject({ code: 'provider_auth' }); + + expect(streamed).toBe(false); + expect(releases).toBe(1); + expect(conservativeSettlements).toBe(0); + }); + + it('releases an admission when cancellation lands during credential resolution', async () => { + const aborted = { + aborted: false, + addEventListener: () => undefined, + removeEventListener: () => undefined, + }; + let streamed = false; + const provider: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('generate not used in these tests'); + }, + stream: (): AsyncIterable => { + streamed = true; + return streamOf([{ type: 'text_delta', text: 'must not run' }, STOP()]); + }, + }; + let resolveKey: ((key: string) => void) | undefined; + const deferredKey = new Promise((resolve) => { + resolveKey = resolve; + }); + let signalKeyRequested: (() => void) | undefined; + const keyRequested = new Promise((resolve) => { + signalKeyRequested = resolve; + }); + let releases = 0; + let conservativeSettlements = 0; + const admission = { + settle: (): void => undefined, + settleAtReservedEstimate: (): void => { + conservativeSettlements += 1; + }, + release: (): void => { + releases += 1; + }, + }; + const pending = runAgentTurn( + baseParams(provider, { + signal: aborted, + chainCapabilities: { + ...CAPABILITIES, + keyFor: () => { + signalKeyRequested?.(); + return deferredKey; + }, + }, + preEgress: () => admission, + }), + ); + await keyRequested; + aborted.aborted = true; + if (resolveKey === undefined) throw new Error('credential lookup was not started'); + resolveKey('test-key'); + + await expect(pending).rejects.toMatchObject({ code: 'cancelled' }); + expect(streamed).toBe(false); + expect(releases).toBe(1); + expect(conservativeSettlements).toBe(0); + }); + + it('admits once at the true provider boundary and settles that matching admission exactly once', async () => { + const provider = scriptedProvider('anthropic', [[{ type: 'text_delta', text: 'ok' }, STOP()]]); + let checks = 0; + let attemptReleases = 0; + const attemptSettlements: number[] = []; + let conservativelySettled = 0; + let settled = false; + const attemptProbe = { + settle: (realizedMicrocents: number): void => { + if (settled) return; + settled = true; + attemptSettlements.push(realizedMicrocents); + }, + settleAtReservedEstimate: (): void => { + if (settled) return; + settled = true; + conservativelySettled += 1; + }, + release: (): void => { + if (settled) return; + settled = true; + attemptReleases += 1; + }, + }; + const params = baseParams(provider, { + preEgress: () => { + checks += 1; + return attemptProbe; + }, + }); + + await expect(runAgentTurn(params)).resolves.toMatchObject({ text: 'ok' }); + expect(checks).toBe(1); // exactly the true FallbackChain provider boundary + expect(attemptReleases).toBe(0); + expect(conservativelySettled).toBe(0); + expect(attemptSettlements).toHaveLength(1); + expect(attemptSettlements[0]).toBeGreaterThan(0); + }); + + it('conservatively settles a true provider-attempt admission when usage is absent', async () => { + const provider = scriptedProvider('anthropic', [ + [ + { + type: 'error', + error: { kind: 'auth', retryable: false, provider: 'anthropic', message: 'bad key' }, + }, + ], + ]); + let checks = 0; + let attemptReleases = 0; + const attemptSettlements: number[] = []; + let conservativeSettlements = 0; + let settled = false; + const attemptProbe = { + settle: (realizedMicrocents: number): void => { + if (settled) return; + settled = true; + attemptSettlements.push(realizedMicrocents); + }, + settleAtReservedEstimate: (): void => { + if (settled) return; + settled = true; + conservativeSettlements += 1; + }, + release: (): void => { + if (settled) return; + settled = true; + attemptReleases += 1; + }, + }; + const params = baseParams(provider, { + preEgress: () => { + checks += 1; + return attemptProbe; + }, + }); + + await expect(runAgentTurn(params)).rejects.toMatchObject({ code: 'provider_auth' }); + expect(checks).toBe(1); + expect(attemptReleases).toBe(0); + expect(conservativeSettlements).toBe(1); + expect(attemptSettlements).toEqual([]); + }); + + it('conservatively settles a clean provider EOF that omits the terminal usage record', async () => { + // FallbackChain treats an iterator ending without a `stop` chunk as a successful empty turn. It may still have + // reached/billed the provider, so this must not be mistaken for the proven pre-egress release path. + const provider = scriptedProvider('anthropic', [[]]); + let releases = 0; + let conservativeSettlements = 0; + const admission = { + settle: (): void => undefined, + settleAtReservedEstimate: (): void => { + conservativeSettlements += 1; + }, + release: (): void => { + releases += 1; + }, + }; + + await expect( + runAgentTurn( + baseParams(provider, { + preEgress: () => admission, + }), + ), + ).resolves.toMatchObject({ text: '' }); + expect(conservativeSettlements).toBe(1); + expect(releases).toBe(0); }); it('maps a pre-egress BudgetExceededError to AgentTurnError(budget_exceeded) — no provider egress', async () => { diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index 6ce1c630..b86befdd 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -56,7 +56,7 @@ import { import { ToolDispatchError } from '../tools/errors.js'; import type { ToolCallPart, ToolDispatchContext, ToolRegistry } from '../tools/types.js'; import { unwrapUntrusted } from '../tools/untrusted.js'; -import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; +import { BudgetExceededError, BudgetPauseError, type BudgetAdmission } from './budget-governor.js'; import type { NodeStreamEvent } from './node-executor.js'; /** @@ -103,6 +103,10 @@ export const DEFAULT_AGENT_TURN_LIMITS: AgentTurnLimits = { * (`budget_exceeded`) to halt. `outputModalities` + `mediaUnitsEstimate` (1.AF/D17) are populated by the * AgentRunner from the node's `output_modalities` + the `[defaults].media_cost_estimate` unit counts so the * governor can fold a media cost estimate into the projected total; both absent ⇒ a text-only turn. + * When the hook returns a {@link BudgetAdmission}, the matching true provider-attempt owner settles it with the + * realized charge, conservatively settles its reserved estimate when the provider may have billed without usage, + * or releases it only when no egress can be attributed. The hook runs at FallbackChain's real attempt boundary; + * there is deliberately no speculative loop-top reservation. */ export type PreEgressHook = (info: { readonly model: string; @@ -113,7 +117,7 @@ export type PreEgressHook = (info: { readonly provider?: ProviderId; readonly outputModalities?: readonly OutputModality[]; readonly mediaUnitsEstimate?: readonly MediaUnitsEstimate[]; -}) => void | Promise; +}) => void | BudgetAdmission | Promise; /** The chain capabilities the host supplies (the platform-level subset of {@link FallbackChainOptions}). */ export type ChainCapabilities = Pick< @@ -611,37 +615,6 @@ async function dispatchToolCalls( return { messages: results, correctable }; } -/** - * Await the pre-egress budget hook before a provider call, mapping a budget-cap failure to the closed turn - * error taxonomy: a {@link BudgetExceededError} (`on_exceed: fail`) → `AgentTurnError('budget_exceeded')`; - * a {@link BudgetPauseError} (`pause_for_approval`) and any other error propagate as-is (the run path maps - * the pause to a `paused` node outcome). Extracted from the turn loop to keep its complexity in budget. - */ -async function awaitPreEgress( - params: AgentTurnParams, - activeModel: string, - activeProvider: ProviderId | undefined, -): Promise { - try { - await params.preEgress?.({ - model: activeModel, - ...(activeProvider === undefined ? {} : { provider: activeProvider }), - ...(params.maxTokens === undefined ? {} : { maxTokens: params.maxTokens }), - ...(params.outputModalities === undefined - ? {} - : { outputModalities: params.outputModalities }), - ...(params.mediaUnitsEstimate === undefined - ? {} - : { mediaUnitsEstimate: params.mediaUnitsEstimate }), - }); - } catch (err) { - if (err instanceof BudgetExceededError) { - throw new AgentTurnError('budget_exceeded', err.message, false); - } - throw err; - } -} - /** * Append a tool-use assistant turn, dispatch its tool calls, append the results to `messages`, and return * the updated correction count. Throws a classified {@link AgentTurnError} on a protocol anomaly (a @@ -747,25 +720,86 @@ async function driveAgentTurn( // overlay (2.5.G S10) lets the tracker price a user-priced model the static registry lacks. const costTracker = new CostTracker(params.resolvePrice); let activeModel = primaryModel; - // The provider that pairs with `activeModel`, updated together in `onAttempt`. A failover moves BOTH, so the - // pre-egress endpoint estimate keys on the routing provider actually in play, not the model's catalog provider - // (review M2). Starts on the primary entry's provider. - let activeProvider: ProviderId | undefined = params.planEntries[0]?.provider.id; let nonSkippedAttempts = 0; + const preEgress = params.preEgress; + // A chain executes one provider attempt at a time, so one per-turn slot is sufficient. It is deliberately NOT a + // governor-global FIFO: parallel nodes can settle out of order, and a failed/no-usage attempt has no cost event + // from which a global queue could infer which reservation to release. + let attemptAdmission: BudgetAdmission | undefined; + let admissionPending = false; + // A chain record can be emitted for a materialization failure or a rejected pre-attempt hook, both of which are + // before provider egress. Only a hook that returned successfully arms this flag, so a budget refusal/cancel does + // not falsely count as an engaged provider turn or settle a nonexistent bill. + let attemptReady = preEgress === undefined; + // `FallbackChain` resolves credentials after its pre-attempt hook. The hook can therefore reserve capacity before + // a host key lookup rejects, but a failed lookup is PROVEN pre-provider and must release that reservation rather + // than becoming a permanent conservative debit. The wrapped `keyFor` below flips this only after it resolved and + // a post-resolution cancellation check still permits the seam call. + let credentialResolvedForAttempt = preEgress === undefined; + const settleUnreportedAttemptAdmission = (): void => { + attemptReady = preEgress === undefined; + credentialResolvedForAttempt = preEgress === undefined; + if (!admissionPending) return; + admissionPending = false; + const active = attemptAdmission; + attemptAdmission = undefined; + // A prior true-boundary admission with no matching AttemptRecord means the consumer/iterator failed after the + // chain might have handed work to a provider. A release would reopen capacity on an unknown bill; retain the + // bounded estimate instead. Proven pre-provider cancellation releases its just-returned admission before this + // slot is ever armed (below). + active?.settleAtReservedEstimate(); + }; + const takeAttemptAdmission = (): BudgetAdmission | undefined => { + if (!admissionPending) return undefined; + admissionPending = false; + const active = attemptAdmission; + attemptAdmission = undefined; + return active; + }; const onAttempt = (record: AttemptRecord): void => { // A SKIPPED entry (cooldown / capability) was not invoked — it must not become `activeModel`, or // the next entry's streamed tokens would be mis-attributed to a provider that never ran. if (record.outcome === 'skipped') return; + const providerMayHaveEngaged = attemptReady && credentialResolvedForAttempt; + // With no governor hook, preserve the established FallbackChain contract: every non-skipped record represents + // the chain's best available attempt trace. With a governor hook, only its successful true-boundary callback + // proves the provider could have been reached. + if (preEgress !== undefined) { + attemptReady = false; + credentialResolvedForAttempt = false; + } + const admission = takeAttemptAdmission(); + if (!providerMayHaveEngaged) { + // A successful pre-attempt check followed by a credential failure/cancellation never reached a provider. + // This is the one path where the held admission is conclusively safe to release after the hook returned. + admission?.release(); + return; + } activeModel = record.model; - activeProvider = record.provider; nonSkippedAttempts += 1; usage.engaged = true; // a non-skipped attempt RAN — mark engaged even if it then errored at zero usage - if (record.usage === undefined) return; + if (record.usage === undefined) { + // A clean EOF and a partial-stream failure can both omit terminal usage AFTER provider egress. Dropping the + // reservation would silently reopen cap capacity for money that may already be owed, so fail closed at the + // bounded estimate. A credential/materialization failure before the true attempt boundary never reaches here. + admission?.settleAtReservedEstimate(); + return; + } usage.input += record.usage.inputTokens; usage.output += record.usage.outputTokens; // The chain already folded this attempt's usage into our `costTracker` and put the per-attempt // figure on `record.cost`; read it rather than re-recording (which would double the total). + // Settle BEFORE the event sink: it replaces this attempt's reservation with the realized amount atomically, + // keeping the governor safe even if a re-entrant/throwing sink prevents the authoritative engine update below. + // `FallbackChain` intentionally tolerates a CostTracker failure so a successful response stays usable. When + // that happens there is no trustworthy actual price, not proof of a free call — keep the reservation as the + // conservative charge. For an unpriced model no admission exists, so its existing allow-degrade path remains. + if (record.cost === undefined) { + admission?.settleAtReservedEstimate(); + } else { + admission?.settle(record.cost.costMicrocents); + } params.emit({ type: 'cost:updated', nodeId: params.nodeId, @@ -783,9 +817,26 @@ async function driveAgentTurn( }); }; - const preEgress = params.preEgress; + const chainCapabilities: ChainCapabilities = + preEgress === undefined + ? params.chainCapabilities + : { + ...params.chainCapabilities, + keyFor: async (provider) => { + const key = await params.chainCapabilities.keyFor(provider); + // FallbackChain performs no abort poll between resolving a key and entering the adapter. Keep the + // admission in the known-pre-egress state when cancellation lands in that narrow await window; its + // ensuing attempt record releases the lease in `onAttempt` above. + if (params.signal.aborted) { + throw new Error('request aborted before provider egress'); + } + credentialResolvedForAttempt = true; + return key; + }, + }; + const chain = new FallbackChain([...params.planEntries], { - ...params.chainCapabilities, + ...chainCapabilities, costTracker, onAttempt, // The pre-egress budget hook runs before EVERY provider attempt, not just the first turn, so a failover @@ -796,12 +847,17 @@ async function driveAgentTurn( ...(preEgress === undefined ? {} : { - preAttempt: (info: { + preAttempt: async (info: { readonly model: string; readonly provider: ProviderId; readonly maxTokens?: number; - }) => - preEgress({ + }) => { + // This is the only admitting boundary. Check cancellation on BOTH sides of the awaited governor call: + // a cancellation landing while warning durability/admission is pending must not reach key resolution + // or provider egress, and any just-acquired lease is released before the cancellation propagates. + settleUnreportedAttemptAdmission(); + throwIfAborted(params.signal); + const nextAdmission = await preEgress({ ...info, ...(params.outputModalities === undefined ? {} @@ -809,7 +865,17 @@ async function driveAgentTurn( ...(params.mediaUnitsEstimate === undefined ? {} : { mediaUnitsEstimate: params.mediaUnitsEstimate }), - }), + }); + if (params.signal.aborted) { + nextAdmission?.release(); + throwIfAborted(params.signal); + } + if (nextAdmission !== undefined) { + attemptAdmission = nextAdmission; + admissionPending = true; + } + attemptReady = true; + }, }), }); @@ -818,66 +884,26 @@ async function driveAgentTurn( content: [...m.content], })); - // Inline media-out (1.AG/[ADR-0046]): a node requesting a non-text output modality runs a single-shot - // `generate()` (the chain's existing non-streaming path) — terminal, NO tool loop (a media turn is the - // agent's final artifact and `generate()` is one round-trip). The two budget gates below mirror the text - // path: `awaitPreEgress` (primary-model, zero-egress-on-cancel) then the chain's per-attempt `preAttempt`. - if (requestsMediaOutput(params)) { - await awaitPreEgress(params, activeModel, activeProvider); - throwIfAborted(params.signal); - const turn = await generateOneTurn(chain, messages, params); - throwIfAborted(params.signal); // cancel-wins independent of adapter cooperation (mirrors the stream path) - if (turn.stopReason === 'tool_use') { - // A media-output turn is single-shot/terminal (ADR-0046): generate() is one round-trip with no tool - // loop, and `buildRequest` offers it no tools. A `tool_use` stop is therefore a provider PROTOCOL - // ANOMALY — the provider signalled a tool call we never offered and cannot run — so it maps to - // `provider_unavailable`, exactly as the stream path's `tool_use`-stop-with-nothing-runnable guard does. - throw new AgentTurnError( - 'provider_unavailable', - 'a media-output turn signalled a tool_use stop but cannot run a tool round (ADR-0046)', - false, - ); - } - return { - content: turn.content, - text: textOf(turn.content), - usage: { input: usage.input, output: usage.output }, - model: activeModel, - stopReason: turn.stopReason, - }; - } - - let corrections = 0; - - for (let toolTurn = 0; ; toolTurn += 1) { - throwIfAborted(params.signal); - if (toolTurn > params.limits.maxToolTurns) { - throw new AgentTurnError( - 'turn_limit', - `agent exceeded the ${params.limits.maxToolTurns}-turn tool-call limit`, - false, - ); - } - // Two distinct budget gates, by design — NOT a duplicate of the chain's per-attempt check: - // • This loop-top `awaitPreEgress` runs ONCE per tool turn against the PRIMARY model. It is the - // zero-egress-on-cancel guarantee — a cancel landing inside its async check is caught by the - // re-check below, before any provider is engaged. - // • `FallbackChain.preAttempt` then runs again per chain attempt against the ACTUAL (possibly - // failed-over) model, so a failover to a pricier model is still enforced. `streamOneTurn` maps a - // chain-path Budget*Error back into this taxonomy via `chunk.error.cause`. - await awaitPreEgress(params, activeModel, activeProvider); - // The preEgress hook is awaited (its budget check may be async), so the signal can fire during - // that await. Re-check before engaging the provider so a cancel there costs no egress — symmetric - // with the post-stream re-check below. - throwIfAborted(params.signal); - - const turn = await streamOneTurn(chain, messages, params, () => activeModel); - // Cancel-wins independent of adapter cooperation: if the signal fired mid-stream but a - // non-signal-honoring adapter still settled cleanly, fail `cancelled` rather than return a - // stray completed result (mirrors the registry's post-await re-check). - throwIfAborted(params.signal); - - if (turn.stopReason !== 'tool_use') { + try { + // Inline media-out (1.AG/[ADR-0046]): a node requesting a non-text output modality runs a single-shot + // `generate()` (the chain's existing non-streaming path) — terminal, NO tool loop (a media turn is the + // agent's final artifact and `generate()` is one round-trip). Its sole budget gate is the chain's true + // per-attempt `preAttempt`, which retains the admission through the matching attempt record. + if (requestsMediaOutput(params)) { + throwIfAborted(params.signal); + const turn = await generateOneTurn(chain, messages, params); + throwIfAborted(params.signal); // cancel-wins independent of adapter cooperation (mirrors the stream path) + if (turn.stopReason === 'tool_use') { + // A media-output turn is single-shot/terminal (ADR-0046): generate() is one round-trip with no tool + // loop, and `buildRequest` offers it no tools. A `tool_use` stop is therefore a provider PROTOCOL + // ANOMALY — the provider signalled a tool call we never offered and cannot run — so it maps to + // `provider_unavailable`, exactly as the stream path's `tool_use`-stop-with-nothing-runnable guard does. + throw new AgentTurnError( + 'provider_unavailable', + 'a media-output turn signalled a tool_use stop but cannot run a tool round (ADR-0046)', + false, + ); + } return { content: turn.content, text: textOf(turn.content), @@ -887,16 +913,53 @@ async function driveAgentTurn( }; } - // A tool-use turn: append the assistant turn + dispatch its calls (extracted to keep this loop within - // the cognitive-complexity budget). Returns the updated correction count; throws on a protocol anomaly - // or an exhausted correction budget. - corrections = await dispatchToolUseTurn( - turn.content, - messages, - params, - () => activeModel, - nonSkippedAttempts, - corrections, - ); + let corrections = 0; + + for (let toolTurn = 0; ; toolTurn += 1) { + throwIfAborted(params.signal); + if (toolTurn > params.limits.maxToolTurns) { + throw new AgentTurnError( + 'turn_limit', + `agent exceeded the ${params.limits.maxToolTurns}-turn tool-call limit`, + false, + ); + } + // The FallbackChain hook is the sole true provider-attempt gate (including each failover). It performs its + // own post-await cancellation re-check before credential resolution, so there is no speculative loop-top + // reservation that can deny a concurrent branch without ever reaching egress. + throwIfAborted(params.signal); + + const turn = await streamOneTurn(chain, messages, params, () => activeModel); + // Cancel-wins independent of adapter cooperation: if the signal fired mid-stream but a + // non-signal-honoring adapter still settled cleanly, fail `cancelled` rather than return a + // stray completed result (mirrors the registry's post-await re-check). + throwIfAborted(params.signal); + + if (turn.stopReason !== 'tool_use') { + return { + content: turn.content, + text: textOf(turn.content), + usage: { input: usage.input, output: usage.output }, + model: activeModel, + stopReason: turn.stopReason, + }; + } + + // A tool-use turn: append the assistant turn + dispatch its calls (extracted to keep this loop within + // the cognitive-complexity budget). Returns the updated correction count; throws on a protocol anomaly + // or an exhausted correction budget. + corrections = await dispatchToolUseTurn( + turn.content, + messages, + params, + () => activeModel, + nonSkippedAttempts, + corrections, + ); + } + } finally { + // An iterator consumer/fold/event sink can throw before FallbackChain emits its record. The provider may + // already have received work, so close that half-open admission conservatively rather than freeing capacity. + settleUnreportedAttemptAdmission(); } } diff --git a/packages/core/src/engine/budget-governor.test.ts b/packages/core/src/engine/budget-governor.test.ts index 38ddf896..3cf53c0c 100644 --- a/packages/core/src/engine/budget-governor.test.ts +++ b/packages/core/src/engine/budget-governor.test.ts @@ -7,7 +7,12 @@ import { } from '@relavium/llm'; import type { Budget } from '@relavium/shared'; -import { BudgetExceededError, BudgetGovernor, BudgetPauseError } from './budget-governor.js'; +import { + BudgetExceededError, + BudgetGovernor, + BudgetPauseError, + type BudgetAdmission, +} from './budget-governor.js'; import type { RunEventDraft } from './event-bus.js'; describe('BudgetGovernor', () => { @@ -48,19 +53,260 @@ describe('BudgetGovernor', () => { it('allows a call whose estimate stays within the cap', async () => { const { governor, warnings } = makeGovernor(); governor.updateCost(0); - await expect(governor.checkPreEgress('claude-haiku-4-5', 1000)).resolves.toBeUndefined(); + const admission = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(admission).toBeDefined(); + admission?.release(); expect(warnings).toHaveLength(0); }); - it('warns once when the projected cost exceeds the cap', async () => { + it('dedupes an unchanged overage, then re-arms after realized spend with the projected threshold', async () => { const { governor, warnings } = makeGovernor(); governor.updateCost(900_000); - await governor.checkPreEgress('claude-sonnet-4-6', 10_000); + const first = await governor.checkPreEgress('claude-sonnet-4-6', 10_000); expect(warnings).toHaveLength(1); - expect(warnings[0]?.thresholdPct).toBe(90); - // A second check does not emit another warning. - await governor.checkPreEgress('claude-sonnet-4-6', 10_000); + expect(warnings[0]?.spentMicrocents).toBe(900_000); + // The estimate carries the projection far beyond the cap, so the advisory percentage describes the proposed + // post-call state rather than the misleading pre-call 90% value. + expect(warnings[0]?.thresholdPct).toBe(100); + + // The same standing condition is one notice, even though warn mode admits both calls. + const duplicate = await governor.checkPreEgress('claude-sonnet-4-6', 10_000); expect(warnings).toHaveLength(1); + duplicate?.release(); + + // A successful attempt replaces its reservation with actual spend. That new spend re-arms the advisory path + // for a long-lived session, rather than leaving it permanently silent after its first overage. + first?.settle(100_000); + governor.updateCost(1_000_000); // mirrors the authoritative engine/session cost event after settlement + const continued = await governor.checkPreEgress('claude-sonnet-4-6', 10_000); + expect(warnings).toHaveLength(2); + expect(warnings[1]?.spentMicrocents).toBe(1_000_000); + expect(warnings[1]?.thresholdPct).toBe(100); + continued?.release(); + }); + + it('makes concurrent warning admissions await one durable write, including a synchronous re-entry', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + let emits = 0; + let resolveEmission: (() => void) | undefined; + const pendingEmission = new Promise((resolve) => { + resolveEmission = () => resolve(); + }); + let reentrant: Promise | undefined; + const governor = new BudgetGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'warn', + }, + emit: () => { + emits += 1; + // A durable sink can synchronously call back into the governor (for example through an event listener). + // The in-flight promise must already be installed, or this becomes recursive duplicate emission/egress. + if (reentrant === undefined) { + reentrant = governor.checkPreEgress('claude-haiku-4-5', 1000); + } + return pendingEmission; + }, + }); + governor.updateCost(Math.floor(estimatedCallCost / 2) + 1); + + const outer = governor.checkPreEgress('claude-haiku-4-5', 1000); + await Promise.resolve(); // enter the deferred durable sink + expect(emits).toBe(1); + if (reentrant === undefined || resolveEmission === undefined) { + throw new Error('expected the warning sink to create one re-entrant admission'); + } + + let reentrantSettled = false; + void reentrant.then(() => { + reentrantSettled = true; + }); + await Promise.resolve(); + expect(reentrantSettled).toBe(false); // it cannot egress while the first durable write may still fail + + resolveEmission(); + const [outerAdmission, reentrantAdmission] = await Promise.all([outer, reentrant]); + expect(emits).toBe(1); + outerAdmission?.release(); + reentrantAdmission?.release(); + }); + + it('rolls every concurrent warning reservation back when their shared durable write rejects', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + let emits = 0; + let rejectEmission: ((error: Error) => void) | undefined; + const pendingEmission = new Promise((_resolve, reject) => { + rejectEmission = (error) => reject(error); + }); + const governor = new BudgetGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'warn', + }, + emit: () => { + emits += 1; + return pendingEmission; + }, + }); + governor.updateCost(Math.floor(estimatedCallCost / 2) + 1); + + const first = governor.checkPreEgress('claude-haiku-4-5', 1000); + const second = governor.checkPreEgress('claude-haiku-4-5', 1000); + await Promise.resolve(); + if (rejectEmission === undefined) throw new Error('expected the warning sink to be pending'); + rejectEmission(new Error('durable warning sink failed')); + await expect(Promise.all([first, second])).rejects.toThrow('durable warning sink failed'); + expect(emits).toBe(1); + + // Both failed callers released their separate reservations. A below-cap retry must not see a ghost lease. + governor.updateCost(0); + const retry = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(retry).toBeDefined(); + retry?.release(); + }); + + it('keeps a newly re-armed warning armed when an older warning finishes persisting', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + let emits = 0; + let resolveEmission: (() => void) | undefined; + const pendingEmission = new Promise((resolve) => { + resolveEmission = () => resolve(); + }); + const governor = new BudgetGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'warn', + }, + emit: () => { + emits += 1; + return emits === 1 ? pendingEmission : Promise.resolve(); + }, + }); + + // First call is under the cap and holds a live reservation. The second crosses the cap and starts a durable + // warning write. Settling the first with real spend re-arms the latch WHILE that write is still in flight. + const first = await governor.checkPreEgress('claude-haiku-4-5', 1000); + const warning = governor.checkPreEgress('claude-haiku-4-5', 1000); + await Promise.resolve(); + first?.settle(estimatedCallCost); + if (resolveEmission === undefined) + throw new Error('expected the first warning sink to be pending'); + resolveEmission(); + const warningAdmission = await warning; + warningAdmission?.release(); + + const continued = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(emits).toBe(2); + continued?.release(); + }); + + it('atomically reserves a priced call so a concurrent admission cannot overspend the cap', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + const { governor } = makeGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'fail', + }, + }); + + const first = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(first).toBeDefined(); + await expect(governor.checkPreEgress('claude-haiku-4-5', 1000)).rejects.toBeInstanceOf( + BudgetExceededError, + ); + + // A failed/cancelled attempt has no attributable charge, so its reservation releases capacity for the next one. + first?.release(); + const retry = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(retry).toBeDefined(); + retry?.release(); + }); + + it('reconciles one reservation with realized cost exactly once before the engine syncs its total', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + const { governor } = makeGovernor({ + budget: { max_cost_microcents: estimatedCallCost + 100_000, on_exceed: 'fail' }, + }); + + const first = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(first).toBeDefined(); + first?.settle(100_000); + first?.settle(999_999); // idempotent: an accidental second completion cannot inflate the ledger + first?.release(); + + // The reservation is gone, but its actual charge remains. The later authoritative sync is the same total, + // not a second charge, so a next call exactly at the cap is still admitted. + governor.updateCost(100_000); + const next = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(next).toBeDefined(); + next?.release(); + }); + + it('keeps an uncertain billed attempt as a separate conservative debit across lower durable totals', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + const { governor } = makeGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'fail', + }, + }); + + const first = await governor.checkPreEgress('claude-haiku-4-5', 1000); + first?.settleAtReservedEstimate(); + // A usage-less provider attempt has no corresponding durable cost event. An unrelated/older zero total must + // not erase its bounded debit and allow a second worst-case request through the cap. + governor.updateCost(0); + await expect(governor.checkPreEgress('claude-haiku-4-5', 1000)).rejects.toBeInstanceOf( + BudgetExceededError, + ); + }); + + it('restores a priced reservation for egress already submitted before a checkpoint resume', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + const { governor } = makeGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'fail', + }, + }); + + const restored = governor.reserveCommittedEgress('claude-haiku-4-5', 1000); + expect(restored).toBeDefined(); + await expect(governor.checkPreEgress('claude-haiku-4-5', 1000)).rejects.toBeInstanceOf( + BudgetExceededError, + ); + restored?.release(); + }); + + it('rolls back a warning-path reservation when warning durability fails', async () => { + const estimatedCallCost = estimateMaxNextCost('claude-haiku-4-5', 1000); + let shouldFail = true; + let emits = 0; + const governor = new BudgetGovernor({ + budget: { + max_cost_microcents: estimatedCallCost + Math.floor(estimatedCallCost / 2), + on_exceed: 'warn', + }, + emit: () => { + emits += 1; + return shouldFail + ? Promise.reject(new Error('durable warning sink failed')) + : Promise.resolve(); + }, + }); + governor.updateCost(Math.floor(estimatedCallCost / 2) + 1); // pushes the next call into warn mode + + await expect(governor.checkPreEgress('claude-haiku-4-5', 1000)).rejects.toThrow( + 'durable warning sink failed', + ); + + // Restore a below-cap authoritative state. A leaked reservation would turn this otherwise-allowed call back + // into warn mode and invoke the sink a second time. + shouldFail = false; + governor.updateCost(0); + const admission = await governor.checkPreEgress('claude-haiku-4-5', 1000); + expect(emits).toBe(1); + admission?.release(); }); it('fails when on_exceed is fail', async () => { @@ -82,6 +328,7 @@ describe('BudgetGovernor', () => { // 900_000 already spent (the estimate is output-only). expect(err.projectedMicrocents).toBe(900_000 + 15_000_000); expect(err.projectedMicrocents).toBeGreaterThan(err.limitMicrocents); + expect(err.reason).toBe('projected_over_cap'); }); it('pauses when on_exceed is pause_for_approval', async () => { @@ -104,14 +351,16 @@ describe('BudgetGovernor', () => { }); // At exactly the cap minus the default estimate, the call is allowed. governor.updateCost(1_000_000 - 500); // haiku output is 500_000_000 micro-cents/MTok - await governor.checkPreEgress('claude-haiku-4-5', undefined); + const admission = await governor.checkPreEgress('claude-haiku-4-5', undefined); + admission?.release(); expect(warnings).toHaveLength(0); }); it('clamps thresholdPct to [0, 100]', async () => { const { governor, warnings } = makeGovernor(); governor.updateCost(2_000_000); - await governor.checkPreEgress('claude-sonnet-4-6', 1000); + const admission = await governor.checkPreEgress('claude-sonnet-4-6', 1000); + admission?.release(); expect(warnings[0]?.thresholdPct).toBe(100); }); @@ -126,11 +375,10 @@ describe('BudgetGovernor', () => { expect(warnings).toHaveLength(0); }); - it('degrades to allow (does not crash the run) for an unlisted/unpriced model — H4', async () => { - // An unpriced custom/self-hosted model id has no pricing row → estimateMaxNextCost throws - // UnknownModelError. The pre-egress governor must NOT hard-fail an otherwise-valid run on it; it - // degrades to `allow` (mirrors the FallbackChain's unpriced⇒no-cost policy). Even with on_exceed: fail - // and the run already over a notional cap, an unpriced model resolves rather than throwing. + it('degrades to allow (does not crash the run) for any unlisted/unpriced model when strict mode is off', async () => { + // An unpriced model id has no pricing row → estimateMaxNextCost throws UnknownModelError. With strict mode off, + // the governor deliberately permits BOTH custom/self-hosted ids and first-party catalog gaps, because it has no + // trustworthy price estimate for either. It says so once rather than silently pretending the cap applied. const { governor, warnings, unpriced } = makeGovernor({ budget: { ...budget, on_exceed: 'fail' }, }); @@ -143,17 +391,21 @@ describe('BudgetGovernor', () => { }); describe('strict_cost_cap (ADR-0071 §K7)', () => { - it('BLOCKS an unpriced model when on — "if you cannot price it, do not run it"', () => { - // The opt-in for a user who set a cap SPECIFICALLY to bound an untrusted model. The silent degrade-to-allow - // is the wrong trade for them: an unpriced model is a hole in the cap, and they would rather refuse the turn. + it('BLOCKS every unpriced model when on — "if you cannot price it, do not run it"', () => { + // Strict mode intentionally makes no provenance distinction at this seam: both a self-hosted id and a + // first-party catalog gap are unpriced, so both would leave a user-requested cap unenforceable. const { governor } = makeGovernor({ budget: { ...budget, on_exceed: 'fail', strict_cost_cap: true }, }); - const result = governor.evaluatePreEgress('my-self-hosted-model', 10_000); - expect(result.kind).toBe('fail'); - if (result.kind === 'fail') { - expect(result.error.message).toContain('no price'); - expect(result.error.message).toContain('strict_cost_cap'); + for (const model of ['my-self-hosted-model', 'unlisted-first-party-model']) { + const result = governor.evaluatePreEgress(model, 10_000); + expect(result.kind).toBe('fail'); + if (result.kind === 'fail') { + expect(result.error.message).toContain('no price'); + expect(result.error.message).toContain('strict_cost_cap'); + expect(result.error.projectedMicrocents).toBeUndefined(); + expect(result.error.reason).toBe('unpriced_model'); + } } }); @@ -165,7 +417,22 @@ describe('BudgetGovernor', () => { expect(governor.evaluatePreEgress('claude-haiku-4-5', 1000).kind).toBe('allow'); }); - it('OFF (the default) degrades to allow with a notice, not a block', async () => { + it('does not fabricate an over-cap projection for an unpriced strict refusal while another lease is live', async () => { + const { governor } = makeGovernor({ + budget: { ...budget, on_exceed: 'fail', strict_cost_cap: true }, + }); + const live = await governor.checkPreEgress('claude-haiku-4-5', 1000); + const result = governor.evaluatePreEgress('unlisted-first-party-model', 1000); + expect(result.kind).toBe('fail'); + if (result.kind === 'fail') { + expect(result.error.spentMicrocents).toBe(0); + expect(result.error.projectedMicrocents).toBeUndefined(); + expect(result.error.reason).toBe('unpriced_model'); + } + live?.release(); + }); + + it('OFF (the default) permits every unpriced model with a notice, not a block', async () => { const { governor, unpriced } = makeGovernor({ budget: { ...budget, on_exceed: 'fail' } }); // `evaluatePreEgress` classifies; `checkPreEgress` is what APPLIES the result and fires the sink. Drive the // applying path, so the "with a notice" in this test's name is actually asserted. @@ -175,6 +442,16 @@ describe('BudgetGovernor', () => { ).resolves.toBeUndefined(); expect(unpriced).toEqual(['my-self-hosted-model']); }); + + it('is inert without a positive cap, even when strict mode is on', async () => { + const { governor, unpriced } = makeGovernor({ + budget: { max_cost_microcents: 0, on_exceed: 'fail', strict_cost_cap: true }, + }); + await expect( + governor.checkPreEgress('unlisted-first-party-model', 10_000), + ).resolves.toBeUndefined(); + expect(unpriced).toEqual([]); + }); }); it('notifies UNPRICED only once per model — a loop must not repeat it every turn', async () => { @@ -185,6 +462,22 @@ describe('BudgetGovernor', () => { expect(unpriced).toEqual(['my-self-hosted-model', 'another-unpriced-one']); // deduped per model }); + it('does not let a throwing unpriced advisory sink veto the deliberate allow-degrade decision', async () => { + let notices = 0; + const governor = new BudgetGovernor({ + budget: { ...budget, on_exceed: 'fail' }, + emit: () => Promise.resolve(), + onUnpriced: () => { + notices += 1; + throw new Error('renderer unavailable'); + }, + }); + + await expect(governor.checkPreEgress('my-self-hosted-model', 1000)).resolves.toBeUndefined(); + await expect(governor.checkPreEgress('my-self-hosted-model', 1000)).resolves.toBeUndefined(); + expect(notices).toBe(1); + }); + it('accepts a media-unit estimate and folds it as a disjoint addend (1.AF/D17)', () => { // No 1.AF model carries a media rate, so the media estimate degrades to 0 — the decision matches the // token-only projection (the units×rate math is covered in mediaCost/estimateMediaCost). This pins the @@ -252,7 +545,9 @@ describe('BudgetGovernor', () => { resolvePrice: OVERLAY, }); governor.updateCost(0); - await expect(governor.checkPreEgress('acme-custom-1', 1_000)).resolves.toBeUndefined(); + const admission = await governor.checkPreEgress('acme-custom-1', 1_000); + expect(admission).toBeDefined(); + admission?.release(); expect(warnings).toHaveLength(0); }); }); diff --git a/packages/core/src/engine/budget-governor.ts b/packages/core/src/engine/budget-governor.ts index 70402653..07e7be48 100644 --- a/packages/core/src/engine/budget-governor.ts +++ b/packages/core/src/engine/budget-governor.ts @@ -19,24 +19,38 @@ import type { GateRequest } from './node-executor.js'; */ export const DEFAULT_MAX_TOKENS_ESTIMATE = 4096; +/** Why a strict budget check refused the prospective call. */ +export type BudgetExceededReason = 'projected_over_cap' | 'unpriced_model'; + /** - * Thrown when a pre-egress check would exceed the configured cost cap and `on_exceed: fail` is set. - * The turn/run adapter maps this to the `budget_exceeded` `ErrorCode`. + * Thrown when a priced pre-egress projection exceeds a configured `on_exceed: fail` cap, or when strict-cost mode + * refuses an unpriced prospective call. The turn/run adapter maps this to the `budget_exceeded` `ErrorCode`. */ export class BudgetExceededError extends Error { override readonly name = 'BudgetExceededError'; constructor( readonly spentMicrocents: number, readonly limitMicrocents: number, - readonly projectedMicrocents: number, + /** + * The priced post-call projection when one exists. It is deliberately absent for a strict-cap refusal of an + * unpriced model: inventing `cap + 1` there would falsely claim that we measured a cost we explicitly could + * not measure. + */ + readonly projectedMicrocents: number | undefined, // A caller-supplied message for the case the cap fails NOT because spend exceeded it, but because it could not // be enforced at all — an unpriced model under `strict_cost_cap` (ADR-0071 §K7). Absent ⇒ the projection line. message?: string, + readonly reason: BudgetExceededReason = projectedMicrocents === undefined + ? 'unpriced_model' + : 'projected_over_cap', ) { super( message ?? - `pre-egress budget check failed: projected ${projectedMicrocents} micro-cents exceeds ` + - `the cap of ${limitMicrocents} micro-cents (spent ${spentMicrocents})`, + (projectedMicrocents === undefined + ? `pre-egress budget check failed: the cap of ${limitMicrocents} micro-cents cannot be enforced ` + + `because the prospective call has no price (spent ${spentMicrocents})` + : `pre-egress budget check failed: projected ${projectedMicrocents} micro-cents exceeds ` + + `the cap of ${limitMicrocents} micro-cents (spent ${spentMicrocents})`), ); } } @@ -95,10 +109,40 @@ export type BudgetCheckResult = | { readonly kind: 'fail'; readonly error: BudgetExceededError } | { readonly kind: 'pause'; readonly error: BudgetPauseError }; +/** + * One priced provider-call admission. The caller owns its lifetime: settle it with a known actual charge, retain + * its bounded estimate when egress may have happened without trustworthy usage, or release it only when no egress + * can be attributed. All operations are idempotent so every error path can safely close the lease exactly once. + */ +export interface BudgetAdmission { + /** Reconcile the reserved estimate with an attempt's realized, priced charge. */ + readonly settle: (realizedMicrocents: number) => void; + /** + * Conservatively account for the full reservation when the provider may already have accepted/billed a call but + * did not supply trustworthy usage. This is intentionally distinct from {@link release}: an uncertain bill must + * never reopen capacity under a strict cap. + */ + readonly settleAtReservedEstimate: () => void; + /** Drop an admission that produced no attributable realized cost. Equivalent to `settle(0)`. */ + readonly release: () => void; +} + +/** Internal evaluation detail: the public verdict stays compact while admission keeps the exact priced estimate. */ +interface BudgetEvaluation { + readonly result: BudgetCheckResult; + /** Present only for a priced, bounded call; absent means there is nothing meaningful to reserve. */ + readonly estimateMicrocents?: number; +} + +/** The pricing lookup is deliberately separate from the policy verdict so an already-submitted job can be restored. */ +type EstimateResult = + | { readonly kind: 'priced'; readonly estimateMicrocents: number } + | { readonly kind: 'unpriced' }; + /** * The pre-egress budget governor (ADR-0028, 1.AC). It is stateful per run: it tracks the current - * cumulative cost, emits at most one `budget:warning` event per run, and throws a typed error for - * `fail` / `pause_for_approval`. All cost figures are integer micro-cents. + * cumulative cost plus in-flight admissions, re-arms warn-mode notices after new realized spend, and throws a + * typed error for `fail` / `pause_for_approval`. All cost figures are integer micro-cents. */ export class BudgetGovernor { readonly #budget: Budget; @@ -108,8 +152,22 @@ export class BudgetGovernor { ) => Promise; readonly #overlay: PricingOverlay | undefined; readonly #resolveEndpoint: ((provider: ProviderId) => EndpointKind) | undefined; + /** The durable/realized total reported by the engine or session cost stream. */ #cumulativeCostMicrocents = 0; - #warningEmitted = false; + /** + * Bounded estimates committed for attempts that may have billed but supplied no trustworthy usage. This is kept + * separate from the durable total: an unrelated lower/older cost event must never erase an uncertain bill and + * silently reopen strict-cap capacity. + */ + #conservativeCostMicrocents = 0; + #reservedCostMicrocents = 0; + #nextAdmissionId = 0; + readonly #reservedAdmissions = new Map(); + // A warning is deduped while no money has changed across concurrent true attempt admissions, then re-armed by a + // positive realized-cost update so a long-lived warn-mode session does not go permanently silent. + #warningArmed = true; + /** The one durable warning currently being written. Concurrent admissions await this exact promise. */ + #warningInFlight: Promise | undefined; readonly #onUnpriced: ((model: string, capMicrocents: number) => void) | undefined; readonly #unpricedNotified = new Set(); // once per model — a standing condition, not a per-turn event @@ -154,8 +212,11 @@ export class BudgetGovernor { this.#resolveEndpoint = params.resolveEndpoint; } - /** Update the governor with the engine's authoritative running cumulative cost. */ + /** Update the governor with the engine's durable running cumulative cost. Conservative unknown-usage debits stay separate. */ updateCost(cumulativeCostMicrocents: number): void { + if (cumulativeCostMicrocents > this.#cumulativeCostMicrocents) { + this.#warningArmed = true; + } this.#cumulativeCostMicrocents = cumulativeCostMicrocents; } @@ -170,129 +231,254 @@ export class BudgetGovernor { mediaUnitsEstimate?: readonly MediaUnitsEstimate[], provider?: ProviderId, ): BudgetCheckResult { + return this.#evaluate(model, maxTokens, mediaUnitsEstimate, provider).result; + } + + /** Evaluate against the authoritative realized total plus every live admission, without mutating either. */ + #evaluate( + model: string, + maxTokens: number | undefined, + mediaUnitsEstimate: readonly MediaUnitsEstimate[] | undefined, + provider: ProviderId | undefined, + ): BudgetEvaluation { // A cap of 0 means UNBOUNDED (`[chat].max_cost_microcents`: "0 = unbounded"): never block, and never // reach the `thresholdPct` division below (which would be `/0` → NaN). A workflow `BudgetSchema` forbids // 0 (`positiveInt`), but the governor is reused for the `[chat]`/session path where 0 is valid. This // short-circuit stays BEFORE any estimate (ADR-0044 §3 — no `/0`, no estimate work when unbounded). if (this.#budget.max_cost_microcents <= 0) { - return { kind: 'allow' }; + return { result: { kind: 'allow' } }; } - let estimate: number; - try { - // Token estimate + the disjoint media estimate (ADR-0044 §3). estimateMediaCost prices only the - // modalities the model rates (a missing rate degrades to 0); both share the UnknownModelError - // degrade-to-allow below, so an unpriced model never hard-fails the run. - estimate = - estimateMaxNextCost( - model, - maxTokens ?? this.#defaultMaxTokensEstimate, - this.#overlay, - // Key the endpoint on the routing provider (review M2). A media-only gate omits it (`maxTokens: 0` - // makes the token estimate 0 regardless), so `official` is a harmless default there. - (provider === undefined ? undefined : this.#resolveEndpoint?.(provider)) ?? 'official', - ) + - (mediaUnitsEstimate === undefined - ? 0 - : estimateMediaCost(model, mediaUnitsEstimate, this.#overlay)); - } catch (err) { - // An unpriced model (e.g. a custom base-URL / self-hosted id with no pricing row) throws - // UnknownModelError. The pre-egress governor must NOT hard-fail an otherwise-valid run on it — - // degrade to `allow`, mirroring the FallbackChain's "unpriced ⇒ no-cost" policy (H4). A self-hosted - // model has ~no metered cost, so the cap simply does not constrain it. Any other error is a real bug. - if (err instanceof UnknownModelError) { - // A model with no price. The cap CANNOT bound it — we do not know what a turn costs. Two ways to treat that: - if (this.#budget.strict_cost_cap === true) { - // The user asked for a hard cap. If we cannot price it, we do not run it — that is what "strict" means. - return { + const estimateResult = this.#estimate(model, maxTokens, mediaUnitsEstimate, provider); + if (estimateResult.kind === 'unpriced') { + // An unpriced model id (a custom/self-hosted id OR a first-party catalog gap) cannot be distinguished at + // this seam. The regular cap degrades to allow with one notice for EVERY unpriced id; strict_cost_cap blocks + // EVERY unpriced id. The uniform policy is the only honest one when the charge cannot be estimated. + if (this.#budget.strict_cost_cap === true) { + return { + result: { kind: 'fail', error: new BudgetExceededError( this.#cumulativeCostMicrocents, this.#budget.max_cost_microcents, - this.#cumulativeCostMicrocents, + undefined, `model '${model}' has no price, so the ${this.#budget.max_cost_microcents}-micro-cent cap cannot be enforced on it (strict_cost_cap is on). Price it with \`relavium models pricing ${model}\`, or turn strict_cost_cap off.`, ), - }; - } - // The ordinary trade (ADR-0028 H4): a self-hosted model has ~no metered cost, and refusing an otherwise - // valid run over a missing price is worse than the small risk. Allow — but SAY it is unpriced, once. - return { kind: 'unpriced', model }; + }, + }; } - throw err; + // The ordinary trade (ADR-0028 H4): an unpriced call cannot be bounded, but refusing it merely because a + // price row is missing is worse when the caller did not opt into strict mode. Allow — but SAY so once. + return { result: { kind: 'unpriced', model } }; } - const projected = this.#cumulativeCostMicrocents + estimate; + const estimate = estimateResult.estimateMicrocents; + // This is the admission-control invariant: a later concurrent branch sees every already-authorized worst-case + // call, not merely the last durable `cost:updated` snapshot. There is deliberately no await between this read + // and the ledger insertion in `checkPreEgress` below. + const projected = + this.#cumulativeCostMicrocents + + this.#conservativeCostMicrocents + + this.#reservedCostMicrocents + + estimate; if (projected <= this.#budget.max_cost_microcents) { - return { kind: 'allow' }; + return { result: { kind: 'allow' }, estimateMicrocents: estimate }; } - const thresholdPct = clampPct( - Math.round((this.#cumulativeCostMicrocents / this.#budget.max_cost_microcents) * 100), - ); + const thresholdPct = clampPct(Math.round((projected / this.#budget.max_cost_microcents) * 100)); switch (this.#budget.on_exceed) { case 'warn': return { - kind: 'warn', - spentMicrocents: this.#cumulativeCostMicrocents, - limitMicrocents: this.#budget.max_cost_microcents, - thresholdPct, + result: { + kind: 'warn', + spentMicrocents: this.#cumulativeCostMicrocents, + limitMicrocents: this.#budget.max_cost_microcents, + thresholdPct, + }, + estimateMicrocents: estimate, }; case 'fail': return { - kind: 'fail', - error: new BudgetExceededError( - this.#cumulativeCostMicrocents, - this.#budget.max_cost_microcents, - projected, - ), + result: { + kind: 'fail', + error: new BudgetExceededError( + this.#cumulativeCostMicrocents, + this.#budget.max_cost_microcents, + projected, + ), + }, }; case 'pause_for_approval': return { - kind: 'pause', - error: new BudgetPauseError( - this.#cumulativeCostMicrocents, - this.#budget.max_cost_microcents, - thresholdPct, - ), + result: { + kind: 'pause', + error: new BudgetPauseError( + this.#cumulativeCostMicrocents, + this.#budget.max_cost_microcents, + thresholdPct, + ), + }, }; } } /** - * Apply the pre-egress check: emits a one-time `budget:warning` on the warn path, returns on allow, - * and throws `BudgetExceededError` / `BudgetPauseError` for fail / pause. Async because the warning - * event is durable. + * Atomically admit one true provider attempt: price it against realized spend plus all live reservations, insert + * its reservation before the first await, then emit a re-armable warning or throw the typed fail/pause outcome. + * The returned admission MUST be settled, conservatively committed, or released exactly once by the attempt owner. */ async checkPreEgress( model: string, maxTokens: number | undefined, mediaUnitsEstimate?: readonly MediaUnitsEstimate[], provider?: ProviderId, - ): Promise { - const result = this.evaluatePreEgress(model, maxTokens, mediaUnitsEstimate, provider); - if (result.kind === 'allow') return; + ): Promise { + const evaluation = this.#evaluate(model, maxTokens, mediaUnitsEstimate, provider); + const { result } = evaluation; + if (result.kind === 'allow') return this.#admit(evaluation.estimateMicrocents); if (result.kind === 'unpriced') { // Once per model — a standing condition, not an event (a `loop` over an unpriced model must not repeat it // every iteration). The engine cannot print; the host is told and decides where the sentence goes. if (!this.#unpricedNotified.has(result.model)) { this.#unpricedNotified.add(result.model); - this.#onUnpriced?.(result.model, this.#budget.max_cost_microcents); + // The advisory surface is not allowed to turn an explicit allow-degrade policy into a hidden block. The + // condition remains deduped even if a host renderer/logger fails, preventing an exception storm. + try { + this.#onUnpriced?.(result.model, this.#budget.max_cost_microcents); + } catch { + // Best-effort host notice; the governed decision remains allow for non-strict unpriced models. + } } - return; + return undefined; } if (result.kind === 'warn') { - if (!this.#warningEmitted) { - this.#warningEmitted = true; - await this.#emit({ - type: 'budget:warning', - spentMicrocents: result.spentMicrocents, - limitMicrocents: result.limitMicrocents, - thresholdPct: result.thresholdPct, - }); + const admission = this.#admit(evaluation.estimateMicrocents); + try { + await this.#emitWarning(result); + return admission; + } catch (error) { + admission?.release(); + throw error; } - return; } throw result.error; } + + /** + * Recreate a reservation for egress that is already irrevocably submitted (currently an async media job during + * checkpoint resume). It intentionally bypasses cap policy and warnings: rejecting a job after its provider has + * accepted it cannot prevent spend. Unknown prices have no meaningful reservation and preserve the normal + * allow-degrade behavior. + */ + reserveCommittedEgress( + model: string, + maxTokens: number | undefined, + mediaUnitsEstimate?: readonly MediaUnitsEstimate[], + provider?: ProviderId, + ): BudgetAdmission | undefined { + const estimate = this.#estimate(model, maxTokens, mediaUnitsEstimate, provider); + return estimate.kind === 'priced' ? this.#admit(estimate.estimateMicrocents) : undefined; + } + + /** Calculate a price without applying cap policy; shared by prospective admission and committed-job restoration. */ + #estimate( + model: string, + maxTokens: number | undefined, + mediaUnitsEstimate: readonly MediaUnitsEstimate[] | undefined, + provider: ProviderId | undefined, + ): EstimateResult { + try { + // Token estimate + the disjoint media estimate (ADR-0044 §3). estimateMediaCost prices only the modalities + // the model rates (a missing rate degrades to 0); an unknown model follows the uniform policy in #evaluate. + return { + kind: 'priced', + estimateMicrocents: + estimateMaxNextCost( + model, + maxTokens ?? this.#defaultMaxTokensEstimate, + this.#overlay, + // Key the endpoint on the routing provider (review M2). A media-only gate omits it (`maxTokens: 0` + // makes the token estimate 0 regardless), so `official` is a harmless default there. + (provider === undefined ? undefined : this.#resolveEndpoint?.(provider)) ?? 'official', + ) + + (mediaUnitsEstimate === undefined + ? 0 + : estimateMediaCost(model, mediaUnitsEstimate, this.#overlay)), + }; + } catch (err) { + if (err instanceof UnknownModelError) return { kind: 'unpriced' }; + throw err; + } + } + + /** Insert one reservation synchronously. Zero/unpriced/unbounded evaluations intentionally carry no lease. */ + #admit(estimateMicrocents: number | undefined): BudgetAdmission | undefined { + if (estimateMicrocents === undefined || estimateMicrocents <= 0) return undefined; + const id = this.#nextAdmissionId++; + this.#reservedAdmissions.set(id, estimateMicrocents); + this.#reservedCostMicrocents += estimateMicrocents; + let settled = false; + const settle = (realizedMicrocents: number): void => { + if (settled) return; + settled = true; + const reserved = this.#reservedAdmissions.get(id); + if (reserved === undefined) return; + this.#reservedAdmissions.delete(id); + this.#reservedCostMicrocents -= reserved; + // A cost event immediately follows on the normal path and assigns the same authoritative total. Advancing + // here keeps admissions sound even if an event sink throws after the provider has already charged the call. + if (realizedMicrocents > 0) { + this.#cumulativeCostMicrocents += realizedMicrocents; + this.#warningArmed = true; + } + }; + return { + settle, + settleAtReservedEstimate: () => { + if (settled) return; + settled = true; + const reserved = this.#reservedAdmissions.get(id); + if (reserved === undefined) return; + this.#reservedAdmissions.delete(id); + this.#reservedCostMicrocents -= reserved; + this.#conservativeCostMicrocents += reserved; + this.#warningArmed = true; + }, + release: () => settle(0), + }; + } + + /** + * Emit at most once until positive realized spend re-arms the condition. A second concurrent warning admission + * awaits the first persistence promise instead of egressing while that write could still fail; a failed write + * re-arms the latch and makes every waiter roll its own reservation back. + */ + #emitWarning(result: Extract): Promise { + const inFlight = this.#warningInFlight; + if (inFlight !== undefined) return inFlight; + if (!this.#warningArmed) return Promise.resolve(); + this.#warningArmed = false; + const emitted = Promise.resolve().then(() => + this.#emit({ + type: 'budget:warning', + spentMicrocents: result.spentMicrocents, + limitMicrocents: result.limitMicrocents, + thresholdPct: result.thresholdPct, + }), + ); + this.#warningInFlight = emitted; + void emitted.then( + () => { + if (this.#warningInFlight === emitted) this.#warningInFlight = undefined; + }, + () => { + if (this.#warningInFlight === emitted) { + this.#warningInFlight = undefined; + this.#warningArmed = true; + } + }, + ); + return emitted; + } } function clampPct(value: number): number { diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index e630d0ad..16062855 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -61,7 +61,11 @@ import type { WorkflowDefinition } from '../parser.js'; import { EngineStateError } from './errors.js'; import { RunEventBus, type RunEventDraft } from './event-bus.js'; import { RunLoopInvariantError } from './invariant-error.js'; -import { BudgetGovernor, DEFAULT_MAX_TOKENS_ESTIMATE } from './budget-governor.js'; +import { + BudgetGovernor, + DEFAULT_MAX_TOKENS_ESTIMATE, + type BudgetAdmission, +} from './budget-governor.js'; import type { CheckpointState } from './checkpoint.js'; import type { AbortControllerLike, ExecutionHost } from './execution-host.js'; import type { @@ -74,7 +78,12 @@ import type { NodeStreamEvent, } from './node-executor.js'; import { codeForLlmError } from './agent-turn.js'; -import { DEFAULT_MEDIA_UNIT_ESTIMATE, generativeUnits, realizedMediaCost } from './agent-runner.js'; +import { + DEFAULT_MEDIA_UNIT_ESTIMATE, + generativeUnits, + realizedMediaCost, + takeMediaJobAdmission, +} from './agent-runner.js'; import { createClosedRunHandle, createRunHandle, type RunHandle } from './run-handle.js'; /** A vertex's live status in one run. `paused` (at a gate) and `running` are not yet *settled*. */ @@ -108,6 +117,10 @@ interface ParkedMediaJob { readonly submittedAtMs: number; /** The current poll interval (ms), grown exponentially per `pending` poll, capped at `pollMaxMs`. */ backoffMs: number; + /** Process-local cost-cap reservation for an already-submitted job; reconstructed on checkpoint resume. */ + admission?: BudgetAdmission; + /** Guards every terminal/error sweep from emitting the paid-job addend more than once. */ + costAccounted?: boolean; } /** A vertex status counts as *settled* (its dependents can evaluate) when it is one of these. */ @@ -299,6 +312,7 @@ class RunExecution { readonly #onSettled: (runId: string) => void; readonly #resolverCapabilities: ResolverCapabilities; readonly #maxTokensEstimate: number; + readonly #resolvePrice: PricingOverlay | undefined; /** The resolved workflow `context:` (`ctx.*`), folded once at run start (or re-resolved on resume). */ #resolvedContext: Readonly> = {}; @@ -380,6 +394,7 @@ class RunExecution { this.#secretInputNames = secretNames; this.#maskedInputs = maskInputs(params.inputs, secretNames); this.#maxTokensEstimate = params.maxTokensEstimate ?? DEFAULT_MAX_TOKENS_ESTIMATE; + this.#resolvePrice = params.resolvePrice; if (params.plan.budget !== undefined) { this.#budgetGovernor = new BudgetGovernor({ budget: params.plan.budget, @@ -554,6 +569,13 @@ class RunExecution { isBudgetGate: gate.isBudgetGate, }); } + // Re-seed totals BEFORE restoring submitted-job reservations: a committed job must reserve alongside the + // checkpoint's known spend, and its reservation must exist before the first re-armed poll/schedule can run. + this.#totalInputTokens = cp.totalInputTokens; + this.#totalOutputTokens = cp.totalOutputTokens; + this.#cumulativeCostMicrocents = cp.cumulativeCostMicrocents; + this.#budgetGovernor?.updateCost(cp.cumulativeCostMicrocents); + // Re-attach each parked async media job (MJ-1, ADR-0045 §3): re-register it + RE-ARM a poll of the // persisted opaque jobId. NEVER re-call generateMedia — the node is `'paused'` (applyMediaJobEvent set it), // not absent, so it is not re-run via the `'pending'` path; this overrides the checkpoint @@ -574,6 +596,15 @@ class RunExecution { vertex?.config.kind === 'agent' ? generativeUnits(job.modality, vertex.config.node) : DEFAULT_MEDIA_UNIT_ESTIMATE[job.modality]; + // This provider job was accepted before the crash, so normal cap policy cannot retroactively refuse it. + // Recreate only its priced reservation (if any) before poll scheduling; the eventual terminal media-cost + // event reconciles it with the realized charge exactly as in the original process. + const admission = this.#budgetGovernor?.reserveCommittedEgress( + job.model, + 0, + [{ modality: job.modality, units }], + job.provider, + ); this.#pendingMediaJobs.set(job.nodeId, { jobId: job.jobId, provider: job.provider, @@ -586,6 +617,7 @@ class RunExecution { // is the same value seeded into `#startEpochMs` below. submittedAtMs: Date.parse(job.startedAt) - cp.startedAtMs, backoffMs: MEDIA_JOB_POLL_DEFAULTS.pollInitialMs, + ...(admission === undefined ? {} : { admission }), }); this.#armMediaPoll(job.nodeId); } @@ -601,13 +633,6 @@ class RunExecution { for (const gateId of cp.resolvedGateIds) { this.#resolvedGates.add(gateId); } - this.#totalInputTokens = cp.totalInputTokens; - this.#totalOutputTokens = cp.totalOutputTokens; - this.#cumulativeCostMicrocents = cp.cumulativeCostMicrocents; - // Re-seed the budget governor with the restored cumulative cost (H2): it starts at 0 and only advances - // on `cost:updated`, which a resume does NOT replay — so without this a resumed budgeted run would - // under-block by up to ~a full cap on its first post-resume pre-egress check (ADR-0028). - this.#budgetGovernor?.updateCost(cp.cumulativeCostMicrocents); // Post-resume events continue gap-free from the last persisted sequence number. bus.seedSequence(runId, cp.lastSequenceNumber + 1); // Keep measuring durationMs from the ORIGINAL start, so a resumed run's terminal reports total @@ -1081,7 +1106,7 @@ class RunExecution { /** * Build the pre-egress hook for this run, when a budget is configured. The hook is stateful: - * it sees the run's current cumulative cost and may emit a one-time `budget:warning` or throw + * it sees the run's current cumulative cost and may emit a re-armable `budget:warning` or throw * `BudgetExceededError` / `BudgetPauseError` for `fail` / `pause_for_approval`. */ #makePreEgressHook(): import('./agent-turn.js').PreEgressHook | undefined { @@ -1383,6 +1408,9 @@ class RunExecution { const deadlineAt = new Date( Date.parse(startedAt) + (job.deadlineMs ?? MEDIA_JOB_POLL_DEFAULTS.deadlineMs), ).toISOString(); + // Consume the exact submission object before the first await. The runner's WeakMap never crosses persistence; + // after this point the parked-job record owns the lease through poll, cancel, failure and completion. + const admission = takeMediaJobAdmission(job); this.#pendingMediaJobs.set(vertex.id, { jobId: job.jobId, provider: job.provider, @@ -1394,6 +1422,7 @@ class RunExecution { // recompute (from the checkpoint slot) yields an identical value (M2). submittedAtMs: Date.parse(startedAt) - this.#startEpochMs, backoffMs: MEDIA_JOB_POLL_DEFAULTS.pollInitialMs, + ...(admission === undefined ? {} : { admission }), }); await this.#emitDurable({ type: 'media_job:submitted', @@ -1445,13 +1474,30 @@ class RunExecution { * streamed by `#nodeEmit`. Emitted exactly once per job: at `done`, or — for a paid job abandoned by a * fail/deadline/cancel — at that settle (the provider bills regardless, a cost-integrity requirement). */ #emitMediaJobCost(nodeId: string, job: ParkedMediaJob): void { + if (job.costAccounted === true) { + return; + } + // Mark before the first side effect. A terminal/error path may re-enter while a sink is unwinding; the provider + // has only one submitted job, so the engine must never manufacture a second billed addend for it. + job.costAccounted = true; + const costMicrocents = realizedMediaCost( + job.model, + job.modality, + job.units, + this.#resolvePrice, + ); + // Reconcile the lease BEFORE publishing the engine cost event. If event delivery faults after a provider-paid + // job, the reservation cannot be released as though the submission were free. Clear the process-local handle + // after its idempotent settle so every terminal sweep remains exactly-once from the governor's perspective. + job.admission?.settle(costMicrocents); + delete job.admission; this.#nodeEmit({ type: 'cost:updated', nodeId, model: job.model, inputTokens: 0, outputTokens: 0, - costMicrocents: realizedMediaCost(job.model, job.modality, job.units), + costMicrocents, cumulativeCostMicrocents: 0, // #nodeEmit overwrites with the authoritative run-wide total }); } @@ -1555,7 +1601,18 @@ class RunExecution { // Exponential backoff (no jitter) capped at pollMaxMs, then re-arm. Progress is TRANSIENT (ADR-0045 // §2) — never persisted; no run event today. job.backoffMs = Math.min(job.backoffMs * 2, MEDIA_JOB_POLL_DEFAULTS.pollMaxMs); - this.#armMediaPoll(vertex.id); + try { + this.#armMediaPoll(vertex.id); + } catch { + // The job was already accepted by the provider. A host timer fault is an engine failure, not evidence + // that the job was free: settle its single paid addend before failing the node, so both the admission and + // durable terminal total survive this otherwise easy-to-miss out-of-band path. + await this.#settleMediaJobFailed(vertex, job, { + code: 'internal', + message: 'the media job poll timer could not be re-armed', + retryable: false, + }); + } return; } case 'done': diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 8c76b0da..73ae5c51 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -30,6 +30,7 @@ import type { CapabilityFlags, LlmProvider, MediaJobStatus, + PricingOverlay, ProviderId, StreamChunk, } from '@relavium/llm'; @@ -502,6 +503,68 @@ workflow: `, ); +/** Two independent async-media submissions: a parked job is slot-free but its priced admission must stay live. */ +const BUDGETED_ASYNC_MEDIA_PARALLEL = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-budgeted-async-media-parallel + max_parallel: 1 + budget: + max_cost_microcents: 1500 + on_exceed: fail + agents: + - id: painter + model: budget-media-model + provider: openai + system_prompt: You make images. + nodes: + - { id: n1, type: agent, agent_ref: painter, prompt_template: 'Draw one', output_modalities: [image], count: 1 } + - { id: n2, type: agent, agent_ref: painter, prompt_template: 'Draw two', output_modalities: [image], count: 1 } + edges: [] +`, +); + +/** A parked media job and a human gate; resuming the gate reveals whether the parked-job reservation was restored. */ +const BUDGETED_ASYNC_MEDIA_RESUME = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-budgeted-async-media-resume + max_parallel: 2 + budget: + max_cost_microcents: 1500 + on_exceed: fail + agents: + - id: painter + model: budget-media-model + provider: openai + system_prompt: You make images. + nodes: + - { id: gen, type: agent, agent_ref: painter, prompt_template: 'Draw one', output_modalities: [image], count: 1 } + - { id: g, type: human_gate, gate_type: approval } + - { id: after, type: agent, agent_ref: painter, prompt_template: 'Draw two', output_modalities: [image], count: 1 } + edges: + - { from: g, to: after } +`, +); + +/** A synthetic user-priced generative model: the shipped catalog intentionally has no media rates yet. */ +const BUDGET_MEDIA_PRICING: PricingOverlay = new Map([ + [ + 'budget-media-model', + { + provider: 'openai', + nativeId: 'budget-media-model', + displayName: 'Budget media model', + contextWindowTokens: 1, + maxOutputTokens: 1, + inputPerMtokMicrocents: 0, + outputPerMtokMicrocents: 0, + cachedInputPerMtokMicrocents: 0, + mediaOutputRates: { image: 1000 }, + }, + ], +]); + const INPUTS = { topic: 'the report' } as const; // --- The reusable driver --------------------------------------------------------------------------- @@ -512,14 +575,17 @@ function buildEngine( host: Host, resolveProvider: (id: ProviderId) => LlmProvider | undefined, resolveMediaSurface?: (model: string) => 'chat' | 'generative' | undefined, + resolvePrice?: PricingOverlay, ): WorkflowEngine { return new WorkflowEngine({ host, + ...(resolvePrice === undefined ? {} : { resolvePrice }), executor: createStandardNodeExecutor({ sandbox, agent: { resolveProvider, ...(resolveMediaSurface === undefined ? {} : { resolveMediaSurface }), + ...(resolvePrice === undefined ? {} : { resolvePrice }), registry: echoRegistry, tools: [echoToolDef], keyFor: () => 'k', @@ -751,6 +817,40 @@ describe('M2 — end-to-end Node harness (1.U)', () => { expect(costsOf(events).filter((c) => c.nodeId === 'work')).toHaveLength(1); }); + it('async media budget: a slot-free parked job retains its priced admission and blocks the next submission (G38)', async () => { + // `max_parallel:1` deliberately frees the execution slot when n1 parks. The money reservation must NOT follow + // that scheduling slot: n2's same-priced submission would take the total from 1,000 to 2,000µ¢ over a 1,500µ¢ + // cap. Before the retained lease, both generateMedia calls ran and the workflow completed after both polls. + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + const job = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + BUDGET_MEDIA_PRICING, + ); + + const events = await driveMediaRun( + engine.start({ workflow: BUDGETED_ASYNC_MEDIA_PARALLEL, inputs: INPUTS }), + host, + ); + + const terminal = events.at(-1); + expect(terminal?.type).toBe('run:failed'); + expect(terminal?.type === 'run:failed' && terminal.error.code).toBe('budget_exceeded'); + expect(terminal?.type === 'run:failed' && terminal.cumulativeCostMicrocents).toBe(1000); + expect(job.generateCalls()).toBe(1); + expect(events.filter((event) => event.type === 'media_job:submitted')).toHaveLength(1); + const costs = costsOf(events); + expect(costs).toHaveLength(1); + expect(costs[0]?.costMicrocents).toBe(1000); + assertGapFreeSeq(events); + assertCanonicalSchema(events); + }); + it('async media job: a cross-process resume RE-ATTACHES (re-polls) the persisted jobId — never re-submits (MJ-1)', async () => { const store = new InMemoryRunStore(); // Process 1: submit + park, then "crash" (break at run:paused; the poll timer is never fired). @@ -793,6 +893,65 @@ describe('M2 — end-to-end Node harness (1.U)', () => { expect(costsOf(events2).filter((c) => c.nodeId === 'work')).toHaveLength(1); }); + it('async media budget: checkpoint resume reconstructs a parked-job reservation before gate-unblocked egress', async () => { + const store = new InMemoryRunStore(); + // Process 1: submit the priced job and park it alongside a human gate. Its timer is intentionally not fired; + // the checkpoint is the only thing process 2 may use to reconstruct the already-paid commitment. + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + BUDGET_MEDIA_PRICING, + ); + const { + events: events1, + gateId, + lastSeq, + } = await drive( + engine1.start({ workflow: BUDGETED_ASYNC_MEDIA_RESUME, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + const runId = events1[0]?.runId ?? ''; + if (gateId === undefined) + throw new Error('expected the process-1 checkpoint to contain a human gate'); + expect(job1.generateCalls()).toBe(1); + + // Process 2 approves the gate, making `after` ready. It must see gen's reconstructed 1,000µ¢ reservation and + // fail before submit; normal `checkPreEgress` is intentionally NOT rerun for gen itself because it is paid. + const job2 = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const host2 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine2 = buildEngine( + host2, + () => job2.provider, + () => 'generative', + BUDGET_MEDIA_PRICING, + ); + const events2 = await driveMediaRun( + await engine2.resumeFromCheckpoint({ + runId, + workflow: BUDGETED_ASYNC_MEDIA_RESUME, + inputs: INPUTS, + gateId, + decision: { decision: 'approved', decidedBy: 'human' }, + }), + host2, + ); + + const terminal = events2.at(-1); + expect(terminal?.type).toBe('run:failed'); + expect(terminal?.type === 'run:failed' && terminal.error.code).toBe('budget_exceeded'); + expect(terminal?.type === 'run:failed' && terminal.cumulativeCostMicrocents).toBe(1000); + expect(job2.generateCalls()).toBe(0); // gen re-attached; `after` never submitted + expect(costsOf(events2).filter((cost) => cost.nodeId === 'gen')).toHaveLength(1); + // Resume keeps the prior process's sequence space; the resumed segment starts at the persisted last+1 and is + // gap-free from there (rather than restarting at 0 like a fresh run). + events2.forEach((event, index) => expect(event.sequenceNumber).toBe(lastSeq + index + 1)); + assertCanonicalSchema(events2); + }); + it('async media job: a run cancel aborts the in-flight poll → run:cancelled (ADR-0045 §4)', async () => { const host = createInMemoryHost({ store: new InMemoryRunStore(), @@ -1175,6 +1334,44 @@ describe('M2 — end-to-end Node harness (1.U)', () => { expect(done?.type === 'node:completed' && done.durationMs > 0).toBe(true); }); + it('async media job: a poll re-arm timer fault still accounts the paid submission exactly once', async () => { + const baseHost = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + let timerCalls = 0; + const host: Host = { + ...baseHost, + setTimer: (ms, onFire) => { + timerCalls += 1; + // The first timer parks the submitted job. Its first pending poll then attempts the second arm, which + // models a host timer failure outside the normal executor/adapter path. + if (timerCalls === 2) throw new Error('timer unavailable'); + return baseHost.setTimer(ms, onFire); + }, + }; + const job = asyncMediaProvider([{ state: 'pending' }]); + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + ); + const events = await driveMediaRun( + engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:failed'); + const costs = costsOf(events).filter((event) => event.nodeId === 'work'); + expect(costs).toHaveLength(1); + const terminal = events.at(-1); + expect(terminal?.type === 'run:failed' && terminal.cumulativeCostMicrocents).toBe( + costs[0]?.costMicrocents, + ); + assertGapFreeSeq(events); + assertCanonicalSchema(events); + }); + it('flagship: one run — retry then failover, pause at the gate, cross-process resume reproduces the final output', async () => { // The primary (anthropic) ALWAYS errors retryably pre-content; the fallback (openai) fails the first // dispatch then succeeds — so dispatch 1 exhausts the chain (→ node retry), dispatch 2 fails over to the diff --git a/packages/core/src/tools/registry.test.ts b/packages/core/src/tools/registry.test.ts index d67324f6..4d177304 100644 --- a/packages/core/src/tools/registry.test.ts +++ b/packages/core/src/tools/registry.test.ts @@ -779,6 +779,118 @@ describe('ToolRegistry — per-tool approval (ADR-0057 EA3)', () => { expect(confirm.mock.calls[0]?.[0]?.preview).toEqual({ command: 'ls' }); }); + /* --- #91: the approval preview is secret-redacted, like sanitizeInput() already is (ADR-0029(c)) --- */ + + // A syntactically valid but FAKE credential: 20 alphanumerics after `sk-`, which is what + // `redactSecretShapedText`'s standalone-shape alternation matches (`sk-[A-Za-z0-9]{16,}`). + const FAKE_KEY = 'sk-ABCDEFGHIJKLMNOP1234'; + + it('redacts a secret-shaped segment of an fs_write path — and dispatches on the REAL, unscrubbed target', async () => { + const confirm = approving(); + const { host, writeFile } = hostWithWriteSpy(); + await createToolRegistry({ tools: BUILTIN_TOOLS, host }).dispatch( + call('write_file', { path: `./${FAKE_KEY}/out.txt`, content: 'hi' }), + ctx({ approval: { confirm } }), + ); + expect(confirm.mock.calls[0]?.[0]?.preview).toEqual({ path: './[redacted]/out.txt' }); + // The preview is DISPLAY-only: the side effect must still run against the path the model asked for, or + // redaction would have silently changed what the tool does. + expect(writeFile.mock.calls[0]?.[0]).toBe(`./${FAKE_KEY}/out.txt`); + }); + + it('scrubs the egress host arm too — the defence-in-depth branch, which nothing else exercises', async () => { + // An FQDN whose leftmost label is itself credential-SHAPED: legal DNS, and the only input that tells a + // scrubbed host arm apart from an unscrubbed one. Without it, reverting that arm leaves the suite green. + const confirm = approving(); + const secretHost = 'sk-abcdefghijklmnopqrst.example.com'; + const egressHost = stubHost({ + egress: { fetch: () => Promise.resolve({ status: 200, headers: {}, body: 'ok' }) }, + }); + await createToolRegistry({ tools: BUILTIN_TOOLS, host: egressHost }).dispatch( + call('http_request', { url: `https://${secretHost}/x`, method: 'GET' }), + ctx({ approval: { confirm }, toolPolicy: { allowedDomains: [secretHost] } }), + ); + expect(confirm.mock.calls[0]?.[0]?.preview).toEqual({ host: '[redacted].example.com' }); + }); + + it('redacts a credential carried in a run_command arg before it reaches the preview', async () => { + const confirm = approving(); + await registry().dispatch( + call('run_command', { command: 'curl', args: ['-H', `Authorization: Bearer ${FAKE_KEY}`] }), + ctx({ approval: { confirm }, toolPolicy: { allowedCommandGlobs: ['curl *'] } }), + ); + // Assert the SECURITY property (the credential is gone, something was scrubbed) rather than pinning the + // exact output — the redactor's pattern set is free to tighten without dragging this test with it. + const command = confirm.mock.calls[0]?.[0]?.preview.command; + expect(command).not.toContain(FAKE_KEY); + expect(command).toContain('[redacted]'); + // The other half of the property, and the one now most at risk: the preview must stay REVIEWABLE. Without + // this, `command: '[redacted]'` (redact everything) would pass — and the approval prompt would be useless. + expect(command).toContain('curl'); + }); + + it('redacts the EA5 agent:approval_requested preview too — the --json/observability copy, not just the prompt', async () => { + const confirm = approving(); + const emitApprovalRequested = vi.fn<(req: ToolApprovalRequest) => void>(); + const { host } = hostWithWriteSpy(); + await createToolRegistry({ tools: BUILTIN_TOOLS, host }).dispatch( + call('write_file', { path: `./${FAKE_KEY}.txt`, content: 'hi' }), + ctx({ approval: { confirm, emitApprovalRequested } }), + ); + expect(emitApprovalRequested.mock.calls[0]?.[0]?.preview).toEqual({ path: './[redacted].txt' }); + }); + + it('leaves an ordinary path untouched — no over-redaction of a normal target', async () => { + const confirm = approving(); + const { host } = hostWithWriteSpy(); + await createToolRegistry({ tools: BUILTIN_TOOLS, host }).dispatch( + call('write_file', { path: './config/.env', content: 'hi' }), + ctx({ approval: { confirm } }), + ); + expect(confirm.mock.calls[0]?.[0]?.preview).toEqual({ path: './config/.env' }); + }); + + it.each([ + // The credential whose value SPANS a separator — the case a per-segment scrub misses entirely, because + // the part before the `/` falls under the key=value pattern's own 6-char value floor. + ['./api_key=AAAAA/BBBBBB.txt', './[redacted]'], + ['./secret=abc/def.txt', './[redacted]'], + [`./${FAKE_KEY}/.git/config`, './[redacted]/.git/config'], + ])('scrubs a credential that spans a path separator: %s', async (input, expected) => { + const confirm = approving(); + const { host } = hostWithWriteSpy(); + await createToolRegistry({ tools: BUILTIN_TOOLS, host }).dispatch( + call('write_file', { path: input, content: 'hi' }), + ctx({ approval: { confirm } }), + ); + expect(confirm.mock.calls[0]?.[0]?.preview).toEqual({ path: expected }); + }); + + it('hands the confirm hook the UNREDACTED target alongside the redacted preview (#91)', async () => { + // The whole point of splitting the two uses: the display copy can be scrubbed as hard as secrecy needs, + // because the host's protected-path CLASSIFIER reads `target`, not `preview`. Here the scrub collapses + // the `.ssh` segment out of the preview — and `target` still carries the real path. + const confirm = approving(); + const { host } = hostWithWriteSpy(); + const path = './Access Token Backupxyz/.ssh/authorized_keys'; + await createToolRegistry({ tools: BUILTIN_TOOLS, host }).dispatch( + call('write_file', { path, content: 'hi' }), + ctx({ approval: { confirm } }), + ); + const request = confirm.mock.calls[0]?.[0]; + expect(request?.unredactedPreview).toEqual({ path }); + expect(request?.preview.path).not.toBe(path); // the display copy really did lose information + }); + + it('sets unredactedPreview on EVERY governed class, so a classifier can always prefer it', async () => { + const confirm = approving(); + await registry().dispatch( + call('run_command', { command: 'ls' }), + ctx({ approval: { confirm }, toolPolicy: { allowedCommands: ['ls'] } }), + ); + expect(confirm.mock.calls[0]?.[0]?.unredactedPreview).toEqual({ command: 'ls' }); + }); + it('forwards ctx.signal to the confirm hook as its second argument', async () => { const signal = mutableSignal(); const confirm = approving(); diff --git a/packages/core/src/tools/registry.ts b/packages/core/src/tools/registry.ts index b694137a..0b965bdc 100644 --- a/packages/core/src/tools/registry.ts +++ b/packages/core/src/tools/registry.ts @@ -9,7 +9,12 @@ import { extractHttpsHost, type ToolActionClass } from '@relavium/shared'; -import { boundForModel, redactInlineMedia, redactSecretShapedValue } from './bounding.js'; +import { + boundForModel, + redactInlineMedia, + redactSecretShapedText, + redactSecretShapedValue, +} from './bounding.js'; import { ToolArgsInvalidError, ToolCancelledError, @@ -362,6 +367,9 @@ async function confirmDispatch( toolId: def.id, action, preview: previewFor(action, target), + // The same field selection, UNSCRUBBED, for CLASSIFICATION only (see the field's own doc). In-process: + // it never reaches the event, so `preview` stays the one thing crossing the observability boundary. + unredactedPreview: rawPreviewFor(action, target), }; // EA5: emit the observability event for EVERY governed dispatch that reaches this gate, just before the host // decides — a durable "a governed action was gated" trace on the session / `--json` stream (the session @@ -432,8 +440,36 @@ export function governedAction(def: ToolDef, target: PolicyTarget): ToolActionCl return undefined; } -/** A secret-free preview for the approval prompt: the resolved path / command / host (never a full URL). */ -function previewFor(action: ToolActionClass, target: PolicyTarget): ToolActionPreview { +/** + * A secret-free preview for the approval prompt: the resolved path / command / host (never a full URL). + * + * "Secret-free" is **enforced**, not merely asserted: every field is scrubbed with the SAME + * `redactSecretShapedText` detector `sanitizeInput()` applies to `toolInput` moments later in this dispatch + * (ADR-0029(c) — the taint gate only covers KNOWN `secret`-typed args, so a model-placed credential in a + * `run_command` arg or a `write_file` path would otherwise ride this preview verbatim onto the approval + * prompt and the `agent:approval_requested` / `--json` observability stream). + * + * Display-only, in both directions: the dispatch has already resolved and policy-checked the REAL target + * (`enforcePolicy` runs before `confirmDispatch`), so scrubbing here can never change which side effect runs. + * + * The scrub is deliberately WHOLE-STRING rather than per-path-segment, and that is only safe because of + * {@link ToolApprovalRequest.target}. Two of the detector's patterns (`bearer|basic|token ` and + * `=`) have value classes containing `/`, which cuts both ways: whole-string catches a + * credential whose value SPANS a separator (`./api_key=AAAAA/BBBBBB.txt` — a per-segment scrub misses it, + * because the visible part of the value falls under the pattern's own length floor), but one match can also + * swallow the rest of the path (`./Access Token Backup/.ssh/authorized_keys` → `./Access Token [redacted]`). + * No single pattern gives both. So the two USES are split instead: this projection is scrubbed as hard as + * secrecy requires, and the host's protected-path CLASSIFIER reads the unredacted `target` — where a swallowed + * `.ssh` segment cannot change the answer, because it never sees this string. + * + * One limitation remains, and it is a property of redaction itself rather than of this choice: `command` is + * fully model-controlled and the detector's patterns are public, so a prompt-injected model can shape a + * payload to MATCH a pattern (`sh -c token:evil.example/x|sh`) and be shown `sh -c [redacted]` — which reads + * as "we protected you" rather than "you cannot see this". Bounded by `allowedCommands` / + * `allowedCommandGlobs` being deny-all by default and by the fs/egress floors, but real; surfacing "this was + * redacted" to the prompt needs a schema field and is the open follow-up. + */ +function rawPreviewFor(action: ToolActionClass, target: PolicyTarget): ToolActionPreview { switch (action) { case 'fs_write': return target.path === undefined ? {} : { path: target.path }; @@ -443,6 +479,9 @@ function previewFor(action: ToolActionClass, target: PolicyTarget): ToolActionPr if (target.url === undefined) { return {}; // web_search / mcp_call expose no pre-dispatch URL target — the action class is enough } + // The HOST only, never `target.url` — the raw URL carries the query string, which is exactly where a + // credential lives (`?token=…`). That exclusion is why an UNREDACTED copy of this shape is safe to hand + // the host at all; it is the one field selection both copies share. const parsed = extractHttpsHost(target.url); return parsed === null ? {} : { host: parsed.host }; } @@ -459,6 +498,16 @@ function previewFor(action: ToolActionClass, target: PolicyTarget): ToolActionPr } } +/** The DISPLAY copy: {@link rawPreviewFor}'s field selection with every string scrubbed. */ +function previewFor(action: ToolActionClass, target: PolicyTarget): ToolActionPreview { + const raw = rawPreviewFor(action, target); + return { + ...(raw.path === undefined ? {} : { path: redactSecretShapedText(raw.path) }), + ...(raw.command === undefined ? {} : { command: redactSecretShapedText(raw.command) }), + ...(raw.host === undefined ? {} : { host: redactSecretShapedText(raw.host) }), + }; +} + function enforceHttpEgress(toolId: ToolId, url: string, ctx: ToolDispatchContext): void { const parsed = extractHttpsHost(url); if (parsed === null || parsed.hasCredentials) { diff --git a/packages/core/src/tools/types.ts b/packages/core/src/tools/types.ts index 00760828..1f983595 100644 --- a/packages/core/src/tools/types.ts +++ b/packages/core/src/tools/types.ts @@ -294,6 +294,30 @@ export interface ToolApprovalRequest { readonly toolId: ToolId; readonly action: ToolActionClass; readonly preview: ToolActionPreview; + /** + * The same fields as {@link ToolApprovalRequest.preview}, UNSCRUBBED — for **classification only**, never + * for display. + * + * `preview` is a lossy display projection: it is scrubbed (#91), so a credential-shaped run inside a path + * can be replaced by `[redacted]`. A host that CLASSIFIES the target — the CLI's `auto`-mode protected-path + * check is the one that does — must never read that projection, because the scrub can change the answer: + * `./Access Token Backup/.ssh/authorized_keys` collapses to `./Access Token [redacted]`, and a protected + * path classifies as unprotected. Splitting the two uses removes the trade-off entirely — the display copy + * can be scrubbed as hard as secrecy requires without ever weakening a security decision. + * + * **In-process only.** The confirm hook runs in the same process; this field is deliberately absent from + * `agent:approval_requested` (`packages/shared/src/run-event.ts`, whose preview object is `.strict()`), so + * it never crosses the event / IPC / `--json` boundary. A host must not log, persist or forward it. + * + * Optional only so a HAND-BUILT request (a test fixture, a surface stub) stays constructible; the engine — + * the sole production producer, `confirmDispatch` — always sets it, which `registry.test.ts` pins. A + * classifier should prefer it and fall back to `preview` only when it is absent. + * + * Note it is the preview's FIELD SELECTION, not the raw `PolicyTarget`: an egress entry carries the host + * and never `target.url`, because a URL's query string is exactly where a credential lives. Withholding it + * is what makes an unredacted copy safe to hand out at all. + */ + readonly unredactedPreview?: ToolActionPreview; } /** The host's verdict. `reject.reason` is an optional, secret-free, display-safe label echoed in the error. */ diff --git a/packages/db/src/client.test.ts b/packages/db/src/client.test.ts index 89cf4c50..15e26932 100644 --- a/packages/db/src/client.test.ts +++ b/packages/db/src/client.test.ts @@ -1,5 +1,5 @@ import { randomUUID } from 'node:crypto'; -import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -8,7 +8,7 @@ import { EXECUTION_MODES, RunStatusSchema } from '@relavium/shared'; import { eq, sql } from 'drizzle-orm'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; -import { createClient, runMigrations, type DbClient } from './client.js'; +import { createClient, DbOpenError, runMigrations, type DbClient } from './client.js'; import { mediaObjects, mediaReferences, @@ -454,6 +454,48 @@ describe('@relavium/db migration + constraint invariants', () => { ); }); + /* --- #104: a typed, code-discriminated open failure with the path as a FIELD, not in the message --- */ + + it('throws a typed DbOpenError with code uri_unsupported, carrying the path off-message', () => { + let caught: unknown; + try { + createClient('file::memory:?cache=shared'); + } catch (err) { + caught = err; + } + if (!(caught instanceof DbOpenError)) throw new Error('expected a DbOpenError'); + const error = caught; + expect(error.code).toBe('uri_unsupported'); // callers narrow on this, never on the message + expect(error.name).toBe('DbOpenError'); + expect(error.path).toBe('file::memory:?cache=shared'); + // The path is a structured field, NOT interpolated — this message is surfaced verbatim to the user by + // the CLI's history openers, and an absolute db path carries the OS username. + expect(error.message).not.toContain('file::memory:'); + }); + + it('throws a typed DbOpenError with code open_failed and the original cause when the driver fails', () => { + const dir = mkdtempSync(join(tmpdir(), 'relavium-open-fail-')); + try { + // A DIRECTORY where the database file should be: mkdir succeeds, `new Database()` then fails. + const path = join(dir, 'history.db'); + mkdirSync(path); + let caught: unknown; + try { + createClient(path); + } catch (err) { + caught = err; + } + if (!(caught instanceof DbOpenError)) throw new Error('expected a DbOpenError'); + const error = caught; + expect(error.code).toBe('open_failed'); + expect(error.path).toBe(path); + expect(error.cause).toBeDefined(); // wrap-and-rethrow preserves the root cause + expect(error.message).not.toContain(path); + } finally { + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }); + it('rejects a duplicate (run_id, seq) in run_events (the unique gap-detection invariant)', () => { const workflowId = randomUUID(); const runId = randomUUID(); diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index c8c8c1a5..ed62d2be 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -6,6 +6,7 @@ import Database from 'better-sqlite3'; import { drizzle, type BetterSQLite3Database } from 'drizzle-orm/better-sqlite3'; import { migrate } from 'drizzle-orm/better-sqlite3/migrator'; +import { withMigrationLock } from './migrate-lock.js'; import * as schema from './schema.js'; /** @@ -26,45 +27,103 @@ export interface DbClient { readonly db: Db; /** The underlying better-sqlite3 connection (call `.close()` when done). */ readonly sqlite: Database.Database; + /** + * The path this client was opened on (`':memory:'` for the default). Carried so a caller can pass it to + * {@link runMigrations} for the ADR-0073 cross-process migration lock without having to remember it + * separately from the handle. + */ + readonly path: string; +} + +/** Why {@link createClient} refused or failed to open a database — narrow on this, never on the message. */ +export type DbOpenErrorCode = + /** A `file:…` SQLite URI filename, which this factory deliberately does not support. */ + | 'uri_unsupported' + /** The driver or the parent-directory `mkdir` failed (locked, corrupt, permission, missing dir). */ + | 'open_failed'; + +/** + * A typed `history.db` open failure — the package convention (`SafeEgressError`, `MediaWriteError`), so a + * caller narrows on `.code` rather than matching a message (`docs/standards/error-handling.md`). + * + * The `message` names a **reason only**: the database path rides as the structured {@link DbOpenError.path} + * field instead of being interpolated into it. That is the same rule the sibling errors follow, and it matters + * here because this message is surfaced verbatim to the user by the CLI's history openers — an absolute path + * carries the OS username, which a user-facing error is not supposed to leak. A caller that wants to show it + * still can, redacted, from the field. + */ +export class DbOpenError extends Error { + readonly code: DbOpenErrorCode; + /** The database path that was requested. A diagnostic field — deliberately NOT part of `message`. */ + readonly path: string; + constructor(code: DbOpenErrorCode, message: string, path: string, options?: { cause?: unknown }) { + super(message, options); + this.name = 'DbOpenError'; + this.code = code; + this.path = path; + } } /** * Open a SQLite database and return a schema-bound Drizzle client. `path` defaults to a * private in-memory database; pass a filesystem path for a persistent local store. + * Throws a typed {@link DbOpenError} on a rejected or failed open. + * + * Applies **all four** project PRAGMAs — the canonical home for what they are for is + * [database-schema.md §Concurrency & transaction behavior](../../../docs/reference/shared-core/database-schema.md#concurrency--transaction-behavior): * - * Applies the project PRAGMAs: `journal_mode = WAL` (concurrent reads while a run writes; - * a no-op for in-memory) and `foreign_keys = ON` (SQLite does not enforce FKs per - * connection by default — the CASCADE rules in the schema depend on it). + * - `journal_mode = WAL` — readers never block the single writer, and vice-versa (a no-op for in-memory). + * - `foreign_keys = ON` — SQLite does not enforce FKs per connection by default, and the schema's CASCADE + * rules depend on it. + * - `busy_timeout = 5000` — SQLite's built-in busy handler waits up to 5 s for a contended lock before + * returning `SQLITE_BUSY`. Load-bearing for the concurrent-process write path, and the term that dominates + * `withBusyRetry`'s worst case ([retry.ts](./retry.ts)). + * - `synchronous = NORMAL` — the recommended durability/throughput trade-off under WAL. */ export function createClient(path = ':memory:'): DbClient { // SQLite URI filenames (`file:…`) are NOT supported: better-sqlite3 needs `{ uri: true }` to // interpret them and otherwise silently creates a literal file of that name. Reject up front // rather than open the wrong database — pass ':memory:' or a plain filesystem path. if (path.startsWith('file:')) { - throw new Error( - `SQLite URI paths are not supported by createClient — pass ':memory:' or a filesystem path (got '${path}')`, + throw new DbOpenError( + 'uri_unsupported', + "SQLite URI paths are not supported by createClient — pass ':memory:' or a filesystem path", + path, ); } - let sqlite: Database.Database; + let sqlite: Database.Database | undefined; try { // Create the parent directory for a real file path so a first-run open doesn't fail on a - // missing folder (inside the try so a filesystem error gets the same path-rich message). + // missing folder (inside the try so a filesystem error is classified the same way). if (path !== ':memory:') { mkdirSync(dirname(path), { recursive: true }); } sqlite = new Database(path); + // The PRAGMAs and the Drizzle bind are INSIDE the try: any of them can fail (a corrupt header surfaces on + // the first `journal_mode` write, not on open), and outside it a failure would leak the just-opened + // connection for the process lifetime AND escape as an untyped driver error — past both this factory's + // typed-error contract and `openLocalDb`'s cleanup/at-rest handling. + sqlite.pragma('journal_mode = WAL'); // concurrent reads while a run writes (no-op in memory) + sqlite.pragma('foreign_keys = ON'); // SQLite does not enforce FKs per connection by default + sqlite.pragma('busy_timeout = 5000'); // wait up to 5s for a writer lock instead of erroring + sqlite.pragma('synchronous = NORMAL'); // the recommended durability/throughput trade-off with WAL + const db = drizzle(sqlite, { schema }); + return { db, sqlite, path }; } catch (err) { - // Rethrow with the resolved path + reason (locked/corrupt/permission/missing dir), - // preserving the original via `cause` — a bare fs/driver error has no path context. - const reason = err instanceof Error ? err.message : String(err); - throw new Error(`failed to open SQLite database at '${path}': ${reason}`, { cause: err }); + // Close what we opened before propagating, or the connection leaks on every failed setup. + try { + sqlite?.close(); + } catch { + /* already failing; a secondary close error must not replace the real cause */ + } + // A STABLE, generic message: `err.message` from better-sqlite3 routinely contains the absolute file path, + // and this message is surfaced verbatim to the user by the CLI's history openers — interpolating it would + // put the OS username back in user-facing output, which is the very thing moving the path to `.path` was + // for. The driver's own error is preserved in full via `cause` for diagnosis. + throw new DbOpenError('open_failed', 'could not open the SQLite database', path, { + cause: err, + }); } - sqlite.pragma('journal_mode = WAL'); // concurrent reads while a run writes (no-op in memory) - sqlite.pragma('foreign_keys = ON'); // SQLite does not enforce FKs per connection by default - sqlite.pragma('busy_timeout = 5000'); // wait up to 5s for a writer lock instead of erroring - sqlite.pragma('synchronous = NORMAL'); // the recommended durability/throughput trade-off with WAL - const db = drizzle(sqlite, { schema }); - return { db, sqlite }; } /** The packaged migration set (`./drizzle`), resolved relative to this module so it works @@ -72,10 +131,30 @@ export function createClient(path = ':memory:'): DbClient { * the package root. */ const MIGRATIONS_DIR = fileURLToPath(new URL('../drizzle', import.meta.url)); +/** Options for {@link runMigrations}. */ +export interface RunMigrationsOptions { + /** + * The database file, so the batch can be serialized across processes by the ADR-0073 migration lock — + * pass {@link DbClient.path}. Omitted (or `':memory:'`) runs unlocked, which is correct for a private + * in-memory database and is what every unit test wants. + * + * Optional rather than required deliberately: making it required would be a breaking change for every + * existing caller, and the ones that matter — the real `history.db` openers — are a short, reviewable list. + */ + readonly dbPath?: string; +} + /** * Apply every pending `drizzle-kit` migration to the given client. Idempotent: Drizzle * tracks applied migrations, so re-running is a no-op. Surfaces call this on first use. + * + * With `dbPath` set, the whole batch runs under the cross-process lock described in + * [ADR-0073](../../../docs/decisions/0073-history-db-migration-lock.md) — because drizzle's migrator decides + * what to apply in a `SELECT` OUTSIDE its own transaction, two processes racing a fresh file would otherwise + * both apply the full set and one would die on `CREATE TABLE` (finding #99). */ -export function runMigrations(db: Db): void { - migrate(db, { migrationsFolder: MIGRATIONS_DIR }); +export function runMigrations(db: Db, options: RunMigrationsOptions = {}): void { + withMigrationLock(options.dbPath, () => { + migrate(db, { migrationsFolder: MIGRATIONS_DIR }); + }); } diff --git a/packages/db/src/fixtures/migrate-racer.mjs b/packages/db/src/fixtures/migrate-racer.mjs new file mode 100644 index 00000000..0fb50210 --- /dev/null +++ b/packages/db/src/fixtures/migrate-racer.mjs @@ -0,0 +1,27 @@ +// A child process for the ADR-0073 / #99 two-process migration race (migrate-lock.e2e.test.ts). It opens a +// FRESH, UNMIGRATED `history.db` via the BUILT `@relavium/db` and runs `runMigrations` against it. Two of these +// starting together exercise the real cross-process race a single Node process cannot reproduce: drizzle's +// migrator decides what to apply in a `SELECT` outside its own transaction, so without the lock both children +// conclude the full set is pending, one commits, and the other dies on a duplicate `CREATE TABLE`. +// +// It prints `OK` on success and nothing on failure (the non-zero exit is the signal), so the parent can assert +// that BOTH children succeeded rather than just that one did. +// +// argv: [node, thisFile, , ] +/* global process -- a Node child-process fixture (not TS source); it uses only this Node global. */ +const [, , distPath, dbPath] = process.argv; + +let client; +try { + // test-harness mechanism: the child cannot use vitest's source resolution, so it imports the BUILT + // @relavium/db by an argv-provided abs path. Not a seam bypass — @relavium/db carries no provider SDK. + // eslint-disable-next-line no-restricted-syntax + const { createClient, runMigrations } = await import(distPath); + client = createClient(dbPath); + // The whole point: `dbPath` is what engages the cross-process lock. Omitting it here would make the test + // pass for the wrong reason on a fast machine and fail intermittently on a slow one. + runMigrations(client.db, { dbPath: client.path }); + process.stdout.write('OK'); +} finally { + client?.sqlite.close(); +} diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index c5b07eee..04da1358 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -53,7 +53,15 @@ export type { NewMediaReferenceRow, } from './schema.js'; -export { createClient, runMigrations, type Db, type DbClient } from './client.js'; +export { + createClient, + runMigrations, + DbOpenError, + type Db, + type DbClient, + type DbOpenErrorCode, + type RunMigrationsOptions, +} from './client.js'; // Session persistence (1.X) — the directly-stored, append-only transcript layer over the // agent_sessions + session_messages tables. The domain ↔ row mappers double as the validation diff --git a/packages/db/src/migrate-lock.e2e.test.ts b/packages/db/src/migrate-lock.e2e.test.ts new file mode 100644 index 00000000..94f742eb --- /dev/null +++ b/packages/db/src/migrate-lock.e2e.test.ts @@ -0,0 +1,114 @@ +import { spawn } from 'node:child_process'; +import { existsSync, mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import Database from 'better-sqlite3'; +import { sql } from 'drizzle-orm'; +import { describe, expect, it } from 'vitest'; + +import { createClient, runMigrations } from './client.js'; + +/** + * ADR-0073 / finding #99 — the REAL two-process migration race the acceptance clause names: *"two processes + * racing `runMigrations` against a fresh `history.db` no longer crash either one, proven by a two-process + * regression test."* + * + * A single Node process cannot reproduce it. `better-sqlite3` is synchronous, so two in-process + * `runMigrations` calls can only ever run one after the other — the interleaving that makes drizzle's + * decide-outside-the-transaction migrator unsafe requires two OS processes with their own SQLite instances and + * real OS file locks. So: two children spawned together against one brand-new file. + * + * It needs the BUILT `@relavium/db` (the child cannot use vitest's source resolution) and is visibly SKIPPED — + * never silently passed — when the dist is absent, mirroring `apps/cli/src/harness/concurrency.e2e.test.ts`. + * A `pnpm turbo run build` produces it; CI builds upstream packages first. + * + * **What this proves, precisely.** With the lock it passes every run. WITHOUT the lock it fails roughly one + * run in three — measured, not assumed — because the collision depends on how closely the two `spawn`s land. + * That is the same flakiness that let #99 ship, so this is a coexistence SMOKE, not the clause-guard: the + * deterministic guards for every branch of the protocol (wait, stale takeover, garbage lock, reconcile, + * fail-loud) are the injected-clock unit tests in `migrate-lock.test.ts`. Exactly the division of labour + * `concurrency.e2e.test.ts` documents for `withBusyRetry`. A READY-handshake barrier would make the collision + * deterministic, as it does there; it is a worthwhile follow-up, not a blocker for the fix itself. + */ + +// The child receives the built db as a file:// URL (its `import()` needs a URL — a bare Windows path like +// `C:\…` is not a valid import specifier); the path form is only for the existence gate. +const DB_DIST_URL = new URL('../dist/index.js', import.meta.url); +const DB_DIST_PATH = fileURLToPath(DB_DIST_URL); +const CHILD_SCRIPT = fileURLToPath(new URL('./fixtures/migrate-racer.mjs', import.meta.url)); + +/** Run the racer child, resolving its stdout and exit code. Never rejects — the parent asserts on both. */ +function race(dbPath: string): Promise<{ code: number | null; out: string; err: string }> { + return new Promise((resolve) => { + const child = spawn(process.execPath, [CHILD_SCRIPT, DB_DIST_URL.href, dbPath], { + stdio: ['ignore', 'pipe', 'pipe'], + }); + let out = ''; + let err = ''; + child.stdout.on('data', (chunk: Buffer) => { + out += chunk.toString('utf8'); + }); + child.stderr.on('data', (chunk: Buffer) => { + err += chunk.toString('utf8'); + }); + child.on('close', (code) => resolve({ code, out, err })); + }); +} + +describe('runMigrations — the two-process race (#99)', () => { + it.skipIf(!existsSync(DB_DIST_PATH))( + 'two concurrent processes BOTH succeed against one fresh history.db', + async () => { + const dir = mkdtempSync(join(tmpdir(), 'relavium-migrate-race-')); + const dbPath = join(dir, 'history.db'); + try { + // Started together, before either has awaited — as close to simultaneous as spawn allows. Without the + // lock, the loser dies with a DrizzleError on a duplicate CREATE TABLE. + const [first, second] = await Promise.all([race(dbPath), race(dbPath)]); + + for (const [label, result] of [ + ['first', first], + ['second', second], + ] as const) { + // Assert on stdout too, not just the exit code: a child that silently did nothing would also exit 0. + expect(result.out, `${label} child stderr: ${result.err}`).toBe('OK'); + expect(result.code, `${label} child stderr: ${result.err}`).toBe(0); + } + + // The schema really landed, exactly once, and a third pass is still a clean no-op. + const client = createClient(dbPath); + try { + expect(() => runMigrations(client.db, { dbPath: client.path })).not.toThrow(); + const tables = client.db + .all<{ name: string }>(sql`select name from sqlite_master where type = 'table'`) + .map((row) => row.name); + expect(tables).toContain('runs'); + expect(tables).toContain('agent_sessions'); + // One row per migration, no duplicates — the loser must have applied NOTHING, not re-applied. + const applied = client.db.all<{ n: number }>( + sql`select count(*) as n from __drizzle_migrations`, + ); + expect(applied[0]?.n).toBeGreaterThan(0); + } finally { + client.sqlite.close(); + } + // The lock file is a SQLite database, so it legitimately REMAINS on disk after a clean pair of runs — + // what must not remain is a held lock. Prove it is released by acquiring it: a stuck EXCLUSIVE would + // make this throw SQLITE_BUSY instead. + const probe = new Database(`${dbPath}.migrate.lock`); + try { + probe.pragma('busy_timeout = 0'); + expect(() => probe.exec('BEGIN EXCLUSIVE')).not.toThrow(); + probe.exec('COMMIT'); + } finally { + probe.close(); + } + } finally { + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }, + 60_000, + ); +}); diff --git a/packages/db/src/migrate-lock.test.ts b/packages/db/src/migrate-lock.test.ts new file mode 100644 index 00000000..1c179c14 --- /dev/null +++ b/packages/db/src/migrate-lock.test.ts @@ -0,0 +1,134 @@ +import { existsSync, mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import Database from 'better-sqlite3'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import { withMigrationLock } from './migrate-lock.js'; + +/** + * The ADR-0073 migration lock. It is an OS advisory lock (`BEGIN EXCLUSIVE` on a dedicated SQLite lock file), + * so most of what the previous lock-file protocol needed tests for — staleness, takeover, pid handling — no + * longer exists: the kernel releases the lock on process death. What remains testable in-process is the + * mutual-exclusion contract, the release, and the two fall-through paths. + * + * The REAL two-process race (#99's acceptance clause) lives in `migrate-lock.e2e.test.ts`. + */ + +let dir: string; +let dbPath: string; +let lockPath: string; + +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'relavium-migrate-lock-')); + dbPath = join(dir, 'history.db'); + lockPath = `${dbPath}.migrate.lock`; +}); + +afterEach(() => { + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); +}); + +describe('withMigrationLock (ADR-0073)', () => { + it('runs the body and releases the lock afterwards', () => { + expect(withMigrationLock(dbPath, () => 'ok')).toBe('ok'); + // Released: a second acquisition succeeds promptly rather than waiting out the timeout. + const started = Date.now(); + expect(withMigrationLock(dbPath, () => 'again')).toBe('again'); + expect(Date.now() - started).toBeLessThan(2_000); + }); + + it('releases the lock even when the body throws — a failed migration must not wedge the install', () => { + expect(() => + withMigrationLock(dbPath, () => { + throw new Error('disk full'); + }), + ).toThrow('disk full'); + expect(withMigrationLock(dbPath, () => 'ok')).toBe('ok'); + }); + + it('EXCLUDES a concurrent holder — the body never runs while another connection holds the lock', () => { + // A live holder, exactly as a second process would be: a real EXCLUSIVE transaction on the lock file. + const holder = new Database(lockPath); + try { + holder.exec('BEGIN EXCLUSIVE'); + let attempts = 0; + // Reaching `runReconciling` — TWO attempts, not one — is the observable proof that acquisition was + // refused. Under the old rename-based takeover this same shape let both parties into the body. + const result = withMigrationLock( + dbPath, + () => { + attempts += 1; + if (attempts === 1) throw new Error('table `runs` already exists'); + return 'reconciled'; + }, + // timeoutMs 0 surfaces SQLITE_BUSY at once — what a waiter past the real 10 s timeout sees, + // without making the suite wait 10 s for it. + { timeoutMs: 0 }, + ); + expect(result).toBe('reconciled'); + expect(attempts).toBe(2); + } finally { + holder.exec('COMMIT'); + holder.close(); + } + }); + + it('is a NO-OP for an in-memory database — a `:memory:.migrate.lock` would be nonsense', () => { + let ran = false; + withMigrationLock(':memory:', () => { + ran = true; + }); + expect(ran).toBe(true); + expect(existsSync(':memory:.migrate.lock')).toBe(false); + }); + + it('is a NO-OP when no path is supplied (every unit test takes this path)', () => { + let ran = false; + withMigrationLock(undefined, () => { + ran = true; + }); + expect(ran).toBe(true); + }); + + it('reconciles when the lock file cannot be opened at all (a read-only directory)', () => { + let attempts = 0; + const result = withMigrationLock( + dbPath, + () => { + attempts += 1; + if (attempts === 1) throw new Error('table `runs` already exists'); + return 'reconciled'; + }, + { + openLock: () => { + throw new Error('EROFS: read-only file system'); + }, + }, + ); + expect(result).toBe('reconciled'); + expect(attempts).toBe(2); + }); + + it('still FAILS LOUD through the fallback, rethrowing the FIRST error, not the second', () => { + let attempts = 0; + expect( + () => + withMigrationLock( + dbPath, + () => { + attempts += 1; + throw new Error(attempts === 1 ? 'the real cause' : 'a vaguer second failure'); + }, + { + openLock: () => { + throw new Error('EROFS: read-only file system'); + }, + }, + ), + // Reconcile must not launder a genuine migration failure into a different message. + ).toThrow('the real cause'); + expect(attempts).toBe(2); // exactly one retry — bounded, never a loop + }); +}); diff --git a/packages/db/src/migrate-lock.ts b/packages/db/src/migrate-lock.ts new file mode 100644 index 00000000..fd29da07 --- /dev/null +++ b/packages/db/src/migrate-lock.ts @@ -0,0 +1,103 @@ +import Database from 'better-sqlite3'; + +/** + * Cross-process serialization for the `runMigrations` batch + * ([ADR-0073](../../../docs/decisions/0073-history-db-migration-lock.md), finding `#99`). + * + * **Why a lock at all.** drizzle's synchronous SQLite migrator runs the `SELECT` that DECIDES which migrations + * are pending *outside* its own transaction, then opens a plain DEFERRED `BEGIN`. So the read that decides and + * the write that applies are not atomic whatever `BEGIN` mode is used — two processes on a fresh `history.db` + * both conclude the full set is pending, one commits, and the other dies on `CREATE TABLE runs` with a raw + * `DrizzleError`. We also cannot hoist `migrate()` into our own transaction, because its raw `BEGIN` would + * throw inside one. + * + * **The mechanism: a real advisory lock, via a dedicated SQLite lock database.** `BEGIN EXCLUSIVE` on a + * separate file takes an OS-level `fcntl` write lock for the life of the transaction, and — the property that + * matters — **the kernel releases it when the process dies, however it dies.** ADR-0073's original lock-file + * protocol could not achieve that: it needed a staleness threshold, and breaking a stale lock with plain file + * operations is unsound (two takers can each believe they won, and a superseded holder can delete the new + * owner's lock). Review found exactly that; the ADR carries a dated note. + * + * A separate file, not `history.db` itself: taking `BEGIN EXCLUSIVE` on the database drizzle is about to + * migrate would deadlock it against our own connection. + * + * On contention the waiter blocks inside SQLite's busy handler for up to {@link LOCK_TIMEOUT_MS}; past that it + * falls through to run-then-reconcile rather than hang or crash. + */ + +/** + * How long a waiter blocks for the lock before reconciling instead. Generous against a millisecond-scale + * batch, and bounded so a pathological holder cannot hang first-run startup — the one thing this protects. + */ +const LOCK_TIMEOUT_MS = 10_000; + +/** + * Run `migrate` once; on ANY failure, run it exactly once more. The second pass observes the winner's + * committed `__drizzle_migrations` row and applies nothing, so a lost race resolves cleanly — while a genuine + * migration failure fails the same way twice and the ORIGINAL error is rethrown (fail-loud, never the vaguer + * second one). Reachable only when the lock could not be taken: a read-only directory, a filesystem whose + * `fcntl` locking does not work (some network mounts), or a holder that outlasted the timeout. Crashing the + * loser of a race there would be strictly worse. + */ +function runReconciling(run: () => T): T { + try { + return run(); + } catch (error_) { + try { + return run(); + } catch { + throw error_; + } + } +} + +/** + * Run `migrate` under the cross-process migration lock. `dbPath` is the database file; `undefined` or + * `':memory:'` skips the lock entirely — a private in-memory database cannot be contended by construction, and + * it keeps the test suite's hundreds of in-memory migrations off this path. `deps.openLock` is a test seam for + * driving the could-not-acquire branch without a real second process. + */ +export function withMigrationLock( + dbPath: string | undefined, + run: () => T, + deps: { + readonly openLock?: (path: string) => Database.Database; + /** Acquisition timeout override — tests drive the refused-acquisition branch without a real 10 s wait. */ + readonly timeoutMs?: number; + } = {}, +): T { + if (dbPath === undefined || dbPath === ':memory:') { + return run(); + } + const openLock = deps.openLock ?? ((path: string) => new Database(path)); + let lock: Database.Database; + try { + lock = openLock(`${dbPath}.migrate.lock`); + } catch { + // The lock file cannot be opened at all (a read-only directory). Reconciling beats refusing to start. + return runReconciling(run); + } + try { + lock.pragma(`busy_timeout = ${deps.timeoutMs ?? LOCK_TIMEOUT_MS}`); + try { + // The acquisition. `BEGIN EXCLUSIVE` is the only statement here that can block or fail on contention. + lock.exec('BEGIN EXCLUSIVE'); + } catch { + // SQLITE_BUSY past the timeout, or a filesystem whose locking does not work. Either way: do not hang. + return runReconciling(run); + } + try { + return run(); + } finally { + // Release. A throw here must not mask the migration's own outcome, and `close()` below releases the OS + // lock regardless — so this is best-effort by design, not a swallowed failure. + try { + lock.exec('COMMIT'); + } catch { + /* the close below releases the OS lock either way */ + } + } + } finally { + lock.close(); + } +} diff --git a/packages/db/src/retry.test.ts b/packages/db/src/retry.test.ts index f4c2fd85..46a80853 100644 --- a/packages/db/src/retry.test.ts +++ b/packages/db/src/retry.test.ts @@ -5,11 +5,12 @@ import { join } from 'node:path'; import { describe, expect, it, vi } from 'vitest'; import { createClient, runMigrations } from './client.js'; -import { withBusyRetry } from './retry.js'; +import { withBusyRetry, withBusyRetryAsync } from './retry.js'; /** A `better-sqlite3`-shaped lock error: an `Error` with the string `.code` the driver sets. */ -const lockError = (code: 'SQLITE_BUSY' | 'SQLITE_LOCKED'): Error => - Object.assign(new Error('database is locked'), { code }); +const lockError = ( + code: 'SQLITE_BUSY' | 'SQLITE_LOCKED' | 'SQLITE_BUSY_SNAPSHOT' | 'SQLITE_BUSY_RECOVERY', +): Error => Object.assign(new Error('database is locked'), { code }); describe('withBusyRetry — unit (2.5.I)', () => { it('returns the value on first success (no retry, no sleep)', () => { @@ -54,19 +55,29 @@ describe('withBusyRetry — unit (2.5.I)', () => { expect(sleeps).toEqual([25, 50, 75, 100]); }); - it('retries SQLITE_LOCKED as well', () => { - let calls = 0; - const result = withBusyRetry( - () => { - calls += 1; - if (calls < 2) throw lockError('SQLITE_LOCKED'); - return 'ok'; - }, - { sleep: () => {} }, - ); - expect(result).toBe('ok'); - expect(calls).toBe(2); - }); + // One parameterized case per retryable code rather than three near-identical blocks. Each code is in the + // set for its own reason, so the reasons live here: `SQLITE_LOCKED` is the plain table-lock fault; + // `SQLITE_BUSY_SNAPSHOT` is the stale-DEFERRED upgrade failure `busy_timeout` does NOT cover, which arrives + // as its own EXTENDED code string rather than as a `SQLITE_BUSY` prefix — the whole substance of #100; and + // `SQLITE_BUSY_RECOVERY` is a WAL-index rebuild after a crash, which the busy handler's loop exits WITH + // rather than downgrading, so a first-run-after-crash write would otherwise fail loud on a condition that + // clears in milliseconds. + it.each(['SQLITE_LOCKED', 'SQLITE_BUSY_SNAPSHOT', 'SQLITE_BUSY_RECOVERY'] as const)( + 'retries %s', + (code) => { + let calls = 0; + const result = withBusyRetry( + () => { + calls += 1; + if (calls < 2) throw lockError(code); + return 'ok'; + }, + { sleep: () => {} }, + ); + expect(result).toBe('ok'); + expect(calls).toBe(2); + }, + ); it('rethrows a NON-lock error immediately, unchanged (no retry, no sleep)', () => { const sleep = vi.fn(); @@ -139,6 +150,133 @@ describe('withBusyRetry — unit (2.5.I)', () => { }); }); +describe('withBusyRetryAsync — the async twin (#226)', () => { + it('returns the value on first success (no retry, no sleep)', async () => { + const sleep = vi.fn(() => Promise.resolve()); + await expect(withBusyRetryAsync(() => 42, { sleep })).resolves.toBe(42); + expect(sleep).not.toHaveBeenCalled(); + }); + + it('awaits an async fn and resolves its value', async () => { + await expect( + withBusyRetryAsync(() => Promise.resolve('ok'), { sleep: () => Promise.resolve() }), + ).resolves.toBe('ok'); + }); + + it('follows the SAME deterministic linear schedule as the sync twin (no jitter)', async () => { + const sleeps: number[] = []; + let caught: unknown; + try { + await withBusyRetryAsync( + () => { + throw lockError('SQLITE_BUSY'); + }, + { + baseDelayMs: 25, + sleep: (ms) => { + sleeps.push(ms); + return Promise.resolve(); + }, + }, + ); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(Error); + // Byte-identical to the sync twin's schedule assertion above — the two must never drift apart. + expect(sleeps).toEqual([25, 50, 75, 100]); + }); + + it('retries a REJECTED lock fault (not just a synchronous throw) and then succeeds', async () => { + let calls = 0; + const result = await withBusyRetryAsync( + () => { + calls += 1; + return calls < 3 + ? Promise.reject(lockError('SQLITE_BUSY_SNAPSHOT')) + : Promise.resolve('ok'); + }, + { sleep: () => Promise.resolve() }, + ); + expect(result).toBe('ok'); + expect(calls).toBe(3); + }); + + it('rejects with the ORIGINAL non-lock error immediately (no retry, no sleep)', async () => { + const sleep = vi.fn(() => Promise.resolve()); + const constraint = Object.assign(new Error('constraint failed'), { code: 'SQLITE_CONSTRAINT' }); + let calls = 0; + await expect( + withBusyRetryAsync( + () => { + calls += 1; + throw constraint; + }, + { sleep }, + ), + ).rejects.toBe(constraint); + expect(calls).toBe(1); + expect(sleep).not.toHaveBeenCalled(); + }); + + it('fails loud after exhausting maxAttempts, rejecting with the ORIGINAL lock error', async () => { + const busy = lockError('SQLITE_BUSY'); + let calls = 0; + await expect( + withBusyRetryAsync( + () => { + calls += 1; + throw busy; + }, + { maxAttempts: 4, sleep: () => Promise.resolve() }, + ), + ).rejects.toBe(busy); // never swallowed — a dropped write is silent data loss (ADR-0050) + expect(calls).toBe(4); + }); + + it('a stray maxAttempts of 0 is floored to one attempt, matching the sync twin', async () => { + const sleep = vi.fn(() => Promise.resolve()); + let calls = 0; + await expect( + withBusyRetryAsync( + () => { + calls += 1; + throw lockError('SQLITE_BUSY'); + }, + { maxAttempts: 0, sleep }, + ), + ).rejects.toThrow(); + expect(calls).toBe(1); + expect(sleep).not.toHaveBeenCalled(); + }); + + it('actually YIELDS the event loop between attempts — the whole point of the twin', async () => { + // With the default (real `setTimeout`) sleep, other work must be able to interleave during the backoff. + // The sync twin's `Atomics.wait` would park the thread and make this impossible. + // A MACROTASK, armed from inside attempt 1, is the strong form of the claim: `Atomics.wait` parks the + // thread and drains no queue at all, so a timer armed before the backoff could not fire until the whole + // synchronous retry returned. A 0 ms timer (clamped to 1 ms) against a 20 ms backoff leaves a 19 ms + // ordering margin, so this is deterministic rather than a race. + const observed: string[] = []; + let calls = 0; + await expect( + withBusyRetryAsync( + () => { + calls += 1; + observed.push(`attempt${calls}`); + if (calls < 2) { + setTimeout(() => observed.push('other-work'), 0); + throw lockError('SQLITE_BUSY'); + } + return 'ok'; + }, + { baseDelayMs: 20 }, // the REAL sleepAsync — no injected sleep + ), + ).resolves.toBe('ok'); + expect(observed).toEqual(['attempt1', 'other-work', 'attempt2']); + }); +}); + /** * Real SQLITE_BUSY contention: two connections on one file, one holding the single WAL write lock. The * injected `sleep` is the interleave hook — releasing the lock during the backoff lets the retry succeed, diff --git a/packages/db/src/retry.ts b/packages/db/src/retry.ts index 675a988f..8bfad719 100644 --- a/packages/db/src/retry.ts +++ b/packages/db/src/retry.ts @@ -20,29 +20,74 @@ * `replaceProviderModels` bulk-upsert + `upsert`, the provider `upsert`) are idempotent given the same input + * DB state. * + * **Two twins, one policy.** {@link withBusyRetry} is synchronous, because `better-sqlite3` is; + * {@link withBusyRetryAsync} is for a call site whose caller is genuinely async — today only `persistEvent` + * (run history) — where the backoff can yield the event loop instead of parking the thread (#226). The chat + * persister's writers are NOT on it and cannot be: they run inside a synchronous `RunEventBus` subscriber, so + * an async write would escape `deliver()`'s catch as a floating promise and never reach the listener sink. They share the retryable-code set, the budget, the + * schedule and the fail-loud exhaustion — only the sleep differs. Read the async twin's own doc for what that + * does and, more importantly, what it does NOT buy: each attempt still blocks inside the synchronous driver for + * up to `busy_timeout`, so the backoff is the small term, not the dominant one. + * * CAVEAT: `{ behavior: 'immediate' }` only applies to the OUTERMOST `BEGIN`. If a wrapped store method is ever * called INSIDE another `db.transaction`, better-sqlite3 demotes it to a `SAVEPOINT` and the IMMEDIATE behavior - * is silently ignored. All current call sites invoke these as top-level store methods; a future batch-in-one- - * transaction caller must take the outer `BEGIN IMMEDIATE` itself. + * is silently ignored — and the retry cannot rescue that: with the outer transaction still open, a retry cannot + * refresh the connection's read snapshot, so every attempt fails identically and the budget is burned for + * nothing. All current call sites invoke these as top-level store methods; a future batch-in-one-transaction + * caller must take the outer `BEGIN IMMEDIATE` itself. */ -/** Driver error codes we wait out: a lock we can retry. Anything else is a real fault → rethrow. */ -const RETRYABLE_CODES: ReadonlySet = new Set(['SQLITE_BUSY', 'SQLITE_LOCKED']); +/** + * Driver error codes we wait out: a lock we can retry. Anything else is a real fault → rethrow. + * + * `better-sqlite3` reports the **extended** result code, so the stale-`BEGIN DEFERRED` upgrade failure this + * module's header names arrives as `SQLITE_BUSY_SNAPSHOT` — a distinct string, not a `SQLITE_BUSY` prefix + * match. `BEGIN IMMEDIATE` avoids it on every writer we own; this entry is the belt for one that ever escapes + * (a future caller that opens its own DEFERRED transaction — each retry re-`BEGIN`s, so the snapshot + * refreshes). It is NOT a belt for a store method demoted to a `SAVEPOINT` inside an outer transaction: a + * retry there cannot refresh the connection's read snapshot while the OUTER transaction is still open, so + * every attempt fails identically. That case is unretryable by construction and must be fixed by the outer + * caller taking `BEGIN IMMEDIATE` itself. `SQLITE_BUSY_RECOVERY` is the same class and arrives by the same route: SQLite + * returns it when another process is actively rebuilding the WAL index — the first open after a crash — and + * `busy_timeout`'s handler loop exits with that code still set rather than downgrading it. Genuinely + * transient, so retrying is exactly right; without it a first-run-after-crash write fails loud. + * + * Matched exactly, never by prefix, so an unrelated future `SQLITE_BUSY_*` code is not silently swept in + * (#100). The DELIBERATE exclusions, recorded so the set is auditable rather than arbitrary: + * `SQLITE_LOCKED_SHAREDCACHE` cannot occur (this build sets `SQLITE_OMIT_SHARED_CACHE`) and + * `SQLITE_BUSY_TIMEOUT` requires `SQLITE_ENABLE_SETLK_TIMEOUT`, which is not defined — better-sqlite3 does + * not even map it, so were the build to change it would surface as `UNKNOWN_SQLITE_ERROR_773`, not silently. + */ +const RETRYABLE_CODES: ReadonlySet = new Set([ + 'SQLITE_BUSY', + 'SQLITE_LOCKED', + 'SQLITE_BUSY_SNAPSHOT', + 'SQLITE_BUSY_RECOVERY', +]); /** Default total attempts (the first try + up to 4 retries). */ const DEFAULT_MAX_ATTEMPTS = 5; /** Default linear-backoff base; the nth retry sleeps `base × n` ms. */ const DEFAULT_BASE_DELAY_MS = 25; -export interface BusyRetryOptions { +/** The attempt budget + backoff schedule — identical for both twins, so the policy has one home. */ +interface BusyRetryBudget { /** Total attempts INCLUDING the first (default {@link DEFAULT_MAX_ATTEMPTS}). Must be ≥ 1. */ readonly maxAttempts?: number; /** Linear-backoff base in ms (default {@link DEFAULT_BASE_DELAY_MS}); the nth retry sleeps `base × n`. */ readonly baseDelayMs?: number; +} + +export interface BusyRetryOptions extends BusyRetryBudget { /** Injectable synchronous sleep — tests pass a no-op/recorder so they never actually block. */ readonly sleep?: (ms: number) => void; } +export interface AsyncBusyRetryOptions extends BusyRetryBudget { + /** Injectable async sleep — tests pass a resolved-promise recorder so they never actually wait. */ + readonly sleep?: (ms: number) => Promise; +} + /** A `better-sqlite3` `SqliteError` carries a string `.code` (e.g. `SQLITE_BUSY`); match structurally. */ function isRetryableLockError(err: unknown): boolean { return ( @@ -65,25 +110,98 @@ function sleepSync(ms: number): void { } /** - * Run `fn`, retrying only on `SQLITE_BUSY`/`SQLITE_LOCKED` up to `maxAttempts`, with a deterministic linear - * backoff between attempts. Returns `fn`'s value on success; rethrows the original error on a non-lock fault - * or an exhausted budget (fail-loud). Synchronous — it wraps a synchronous `db.transaction(...)` call. + * Async sleep — the {@link sleepSync} twin for {@link withBusyRetryAsync}. A plain `setTimeout`, deliberately + * NOT `.unref()`d: an unref'd timer would let Node exit mid-backoff and abandon a write the caller is still + * awaiting, which is the silent data loss this module exists to prevent. The wait is bounded by the linear + * schedule (≤ 100 ms at the defaults), so holding the loop open for it costs nothing at shutdown. + */ +function sleepAsync(ms: number): Promise { + if (ms <= 0) return Promise.resolve(); + return new Promise((resolve) => { + setTimeout(resolve, ms); + }); +} + +/** + * The shared retry PREDICATE — one definition, so a change to what counts as retryable cannot apply to one + * twin and not the other. Returns a boolean rather than throwing: the loop bound stays visible IN the loop, so + * a mutation here degrades to a fail-loud rethrow, never to an infinite CPU-burning loop (the shape the + * previous `throwUnlessRetryable` had — breaking it hung the test worker to an OOM instead of failing). + */ +function shouldRetry(err: unknown, attempt: number, maxAttempts: number): boolean { + return attempt < maxAttempts && isRetryableLockError(err); +} + +/** Resolve the budget once, identically for both twins. Floors guard a stray `0`/negative: `maxAttempts` + * so it can never disable the first attempt, `baseDelayMs` so the async twin cannot spin the microtask + * queue without ever yielding a macrotask — which would defeat the twin's entire purpose. */ +function resolveBudget(options: BusyRetryBudget): { maxAttempts: number; baseDelayMs: number } { + return { + maxAttempts: Math.max(1, options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS), + baseDelayMs: Math.max(1, options.baseDelayMs ?? DEFAULT_BASE_DELAY_MS), + }; +} + +/** + * Run `fn`, retrying only on a {@link RETRYABLE_CODES} lock fault up to `maxAttempts`, with a deterministic + * linear backoff between attempts. Returns `fn`'s value on success; rethrows the original error on a non-lock + * fault or an exhausted budget (fail-loud). Synchronous — it wraps a synchronous `db.transaction(...)` call. */ export function withBusyRetry(fn: () => T, options: BusyRetryOptions = {}): T { - // Floor at 1 so a stray `0`/negative can never disable the first attempt (or spin) — always at least one try. - const maxAttempts = Math.max(1, options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS); - const baseDelayMs = options.baseDelayMs ?? DEFAULT_BASE_DELAY_MS; + const { maxAttempts, baseDelayMs } = resolveBudget(options); const sleep = options.sleep ?? sleepSync; - for (let attempt = 1; ; attempt += 1) { + for (let attempt = 1; attempt <= maxAttempts; attempt += 1) { try { return fn(); } catch (err) { // Fail loud: the last attempt, or any non-lock fault, rethrows the ORIGINAL error unchanged. - if (attempt >= maxAttempts || !isRetryableLockError(err)) { - throw err; - } + if (!shouldRetry(err, attempt, maxAttempts)) throw err; // Deterministic linear backoff (no jitter): let the contending writer commit before we re-take the lock. sleep(baseDelayMs * attempt); } } + // Unreachable: the final attempt either returns or rethrows above. Present so the loop bound is structural. + throw new Error('withBusyRetry: exhausted the attempt budget without returning or throwing'); +} + +/** + * The **async twin** of {@link withBusyRetry}, for the call sites whose caller is genuinely async — today + * `persistEvent` (run history). Identical budget, identical deterministic + * linear backoff, identical fail-loud exhaustion; the only difference is that the backoff **yields the event + * loop** instead of parking the thread on `Atomics.wait` (#226). + * + * Scope, stated plainly so this is not mistaken for more than it is: `better-sqlite3` is a **synchronous** + * driver and each attempt's `BEGIN IMMEDIATE` can itself block the thread for up to `busy_timeout` (5 s). The + * backoff is therefore the SMALL term — the sub-300 ms of a ~25 s worst case documented in + * [database-schema.md](../../../docs/reference/shared-core/database-schema.md#concurrency--transaction-behavior). + * This twin removes that small term where it is free to remove; it does not, and cannot, make the write path + * non-blocking. Converting the remaining twelve synchronous call sites (across five store interfaces) would not + * change that either — which is why they stay on {@link withBusyRetry} rather than growing an async API for + * no gain. Note the "small term" claim is precise about WHICH code: for `SQLITE_BUSY`/`SQLITE_LOCKED` — the + * codes that actually fire — an attempt blocks inside the driver for up to `busy_timeout` (~5.2 s measured), + * so the backoff is small. `SQLITE_BUSY_SNAPSHOT` returns IMMEDIATELY, so for that code the backoff is the + * whole cost; it cannot reach a synchronous call site today because every one of them is `IMMEDIATE`, + * single-statement, or read-only. + * + * One consequence the sync twin does not have: the `await` between attempts is an **observable yield**, so a + * sibling writer may commit during the backoff. That is precisely the outcome the backoff exists to allow — + * but it makes the re-runnability requirement on `fn` load-bearing rather than incidental. `fn` MUST be one + * self-contained, idempotent transaction, never a partially-applied sequence: at the moment of the yield our + * own transaction has already rolled back, so nothing of ours is in flight. + */ +export async function withBusyRetryAsync( + fn: () => T | Promise, + options: AsyncBusyRetryOptions = {}, +): Promise { + const { maxAttempts, baseDelayMs } = resolveBudget(options); + const sleep = options.sleep ?? sleepAsync; + for (let attempt = 1; attempt <= maxAttempts; attempt += 1) { + try { + return await fn(); + } catch (err) { + if (!shouldRetry(err, attempt, maxAttempts)) throw err; + await sleep(baseDelayMs * attempt); + } + } + throw new Error('withBusyRetryAsync: exhausted the attempt budget without returning or throwing'); } diff --git a/packages/db/src/run-history-store.test.ts b/packages/db/src/run-history-store.test.ts index 5ac81049..1af5a6d8 100644 --- a/packages/db/src/run-history-store.test.ts +++ b/packages/db/src/run-history-store.test.ts @@ -71,6 +71,32 @@ describe('createRunHistoryStore', () => { client.sqlite.close(); }); + it('persistEvent YIELDS the event loop between retries — it is on the async twin (#226)', async () => { + // The central behavioural change of #226 had no regression guard: reverting `persistEvent` to the + // synchronous `withBusyRetry` left the whole packages/db suite green. This is the assertion that dies. + const workflowId = await store.resolveWorkflowId('demo'); + const observed: string[] = []; + let calls = 0; + // Fail the first transaction with a lock fault so the retry path (and therefore the backoff) is entered. + vi.spyOn(client.db, 'transaction').mockImplementationOnce(() => { + calls += 1; + observed.push('attempt1'); + // Armed BEFORE the backoff: under `Atomics.wait` the thread is parked and this cannot fire until the + // whole synchronous retry has returned, so its position in `observed` is the proof. + setTimeout(() => observed.push('other-work'), 0); + throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + }); + + await store.persistEvent( + ev('run:started', 0, { workflowId, inputs: { n: 3 }, executionMode: 'local' }), + ); + + expect(calls).toBe(1); // the first attempt really did fail… + expect(observed).toEqual(['attempt1', 'other-work']); // …and the loop ran during the backoff + // The write still landed on the retry — a yielding backoff must not cost durability. + expect(client.db.select().from(runEvents).all()).toHaveLength(1); + }); + it('persistEvent opens an IMMEDIATE write transaction (2.5.I — guards against a DEFERRED regression)', async () => { const workflowId = await store.resolveWorkflowId('demo'); const txnSpy = vi.spyOn(client.db, 'transaction'); diff --git a/packages/db/src/run-history-store.ts b/packages/db/src/run-history-store.ts index 0fa2a967..d359f413 100644 --- a/packages/db/src/run-history-store.ts +++ b/packages/db/src/run-history-store.ts @@ -7,7 +7,7 @@ import { import { and, asc, desc, eq, getTableColumns, inArray, isNull, notInArray, sql } from 'drizzle-orm'; import type { Db } from './client.js'; -import { withBusyRetry } from './retry.js'; +import { withBusyRetryAsync } from './retry.js'; import { runCosts, runEvents, @@ -425,10 +425,11 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis return Promise.resolve(find() ?? id); }, - persistEvent: (event) => { - // Synchronous (better-sqlite3) but Promise-returning to honor the async RunStore port — a fault (bad + persistEvent: async (event) => { + // The DB work is synchronous (better-sqlite3) but this honors the async RunStore port — a fault (bad // event, UNIQUE(run_id, seq), FK, disk) becomes a REJECTED promise, never a synchronous throw, so the // engine's `await persistEvent(...)` (durability-first: ADR-0050 fatal posture) and any `.catch` see it. + // `async` guarantees that on its own: an async function can only ever reject, never throw synchronously. try { const parsed = RunEventSchema.parse(event); // validate on the way in (round-trip + envelope) const runId = parsed.runId; @@ -439,15 +440,22 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis const ts = isoToEpochMs(parsed.timestamp); // One IMMEDIATE transaction per event: the run_events append and its derived rows land atomically, so a // crash can never leave a derived row without its event (or vice-versa). `BEGIN IMMEDIATE` takes the - // write lock up front (never a DEFERRED read→write upgrade race), and `withBusyRetry` waits out residual + // write lock up front (never a DEFERRED read→write upgrade race), and the retry waits out residual // cross-process lock contention — fail-loud, so a swallowed write can never be silent data loss // (ADR-0050; the ADR-0064 amendment note — DB write-path concurrency). - withBusyRetry(() => + // + // The ASYNC twin (#226): this caller is already async, so the backoff between attempts yields the event + // loop instead of parking the thread on `Atomics.wait`. The yield is observable — a sibling branch's + // `persistEvent` may commit during our backoff — which is exactly what the backoff is for, and is safe + // because each event's fold is ONE self-contained IMMEDIATE transaction that has already rolled back + // before we sleep. Within a single branch the engine awaits these sequentially, so no event can + // overtake its own predecessor. + await withBusyRetryAsync(() => db.transaction(() => fold(parsed, runId, ts), { behavior: 'immediate' }), ); - return Promise.resolve(); } catch (error) { - return Promise.reject(error instanceof Error ? error : new Error(String(error))); + // Preserve the root cause (error-handling.md): never swallow it to rethrow a vaguer one. + throw error instanceof Error ? error : new Error(String(error), { cause: error }); } }, diff --git a/packages/db/src/session-store.test.ts b/packages/db/src/session-store.test.ts index f5a0f09a..17dfe146 100644 --- a/packages/db/src/session-store.test.ts +++ b/packages/db/src/session-store.test.ts @@ -603,6 +603,146 @@ describe('SessionStore — loadFull snapshot isolation (2.5.I)', () => { * the two writes into two transactions, since both would normally succeed and the sums would agree anyway. So the * mechanism is pinned too. */ +/** + * #228: the three single-statement session writers under real cross-process lock contention. Before the fix + * they went straight at the write lock with nothing but `busy_timeout` behind them — so once that 5 s wait + * expired the statement simply threw, and because the chat persister calls these from inside a `RunEventBus` + * subscriber, the throw became an unhandled rejection that killed the CLI after the reply was already shown. + * A held write lock plus `busy_timeout = 0` reproduces that deterministically, with no waiting and no threads. + */ +describe('SessionStore — writeTurn is atomic (#228)', () => { + // One test spies `db.transaction` to pin the lock MODE; the suite restores no mocks globally, and the fresh + // per-test client makes leakage impossible only for spies on THAT object — restore explicitly. + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('rolls the WHOLE turn back when a later message violates a constraint', () => { + store.createSession(makeSession()); + // Two rows at the SAME sequence number: the second trips the (session_id, sequence_number) UNIQUE index + // — a real constraint, not an injected fault, so this exercises the actual transaction. + expect(() => + store.writeTurn({ + messages: [{ message: makeMessage(0) }, { message: makeMessage(0, { id: 'msg-dup' }) }], + session: makeSession({ status: 'ended' }), + }), + ).toThrow(/UNIQUE/i); + // All of it, or none of it: the FIRST message must not have survived either. + expect(store.loadMessages('sess-1')).toHaveLength(0); + // …and the session flush in the same transaction is rolled back with it. + expect(store.loadSession('sess-1')?.status).toBe('active'); + }); + + it('opens an IMMEDIATE transaction — guards against a silent DEFERRED regression', () => { + // A mutation that dropped `{ behavior: 'immediate' }` left the whole suite GREEN, so atomicity was pinned + // but the LOCK MODE was not. `BEGIN IMMEDIATE` is what closes the read→write upgrade race (ADR-0064's + // 2.5.I convention); DEFERRED would still be atomic and still be wrong. Same spy technique `loadFull` + // already uses to guard its read transaction. + store.createSession(makeSession()); + const txn = vi.spyOn(client.db, 'transaction'); + store.writeTurn({ messages: [{ message: makeMessage(0) }], session: makeSession() }); + expect(txn).toHaveBeenCalledWith(expect.any(Function), { behavior: 'immediate' }); + }); + + it('commits messages and the session flush together on success', () => { + store.createSession(makeSession()); + store.writeTurn({ + messages: [{ message: makeMessage(0) }, { message: makeMessage(1, { role: 'assistant' }) }], + session: makeSession({ status: 'ended', title: 'done' }), + }); + expect(store.loadMessages('sess-1').map((m) => m.role)).toEqual(['user', 'assistant']); + expect(store.loadSession('sess-1')?.title).toBe('done'); + }); + + it('accepts an empty message list — a row-less flush is still one transaction', () => { + store.createSession(makeSession()); + store.writeTurn({ messages: [], session: makeSession({ status: 'ended' }) }); + expect(store.loadSession('sess-1')?.status).toBe('ended'); + }); + + it('never SETs total_cost_microcents — recordSessionCost stays its single writer (ADR-0070)', () => { + // writeTurn shares `mutableSessionColumns` with updateSession precisely so this invariant cannot be + // reintroduced on a second write path. + store.createSession(makeSession()); + store.recordSessionCost({ + id: 'c1', + sessionId: 'sess-1', + model: 'm', + inputTokens: 1, + outputTokens: 1, + costMicrocents: 500, + priced: true, + ts: 1, + }); + store.writeTurn({ messages: [], session: makeSession({ totalCostMicrocents: 0 }) }); + expect(store.loadSession('sess-1')?.totalCostMicrocents).toBe(500); // not clobbered back to 0 + }); +}); + +describe('SessionStore — the session writers survive lock contention (#228)', () => { + /** A `better-sqlite3`-shaped lock fault: an `Error` carrying the string `.code` the driver sets. */ + const busy = (): Error => Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + + /** + * Inject ONE lock fault into the next `db.()` call, then let the real implementation through. The + * throw lands inside the wrapped `fn`, which is all `withBusyRetry` cares about — so this pins "the wrapper + * is present on this writer" deterministically, in ~25 ms, without a second process or a real held lock. + * (`withBusyRetry`'s behaviour against REAL `SQLITE_BUSY` contention is covered in `retry.test.ts`.) + */ + const failOnce = (verb: 'insert' | 'update'): void => { + vi.spyOn(client.db, verb).mockImplementationOnce(() => { + throw busy(); + }); + }; + + // Defence in depth only: the file's global `beforeEach` builds a FRESH `client` per test, so a spy on + // `client.db` is attached to an object the next test discards — leakage is structurally impossible here, + // not merely prevented. Kept so the hook does not become load-bearing if that setup ever changes. + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('appendMessage retries a lock fault instead of throwing — the lost-turn case', () => { + store.createSession(makeSession()); + failOnce('insert'); + expect(() => store.appendMessage(makeMessage(0))).not.toThrow(); + expect(store.loadMessages('sess-1')).toHaveLength(1); // the row really landed, not just "didn't throw" + }); + + it('createSession retries a lock fault', () => { + failOnce('insert'); + expect(() => store.createSession(makeSession())).not.toThrow(); + expect(store.loadSession('sess-1')).toBeDefined(); + }); + + it('updateSession retries a lock fault', () => { + store.createSession(makeSession()); + failOnce('update'); + expect(() => store.updateSession(makeSession({ status: 'ended' }))).not.toThrow(); + expect(store.loadSession('sess-1')?.status).toBe('ended'); + }); + + it('still FAILS LOUD once the budget is exhausted — a dropped turn is never silent', () => { + // The point of the retry is to absorb a TRANSIENT lock, never to hide a persistent one: a swallowed write + // is silent data loss (ADR-0050). Beyond the budget the ORIGINAL driver error is rethrown, which is exactly + // what the bus's `onListenerError` sink then reports to the user. + store.createSession(makeSession()); + const persistent = busy(); + vi.spyOn(client.db, 'insert').mockImplementation(() => { + throw persistent; + }); + // `toThrow(err)` compares MESSAGES, which cannot tell the original from a same-message wrapper — and + // "the ORIGINAL error is rethrown" is exactly the claim. Catch and compare identity. + let caught: unknown; + try { + store.appendMessage(makeMessage(0)); + } catch (err) { + caught = err; + } + expect(caught).toBe(persistent); + }); +}); + describe('session_costs — the ADR-0070 reconciliation invariant', () => { const sumRows = (sessionId: string): number => store.loadSessionCosts(sessionId).reduce((n, r) => n + r.costMicrocents, 0); diff --git a/packages/db/src/session-store.ts b/packages/db/src/session-store.ts index 0798c621..5e78f671 100644 --- a/packages/db/src/session-store.ts +++ b/packages/db/src/session-store.ts @@ -114,6 +114,29 @@ export function fromAgentSessionRow(row: AgentSessionRow): AgentSessionRecord { return AgentSessionSchema.parse(candidate); } +/** + * The MUTABLE `agent_sessions` columns of a record — what an update may SET. Shared by `updateSession` and the + * atomic `writeTurn` so the two can never disagree about what a flush is allowed to touch. + * + * `created_at` is frozen at creation, so it (and the `id` WHERE-key) are dropped from the SET payload — + * an update overwrites only the mutable columns (status, title, context, exportedWorkflowPath, deletedAt, + * updatedAt, …), never the creation timestamp, regardless of what the caller passes. + * + * `total_cost_microcents` is ALSO dropped (ADR-0070 §2): it has exactly ONE writer, `recordSessionCost`, which + * bumps it ADDITIVELY in the same transaction as the `session_costs` row. It used to be SET blindly here from + * whatever cumulative the caller happened to hold — from four persister call sites and from `chat-export` — so + * any writer with a stale in-memory total (two `chat-resume` processes on one sessionId; a late flush landing + * after a cost write) would permanently break `SUM(session_costs) == total_cost_microcents`. A single owner is + * what makes the invariant a property of the code rather than a hope about call ordering. + */ +function mutableSessionColumns(record: AgentSessionRecord): Partial { + const mutable: Partial = { ...toAgentSessionRow(record) }; + delete mutable.id; + delete mutable.createdAt; + delete mutable.totalCostMicrocents; + return mutable; +} + /** Map a validated {@link SessionMessage} (+ optional denormalized {@link SessionMessageMeta}) to a row. */ export function toSessionMessageRow( message: SessionMessage, @@ -222,6 +245,17 @@ export interface SessionCostRow { readonly isLegacy: boolean; } +/** One atomic turn write: the messages to append (in order) plus the session row to flush. */ +export interface SessionTurnWrite { + /** Transcript rows to append, in ascending `sequenceNumber` order. May be empty (a row-less flush). */ + readonly messages: readonly { + readonly message: SessionMessage; + readonly meta?: SessionMessageMeta; + }[]; + /** The session row to flush in the SAME transaction — `undefined` to append without touching it. */ + readonly session?: AgentSessionRecord; +} + export interface SessionStore { /** Insert a new `agent_sessions` row. */ createSession: (record: AgentSessionRecord) => void; @@ -270,6 +304,22 @@ export interface SessionStore { }; /** Append a transcript message (the caller assigns the next monotonic `sequenceNumber`). */ appendMessage: (message: SessionMessage, meta?: SessionMessageMeta) => void; + /** + * Append a turn's messages AND flush the session row in **one** `BEGIN IMMEDIATE` transaction — all of it, + * or none of it (#228). + * + * The three-statement, auto-committed alternative is not merely slower, it is unsound: a failure between + * the `user` append and the `assistant` append leaves an unanswered `user` row in the durable transcript. + * `resumableMessageSequences` rolls back only a TRAILING one, so once the session survives the failure and + * keeps going, that orphan is buried mid-transcript — and a resume then replays two consecutive `user` + * messages, which a provider rejects. Atomicity is what makes "the durable transcript mirrors the engine's" + * true rather than probable. + * + * This also discharges the per-turn transaction follow-up named in `database-schema.md` + * §"Concurrency & transaction behavior", and cuts the worst-case contended block roughly threefold by + * collapsing three retryable statements into one. + */ + writeTurn: (turn: SessionTurnWrite) => void; /** Load a session's full transcript in `sequenceNumber` order. */ loadMessages: (sessionId: string) => SessionMessage[]; /** Load a session and its ordered transcript — the resume entry point (`undefined` if the session is absent). */ @@ -354,25 +404,45 @@ export function createSessionStore(db: Db): SessionStore { })); return { + // The three single-statement session writers below go through `withBusyRetry` (#228). They are NOT wrapped + // in a transaction — a lone INSERT/UPDATE already takes the write lock immediately, so `BEGIN IMMEDIATE` + // would buy nothing, which is why `database-schema.md` exempts single-statement writes from it. The retry + // is the orthogonal half: `busy_timeout` waits 5 s and then *fails*, and for these three writers a failure + // is a lost chat turn plus — before this — a dead process, because the chat persister calls them from + // inside a `RunEventBus` subscriber whose throw became an unhandled rejection. createSession: (record) => { - db.insert(agentSessions).values(toAgentSessionRow(record)).run(); + withBusyRetry(() => db.insert(agentSessions).values(toAgentSessionRow(record)).run()); }, + writeTurn: ({ messages, session }) => { + // ONE IMMEDIATE transaction for the whole turn. `BEGIN IMMEDIATE` takes the write lock up front (never + // a DEFERRED read→write upgrade race) and `withBusyRetry` waits out residual cross-process contention — + // the ADR-0064 §2.5.I convention, now covering the transcript the way it already covered the cost row. + withBusyRetry(() => + db.transaction( + (tx) => { + for (const { message, meta } of messages) { + tx.insert(sessionMessages).values(toSessionMessageRow(message, meta)).run(); + } + if (session !== undefined) { + tx.update(agentSessions) + .set(mutableSessionColumns(session)) + .where(eq(agentSessions.id, session.id)) + .run(); + } + }, + { behavior: 'immediate' }, + ), + ); + }, + updateSession: (record) => { - // `created_at` is frozen at creation, so it (and the `id` WHERE-key) are dropped from the SET payload — - // an update overwrites only the mutable columns (status, title, context, exportedWorkflowPath, deletedAt, - // updatedAt, …), never the creation timestamp, regardless of what the caller passes. - // - // `total_cost_microcents` is ALSO dropped (ADR-0070 §2): it has exactly ONE writer, `recordSessionCost`, which - // bumps it ADDITIVELY in the same transaction as the `session_costs` row. It used to be SET blindly here from - // whatever cumulative the caller happened to hold — from four persister call sites and from `chat-export` — so - // any writer with a stale in-memory total (two `chat-resume` processes on one sessionId; a late flush landing - // after a cost write) would permanently break `SUM(session_costs) == total_cost_microcents`. A single owner is - // what makes the invariant a property of the code rather than a hope about call ordering. - const mutable: Partial = { ...toAgentSessionRow(record) }; - delete mutable.id; - delete mutable.createdAt; - delete mutable.totalCostMicrocents; - db.update(agentSessions).set(mutable).where(eq(agentSessions.id, record.id)).run(); + withBusyRetry(() => + db + .update(agentSessions) + .set(mutableSessionColumns(record)) + .where(eq(agentSessions.id, record.id)) + .run(), + ); }, recordSessionCost: (entry) => { @@ -443,7 +513,9 @@ export function createSessionStore(db: Db): SessionStore { loadSession, listSessions, appendMessage: (message, meta) => { - db.insert(sessionMessages).values(toSessionMessageRow(message, meta)).run(); + withBusyRetry(() => + db.insert(sessionMessages).values(toSessionMessageRow(message, meta)).run(), + ); }, loadMessages, loadFull: (sessionId) => diff --git a/packages/llm/src/adapters/anthropic.test.ts b/packages/llm/src/adapters/anthropic.test.ts index a63cb7e8..7f6c48f9 100644 --- a/packages/llm/src/adapters/anthropic.test.ts +++ b/packages/llm/src/adapters/anthropic.test.ts @@ -935,6 +935,73 @@ describe('AnthropicAdapter — stream edge cases', () => { const sse = (body: string): Response => new Response(body, { status: 200, headers: { 'content-type': 'text/event-stream' } }); + it("carries the provider's Retry-After from a real SDK 429 into the LlmError (#279)", async () => { + // The adapter→chain wire, which nothing pinned: severing `readRetryAfter` in either adapter left all 718 + // llm tests green even though the feature works in production. A real 429 WITH the header is the only + // thing that proves the SDK's header container is read rather than the shape a hand-built error happens + // to have. + const adapter = createAnthropicAdapter({ + fetch: () => + Promise.resolve( + new Response( + JSON.stringify({ type: 'error', error: { type: 'rate_limit_error', message: 'rl' } }), + { + status: 429, + headers: { 'content-type': 'application/json', 'retry-after': '3' }, + }, + ), + ), + }); + const chunks = await collect(adapter.stream(REQ, 'k')); + const first = chunks[0]; + expect(first?.type).toBe('error'); + if (first?.type === 'error') { + expect(first.error.kind).toBe('rate_limit'); + expect(first.error.retryAfterMs).toBe(3_000); + } + }); + + it('leaves retryAfterMs absent when the provider sent no header (#279)', async () => { + // Absent must mean "no instruction", never 0 — the chain falls back to its own curve on that distinction. + const adapter = createAnthropicAdapter({ + fetch: () => + Promise.resolve( + new Response( + JSON.stringify({ type: 'error', error: { type: 'rate_limit_error', message: 'rl' } }), + { status: 429, headers: { 'content-type': 'application/json' } }, + ), + ), + }); + const chunks = await collect(adapter.stream(REQ, 'k')); + const first = chunks[0]; + if (first?.type === 'error') { + expect(first.error.retryAfterMs).toBeUndefined(); + } + }); + + it('does NOT retry a 429 inside the SDK — the chain owns retry policy (#276)', async () => { + // Deliberately WITHOUT `maxRetries`, which is the whole point: every other test in this file passes + // `maxRetries: 0` explicitly, so none of them exercised what PRODUCTION gets. Left to the SDK default the + // client retried twice more in here, so `FallbackChain` never saw the first 429 — failover was delayed by + // the SDK's own backoff and a rate limit looked like a slow call instead of a reason to move provider. + let calls = 0; + const adapter = createAnthropicAdapter({ + fetch: () => { + calls += 1; + return Promise.resolve( + new Response( + JSON.stringify({ type: 'error', error: { type: 'rate_limit_error', message: 'rl' } }), + { status: 429, headers: { 'content-type': 'application/json' } }, + ), + ); + }, + }); + const chunks = await collect(adapter.stream(REQ, 'k')); + expect(chunks[0]?.type).toBe('error'); + // EXACTLY one network attempt. Restoring the SDK default makes this 3. + expect(calls).toBe(1); + }); + it('yields a single error chunk when the stream fails to start (429)', async () => { const adapter = createAnthropicAdapter({ fetch: () => diff --git a/packages/llm/src/adapters/anthropic.ts b/packages/llm/src/adapters/anthropic.ts index 6baac389..ef4d9ac6 100644 --- a/packages/llm/src/adapters/anthropic.ts +++ b/packages/llm/src/adapters/anthropic.ts @@ -39,6 +39,7 @@ import { isRecord, positiveModelInt, toModelListing, + readRetryAfter, } from './shared.js'; /** @@ -217,6 +218,8 @@ function mapAnthropicApiError(err: { status?: unknown; type?: unknown; message: string; + /** Structurally typed, so no vendor header type crosses the seam — only `.get` is used (#279). */ + headers?: { readonly get?: unknown } | undefined; }): LlmError { const status = typeof err.status === 'number' ? err.status : undefined; const code = typeof err.type === 'string' && err.type.length > 0 ? err.type : undefined; @@ -225,12 +228,16 @@ function mapAnthropicApiError(err: { const kind = (code === undefined ? undefined : kindFromErrorType(code)) ?? (status === undefined ? 'unknown' : kindFromHttpStatus(status)); + // #279: carry the provider's OWN requested wait when it sent one. A mid-stream `error` event has no + // headers, so this is genuinely optional — absent means "no instruction", never "wait zero". + const retryAfterMs = readRetryAfter(err.headers); return makeLlmError({ provider: PROVIDER, kind, message: err.message, ...(status === undefined ? {} : { status }), ...(code === undefined ? {} : { code }), + ...(retryAfterMs === undefined ? {} : { retryAfterMs }), }); } @@ -622,7 +629,18 @@ function buildRequestOptions(req: LlmRequest): { signal?: AbortSignal } { export interface AnthropicAdapterDeps { /** Inject a `fetch` (the replayer/recorder) in place of the network. */ readonly fetch?: (input: string | URL | Request, init?: RequestInit) => Promise; - /** Override the SDK retry count (the replayer sets 0 for deterministic, fast tests). */ + /** + * Override the SDK's own retry count. **Defaults to `0`** — the vendor SDK's built-in retry is deliberately + * OFF in production, because `FallbackChain` owns the retry/fallback policy (ADR-0011: "the runner — not the + * adapter — owns the retry/fallback policy"). Left at the SDK default it silently retried a 429/5xx two more + * times INSIDE the adapter: the chain never saw the first failure, so failover was delayed by the SDK's own + * backoff, the node retry budget was double-counted, and a rate limit looked like a slow call rather than a + * reason to move to the next provider (#276). Tests that want the SDK's behaviour can still set it. + * + * This governs the **chain-governed** calls (`generate` / `stream`) only. The surfaces `FallbackChain` does + * NOT sit above — live model discovery and the async media-job poll — keep a small SDK retry, because for + * them there is no runner to own the policy — and they run without SDK retry too; see the note below. + */ readonly maxRetries?: number; } @@ -836,12 +854,25 @@ async function* streamChunks(client: Anthropic, req: LlmRequest): AsyncIterable< } /** Build an Anthropic `LlmProvider`. Exposed as `anthropicAdapter`; the factory enables DI for 1.F. */ +/** + * **The off-chain paths deliberately run with NO SDK retry either — and that is a knowing residual, not an + * oversight.** `listModels` and the media-job poll are not governed by `FallbackChain`, so nothing retries a + * transient fault on them: one 500 fails a `/models refresh`, and one 429 on a status poll settles an + * already-billed media node. Giving them the SDK's retry (an earlier attempt at this) was worse: that sleep is + * neither abort-aware nor ceiling-bounded and the SDK parses `Retry-After` itself, so a cancel was ignored for + * the whole wait and a hostile `retry-after-ms` could park the call indefinitely — past both + * `boundedListModels`'s timeout and the seam's `RETRY_AFTER_CEILING_MS`. Resilience here needs a retry WE own, + * abort-aware and ceiling-bounded; until that lands, no retry is the safer of the two failure modes. + */ export function createAnthropicAdapter(deps: AnthropicAdapterDeps = {}): LlmProvider { - const createClient = (key: string): Anthropic => + const createClient = (key: string, maxRetries = deps.maxRetries ?? 0): Anthropic => new Anthropic({ apiKey: key, ...(deps.fetch === undefined ? {} : { fetch: deps.fetch }), - ...(deps.maxRetries === undefined ? {} : { maxRetries: deps.maxRetries }), + // ALWAYS passed, never conditionally: an absent option means the SDK's own default (2), which is + // exactly the pre-emption #276 is about. Explicit beats implicit. Floored, because a negative value + // makes the SDK's retry loop unbounded (`retriesRemaining - 1` stays truthy at -1). + maxRetries: Math.max(0, Math.trunc(maxRetries)), }); return { diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index a8444769..a7b5d3ce 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -715,6 +715,19 @@ export function buildGeminiRequest(req: LlmRequest): GeminiRequest { // --- The default transport (the only place the SDK is used) ----------------------------------- /* v8 ignore start -- the live-only real-SDK transport; the fold it feeds is fully covered offline via an injected GeminiTransport */ + +/** + * **Gemini needs no `maxRetries: 0` (#276) — verified, not assumed.** `@google/genai`'s `ApiClient.apiCall` + * issues a bare `fetch` unless `clientOptions.httpOptions.retryOptions` is supplied at construction, and this + * adapter never supplies `httpOptions`. So `FallbackChain` is already the sole retry authority on every path + * used here (`generateContent`, `generateContentStream`, `models.list`, `generateImages`, `generateVideos`, + * `operations.getVideosOperation`). + * + * Two traps for whoever edits this next. **Never pass `httpOptions.retryOptions`** — the SDK's defaults behind + * it are 5 attempts over 408/429/500/502/503/504. And `client.interactions` / `.webhooks` / `.agents` route + * through a different, Stainless-style client that DOES default to `maxRetries: 2`; moving any call to those + * surfaces silently reopens #276 and would need an explicit 0. + */ const sdkTransport: GeminiTransport = { async generate(request: GeminiRequest, key: string): Promise { const client = new GoogleGenAI({ apiKey: key }); diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 9aa712ac..f361d0c5 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -1668,6 +1668,51 @@ describe('OpenAI-compatible adapter — stream edge cases', () => { messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }], }; + it("carries the provider's Retry-After from a real SDK 429 into the LlmError (#279)", async () => { + // Same unpinned wire as the Anthropic adapter: a real 429 carrying the header is the only thing that + // proves the SDK's own header container is what gets read. + const adapter = createOpenAiAdapter({ + fetch: () => + Promise.resolve( + new Response(JSON.stringify({ error: { message: 'rl', type: 'rate_limit_exceeded' } }), { + status: 429, + headers: { 'content-type': 'application/json', 'retry-after-ms': '2500' }, + }), + ), + }); + const chunks = await collect(adapter.stream(REQ, 'k')); + const first = chunks[0]; + expect(first?.type).toBe('error'); + if (first?.type === 'error') { + expect(first.error.kind).toBe('rate_limit'); + // `retry-after-ms` wins over `retry-after` and is read verbatim in milliseconds. + expect(first.error.retryAfterMs).toBe(2_500); + } + }); + + it('does NOT retry a 429 inside the SDK — the chain owns retry policy (#276)', async () => { + // Deliberately WITHOUT `maxRetries`: every other test here passes 0 explicitly, so none of them exercised + // what PRODUCTION gets. Left to the SDK default the client retried twice more in here, so `FallbackChain` + // never saw the first 429 — failover was delayed by the SDK's own backoff (ADR-0011: the runner, not the + // adapter, owns retry policy). + let calls = 0; + const adapter = createOpenAiAdapter({ + fetch: () => { + calls += 1; + return Promise.resolve( + new Response(JSON.stringify({ error: { message: 'rl', type: 'rate_limit_exceeded' } }), { + status: 429, + headers: { 'content-type': 'application/json' }, + }), + ); + }, + }); + const chunks = await collect(adapter.stream(REQ, 'k')); + expect(chunks[0]?.type).toBe('error'); + // EXACTLY one network attempt. Restoring the SDK default makes this 3. + expect(calls).toBe(1); + }); + it('yields a single error chunk when the stream fails to start (429)', async () => { const adapter = createOpenAiAdapter({ fetch: () => diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index 0a5dc8c9..0a88ba9c 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -56,6 +56,7 @@ import { isRecord, redactKey, toModelListing, + readRetryAfter, } from './shared.js'; /** @@ -371,6 +372,11 @@ function mapOpenAiApiError( ): LlmError { const status = typeof err.status === 'number' ? err.status : undefined; const code = firstNonEmptyString(err.code, err.type); + // #279: the provider's own requested wait, when it sent one. A value object (not a thrown APIError) has no + // headers, so this is genuinely optional — absent means "no instruction", never "wait zero". + const retryAfterMs = readRetryAfter( + (err as { headers?: { readonly get?: unknown } | undefined }).headers, + ); // A content-policy block normalizes to content_filter regardless of HTTP status (a moderation 400 would // otherwise map to bad_request) — the wired image-gen path then delivers the documented taxonomy. let kind: LlmErrorKind; @@ -385,6 +391,7 @@ function mapOpenAiApiError( provider, kind, message: err.message, + ...(retryAfterMs === undefined ? {} : { retryAfterMs }), ...(status === undefined ? {} : { status }), ...(code === undefined ? {} : { code }), }); @@ -434,6 +441,9 @@ export function openaiErrorToLlmError(err: unknown, provider: ProviderId, key?: kind: base.kind, message: redactKey(base.message, key), ...(base.status === undefined ? {} : { status: base.status }), + // Carried through the re-wrap: dropping it here would silently discard the provider's Retry-After on + // exactly the custom-endpoint path that most needs a polite retry (#279). + ...(base.retryAfterMs === undefined ? {} : { retryAfterMs: base.retryAfterMs }), ...(base.code === undefined ? {} : { code: redactKey(base.code, key) }), }); } @@ -1308,11 +1318,32 @@ export interface OpenAiAdapterDeps { readonly baseURL?: string; /** Inject a `fetch` (the replayer/recorder) in place of the network. */ readonly fetch?: (input: string | URL | Request, init?: RequestInit) => Promise; - /** Override the SDK retry count (the replayer sets 0 for deterministic, fast tests). */ + /** + * Override the SDK's own retry count. **Defaults to `0`** — the vendor SDK's built-in retry is deliberately + * OFF in production, because `FallbackChain` owns the retry/fallback policy (ADR-0011: "the runner — not the + * adapter — owns the retry/fallback policy"). Left at the SDK default it silently retried a 429/5xx two more + * times INSIDE the adapter: the chain never saw the first failure, so failover was delayed by the SDK's own + * backoff, the node retry budget was double-counted, and a rate limit looked like a slow call rather than a + * reason to move to the next provider (#276). Tests that want the SDK's behaviour can still set it. + * + * This governs the **chain-governed** calls (`generate` / `stream`) only. The surfaces `FallbackChain` does + * NOT sit above — live model discovery and the async media-job poll — keep a small SDK retry, because for + * them there is no runner to own the policy — and they run without SDK retry too; see the note below. + */ readonly maxRetries?: number; } /** Build an OpenAI-compatible `LlmProvider`. Exposed as `openaiAdapter` / `deepseekAdapter`. */ +/** + * **The off-chain paths deliberately run with NO SDK retry either — and that is a knowing residual, not an + * oversight.** `listModels` and the media-job poll are not governed by `FallbackChain`, so nothing retries a + * transient fault on them: one 500 fails a `/models refresh`, and one 429 on a status poll settles an + * already-billed media node. Giving them the SDK's retry (an earlier attempt at this) was worse: that sleep is + * neither abort-aware nor ceiling-bounded and the SDK parses `Retry-After` itself, so a cancel was ignored for + * the whole wait and a hostile `retry-after-ms` could park the call indefinitely — past both + * `boundedListModels`'s timeout and the seam's `RETRY_AFTER_CEILING_MS`. Resilience here needs a retry WE own, + * abort-aware and ceiling-bounded; until that lands, no retry is the safer of the two failure modes. + */ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { const providerId: OpenAiProviderId = deps.providerId ?? 'openai'; const supports = providerId === 'deepseek' ? DEEPSEEK_SUPPORTS : OPENAI_SUPPORTS; @@ -1334,12 +1365,15 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { const endpoint: EndpointKind = endpointKindFor(providerId, deps.baseURL); // Host-qualified so two custom gateways never share learned param rejections (see `endpointScope`). const rejectionScope = endpointScope(endpoint, deps.baseURL); - const createClient = (key: string): OpenAI => + const createClient = (key: string, maxRetries = deps.maxRetries ?? 0): OpenAI => new OpenAI({ apiKey: key, ...(baseURL === undefined ? {} : { baseURL }), ...(deps.fetch === undefined ? {} : { fetch: deps.fetch }), - ...(deps.maxRetries === undefined ? {} : { maxRetries: deps.maxRetries }), + // ALWAYS passed, never conditionally: an absent option means the SDK's own default (2), which is + // exactly the pre-emption #276 is about. Explicit beats implicit. Floored, because a negative value + // makes the SDK's retry loop unbounded (`retriesRemaining - 1` stays truthy at -1). + maxRetries: Math.max(0, Math.trunc(maxRetries)), }); return { @@ -1498,6 +1532,14 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { }), }; } + // Deliberately maxRetries 0 (the default), NOT the off-chain budget. Giving the poll SDK retry looked + // right — nothing else retries it — but it opened a worse hole than it closed: the SDK's retry sleep is + // neither abort-aware nor ceiling-bounded, and it parses `Retry-After` ITSELF, so the seam's + // RETRY_AFTER_CEILING_MS never runs. A hostile or broken gateway (`base_url` is user-configurable, + // ADR-0065) answering one status poll with `retry-after-ms: 999999999` parked the poll for ~11.6 days, + // ignoring the user's cancel and holding a run slot and an event-loop timer. Resilience here has to be a + // retry WE own — abort-aware and ceiling-bounded — which is the follow-up; an unbounded one is worse + // than none. return pollMediaJobSora(createClient(key), jobId, providerId, signal, key); }, // ADR-0062 context-compaction seam — the shared defaults (covers both OpenAI and DeepSeek via this one diff --git a/packages/llm/src/adapters/shared.ts b/packages/llm/src/adapters/shared.ts index 6d2236a2..f7c2f6c5 100644 --- a/packages/llm/src/adapters/shared.ts +++ b/packages/llm/src/adapters/shared.ts @@ -2,7 +2,7 @@ import type { AbortSignalLike } from '@relavium/shared'; import { mediaSupportReason } from '../capabilities.js'; import { UnsupportedCapabilityError } from '../errors.js'; -import { LlmProviderError, makeLlmError } from '../llm-error.js'; +import { LlmProviderError, makeLlmError, retryAfterMsFromHeaders } from '../llm-error.js'; import { catalogModel } from '../catalog/lookup.js'; import { LlmMessageSchema, ModelListingSchema } from '../types.js'; import type { @@ -347,3 +347,22 @@ export async function boundedListModels(params: { } } } + +/** + * Read a provider's requested wait off an SDK error's headers, without letting a vendor header type cross the + * seam (#279). Structurally typed and fully defensive: an error with no headers, a non-callable `get`, or a + * `get` that throws all yield `undefined` — i.e. "the provider said nothing", so the chain keeps its own + * backoff. Both Stainless SDKs expose a `Headers`-like object here; a mid-stream error event exposes none. + */ +export function readRetryAfter( + headers: { readonly get?: unknown } | undefined, +): number | undefined { + if (headers === undefined || typeof headers.get !== 'function') return undefined; + const get = headers.get.bind(headers) as (name: string) => string | null | undefined; + try { + return retryAfterMsFromHeaders({ get }); + } catch { + // A hostile/exotic header container must never break error CLASSIFICATION — the error still surfaces. + return undefined; + } +} diff --git a/packages/llm/src/conformance/anthropic.conformance.test.ts b/packages/llm/src/conformance/anthropic.conformance.test.ts index 15a97045..8dcb0abc 100644 --- a/packages/llm/src/conformance/anthropic.conformance.test.ts +++ b/packages/llm/src/conformance/anthropic.conformance.test.ts @@ -9,7 +9,7 @@ import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; // the adapter takes a `fetch` override, so the SDK stays inside src/adapters/* (the seam fence). // `replayFor` serves one body (one-shot scenarios) or a sequence (the multi-turn tool loop). const makeReplayAdapter: MakeReplayAdapter = (recorded) => - createAnthropicAdapter({ fetch: replayFor(recorded), maxRetries: 0 }); + createAnthropicAdapter({ fetch: replayFor(recorded) }); defineConformanceSuite('anthropic', makeReplayAdapter, ANTHROPIC_FIXTURES); diff --git a/packages/llm/src/conformance/deepseek.conformance.test.ts b/packages/llm/src/conformance/deepseek.conformance.test.ts index 8f67da5c..e13d2540 100644 --- a/packages/llm/src/conformance/deepseek.conformance.test.ts +++ b/packages/llm/src/conformance/deepseek.conformance.test.ts @@ -9,7 +9,7 @@ import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; // distinct provider id + cache field. The fetch override keeps the SDK inside src/adapters/*. `replayFor` // serves one body (one-shot scenarios) or a sequence (the multi-turn tool loop). const makeReplayAdapter: MakeReplayAdapter = (recorded) => - createOpenAiAdapter({ providerId: 'deepseek', fetch: replayFor(recorded), maxRetries: 0 }); + createOpenAiAdapter({ providerId: 'deepseek', fetch: replayFor(recorded) }); defineConformanceSuite('deepseek', makeReplayAdapter, DEEPSEEK_FIXTURES); diff --git a/packages/llm/src/conformance/effort.conformance.test.ts b/packages/llm/src/conformance/effort.conformance.test.ts index a49465fe..a4f6883c 100644 --- a/packages/llm/src/conformance/effort.conformance.test.ts +++ b/packages/llm/src/conformance/effort.conformance.test.ts @@ -99,7 +99,7 @@ describe('anthropic — effort conformance (live, nightly): the catalog does not it.skipIf(anthropicKey === '' || tiers.length === 0)( `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, async () => { - const adapter = createAnthropicAdapter({ maxRetries: 0 }); + const adapter = createAnthropicAdapter(); for (const tier of probeTiers) { await assertTierAccepted(adapter, model, tier, anthropicKey); } @@ -115,7 +115,7 @@ describe('openai — effort conformance (live, nightly): the catalog does not li it.skipIf(openaiKey === '' || tiers.length === 0)( `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, async () => { - const adapter = createOpenAiAdapter({ providerId: 'openai', maxRetries: 0 }); + const adapter = createOpenAiAdapter({ providerId: 'openai' }); for (const tier of probeTiers) { await assertTierAccepted(adapter, model, tier, openaiKey); } @@ -147,7 +147,7 @@ describe('deepseek — effort conformance (live, nightly): the catalog does not it.skipIf(deepseekKey === '' || tiers.length === 0)( `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, async () => { - const adapter = createOpenAiAdapter({ providerId: 'deepseek', maxRetries: 0 }); + const adapter = createOpenAiAdapter({ providerId: 'deepseek' }); for (const tier of probeTiers) { await assertTierAccepted(adapter, model, tier, deepseekKey); } diff --git a/packages/llm/src/conformance/openai.conformance.test.ts b/packages/llm/src/conformance/openai.conformance.test.ts index 8d901dfa..f13df934 100644 --- a/packages/llm/src/conformance/openai.conformance.test.ts +++ b/packages/llm/src/conformance/openai.conformance.test.ts @@ -9,7 +9,7 @@ import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; // takes a `fetch` override, so the SDK stays inside src/adapters/* (the seam fence). `replayFor` serves // one body (one-shot scenarios) or a sequence (the multi-turn tool loop). const makeReplayAdapter: MakeReplayAdapter = (recorded) => - createOpenAiAdapter({ providerId: 'openai', fetch: replayFor(recorded), maxRetries: 0 }); + createOpenAiAdapter({ providerId: 'openai', fetch: replayFor(recorded) }); defineConformanceSuite('openai', makeReplayAdapter, OPENAI_FIXTURES); diff --git a/packages/llm/src/cost-tracker.test.ts b/packages/llm/src/cost-tracker.test.ts index f532e044..b1a9a104 100644 --- a/packages/llm/src/cost-tracker.test.ts +++ b/packages/llm/src/cost-tracker.test.ts @@ -295,3 +295,41 @@ describe("the priced catalog's invariants (the values seeded into model_catalog) } }); }); + +describe('CostTracker.record — usage bounds at the money call site (#198)', () => { + const good = { inputTokens: 100, outputTokens: 50 }; + + it.each([ + ['NaN inputTokens', { ...good, inputTokens: Number.NaN }], + ['Infinity outputTokens', { ...good, outputTokens: Number.POSITIVE_INFINITY }], + ['negative inputTokens', { ...good, inputTokens: -1 }], + ['a fractional count', { ...good, outputTokens: 1.5 }], + ['beyond the safe-integer range', { ...good, inputTokens: 2 ** 53 }], + ['a bad cacheReadTokens', { ...good, cacheReadTokens: Number.NaN }], + ])('rejects %s rather than folding it into the cumulative total', (_label, usage) => { + const tracker = new CostTracker(); + expect(() => tracker.record('claude-opus-4-8', usage)).toThrow(/non-accountable/); + // The total must be untouched — a rejected value that still moved the running figure would be worse + // than no check at all. + expect(tracker.cumulativeCostMicrocents).toBe(0); + }); + + it('still accepts a legitimate zero and an absent optional count', () => { + const tracker = new CostTracker(); + expect(() => + tracker.record('claude-opus-4-8', { inputTokens: 0, outputTokens: 0 }), + ).not.toThrow(); + }); + + it('NaN would otherwise poison the cap permanently — the reason this is fail-loud', () => { + // Documents the failure mode: NaN is absorbing, so once folded in, every later `cumulative > cap` + // comparison is false and the cost cap silently stops being a cap. + const tracker = new CostTracker(); + expect(() => + tracker.record('claude-opus-4-8', { inputTokens: Number.NaN, outputTokens: 1 }), + ).toThrow(); + tracker.record('claude-opus-4-8', good); + expect(Number.isFinite(tracker.cumulativeCostMicrocents)).toBe(true); + expect(tracker.cumulativeCostMicrocents).toBeGreaterThan(0); + }); +}); diff --git a/packages/llm/src/cost-tracker.ts b/packages/llm/src/cost-tracker.ts index dd924374..8b0329dd 100644 --- a/packages/llm/src/cost-tracker.ts +++ b/packages/llm/src/cost-tracker.ts @@ -120,6 +120,30 @@ export function worstCaseRates(p: ModelPricing): Rates { * this (Anthropic's `input_tokens` is already net; the OpenAI-compatible adapter subtracts the * cached subset). So each token class is billed exactly once. */ +/** + * Reject a `Usage` that cannot be accounted (#198). Every token/unit count must be a finite, non-negative, + * safe integer — the same shape `UsageSchema` pins at the seam, re-asserted at the arithmetic. + * + * `Number.isSafeInteger` rather than `>= 0`: beyond 2^53 integer arithmetic stops being exact, so a cumulative + * total built from such a value is already wrong before any cap compares against it. + */ +function assertAccountableUsage(modelId: string, usage: Usage): void { + const counts: readonly (readonly [string, number | undefined])[] = [ + ['inputTokens', usage.inputTokens], + ['outputTokens', usage.outputTokens], + ['cacheReadTokens', usage.cacheReadTokens], + ['cacheWriteTokens', usage.cacheWriteTokens], + ]; + for (const [field, value] of counts) { + if (value === undefined) continue; + if (!Number.isSafeInteger(value) || value < 0) { + throw new TypeError( + `cost accounting for '${modelId}' received a non-accountable ${field}: expected a non-negative safe integer`, + ); + } + } +} + export function cost(modelId: string, usage: Usage, overlay?: PricingOverlay): number { const p = priceModel(modelId, overlay); const cacheReadTokens = usage.cacheReadTokens ?? 0; @@ -202,8 +226,18 @@ export class CostTracker { this.#overlay = overlay; } - /** Price one attempt's usage and fold it into the running total. */ + /** + * Price one attempt's usage and fold it into the running total. + * + * Validates the usage FIRST (#198). This is the money-accounting call site: a `NaN`, negative, or + * non-finite token count coming off an adapter would otherwise flow straight into the cumulative total and + * poison it permanently — `NaN` is absorbing, so every later comparison against the cost cap silently + * becomes false and the cap stops being a cap. The schema at the seam boundary is the first line of + * defence, but a tracker is also handed to custom hosts and callers, so the invariant is asserted here too, + * where the arithmetic actually happens. Fail loud: a bad number is a defect, never something to round away. + */ record(modelId: string, usage: Usage): CostUpdate { + assertAccountableUsage(modelId, usage); const costMicrocents = cost(modelId, usage, this.#overlay); this.#cumulativeMicrocents += costMicrocents; return { diff --git a/packages/llm/src/fallback-chain.test.ts b/packages/llm/src/fallback-chain.test.ts index 40078ed7..9b69e17b 100644 --- a/packages/llm/src/fallback-chain.test.ts +++ b/packages/llm/src/fallback-chain.test.ts @@ -10,6 +10,7 @@ import { type FallbackChainOptions, type FallbackPlanEntry, } from './fallback-chain.js'; +import { UnknownModelError } from './errors.js'; import { LlmProviderError, makeLlmError } from './llm-error.js'; import type { CapabilityFlags, @@ -108,6 +109,15 @@ const rejects = (provider: ProviderId, kind: LlmErrorKind, message?: string) => (): Promise => Promise.reject(providerError(provider, kind, message)); +/** A rejecting stub whose `LlmError` carries the provider's own requested wait (#279). */ +const rejectsWithRetryAfter = + (provider: ProviderId, retryAfterMs: number) => (): Promise => + Promise.reject( + new LlmProviderError( + makeLlmError({ provider, kind: 'rate_limit', message: 'rl', retryAfterMs }), + ), + ); + async function* streamFrom(chunks: StreamChunk[]): AsyncIterable { await Promise.resolve(); // a real stream awaits I/O; this keeps the fake a true async iterable for (const chunk of chunks) { @@ -832,6 +842,78 @@ describe('FallbackChain — backoff and cooldown', () => { expect(sleeps).toEqual([100, 200]); // backoff before attempts 2 and 3, exponential, no inter-entry delay }); + it('records an unpriced model as priced:false rather than silently as no cost (#194, 2.6.Q)', async () => { + // "No cost" used to be ambiguous between *could not price* and *genuinely free*. A strict cap cannot be + // enforced over spend it cannot see, so the surface has to be able to say which one happened. + const provider = makeProvider({ id: 'anthropic', generate: resolves('ok') }); + const { options, trace } = makeOptions({ + costTracker: { + record: () => { + throw new UnknownModelError('self-hosted-model', ['claude-opus-4-8']); + }, + } as unknown as CostTracker, + }); + const chain = new FallbackChain([entry(provider, 'self-hosted-model')], options); + + await chain.generate(userReq); + + const succeeded = trace.find((r) => r.outcome === 'succeeded'); + expect(succeeded?.priced).toBe(false); + expect(succeeded?.cost).toBeUndefined(); + }); + + it('does NOT swallow a non-pricing failure from the cost tracker (#194)', async () => { + // The catch used to be bare, so a genuine defect in the money path — a bad Usage, a broken overlay, a + // throwing custom tracker — produced "no cost" and looked exactly like an unpriced model. + const provider = makeProvider({ id: 'anthropic', generate: resolves('ok') }); + const { options } = makeOptions({ + costTracker: { + record: () => { + throw new TypeError('cumulative overflowed'); + }, + } as unknown as CostTracker, + }); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); + + await expect(chain.generate(userReq)).rejects.toThrow('cumulative overflowed'); + }); + + it("honours the provider's Retry-After over its own computed backoff (#279)", async () => { + // A rate limit is the one case where the provider knows better than we do: it says when its window + // reopens. Computing our own delay either hammered it early or waited longer than needed. + const primary = makeProvider({ + id: 'anthropic', + generate: rejectsWithRetryAfter('anthropic', 1_500), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('ok') }); + const { options, sleeps } = makeOptions({ backoffBaseMs: 100, backoffMaxMs: 1000 }); + const chain = new FallbackChain( + [{ ...entry(primary, 'claude-opus-4-8'), maxAttempts: 3 }, entry(fallback, 'gpt-5.5')], + options, + ); + + await chain.generate(userReq); + + // 1500 twice — the header, NOT the 100/200 exponential curve this config would otherwise produce. + expect(sleeps).toEqual([1_500, 1_500]); + }); + + it('falls back to its own curve when the provider requested nothing (#279)', async () => { + // `undefined` must mean "no instruction", never "wait zero" — otherwise a provider that omits the + // header would turn the backoff into a tight retry loop. + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'rate_limit') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('ok') }); + const { options, sleeps } = makeOptions({ backoffBaseMs: 100, backoffMaxMs: 1000 }); + const chain = new FallbackChain( + [{ ...entry(primary, 'claude-opus-4-8'), maxAttempts: 3 }, entry(fallback, 'gpt-5.5')], + options, + ); + + await chain.generate(userReq); + + expect(sleeps).toEqual([100, 200]); + }); + it('respects the linear backoff curve and the ceiling', async () => { const provider = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'timeout') }); const { options, sleeps } = makeOptions({ backoffBaseMs: 100, backoffMaxMs: 250 }); diff --git a/packages/llm/src/fallback-chain.ts b/packages/llm/src/fallback-chain.ts index 67db274a..64c33071 100644 --- a/packages/llm/src/fallback-chain.ts +++ b/packages/llm/src/fallback-chain.ts @@ -1,6 +1,7 @@ import type { BackoffStrategy, ContentPart, MediaSource } from '@relavium/shared'; import type { CostTracker, CostUpdate } from './cost-tracker.js'; +import { UnknownModelError } from './errors.js'; import { isRetryable, LlmProviderError, makeLlmError } from './llm-error.js'; import type { LlmError, @@ -102,6 +103,16 @@ export interface AttemptRecord { readonly error?: LlmError; /** Why a `'skipped'` entry was skipped (provider in cooldown, or it can't satisfy the request). */ readonly skipReason?: string; + /** + * `false` when the attempt produced real usage but the model could not be PRICED, so `cost` is absent for a + * reason other than "there was nothing to charge" (2.6.Q's realized-cost half, ADR-0071 §K7). + * + * Without it, "no cost" was ambiguous between *could not price* and *genuinely free* — and a strict cost cap + * cannot be enforced over a model whose spend is unknown, so the surface must be able to say so instead of + * silently reporting zero. Absent (rather than `true`) on the ordinary priced path, so nothing changes shape + * for existing consumers; ADR-0070's durable `unpriced_calls` counter is the same distinction, persisted. + */ + readonly priced?: false; } /** @@ -385,7 +396,7 @@ export class FallbackChain { continue; } if (attempt < budget + bonus) { - await this.#backoff(entry, attempt - 1); + await this.#backoff(entry, attempt - 1, outcome.error); } } return undefined; // budget exhausted → advance to the next entry @@ -464,7 +475,7 @@ export class FallbackChain { continue; } if (attempt < budget + bonus) { - await this.#backoff(entry, attempt - 1); + await this.#backoff(entry, attempt - 1, failure); } } return 'advance'; // budget exhausted → try the next entry @@ -736,13 +747,31 @@ export class FallbackChain { }); } - async #backoff(entry: FallbackPlanEntry, retryIndex: number): Promise { - const delay = backoffDelayMs( - entry.backoff ?? 'exponential', - retryIndex, - this.#backoffBaseMs, - this.#backoffMaxMs, - ); + /** + * Wait before re-attempting an entry. The provider's OWN `Retry-After` wins when it sent one (#279): on a + * rate limit it knows when its window reopens, and computing our own delay was guesswork that either + * hammered it early or waited longer than needed. + * + * Deliberately NO jitter, in either branch. ADR-0040 pins this backoff as deterministic — no jitter, never + * `Math.random` — so a replay reproduces the same schedule; `retry.ts` follows the same convention. The + * thundering-herd risk that jitter would address is therefore accepted and left documented rather than + * silently traded away here: parallel branches that hit one provider's rate limit together will retry + * together. Revisiting that is an ADR-0040 amendment, not a local edit. + * + * The honoured value is already normalized and ceiling-checked at the seam + * ({@link retryAfterMsFromHeaders}), so a hostile `retry-after: 999999999` cannot park a turn — it arrives + * as `undefined` and we fall back to our own schedule. + */ + async #backoff(entry: FallbackPlanEntry, retryIndex: number, error?: LlmError): Promise { + const requested = error?.retryAfterMs; + const delay = + requested ?? + backoffDelayMs( + entry.backoff ?? 'exponential', + retryIndex, + this.#backoffBaseMs, + this.#backoffMaxMs, + ); await this.#sleep(delay); } @@ -773,18 +802,27 @@ export class FallbackChain { this.#emit({ ...record, outcome: 'succeeded' }); return; } - // Best-effort: `priceModel` throws `UnknownModelError` for a model id outside the pricing table - // (a new snapshot, an OpenAI-compatible / self-hosted / custom-base-URL model). The attempt has - // ALREADY succeeded and its tokens were delivered — an unpriced model must degrade to no cost, - // never fail the call. Cost accuracy for unlisted models is a pricing-table concern, not a runtime error. + // `priceModel` throws `UnknownModelError` for a model id outside the pricing table (a new snapshot, an + // OpenAI-compatible / self-hosted / custom-base-URL model). The attempt has ALREADY succeeded and its + // tokens were delivered — an unpriced model must degrade to no cost, never fail the call. Cost accuracy + // for unlisted models is a pricing-table concern, not a runtime error. + // + // NARROW, not bare (#194): the catch used to swallow EVERY exception, so a genuine defect in the money + // path — a bad `Usage`, a broken overlay, a throwing custom tracker — silently produced "no cost" and + // looked exactly like an unpriced model. Anything that is not `UnknownModelError` now propagates, because + // a money bug must be loud (`docs/standards/error-handling.md` — no silent catches). let cost: CostUpdate | undefined; + let unpriced = false; try { cost = this.#options.costTracker?.record(model, usage); - } catch { - cost = undefined; + } catch (error_) { + if (!(error_ instanceof UnknownModelError)) throw error_; + unpriced = true; } this.#emit({ ...record, + // 2.6.Q's realized-cost half: distinguish "could not price" from "nothing to charge". + ...(unpriced ? { priced: false as const } : {}), outcome: 'succeeded', usage, ...(cost === undefined ? {} : { cost }), diff --git a/packages/llm/src/llm-error.test.ts b/packages/llm/src/llm-error.test.ts index adf7efd9..4cfa48c4 100644 --- a/packages/llm/src/llm-error.test.ts +++ b/packages/llm/src/llm-error.test.ts @@ -5,6 +5,7 @@ import { isRetryable, kindFromHttpStatus, makeLlmError, + retryAfterMsFromHeaders, scrubSecrets, } from './llm-error.js'; import { LlmErrorSchema } from './types.js'; @@ -158,3 +159,52 @@ describe('scrubSecrets — defense-in-depth secret backstop (no key/token/baseUR expect(LlmErrorSchema.safeParse(e).success).toBe(true); }); }); + +describe('retryAfterMsFromHeaders (#279)', () => { + /** A minimal `Headers`-like container; only `.get` is used, so no vendor type crosses the seam. */ + const headers = (map: Record) => ({ + get: (name: string): string | null => map[name.toLowerCase()] ?? null, + }); + + it('prefers retry-after-ms, which the Stainless SDKs send with millisecond precision', () => { + expect(retryAfterMsFromHeaders(headers({ 'retry-after-ms': '1500', 'retry-after': '9' }))).toBe( + 1500, + ); + }); + + it('reads retry-after as delta-seconds', () => { + expect(retryAfterMsFromHeaders(headers({ 'retry-after': '2' }))).toBe(2000); + }); + + it('reads the RFC 7231 HTTP-date form', () => { + const at = new Date(Date.now() + 3_000).toUTCString(); + const ms = retryAfterMsFromHeaders(headers({ 'retry-after': at })); + // Second-resolution date, so allow a small window rather than pinning an exact figure. + expect(ms).toBeGreaterThan(1_000); + expect(ms).toBeLessThanOrEqual(4_000); + }); + + it('returns undefined when the provider said nothing — NOT zero', () => { + // The distinction is load-bearing: zero would turn the chain's backoff into a tight retry loop. + expect(retryAfterMsFromHeaders(headers({}))).toBeUndefined(); + expect(retryAfterMsFromHeaders(headers({ 'retry-after': ' ' }))).toBeUndefined(); + }); + + it('CLAMPS an over-ceiling wait rather than ignoring it', () => { + // Dropping it fell back to our own ~250 ms curve — hammering the endpoint that just asked us to back off, + // far harder than a capped 60 s wait would. The ceiling stops a hostile value parking a turn; clamping + // does that AND still respects a real instruction. + expect(retryAfterMsFromHeaders(headers({ 'retry-after': '999999999' }))).toBe(60_000); + expect(retryAfterMsFromHeaders(headers({ 'retry-after': '120' }))).toBe(60_000); + expect(retryAfterMsFromHeaders(headers({ 'retry-after': '30' }))).toBe(30_000); // under it, verbatim + }); + + it('DROPS a value that is not a wait at all', () => { + expect(retryAfterMsFromHeaders(headers({ 'retry-after': '-5' }))).toBeUndefined(); + expect(retryAfterMsFromHeaders(headers({ 'retry-after': 'soon' }))).toBeUndefined(); + expect(retryAfterMsFromHeaders(headers({ 'retry-after-ms': 'NaN' }))).toBeUndefined(); + // A date already in the past is not an instruction to wait. + const past = new Date(Date.now() - 60_000).toUTCString(); + expect(retryAfterMsFromHeaders(headers({ 'retry-after': past }))).toBeUndefined(); + }); +}); diff --git a/packages/llm/src/llm-error.ts b/packages/llm/src/llm-error.ts index f6fb5c7a..c3076926 100644 --- a/packages/llm/src/llm-error.ts +++ b/packages/llm/src/llm-error.ts @@ -72,6 +72,8 @@ interface MakeLlmErrorArgs { readonly status?: number; /** Original error, for debugging only — never re-thrown across the seam. */ readonly cause?: unknown; + /** The provider's own requested wait, normalized to ms (#279). Set only when a header supplied it. */ + readonly retryAfterMs?: number; } /** @@ -116,6 +118,9 @@ export function makeLlmError(args: MakeLlmErrorArgs): LlmError { if (args.status !== undefined) { error.status = args.status; } + if (args.retryAfterMs !== undefined) { + error.retryAfterMs = args.retryAfterMs; + } // `cause` is an INTERNAL diagnostic only (error-handling.md): it is NOT scrubbed and may hold a raw // vendor object, so it must never be logged/serialized raw. The run-event boundary already carries // only { code, message, retryable } (run-event.ts), never `cause`; a future adapter that sets `cause` @@ -125,3 +130,50 @@ export function makeLlmError(args: MakeLlmErrorArgs): LlmError { } return error; } + +/** + * Normalize a provider's requested wait to milliseconds, from either header shape (#279). + * + * Both Anthropic and OpenAI send `retry-after` (RFC 7231: delta-seconds OR an HTTP-date) and the Stainless + * SDKs additionally send `retry-after-ms`. Only a value the provider actually supplied is returned — + * `undefined` means "no instruction", never "wait zero", so the caller keeps its own backoff. + * + * Defensive by design: a header is untrusted input. A non-numeric, negative, or absurd value is dropped + * rather than honoured, because a hostile or broken `retry-after: 999999999` would otherwise park a turn + * indefinitely. The ceiling is the caller's problem too, but rejecting nonsense here keeps one rule in one place. + */ +export function retryAfterMsFromHeaders(headers: { + readonly get: (name: string) => string | null | undefined; +}): number | undefined { + const ms = headers.get('retry-after-ms'); + if (ms !== null && ms !== undefined && ms.trim() !== '') { + const parsed = Number(ms); + if (Number.isFinite(parsed) && parsed >= 0) return clampRetryAfter(parsed); + } + const after = headers.get('retry-after'); + if (after === null || after === undefined || after.trim() === '') return undefined; + const seconds = Number(after); + if (Number.isFinite(seconds) && seconds >= 0) return clampRetryAfter(seconds * 1000); + // An HTTP-date form: honour it only if it is in the future by a sane margin. + const at = Date.parse(after); + if (Number.isNaN(at)) return undefined; + return clampRetryAfter(at - Date.now()); +} + +/** The hard ceiling on a provider-requested wait. Beyond it we wait the ceiling — we do not ignore the ask. */ +const RETRY_AFTER_CEILING_MS = 60_000; + +/** + * CLAMP, never drop. Dropping an over-ceiling value looked safer but is strictly worse: a provider legitimately + * asking for 120 s fell back to our own curve and retried after ~250 ms — hammering the exact endpoint that + * just told us to back off, and far more aggressively than honouring a capped 60 s would. The ceiling exists to + * stop a hostile/broken `retry-after: 999999999` parking a turn, and clamping does that just as well while + * still respecting a real instruction. + * + * Only genuinely uninterpretable values (non-finite, negative — i.e. "this is not a wait") yield `undefined`, + * which means "no instruction" and lets the caller keep its own backoff. + */ +function clampRetryAfter(ms: number): number | undefined { + if (!Number.isFinite(ms) || ms < 0) return undefined; + return Math.round(Math.min(ms, RETRY_AFTER_CEILING_MS)); +} diff --git a/packages/llm/src/types.ts b/packages/llm/src/types.ts index c070f906..21fbe979 100644 --- a/packages/llm/src/types.ts +++ b/packages/llm/src/types.ts @@ -253,6 +253,15 @@ export const LlmErrorSchema = z.object({ retryable: z.boolean(), code: nonEmptyString.optional(), // normalized provider/transport code, e.g. 'rate_limit' status: z.number().int().optional(), // upstream HTTP status, when there was one + /** + * How long the PROVIDER asked us to wait, in milliseconds, normalized from its `Retry-After` / + * `retry-after-ms` header (#279). Present only when the provider actually said so — never guessed. + * + * A plain `number`, so no vendor header type crosses the seam. `FallbackChain` prefers it over its own + * computed backoff, which is the whole point: on a 429 the provider knows when its window reopens and we + * were previously ignoring that and retrying on our own schedule. + */ + retryAfterMs: z.number().int().nonnegative().optional(), provider: ProviderIdSchema, message: z.string(), // human-readable, already redacted of any secret material // INTERNAL diagnostic only — may hold a raw vendor error and is NOT scrubbed by `makeLlmError`