diff --git a/CLAUDE.md b/CLAUDE.md index 4c94dd94..fe12df93 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ A run executes in one of **three execution modes** behind the one `LLMProvider` engine is identical across all three. See [ADR-0012](docs/decisions/0012-managed-inference-dual-mode.md) to [ADR-0015](docs/decisions/0015-managed-mode-data-handling-and-compliance.md) and [docs/architecture/managed-inference.md](docs/architecture/managed-inference.md). -**Status:** Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07); the `FallbackChain` runner (1.K) landed, completing 1.m2 with the cost tracker (PR #13, 2026-06-11); the run loop (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026-06-13) **completing 1.m3** (parse → DAG → run loop emits the canonical event stream), with the built-in `ToolRegistry` (1.T, a 1.m4 component) landing alongside it as the other `AgentRunner` (1.O) join prerequisite; the **`AgentRunner` (1.O) — per-node LLM execution behind the seam — landed (PR #18, 2026-06-14)**; and the **node-type handlers (1.P) — the six non-agent `NodeExecutor` arms (condition / transform / fan_out / fan_in / input / output) behind a dispatching executor — landed (PR #20, 2026-06-14)**; and **checkpoint/resume (1.R) + the human gate (1.Q) landed (PR #22, 2026-06-15)** — the derived `Checkpointer` + cross-process `resumeFromCheckpoint`, and the `human_in_the_loop` gate with the one-shot timeout port; and **node retry (1.S) — the above-chain whole-node retry budget ([ADR-0040](docs/decisions/0040-node-retry-budget-above-the-chain.md), amending ADR-0038) — landed (PR #24, 2026-06-15)**, re-dispatching a whole node on a retryable failure up to `retry.max` attempts (with `node:retrying`, abort-aware backoff, and `retry_on` filtering), with retry-from-node (ADR-0040 Part B) deferred to Phase-2; and the **pre-egress budget governor (1.AC, [ADR-0028](docs/decisions/0028-workflow-resource-governance.md)) + the `AgentSession` agent-first entry point (1.V, [ADR-0024](docs/decisions/0024-agent-first-entry-point-agentsession.md)) landed together (PR #26, 2026-06-16)** — 1.AC was the last 1.m4 component, so **1.m4 is complete** (the full engine stack: node handlers, gate, checkpoint/resume, retry, tools, sandbox, budget governor), and 1.V opens the Lane-C agent-first sub-spine (1.m5); then the **end-to-end Node harness (1.U) landed (PR #27, 2026-06-16), reaching 🎯 M2** — the engine runs end-to-end (live streaming + per-node-boundary checkpointing + cross-process resume + node retry + provider failover, gap-free), **completing the Phase-1 engine critical path**. The remaining Phase-1 work is additive and off the critical path (Lane C: the **`session:*` namespace (1.W) landed (PR #28, 2026-06-17)** — the `SessionEventSink`→`RunEventBus` adapter + per-session `sequenceNumber`, the `SessionHandle`, and the combined `RunOrSessionEventSchema` gate — and **session persistence (1.X) landed (PR #29, 2026-06-17)** — the `agent_sessions`/`session_messages` tables + migration, `SessionMessageSchema`/`AgentSessionSchema`, and the `SessionStore` + domain↔row mappers (data-layer only); then **session checkpoint/resume (1.Y) + export-to-workflow (1.Z) landed (PR #30, 2026-06-17)** — `reconstructSessionState`/`AgentSession.resume` (reload-not-replay; preload the text-only transcript, re-seed turnCount/cost, no `session:started` re-emit) + the `serializeWorkflow`/`sessionToWorkflow` pair (one agent node per completed turn, transcript in `metadata`, secret/signature exclusion structural) — leaving only the **1.AA** chat-regression harness ‖ the 1.m6 multimodal sub-spine, whose first step — **media-input adapters + the shared SSRF policy primitive (1.AE) — landed (PR #32, 2026-06-18)** (base64 image/audio across the three adapters behind per-modality `assertMediaCapabilities`, and the one `@relavium/shared` SSRF policy primitive reused by the provider-`baseURL` + `http_request` callers; the SSRF *mechanism* half + per-modality FallbackChain gating moved into **1.AF**). **1.AF (engine media plumbing) is in progress: P1+P2 merged (PR #33)** — the `MediaStore`/`deInlineMedia` choke point (the active I3 enforcement), per-modality capability gating (`mediaSupportReason`/`requestSupportReason` — replacing coarse vision), the `media_objects`/`media_references` tables, and ADR-0042/0043/0044 — and **P3 + P4/D13 have landed on `development` in a follow-on PR (pending merge):** the binary media-egress + the **SSRF mechanism half** (D9 — the `MediaUrlFetch` re-host hook + the `fetchMediaBytes` SSRF-validated host reference wired at the choke point), the `FallbackChain` resolve-before-egress + the **byte-free re-materialization sidecar** (D8/D7), and the byte-delivery **`Range` gate** (D13 — `MediaStore.readRange` + `validateByteRange`), with the dedicated **P3 egress/SSRF security-review clean** (0 blockers/highs). The rest of **P4** follows: `read_media` + the scope-set authz (D12), the terminal sweep + GC (D11), `save_to` (D16), the `output_modalities` load-check (D15), the per-modality media cost governor (D17), the byte-delivery security-review + the keychain IPC test. **Phase 2 (CLI, M3) is unblocked**. +**Status:** Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07); the `FallbackChain` runner (1.K) landed, completing 1.m2 with the cost tracker (PR #13, 2026-06-11); the run loop (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026-06-13) **completing 1.m3** (parse → DAG → run loop emits the canonical event stream), with the built-in `ToolRegistry` (1.T, a 1.m4 component) landing alongside it as the other `AgentRunner` (1.O) join prerequisite; the **`AgentRunner` (1.O) — per-node LLM execution behind the seam — landed (PR #18, 2026-06-14)**; and the **node-type handlers (1.P) — the six non-agent `NodeExecutor` arms (condition / transform / fan_out / fan_in / input / output) behind a dispatching executor — landed (PR #20, 2026-06-14)**; and **checkpoint/resume (1.R) + the human gate (1.Q) landed (PR #22, 2026-06-15)** — the derived `Checkpointer` + cross-process `resumeFromCheckpoint`, and the `human_in_the_loop` gate with the one-shot timeout port; and **node retry (1.S) — the above-chain whole-node retry budget ([ADR-0040](docs/decisions/0040-node-retry-budget-above-the-chain.md), amending ADR-0038) — landed (PR #24, 2026-06-15)**, re-dispatching a whole node on a retryable failure up to `retry.max` attempts (with `node:retrying`, abort-aware backoff, and `retry_on` filtering), with retry-from-node (ADR-0040 Part B) deferred to Phase-2; and the **pre-egress budget governor (1.AC, [ADR-0028](docs/decisions/0028-workflow-resource-governance.md)) + the `AgentSession` agent-first entry point (1.V, [ADR-0024](docs/decisions/0024-agent-first-entry-point-agentsession.md)) landed together (PR #26, 2026-06-16)** — 1.AC was the last 1.m4 component, so **1.m4 is complete** (the full engine stack: node handlers, gate, checkpoint/resume, retry, tools, sandbox, budget governor), and 1.V opens the Lane-C agent-first sub-spine (1.m5); then the **end-to-end Node harness (1.U) landed (PR #27, 2026-06-16), reaching 🎯 M2** — the engine runs end-to-end (live streaming + per-node-boundary checkpointing + cross-process resume + node retry + provider failover, gap-free), **completing the Phase-1 engine critical path**. The remaining Phase-1 work is additive and off the critical path (Lane C: the **`session:*` namespace (1.W) landed (PR #28, 2026-06-17)** — the `SessionEventSink`→`RunEventBus` adapter + per-session `sequenceNumber`, the `SessionHandle`, and the combined `RunOrSessionEventSchema` gate — and **session persistence (1.X) landed (PR #29, 2026-06-17)** — the `agent_sessions`/`session_messages` tables + migration, `SessionMessageSchema`/`AgentSessionSchema`, and the `SessionStore` + domain↔row mappers (data-layer only); then **session checkpoint/resume (1.Y) + export-to-workflow (1.Z) landed (PR #30, 2026-06-17)** — `reconstructSessionState`/`AgentSession.resume` (reload-not-replay; preload the text-only transcript, re-seed turnCount/cost, no `session:started` re-emit) + the `serializeWorkflow`/`sessionToWorkflow` pair (one agent node per completed turn, transcript in `metadata`, secret/signature exclusion structural) — leaving only the **1.AA** chat-regression harness ‖ the 1.m6 multimodal sub-spine, whose first step — **media-input adapters + the shared SSRF policy primitive (1.AE) — landed (PR #32, 2026-06-18)** (base64 image/audio across the three adapters behind per-modality `assertMediaCapabilities`, and the one `@relavium/shared` SSRF policy primitive reused by the provider-`baseURL` + `http_request` callers; the SSRF *mechanism* half + per-modality FallbackChain gating moved into **1.AF**). **1.AF (engine media plumbing) is ✅ Done (PR #33/#34/#35/#36, 2026-06-20):** the `MediaStore`/`deInlineMedia` choke point (the active I3 enforcement), per-modality capability gating (`mediaSupportReason`/`requestSupportReason` — replacing coarse vision), the `media_objects`/`media_references` tables, and ADR-0042/0043/0044 (P1+P2, PR #33); the binary media-egress + the **SSRF mechanism half** (D9 — the `MediaUrlFetch` re-host hook + the SSRF-validated `fetchMediaBytes` wired at the choke point), the `FallbackChain` resolve-before-egress + the **byte-free re-materialization sidecar** (D8/D7), and the byte-delivery **`Range` gate** (D13) (P3 + P4/D13, PR #34); `read_media` + the scope-set authz (D12), the terminal sweep + grace-window GC (D11), `save_to` (D16), the `output_modalities` load-check (D15), and the per-modality media cost governor (D17) (P4 remainder, PR #35); plus a **13-dimension end-to-end review + an external Sonar/CodeRabbit pass** (PR #36 — H1 `agent:tool_call.toolInput` byte-redaction, H2 the unified exact-membership output-combination gate, the GC grace-basis fix, the `nodeMediaEgressDeps` egress-mechanism test). Each phase passed a dedicated adversarial security-review (0 blockers/highs). The deferred **host-wiring** half (D12 `MediaReadAccess` + session-scope population, the D15 loader, the D17/`resolveForEgress` config) + the keychain no-raw-key IPC test are owned by **1.AH** (recorded in deferred-tasks.md). The remaining Phase-1 work is **1.AG/1.AH**; **Phase 2 (CLI, M3) is unblocked**. Phase 0 (M0, 2026-06-04) landed the monorepo, strict toolchain + CI, `@relavium/shared` (the full Zod contract set), the no-vendor-type seam fence, and `@relavium/db`. Phase 1 has since landed `@relavium/llm` — the `LLMProvider` seam + all three adapters (Anthropic, OpenAI/DeepSeek, @@ -64,7 +64,7 @@ The pre-egress budget governor (1.AC) + the agent-first `AgentSession` (1.V) lan **completing 1.m4**; then the end-to-end Node harness (1.U) landed (PR #27, 2026-06-16) **reaching M2** — the Phase-1 engine critical path is complete. The additive Lane-C agent-first sub-spine is now **complete** (session events **1.W ✅ (PR #28)** + persistence **1.X ✅ (PR #29)** + checkpoint/resume **1.Y** & export **1.Z ✅ (PR #30, 2026-06-17)** + the **1.AA** chat-regression harness ✅ (2026-06-17), closing **1.m5**); -on the 1.m6 multimodal sub-spine, **media-input adapters + the shared SSRF policy primitive (1.AE) landed (PR #32, 2026-06-18)** — after a multi-round + final 8-dimension adversarial review (no SSRF bypass found); the SSRF *mechanism* half + per-modality gating moved into 1.AF. **1.AF (engine media plumbing) is in progress: P1+P2 merged (PR #33)**; **P3 + P4/D13 landed on `development` (pending merge)** — the media-egress + SSRF mechanism half (D9), the `FallbackChain` resolve-before-egress + byte-free re-materialization sidecar (D8/D7), the byte-delivery `Range` gate (D13), and a clean P3 egress/SSRF security-review — with the rest of P4 (D12 `read_media` + authz, D11 GC sweep, D16 `save_to`, D15 load-check, D17 cost governor) to follow. The remaining Phase-1 work is 1.AF (in progress) + 1.AG/1.AH; Phase 2 (CLI) is unblocked. See +on the 1.m6 multimodal sub-spine, **media-input adapters + the shared SSRF policy primitive (1.AE) landed (PR #32, 2026-06-18)** — after a multi-round + final 8-dimension adversarial review (no SSRF bypass found); the SSRF *mechanism* half + per-modality gating moved into 1.AF. **1.AF (engine media plumbing) is ✅ Done (PR #33/#34/#35/#36, 2026-06-20)** — P1+P2 (PR #33); the media-egress + SSRF mechanism half (D9), the `FallbackChain` resolve-before-egress + byte-free re-materialization sidecar (D8/D7), the byte-delivery `Range` gate (D13), and a clean P3 egress/SSRF security-review (P3 + P4/D13, PR #34); D12 `read_media` + authz, D11 GC sweep, D16 `save_to`, D15 load-check, D17 cost governor (P4 remainder, PR #35); and a 13-dimension end-to-end + external Sonar/CodeRabbit review (PR #36 — toolInput byte-redaction, the unified exact-membership output gate, the GC grace-basis fix, the egress-mechanism test). The host-wiring half (D12/D15/D17 mechanism) + the keychain IPC test are deferred to **1.AH**. The remaining Phase-1 work is 1.AG/1.AH; Phase 2 (CLI) is unblocked. See [docs/roadmap/current.md](docs/roadmap/current.md). See [README.md](README.md) for the public overview. ## Non-negotiable rules for AI agents diff --git a/README.md b/README.md index 9f6bdda9..49751460 100644 --- a/README.md +++ b/README.md @@ -53,5 +53,5 @@ both landed in **PR #26 (2026-06-16)**. With the budget governor in, **milestone end-to-end (live streaming + checkpoint + cross-process resume + retry + provider failover), completing the Phase-1 engine critical path. The remaining Phase-1 work is additive (the agent-first sub-spine — **session events 1.W ✅ (PR #28)** + **persistence 1.X ✅ (PR #29)** + **session checkpoint/resume 1.Y & export-to-workflow 1.Z ✅ (PR #30, 2026-06-17)** + **the 1.AA chat-regression harness ✅ (2026-06-17)**, completing **1.m5** — and the -multimodal sub-spine, whose first step — **media-input adapters + the shared SSRF policy primitive 1.AE ✅ (PR #32, 2026-06-18)** — landed); **Phase 2 (the CLI) is unblocked**. See +multimodal sub-spine — **media-input adapters + the shared SSRF policy primitive 1.AE ✅ (PR #32)**, then **engine media plumbing 1.AF ✅ (PR #33/#34/#35/#36, 2026-06-20)** — landed; 1.AG/1.AH remain); **Phase 2 (the CLI) is unblocked**. See [docs/roadmap/current.md](docs/roadmap/current.md) for live status. diff --git a/docs/decisions/0031-llm-seam-shape-amendment-multimodal-io.md b/docs/decisions/0031-llm-seam-shape-amendment-multimodal-io.md index 4e2179c4..42b45b8a 100644 --- a/docs/decisions/0031-llm-seam-shape-amendment-multimodal-io.md +++ b/docs/decisions/0031-llm-seam-shape-amendment-multimodal-io.md @@ -19,6 +19,10 @@ > **Amended 2026-06-18 by [ADR-0042](0042-engine-media-storage-substrate-mediastore-deinline-retention.md), [ADR-0043](0043-media-egress-failover-rematerialization-ssrf.md), and [ADR-0044](0044-media-access-governance-read-media-save-to-cost.md).** The 1.AF wiring of this ADR's reserved media shape, refining (never reversing) it: ADR-0042 lands the `MediaStore` host-port injection + the `deInlineMedia` choke-point ordering + the `media_objects` retention/GC store (promoting this ADR's "Open implementation details" retention default to a decision); ADR-0043 lands the binary media-egress capability + the SSRF mechanism half, and **resolves this ADR's sidecar-ownership wording** (the `(provider, sha256)` sidecar is held per-`FallbackChain` run-instance via a narrow injected hook, not the stateless adapter); ADR-0044 lands the `read_media` scope-set authz + the byte-delivery gate + the `save_to` write port + the per-modality media cost. The seam shape here is unchanged. +> **Amended 2026-06-20 by [ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md).** A refinement, not a reversal: 1.AG wires the reserved `generateMedia`/`pollMediaJob` methods and **additively adds a `signal?` param to `pollMediaJob`** (so a cancel aborts the in-flight poll) — the only shape change to the 1.AD-frozen seam, made while the methods are still un-implemented. The reserved shapes (`MediaGenRequest`/`MediaGenResult`/`MediaJobStatus`, the opaque-jobId rule) stand. + +> **Amended 2026-06-20 by [ADR-0046](0046-inline-media-out-via-generate-streaming-triad-deferred.md).** A refinement, not a reversal: it pins which of §5.1's two inline-media-out paths Phase-1 (1.AG) uses — the non-streaming `generate()` path ("the more common"), de-inlined at the `#emitDurable` choke point — and defers the streaming `media_start`/`media_delta`/`media_end` triad to the host (1.AH). The seam shape (incl. the handle-only `media_end`) is unchanged. + ## Context The `@relavium/llm` seam — the request/result/stream/usage/content shapes in diff --git a/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md b/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md index 0a8e7e93..3448cb6d 100644 --- a/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md +++ b/docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md @@ -6,6 +6,8 @@ > **Amended 2026-06-18 by [ADR-0042](0042-engine-media-storage-substrate-mediastore-deinline-retention.md).** A refinement, not a reversal: ADR-0042 adds an optional `mediaStore?` port to the `ExecutionHost` seam (1.AF) and pins where the async `deInlineMedia` pass sits relative to the single producer-side translation point this ADR defines (the gap-free `sequenceNumber` + persist-before-deliver chokepoint is unchanged). This ADR's substrate decisions stand. +> **Amended 2026-06-20 by [ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md).** A refinement, not a reversal: 1.AG adds a new non-terminal node **suspension state** (a parked async media job) + a durable `media_job:submitted` event, both settling through the single `#emitDurable` choke point this ADR defines (the gap-free `sequenceNumber` + persist-before-deliver are unchanged). This ADR's substrate decisions stand. + ## Context Workstream **1.N** builds the engine run loop: `WorkflowEngine.start(workflowId, input)` / `resume` / `cancel` walking the `RunPlan` (1.M), the `RunEventBus` surfaces subscribe to, and the `RunHandle.events` async iterable every surface consumes. Two of its three contracts are already settled and have a canonical home — the emitted **event shapes** are pinned by [sse-event-schema.md](../reference/contracts/sse-event-schema.md) and the Zod source in `@relavium/shared` (`run-event.ts`), and the **dispatch shape** (a plain async orchestrator over a static topological plan, a dispatch table, state-after-each-node, a *derived* checkpoint, no second state machine, no LangGraph) is pinned by [ADR-0003](0003-pure-ts-engine-not-langgraph-python.md). This ADR does not re-open either; it records the three run-loop **substrate** decisions that no existing ADR covers and that 1.N cannot make implicitly. diff --git a/docs/decisions/0038-agentrunner-llm-call-boundary.md b/docs/decisions/0038-agentrunner-llm-call-boundary.md index 9f1bc1a0..df606006 100644 --- a/docs/decisions/0038-agentrunner-llm-call-boundary.md +++ b/docs/decisions/0038-agentrunner-llm-call-boundary.md @@ -9,6 +9,8 @@ > the chain's own default backoff. Everything else in this ADR stands. (Within-chain same-model retry of the > primary, if wanted, is an optional primary chain-entry `max_attempts`.) +> **Amended 2026-06-20 by [ADR-0046](0046-inline-media-out-via-generate-streaming-triad-deferred.md).** A refinement, not a reversal: the per-node turn loop (stream-only today) gains a single-shot `generate()` branch for a media-output turn (a `media_surface: 'chat'` model whose node requests media `output_modalities`), so inline media-out arrives as an in-flight `media` part in `LlmResult.content` de-inlined at `#emitDurable`. The text-only streaming path + the tool/reasoning/cost machinery are unchanged. + - **Related**: [ADR-0006](0006-os-keychain-for-api-keys.md), [ADR-0011](0011-internal-llm-abstraction.md), [ADR-0018](0018-desktop-execution-and-rust-egress.md), [ADR-0019](0019-cli-node-keychain-library.md), [ADR-0024](0024-agent-first-entry-point-agentsession.md), [ADR-0025](0025-agent-surface-refines-desktop-scope.md), [ADR-0026](0026-session-export-to-workflow.md), [ADR-0028](0028-workflow-resource-governance.md), [ADR-0029](0029-tool-policy-hardening.md), [ADR-0030](0030-llm-seam-shape-amendment-reasoning-response-format-provider-executed.md), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md), [ADR-0037](0037-engine-tool-execution-boundary.md), [ADR-0039](0039-same-provider-reasoning-replay.md), [ADR-0040](0040-node-retry-budget-above-the-chain.md), [llm-provider-seam.md](../reference/shared-core/llm-provider-seam.md), [run-plan.md](../reference/shared-core/run-plan.md), [sse-event-schema.md](../reference/contracts/sse-event-schema.md), [error-handling.md](../standards/error-handling.md), [security-review.md](../standards/security-review.md), [shared-core-engine.md](../architecture/shared-core-engine.md) ## Context diff --git a/docs/decisions/0040-node-retry-budget-above-the-chain.md b/docs/decisions/0040-node-retry-budget-above-the-chain.md index 2da40c4f..70089ee4 100644 --- a/docs/decisions/0040-node-retry-budget-above-the-chain.md +++ b/docs/decisions/0040-node-retry-budget-above-the-chain.md @@ -4,6 +4,8 @@ - **Date**: 2026-06-15 - **Related**: [ADR-0038](0038-agentrunner-llm-call-boundary.md) (amended — see Decision A.2), [ADR-0011](0011-internal-llm-abstraction.md) (the `FallbackChain` owns within-chain policy), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) (run loop + the injected one-shot timer), [ADR-0027](0027-expression-sandbox.md) / [ADR-0029](0029-tool-policy-hardening.md) (failure classification), [error-handling.md](../standards/error-handling.md), [run-plan.md](../reference/shared-core/run-plan.md) and [node-types.md](../reference/shared-core/node-types.md) (which already pre-describe this layer), [expression-sandbox-spec.md](../reference/shared-core/expression-sandbox-spec.md) (the `runId + nodeId + retryCount` idempotency key) +> **Amended 2026-06-20 by [ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md).** A scoped exception, not a reversal: for the async-media node only, ADR-0045 **re-attaches** to a persisted provider job on crash-resume instead of re-running from pending (the default A.6 quotes), and **carves the wall-clock media-poll cadence out of the A.3 deterministic-replay invariant** (an external job result is non-deterministic state already). The node-retry budget + classification rules are otherwise unchanged. + ## Context Reliability in the engine has **two distinct retry concerns**, and only one is built: diff --git a/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md b/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md new file mode 100644 index 00000000..7a37556b --- /dev/null +++ b/docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md @@ -0,0 +1,80 @@ +# ADR-0045: Engine-owned async media-job loop (1.AG/A5) — the `generateMedia`/`pollMediaJob` poll/checkpoint/resume/cancel LRO, the `media_job:submitted` derived state, inline-vs-generative routing, and media-cost realization (amends ADR-0031, ADR-0036, ADR-0040) + +- **Status**: Accepted +- **Date**: 2026-06-20 +- **Related**: [0031-llm-seam-shape-amendment-multimodal-io.md](0031-llm-seam-shape-amendment-multimodal-io.md) (**this ADR amends it** — append-only, ADR-0031 is unchanged in history; it wires the reserved `generateMedia`/`pollMediaJob` methods + additively adds a `signal` param to `pollMediaJob`, the only shape change to the 1.AD-frozen seam, made while the methods are still un-implemented), [0036-run-loop-substrate-event-bus-and-execution-host.md](0036-run-loop-substrate-event-bus-and-execution-host.md) (**amends** — adds a new non-terminal node *suspension* state + the durable `media_job:submitted` event through the existing `#emitDurable` choke point; the gap-free monotone `sequenceNumber` + persist-before-deliver are unchanged), [0040-node-retry-budget-above-the-chain.md](0040-node-retry-budget-above-the-chain.md) (**amends** — overrides, for the async-media node, the checkpoint model's "running-at-crash vertex re-runs from pending" default A.6 quotes (re-attach, not restart) and carves the wall-clock poll cadence out of the A.3 deterministic-replay invariant; reuses the `setTimer`/abort pattern), [0003-pure-ts-engine-not-langgraph-python.md](0003-pure-ts-engine-not-langgraph-python.md) (the derived-from-`run_events` checkpoint rule this stays inside, and the source of the "omitted running vertex re-runs from pending" default — a derived `pendingMediaJobs` slot, **no `media_jobs` table**), [0042-engine-media-storage-substrate-mediastore-deinline-retention.md](0042-engine-media-storage-substrate-mediastore-deinline-retention.md) (the §4 terminal-sweep boundary it explicitly left open for this ADR; the cancel-sweep composes with it), [0043-media-egress-failover-rematerialization-ssrf.md](0043-media-egress-failover-rematerialization-ssrf.md) (the one SSRF-validated media-egress this reuses to re-host a provider output URL — never a second fetch site), [0044-media-access-governance-read-media-save-to-cost.md](0044-media-access-governance-read-media-save-to-cost.md) (the per-modality media cost class + `PreEgressHook` this prices the generate path through), [0011-internal-llm-abstraction.md](0011-internal-llm-abstraction.md) (the seam-purity rule — the opaque jobId, no vendor type across the seam), [0014-managed-metering-quota-and-billing.md](0014-managed-metering-quota-and-billing.md) (the cost-integrity / orphaned-provider-cost caveat on a local-only cancel), [0015-managed-mode-data-handling-and-compliance.md](0015-managed-mode-data-handling-and-compliance.md) (the gateway poll-through / counts-not-content the loop must not foreclose), [../reference/shared-core/llm-provider-seam.md](../reference/shared-core/llm-provider-seam.md), [../reference/contracts/sse-event-schema.md](../reference/contracts/sse-event-schema.md), [../reference/contracts/config-spec.md](../reference/contracts/config-spec.md), [../reference/desktop/database-schema.md](../reference/desktop/database-schema.md), [../reference/contracts/workflow-yaml-spec.md](../reference/contracts/workflow-yaml-spec.md), [../analysis/multimodal-io-design-2026-06-07.md](../analysis/multimodal-io-design-2026-06-07.md) (decision A5). + +## Context + +Workstream **1.AG (Output generation — Phase D of the 1.m6 multimodal sub-spine)** lights up media *output*. [ADR-0031](0031-llm-seam-shape-amendment-multimodal-io.md) froze the seam shape for it at 1.AD: `generateMedia?(req, key): Promise` and `pollMediaJob?(jobId, key): Promise` are **reserved, optional** methods on the one `LlmProvider` seam ([`types.ts`](../../packages/llm/src/types.ts) — `MediaGenRequest`/`MediaGenResult`/`MediaJobStatus`), and the disclaimer reads "**RESERVED at 1.AD; wired at Phase D (1.AG) with its own ADR for the engine-owned poll/checkpoint/resume/cancel loop (A5)**." This is that ADR. + +Three output paths share the seam: **(a) inline media-out** — a chat turn whose `output_modalities` ask for media (Gemini `responseModalities`, OpenAI agentic image-gen via the `providerExecuted`+normalized-media arm, OpenAI inline audio); **(b) `generateMedia` SYNC** — a separate-endpoint one-round-trip generator (gpt-image-1, Imagen, OpenAI TTS) resolving `{ media }`; **(c) `generateMedia` ASYNC** — a minute-scale LRO (Sora, Veo) resolving `{ jobId }` the engine polls. The full provider matrix is in 1.AG scope (Anthropic/DeepSeek have no media output). The inline path is a normal turn; the substrate it needs (the `MediaStore` CAS, `deInlineMedia` at the one `#emitDurable` choke point, the SSRF-validated egress, the `media_references` refcount + terminal sweep, the per-modality cost governor) **all landed at 1.AF** ([ADR-0042](0042-engine-media-storage-substrate-mediastore-deinline-retention.md)/[0043](0043-media-egress-failover-rematerialization-ssrf.md)/[0044](0044-media-access-governance-read-media-save-to-cost.md)). What is genuinely undecided — and a wrong guess forces an ADR-0036/0003-level rewrite — is the **async LRO**: a paid, minute-scale provider job that must **survive an engine restart**, live in the run loop + checkpointer, and cancel cleanly. ADR-0042 §4 deliberately fenced "the minute-scale async-job cancel-sweep … out of 1.AF scope" and gave it "its own ADR; 1.AF's sweep/GC/governor design must not foreclose it, so the sweep is keyed on the terminal *event*, agnostic to job lifetime." Four facts, verified in code, frame the stakes: + +1. **The checkpoint model has no in-flight-job slot.** `CheckpointNodeState` is `completed | failed | skipped | paused` plus `pendingGates`; a running-at-crash vertex is **omitted and re-run** ([`checkpoint.ts`](../../packages/core/src/engine/checkpoint.ts) ~26–34, 156–158). Applied naively to an async-media node, **resume re-calls `generateMedia`** — paying for and orphaning a *fresh* Sora/Veo LRO on every restart while the original keeps generating provider-side. The acceptance ("survives a simulated restart, resumes to completion") is unmeetable without a durable jobId + a re-attach. +2. **`pollMediaJob(jobId, key)` has no `signal` param** ([`types.ts`](../../packages/llm/src/types.ts) ~475) — as shaped, a cancel can only stop scheduling the *next* poll; the in-flight provider HTTP poll leaks past cancel (and may resolve a media part with no run-ref → a CAS orphan the sweep cannot reclaim). (`generateMedia` already carries `signal` on its request, `types.ts` ~416.) +3. **`MediaJobStatus.done` carries a `MediaPart` but no `usage`/`mediaUnits`** ([`types.ts`](../../packages/llm/src/types.ts) ~450) — the realized-cost channel on completion is absent; a generate node also **bypasses the agent-turn `CostTracker`/`PreEgressHook`** (it is not a chat turn), so without explicit handling the minute-scale paid job is neither gated pre-creation nor folded into `cumulativeCostMicrocents` — a silent billing/metering regression. +4. **`content_filter` is not an `ErrorCode`.** `ERROR_CODES` ([`constants.ts`](../../packages/shared/src/constants.ts) ~74–88) lacks it though the `LlmErrorKind` enum has it ([`types.ts`](../../packages/llm/src/types.ts) ~234); `codeForLlmError` collapses `content_filter` → `validation` ([`agent-turn.ts`](../../packages/core/src/engine/agent-turn.ts) ~189–191). The acceptance literally requires "a content-policy job failure maps to `content_filter`." And `model_catalog.media_surface` (the inline-vs-generative discriminator the contract names) does not exist as a column yet ([`schema.ts`](../../packages/db/src/schema.ts) ~94–134). + +These sit on the seam/durability/cost boundaries every surface (CLI, desktop Rust, managed gateway) re-implements, so they warrant a recorded decision rather than an implementer's guess. + +## Decision + +**We record the engine-owned async media-job loop for 1.AG as the pinned choices below, as an amendment to (not a supersession of) [ADR-0031](0031-llm-seam-shape-amendment-multimodal-io.md), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md), and [ADR-0040](0040-node-retry-budget-above-the-chain.md).** The 1.AF storage/egress/cost substrate is reused unchanged. + +### 1. Inline-vs-generative routing is data-driven by `model_catalog.media_surface`; the trigger is the agent node, not a new node type + +We add a strict **`model_catalog.media_surface` enum (`'chat' | 'generative'`, default `'chat'`)** (a net-new column + migration + a `CapabilityFlags` projection, mirroring how the 1.AF media-rate columns project onto `ModelPricing`). An **agent node** whose resolved model is `media_surface: 'generative'` routes to `generateMedia`; a `'chat'` model uses the normal turn with `output_modalities`. The author-controlled volume knobs (`count`, `durationSeconds`) ride the existing **`agent_config`** node block (no new top-level node field) → mapped to `MediaGenRequest` at dispatch; their canonical authored home is recorded in `workflow-yaml-spec.md`/`node-types.md`. No model id or endpoint is ever hardcoded (resilient to Sora/Veo/gpt-image renames). *(Considered: a dedicated `generate_media` authored node type with first-class `count`/`duration_seconds` — **rejected** (maintainer call): it freezes a new public YAML schema for a routing distinction the model capability already carries; the agent-node arm keeps the authored contract minimal and surface-driven. The dedicated type stays available as a later additive option if author ergonomics demand it.)* + +### 2. The durable async-job state is a `media_job:submitted` run-event folded into a derived `pendingMediaJobs` checkpoint slot — **no `media_jobs` table** + +A submitted async job emits a **durable `media_job:submitted` run-event** through the existing single `#emitDurable` choke point (so the gap-free monotone `sequenceNumber` + persist-before-deliver hold — [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) is unchanged). It carries `{ jobId, provider, model, modality, nodeId, startedAt, deadlineAt }` (the `jobId` is the Relavium-opaque id; `provider` the bound `LlmProviderId` — these are the wire field names the canonical `sse-event-schema.md` declares). `reconstructCheckpointState` folds it into a **new `pendingMediaJobs` `CheckpointState` slot** (modeled exactly on `pendingGates`): a `media_job:submitted` adds the descriptor **keyed by `nodeId`** (one in-flight job per node — a node-retry re-dispatch (§3) emits a fresh `media_job:submitted` whose descriptor **replaces** the node's entry, latest-submitted wins; the abandoned prior job, being pre-`done`, has materialized **no** bytes and thus no CAS handle, so it leaves no orphan), and the node's **terminal event** (`node:completed | node:failed | node:skipped`) clears it. A media node cannot become `skipped` *after* a submit (skip-propagation precedes execution), but the clear-set includes it defensively so no `pendingMediaJobs` entry can leak. **Per-poll progress** (`MediaJobStatus.pending.progress`) is **transient** — it rides the streamed SSE/`StreamChunk` channel and is **never persisted** (a durable per-poll event would flood `run_events` with dozens of non-state-bearing rows per job and perturb the fold). *(Considered: a mutable `media_jobs` table — **rejected**: it is a SECOND undeclared ADR-0003 divergence; `media_objects`/ADR-0042 was the justified first, and only for **retention** — job lifecycle is **run** state and belongs to the derived model with every other run-state concern. A durable progress event — **rejected**: non-state-bearing flood.)* + +### 3. Resume **re-attaches** (re-polls the persisted jobId); it never re-submits. The run **parks** like a gate; the poll is host-`SetTimer`-driven + +On resume, a node with a `pendingMediaJobs` entry **re-polls its persisted opaque jobId** via `pollMediaJob`; it **NEVER re-calls `generateMedia`** (which double-bills + orphans the first vendor LRO). This **explicitly overrides the checkpoint model's "running-at-crash vertex is omitted → re-runs from pending" default** ([ADR-0003](0003-pure-ts-engine-not-langgraph-python.md), [`checkpoint.ts`](../../packages/core/src/engine/checkpoint.ts); the behavior [ADR-0040](0040-node-retry-budget-above-the-chain.md) A.6 quotes) — *for the async-media node specifically*, the single load-bearing divergence. **A re-attach is not a retry dispatch:** it consumes no node-retry budget and emits **no new `node:started`** (the node resumes its one suspended execution, like a gate resume), so ADR-0040 A.6's "retry count resets to 0 on crash-resume" is left untouched (a re-attach is not an attempt). **The deadline short-circuits a doomed re-poll:** if on resume `now > deadlineAt`, the engine does **not** re-poll (the provider has surely dropped the job) and goes straight to the deadline path below. **A lost/expired jobId** (the provider no longer knows it) or a **deadline-exceeded** surfaces as a `timeout` `LlmError`, which `codeForLlmError` already maps to the **retryable `provider_unavailable`** `ErrorCode` ([`agent-turn.ts`](../../packages/core/src/engine/agent-turn.ts) ~184–186; `provider_unavailable` is in `RETRYABLE_ERROR_CODES`) — so the node-retry budget ([ADR-0040](0040-node-retry-budget-above-the-chain.md)) **may** re-dispatch the node (a *fresh* paid submit, under its `retry.max`/`retry_on`); the loop itself never silently re-submits, and a node-retry re-dispatch is a new attempt with its own `media_job:submitted` (§2). While a job is pending the run **parks** (the node suspends and the scheduler stays free, so siblings proceed and `#handleIdle` can pause the run), reusing the human-gate suspend/resume machinery; the poll cadence is driven by the injected **`ExecutionHost` one-shot `setTimer`** port (re-armed per poll, abort-aware — the same pattern as `#abortableSleep` and the gate timeout), **never** an ambient `setTimeout`/sleep (engine purity). *(Considered: holding a `max_parallel` slot for the whole minute-scale job — **rejected**: it serializes sibling branches and the run never goes idle, a deep `#claimReady`/`#countRunning` scheduler re-plumb; the gate-like park reuses proven machinery.)* + +### 4. `pollMediaJob` gains a `signal` param; cancel is best-effort **local** abort + the **one** terminal sweep + +We **additively amend the reserved [ADR-0031](0031-llm-seam-shape-amendment-multimodal-io.md) shape**: `pollMediaJob(jobId, key, signal?: AbortSignalLike)`. Because the method is still un-implemented, this is cheap-additive now; adding it after 1.AG ships would be a seam amendment to a live method. A run cancel **aborts the in-flight `pollMediaJob`** via the signal (not merely the next schedule) → emits `run:cancelled` → triggers the **terminal-state media sweep** (`#reclaimRunMedia`, [ADR-0042](0042-engine-media-storage-substrate-mediastore-deinline-retention.md)). An async LRO holds **no artifact until completion**, so a cancel *before* `done` has materialized no bytes — there is nothing to reclaim beyond the run-ref bookkeeping. The one byte-bearing case is the **race** where `pollMediaJob` has just returned `done` and its bytes were de-inlined to a handle but the terminal `run:cancelled` has not yet emitted: that handle's run `run`-ref is recorded **before** the terminal emit, so the sweep reclaims it and the race leaves **no CAS orphan**; the sweep removes only the run's `run`-kind refs and never double-frees a handle a `session`/`workspace` scope still references (ADR-0042 §4). **No vendor-side job cancel is attempted** (maintainer call): the Sora/Veo job keeps generating + billing provider-side and its output is discarded — a recorded cost-integrity ([ADR-0014](0014-managed-metering-quota-and-billing.md)) caveat. *(Considered: a net-new `cancelMediaJob?` seam method calling each vendor's cancel endpoint — **rejected for 1.AG**: not every provider offers one, it adds per-adapter vendor-specific surface, and the local abort + sweep is sufficient; it can be added additively later if cost-sensitivity demands it.)* + +### 5. Cost: a pre-egress estimate **before** job creation (gate only) + a single realized addend at `done`, through the 1.AF governor, **derived from the request** (no frozen-shape touch) + +The generate path is **not exempt** from the [ADR-0044](0044-media-access-governance-read-media-save-to-cost.md) cost governor. **Before** `generateMedia` is called, the **`PreEgressHook` gates the minute-scale paid job**: the per-modality estimate (`count` for image, `durationSeconds` for audio/video — `count × durationSeconds` for a generator that takes both — × the per-model media rate, the [ADR-0044](0044-media-access-governance-read-media-save-to-cost.md) formula) is **checked against the `max_cost_microcents` cap only** (`fail`/`pause`/`warn` *before* the spend is committed) — it is **never added to `cumulativeCostMicrocents`**. The running total gains **exactly one addend per job**: at `done` the realized media units fold in as the disjoint addend ([ADR-0044](0044-media-access-governance-read-media-save-to-cost.md) §3), **derived from the same request volume** (a deterministic generator produces what was requested). On a **cancel / fail-after-paid-submit**, that single addend is the **estimate** (the provider bills regardless — the maintainer's cancel-cost expectation). Because the estimate only gates and the realized (or estimate-on-cancel) is the lone addend, there is no estimate+realized double-count within a node; the resume `Math.max` cumulative fold (which reconciles two durable run snapshots, not within-node figures) is preserved. *(Considered: adding `usage`/`mediaUnits` to `MediaJobStatus.done` — **rejected**: the engine already knows the requested volume and a deterministic generator's billed units equal the request, so deriving avoids touching the frozen union; a provider that reports *true* usage can refine the realized figure later, additively.)* + +### 6. `content_filter` becomes a first-class `ErrorCode`; an in-flight job gets **no** cross-provider failover; the poll is carved out of deterministic replay + +We **add `content_filter` to the closed `ERROR_CODES` union** (`constants.ts`) + the `sse-event-schema.md` error taxonomy + `error-handling.md`, classified **fatal** (not in `RETRYABLE_ERROR_CODES`); `codeForLlmError` maps the `content_filter` `LlmErrorKind` to it (it currently collapses to `validation`). A closed-union add now is cheaper than retrofitting every exhaustive `switch` later. (`content_filter` already names a `StopReason` and an `LlmErrorKind`; this adds the third sense, the `ErrorCode` — three distinct concerns, one name, no collision.) The **async generative path does NOT participate in `FallbackChain` cross-provider failover**: a jobId is provider-bound (failover cannot re-poll it; a re-submit would re-bill), so provider selection happens **once, pre-submit** (the existing skip/select runs before `generateMedia`); once a jobId exists the run is committed to that provider for the job's life. The wall-clock poll cadence is **explicitly carved out of [ADR-0040](0040-node-retry-budget-above-the-chain.md) A.3's deterministic-replay invariant** — a job result is external non-deterministic state already (the no-jitter backoff rule governs retry scheduling, not an external-completion poll). + +### 7. Reuse + purity: the one SSRF egress, the opaque jobId, strip-raw for the new result types, managed-mode poll-through agnostic; the cadence as strict config + +A **provider output URL** (DALL-E/CDN, async-fetch URL) is placed in a durable position and re-hosted to a handle by `deInlineMedia` → the **one [ADR-0043](0043-media-egress-failover-rematerialization-ssrf.md) `fetchMediaBytes`** SSRF-validated egress — **never a second fetch site** (no re-implemented pinning/redirect TOCTOU). The **opaque jobId** is adapter-minted and never the vendor operation-name/poll-URL (I1/[ADR-0011](0011-internal-llm-abstraction.md)); the adapter holds the vendor↔opaque map internally. The **strip-raw discipline extends to the two new seam result types** — `MediaGenResult.raw` and the diagnostics behind `MediaJobStatus` (vendor `b64_json`/`inlineData` live inside `z.unknown()` that refines cannot see) are stripped at every sink/log/checkpoint exactly as `LlmResult.raw`/`LlmError.cause` are; the `done`-arm `MediaPart` de-inlines at the choke point. The **poll loop is provider-impl-agnostic** so managed mode ([ADR-0015](0015-managed-mode-data-handling-and-compliance.md)) swaps the `pollMediaJob` body (gateway poll-through, counts-not-content; the gateway materializes nothing — the engine de-inlines the `done` `MediaPart` to the user's local store, the Phase-2/1.AH reconciliation) without an engine change. The cadence is **strict `[defaults]` config keys** (distinct from `[defaults].media_cost_estimate`): `media_job_poll_initial_ms` (**5 000**), `media_job_poll_max_ms` (**30 000**), `media_job_deadline_ms` (**1 800 000** — 30 min). The **first poll fires at `submit + initial`**; poll *n*'s interval is `min(initial × 2^(n-1), max)` (exponential, no jitter, capped) — until `done`/`failed` or **`deadlineAt`** (computed at submit as `startedAt + media_job_deadline_ms`, carried on the `media_job:submitted` event), at which the job is abandoned as the retryable `timeout` of §3/§6. + +## Consequences + +### Land-time obligations (`@relavium/shared` + contracts, all in the 1.AG PR) + +- **`@relavium/shared`** — `run-event.ts`: a new `media_job:submitted` member on the `RunEvent` discriminated union + the count-test; `constants.ts`: the new `content_filter` `ErrorCode` + `RUN_EVENT_TYPES` entry; `media-deinline.ts`/content schemas: the strip-raw discipline extended to `MediaGenResult.raw` + `MediaJobStatus`. +- **`@relavium/llm`** — `types.ts`: `pollMediaJob(jobId, key, signal?)` (additive); the `CapabilityFlags` `media_surface` projection; `cost-tracker.ts`/`pricing.ts`: the generate-path realized fold. +- **`@relavium/core`** — `checkpoint.ts`: the `pendingMediaJobs` `CheckpointState` slot + its fold/clear; `engine.ts`/`agent-turn.ts`/`agent-runner.ts`: the `media_surface` routing, the LRO (submit/poll/park/re-attach/cancel), the `PreEgressHook` on the generate path, `codeForLlmError(content_filter) → content_filter`. +- **`@relavium/db`** — `schema.ts` + a new Drizzle migration: `model_catalog.media_surface` (`'chat' | 'generative'`, default `'chat'`); the 1.AG migration/seed sets `'generative'` for the generative models (gpt-image-1, Imagen, OpenAI TTS, Sora, Veo). +- **Canonical-home docs** — `llm-provider-seam.md` (reserved → wired + the final `pollMediaJob` signature), `database-schema.md` (`media_surface`), `sse-event-schema.md` (`media_job:submitted`, the transient progress note, the `content_filter` taxonomy entry), `config-spec.md` (the three `media_job_*` defaults), `workflow-yaml-spec.md`/`node-types.md` (the `media_surface`-routed agent-node arm + `agent_config.count`/`durationSeconds`). + +### Positive + +- The paid async LRO is **resumable, cancellable, and cost-governed** without leaving the ADR-0003 derived model: a `media_job:submitted` event + a derived `pendingMediaJobs` slot makes re-attach-not-restart possible, closing the single costliest trap (a restart that re-bills + orphans a fresh vendor job every time). +- The 1.AF substrate (CAS, `deInlineMedia` choke point, SSRF egress, terminal sweep, refcount, cost governor) is **reused, not re-plumbed** — 1.AG adds the loop + the adapters + one column + one event + one error code, and the I3/seam/purity invariants are inherited intact. +- The seam shape stays minimal: one additive `signal` param on a still-reserved method, no `cancelMediaJob`, no `usage` on the frozen union, no new node type — the public/authored surface barely grows, and what grows is additive and frozen-forward-compatible. +- `content_filter` becomes honest (a content-policy block reports the right cause/remediation, meeting the acceptance), and managed mode is not foreclosed (the poll loop is impl-agnostic; cost is counts-not-content). + +### Negative + +- A new non-terminal **suspension state** + a new durable event family is real run-loop surface to own and test (mitigated: it models `pendingGates` exactly, settles through the one `#emitDurable` choke, and the resume/cancel/restart edge cases get explicit acceptance tests — restart-mid-poll, cancel-mid-poll, expired-jobId, double-resume). +- A restart (or pause) that **outlasts the job deadline** (`now > deadlineAt` on resume) cannot re-attach — it fails over to a **fresh paid submit** via the node-retry budget (the "resumes to completion" acceptance then holds via a *new* job, not the original; the original's provider-side spend is the local-only-cancel caveat below). The 30-min default keeps this rare for image/audio/short-video. +- A **local-only cancel** leaves the vendor job running + billing provider-side (its output discarded) — a real cost-integrity caveat ([ADR-0014](0014-managed-metering-quota-and-billing.md)) accepted for 1.AG (mitigated: recorded; a vendor `cancelMediaJob` is an additive later option; the user-facing cost report shows the paid estimate so the spend is never silently hidden). +- The realized cost is **derived from the request**, not a provider-reported usage figure — accurate for deterministic generators but an approximation if a provider over/under-produces (mitigated: a provider that reports true usage can refine the fold additively without a contract change). +- **Three amended ADRs** (0031 shape, 0036 state set, 0040 re-attach + replay carve-out) mean future readers of any of those must also read this (mitigated: the amendments are append-only and each is narrow and named here, with inline notes per the obligation below). + +### Neutral + +- Per ADR-0009's append-only rule, ADR-0031/0036/0040 are unchanged in history; this ADR is the authoritative record for the async media-job loop, the `media_job:submitted` durable state, the `pollMediaJob` signal param, the inline-vs-generative routing, and the generate-path cost realization. **On acceptance, this lands a dated `> Amended … by [ADR-0045]` note on ADR-0031, ADR-0036, and ADR-0040** (a documentation pointer, not a status flip — [documentation-style.md](../standards/documentation-style.md) §7, the same precedent as ADR-0042's note on ADR-0036 and ADR-0040's on ADR-0038). Future readers of the seam, the run loop, or node retry should read those **and** this. +- The canonical-home updates land with the 1.AG code (enumerated under *Land-time obligations* above) — none restated here. diff --git a/docs/decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md b/docs/decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md new file mode 100644 index 00000000..728176d9 --- /dev/null +++ b/docs/decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md @@ -0,0 +1,43 @@ +# ADR-0046: Inline media-out routes through `generate()`; the streaming media triad is host-deferred (amends ADR-0031, ADR-0038) + +- **Status**: Accepted +- **Date**: 2026-06-20 +- **Related**: [0031-llm-seam-shape-amendment-multimodal-io.md](0031-llm-seam-shape-amendment-multimodal-io.md) (**this ADR amends it** — pins which of §5.1's two inline-media-out delivery paths Phase-1 (1.AG) uses; ADR-0031's seam shape is unchanged), [0038-agentrunner-llm-call-boundary.md](0038-agentrunner-llm-call-boundary.md) (**amends** — the per-node turn loop, stream-only today, gains a single-shot `generate()` branch for a media-output turn), [0042-engine-media-storage-substrate-mediastore-deinline-retention.md](0042-engine-media-storage-substrate-mediastore-deinline-retention.md) (the `deInlineMedia` choke point this reuses to turn the in-flight media into a handle), [0043-media-egress-failover-rematerialization-ssrf.md](0043-media-egress-failover-rematerialization-ssrf.md) (the input-side `resolveForEgress` host-hook precedent the deferred streaming path would mirror), [0032-desktop-rust-media-de-inline-amends-0018.md](0032-desktop-rust-media-de-inline-amends-0018.md) (the desktop Rust CAS that owns the deferred streaming-out de-inline), [0045-async-media-job-loop-poll-checkpoint-resume-cancel.md](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) (the 1.AG sibling — separate-endpoint generators; this is the inline half), [../analysis/multimodal-io-design-2026-06-07.md](../analysis/multimodal-io-design-2026-06-07.md) (§5.1). + +## Context + +1.AG Section B wires **inline media-out** — a `media_surface: 'chat'` model returning media *in the chat turn* (Gemini `responseModalities` image/audio, OpenAI inline audio, OpenAI agentic image-gen). [ADR-0031](0031-llm-seam-shape-amendment-multimodal-io.md) §5.1 lists **two** delivery paths and calls non-streaming "**the more common of the two**", but does not pin which one Phase-1 implements. The implemented seam makes the choice load-bearing: + +- The per-node agent turn ([`agent-turn.ts`](../../packages/core/src/engine/agent-turn.ts)) consumes **`chain.stream()` only**; it folds `text` / `tool_call` / `reasoning` chunks into the result — there is no media path in the turn loop, and the turn **never invokes the `FallbackChain`'s existing `generate()`** (present since 1.K, `fallback-chain.ts`): the turn loop is stream-only today. +- `StreamChunk.media_end` is **handle-only** (`DurableMediaPartSchema`, 1.AD-frozen) and the adapter is **pure** (no `MediaStore`). So a Node adapter that receives the provider's base64 media in its stream has **no way to deliver it** — it cannot emit base64 on a `StreamChunk` (none of the media arms carry base64), and it cannot mint a handle (no store). The input-side de-inline (`resolveForEgress`) is called by the **chain before** the adapter ([ADR-0043](0043-media-egress-failover-rematerialization-ssrf.md) §1); there is no symmetric output-side hook. + +So the streaming media triad cannot be driven on Node without a new host-injected de-inline hook reaching the adapter (an output twin of `resolveForEgress`) — a `LlmProvider.stream` signature change to a 1.AD-frozen seam. The non-streaming `generate()` already returns `LlmResult.content`, whose in-flight `MediaPartSchema` **permits base64**, which the engine already de-inlines at the one `#emitDurable` choke point ([ADR-0042](0042-engine-media-storage-substrate-mediastore-deinline-retention.md)). The choice — change the frozen stream seam, or route through the existing `generate()` — is the maintainer's (decided 2026-06-20). + +## Decision + +**We will deliver Phase-1 (1.AG) inline media-out through the non-streaming `generate()` path; the streaming media triad stays reserved/deferred to the host (1.AH).** + +1. **A media-output agent turn calls `chain.generate()`, not `chain.stream()`.** When a turn's resolved model is `media_surface: 'chat'` **and** the node requests media `output_modalities`, the engine issues a single-shot `generate()` whose `LlmResult.content` carries an **in-flight `media` `ContentPart`** (base64). The engine de-inlines it to a `media://sha256-…` handle at the existing `#emitDurable` choke point (the 1.AF `deInlineMedia` pass) on `node:completed.output` — **no new seam shape, no new host hook**. A text-only turn keeps streaming exactly as today. +2. **The agent turn gains a `generate()` routing branch — it calls the `FallbackChain`'s _existing_ `generate()`** (present since 1.K, never invoked by the stream-only turn loop until now), not a new chain method. `generate()` already reuses the same provider skip/select, cross-provider failover, per-attempt cost (`CostTracker`/`PreEgressHook`), `LlmError` classification, **and same-provider reasoning replay** (`beginEntry`, [ADR-0039](0039-same-provider-reasoning-replay.md)) as `stream()`, in one round-trip — so a media-output turn inherits the full chain policy unchanged. *(Considered: drain `stream()` into a synthesised result rather than calling `generate()` — rejected: the media never rides a `StreamChunk`, so a drained stream yields no media; `generate()` is the seam method that returns `LlmResult.content`.)* +3. **The adapter parses provider media output into an in-flight `media` part** (base64), the engine de-inlines it: Gemini `inlineData` → a `media` `ContentPart`; OpenAI inline audio → a `media` part **plus** the transcript text; OpenAI agentic image-gen → a `providerExecuted: true` `tool_result` carrying a **normalized** `media` part (ADR-0031 §4.3/#7). No vendor shape escapes the seam (I1); `LlmResult.raw` is strip-discarded as ever. +4. **The streaming media triad (`media_start`/`media_delta`/`media_end`) stays RESERVED/deferred to 1.AH.** Its Node de-inline needs a host hook reaching the adapter (the output twin of `resolveForEgress`) or the desktop Rust CAS ([ADR-0032](0032-desktop-rust-media-de-inline-amends-0018.md)); recorded in deferred-tasks.md. *Why acceptable:* a media-output turn is typically **terminal/single-shot** (the agent's final artifact), so token-streaming its short accompanying text is low value against a frozen-seam change; ADR-0031 §5.1 itself names non-streaming "the more common." + +The `StopReason` rule is unchanged (ADR-0031 §5.3): a media-only turn reports `'stop'`; the signal is a `media` part in `content`, never a new stop reason — consumers inspect `content`. **Additively (a refinement of ADR-0031 §5.3's content-inspection rule):** the engine compares the produced modalities against the node's authored `output_modalities` and may route/retry on a missing modality (a turn *requested* to emit an image that returned only text), rather than relying on a stop reason. + +## Consequences + +### Positive + +- No change to the 1.AD-frozen `StreamChunk`/seam shape and no new host hook in Phase 1; inline media-out reuses the proven `#emitDurable` `deInlineMedia` choke point (1.AF) — the I3 "no bytes durable" invariant is inherited intact (the handle is the only durable form). +- The `FallbackChain.generate()` method is a small, well-bounded addition that mirrors `stream()`'s policy, and the turn change is one routing branch — the streaming machinery (tool loop, reasoning, cost, failover) is untouched for the common text path. + +### Negative + +- A media-output turn does **not** stream tokens (the short accompanying text arrives whole) — accepted: media turns are typically terminal and the text is incidental. +- True streaming media-out (progressive previews via the triad) is deferred to 1.AH (the host de-inline hook / desktop Rust CAS) — recorded in deferred-tasks.md, not lost. +- A turn that both calls tools **and** emits inline media in one shot is the awkward case — the client-side tool loop is built around `stream()`, and `generate()` is a single round-trip, so a media-output turn cannot run a further tool round. 1.AG handles the terminal media-output turn (no further tool round), consistent with how media-gen is authored (Gemini `responseModalities` and OpenAI's `providerExecuted` image-gen are single-shot). + +### Neutral + +- Per ADR-0009's append-only rule, ADR-0031/0038 are unchanged in history; this ADR is the authoritative record for the inline-media-out delivery path and the streaming-triad deferral. On acceptance this lands a dated `> Amended by [ADR-0046]` note on ADR-0031 and ADR-0038 (documentation-style.md §7). +- The separate-endpoint generators (`generateMedia`/`pollMediaJob`) are the 1.AG sibling decision ([ADR-0045](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)); together they cover 1.AG output generation (inline + generative). diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 60f25d54..b00bb438 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -88,6 +88,8 @@ flowchart TD | 0042 | [Engine media storage substrate — the `MediaStore` host port, the `deInlineMedia` choke-point ordering, and the `media_objects` retention/GC store (amends 0036)](0042-engine-media-storage-substrate-mediastore-deinline-retention.md) | Accepted | 2026-06-18 | | 0043 | [Media egress — the binary media-egress capability, the `FallbackChain`↔`MediaStore` re-materialization contract, and the SSRF mechanism half (amends 0031)](0043-media-egress-failover-rematerialization-ssrf.md) | Accepted | 2026-06-18 | | 0044 | [Media access & spend governance — `read_media` scope-set authz, the byte-delivery gate, the `save_to` write port, and the per-modality media cost (amends 0028 and 0029)](0044-media-access-governance-read-media-save-to-cost.md) | Accepted | 2026-06-18 | +| 0045 | [Engine-owned async media-job loop (1.AG/A5) — the `generateMedia`/`pollMediaJob` poll/checkpoint/resume/cancel LRO, the `media_job:submitted` derived state, inline-vs-generative routing, and media-cost realization (amends 0031, 0036, 0040)](0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) | Accepted | 2026-06-20 | +| 0046 | [Inline media-out routes through `generate()`; the streaming media triad is host-deferred (amends 0031, 0038)](0046-inline-media-out-via-generate-streaming-triad-deferred.md) | Accepted | 2026-06-20 | ## Creating a new ADR diff --git a/docs/reference/contracts/config-spec.md b/docs/reference/contracts/config-spec.md index f8cbb9dd..2025239c 100644 --- a/docs/reference/contracts/config-spec.md +++ b/docs/reference/contracts/config-spec.md @@ -83,6 +83,9 @@ autostart = true model = "claude-sonnet-4-6" # default model for agents that omit one fs_scope = "sandboxed" # sandboxed | project | full (see filesystem tiers) max_tokens_estimate = 4096 # per-call output-token estimate the pre-egress budget governor uses when a node/session omits maxTokens (ADR-0028) — not the model's absolute max, which would over-block +media_job_poll_initial_ms = 5000 # async media-job (generateMedia LRO) first-poll delay + backoff base (1.AG/ADR-0045 §7) +media_job_poll_max_ms = 30000 # backoff cap: poll interval = min(initial × 2^(n-1), max), no jitter +media_job_deadline_ms = 1800000 # abandon a job past this (from submit) as a retryable timeout (30 min) [defaults.media_cost_estimate] # per-modality media-output UNIT-COUNT default for the pre-egress media cost estimate (1.AF/D17, ADR-0044 §3) — a COUNT, not a price; the per-unit price lives in the model catalog. Used when a media-output turn declares no volume. Omit the table for text-only workflows. image = 1 # assumed images per media-output turn diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index 32358e70..ff6a2a43 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -52,6 +52,7 @@ export type RunEvent = | NodeFailedEvent | NodeSkippedEvent | NodeRetryingEvent + | MediaJobSubmittedEvent | HumanGatePausedEvent | HumanGateResumedEvent | RunCompletedEvent @@ -73,14 +74,15 @@ export type RunEvent = | `agent:tool_call` | An agent invoked a tool. | `nodeId`, `model` (the invoking model — so a tool call is attributable across a failover), `toolId`, `toolInput` (sanitized — no secrets), `attemptNumber?` (1-based, matches `cost:updated`) | | `agent:tool_result` | A tool returned. | `nodeId`, `toolId`, `success`, `outputSummary` (truncated for UI), `attemptNumber?` | | `agent:file_patch_proposed` | An agent proposed a file change (**gated — no write until the user accepts**; e.g. the VS Code inline-diff review). | `nodeId`, `patches: [{ uri, unifiedDiff }]` (≥1 — an empty proposal is meaningless), `attemptNumber?` | -| `cost:updated` | A node's token cost was tallied (drives the cost waterfall). | `nodeId`, `model`, `inputTokens`, `outputTokens`, `costMicrocents`, `cumulativeCostMicrocents` (integer micro-cents — canonical unit in [llm-provider-seam.md](../shared-core/llm-provider-seam.md#6-usage); **includes realized media spend**, folded as a disjoint addend per [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3 — the per-unit `Usage.mediaUnits` axis is **not yet a field on this event**, deferred, see [deferred-tasks.md](../../roadmap/deferred-tasks.md)), `attemptNumber?` (1-based **within-chain** FallbackChain attempt — resets per node-retry re-dispatch; **distinct** from `node:*.attemptNumber`, see the [two attemptNumber families](#two-attemptnumber-families) note) | +| `cost:updated` | A node's token cost was tallied (drives the cost waterfall). | `nodeId`, `model`, `inputTokens`, `outputTokens`, `costMicrocents`, `cumulativeCostMicrocents` (integer micro-cents — canonical unit in [llm-provider-seam.md](../shared-core/llm-provider-seam.md#6-usage); **includes realized media spend**, folded as a disjoint addend per [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3 — the per-unit `Usage.mediaUnits` axis is **not yet a field on this event**, deferred, see [deferred-tasks.md](../../roadmap/deferred-tasks.md)), `attemptNumber?` (1-based **within-chain** FallbackChain attempt — resets per node-retry re-dispatch; **distinct** from `node:*.attemptNumber`, see the [two attemptNumber families](#two-attemptnumber-families) note). **Generative-node variant (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5):** a `media_surface: 'generative'` agent node emits **exactly one** `cost:updated` with `inputTokens` / `outputTokens` **= 0** (no token billing — the spend rides entirely in `costMicrocents` as the per-modality media addend) and **no `attemptNumber`** (no FallbackChain on the generative path — one provider, no failover). | | `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1) | | `node:failed` | A node failed (TERMINAL — exactly one per node; emitted when the node-retry budget is exhausted, on a fatal / `retry_on`-excluded failure, **or** when a pending retry is abandoned by a cancel or a sibling abort — see 1.S). | `nodeId`, `error: {code, message, retryable, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `correlationId` is a secret-free id joined to the internal log — ADR-0036), `attemptNumber?` (the last attempt, when a retry budget was spent — 1.S) | | `node:retrying` | A retryable node attempt failed and the engine will re-dispatch the whole node (1.S, [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)) — **non-terminal** (the node continues; `node:failed` is the terminal). | `nodeId`, `attemptNumber` (the attempt that just failed, 1-based), `error: {code, message, retryable}` (the `NodeFailure` shape — **no** `correlationId`; that anchors the terminal failure), `delayMs` (backoff before the next attempt) | | `node:skipped` | A node was skip-propagated (never ran). | `nodeId`, `reason: 'branch_not_taken' \| 'upstream_unreachable'` (`branch_not_taken` = a `condition` routed away from it; `upstream_unreachable` = every in-edge is dead because an upstream was skipped/failed). Emitted so the event log is a **complete, replayable** record — checkpoint/resume reconstructs a skipped vertex from it ([run-plan.md](../shared-core/run-plan.md)) and a surface can render the dimmed path instead of the node silently vanishing. | +| `media_job:submitted` | An async media-generation job was submitted; the engine owns its poll/checkpoint/resume/cancel loop (1.AG, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)) — **non-terminal** (the node parks until its `node:completed`/`node:failed`). **Durable** so a crash-resume re-attaches (re-polls the opaque `jobId`) instead of re-submitting; per-poll progress is **transient** (off this durable stream). | `nodeId`, `jobId` (Relavium-opaque — never the vendor op-name), `provider`, `model`, `modality: 'image' \| 'audio' \| 'video'`, `startedAt`, `deadlineAt` | | `human_gate:paused` | Execution suspended at a human gate. | `nodeId`, `gateId`, `gateType: 'approval' \| 'input' \| 'review'`, `message`, `assignee?`, `timeoutMs?`, `timeoutAction?: 'approve' \| 'reject'` (on-timeout policy, present only with `timeoutMs`), `expiresAt?` | | `human_gate:resumed` | A gate decision was applied; execution continues. | `nodeId`, `decision: 'approved' \| 'rejected' \| 'input_provided'`, `decidedBy`, `payload?` | -| `run:paused` | The run is suspended with **≥1 gate pending** — the multi-gate aggregate that backs the pending-gate queue (parallel branches may each reach a gate). | `pendingGateCount`, `gateIds[]` | +| `run:paused` | The run is suspended on **≥1 gate AND/OR ≥1 async media job** — the multi-suspension aggregate (parallel branches may each reach a gate or a media job). `pendingGateCount` is the count of `gateIds[]` (they must agree) and both are `0`/empty for a media-only park; `pendingMediaJobNodeIds` lists nodes parked on the engine-owned `pollMediaJob` loop (1.AG Section D, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §2). At least one suspension reason (a gate or a media job) always holds. A resume disambiguates by registry: a gate by `gateId` (a decision), a media job by `nodeId` (a re-attach). | `pendingGateCount`, `gateIds[]`, `pendingMediaJobNodeIds[]?` | | `run:completed` | The run finished. | `outputs` (a record **keyed by each terminal `output` vertex's node id**, the value being that vertex's captured output — see [run-plan.md §output capture](../shared-core/run-plan.md)), `totalTokensUsed`, `totalCostMicrocents` (integer micro-cents closing total for the whole run), `durationMs` | | `run:failed` | The run failed. | `error: {code, message, retryable, nodeId?, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `nodeId` is the root-cause node; `correlationId` joins to the internal log — ADR-0036), `partialOutputs` | | `run:cancelled` | The run was cancelled. | (base only) | @@ -153,6 +155,17 @@ export interface NodeRetryingEvent extends BaseEvent { delayMs: number; // backoff before the next attempt } +export interface MediaJobSubmittedEvent extends BaseEvent { + type: 'media_job:submitted'; // 1.AG/ADR-0045 §2 — an async media job was submitted; the node PARKS (non-terminal suspension). DURABLE (resume re-attaches). + nodeId: string; + jobId: string; // the Relavium-opaque job id the engine re-polls — never the vendor operation-name (ADR-0011 I1) + provider: 'anthropic' | 'openai' | 'gemini' | 'deepseek'; // the bound LlmProviderId (closed z.enum(LLM_PROVIDERS); failover does not apply to an in-flight job) + model: string; // canonical model id + modality: 'image' | 'audio' | 'video'; + startedAt: string; // ISO-8601 submit time + deadlineAt: string; // ISO-8601 = startedAt + [defaults].media_job_deadline_ms; on resume now > deadlineAt short-circuits a doomed re-poll +} + export interface HumanGatePausedEvent extends BaseEvent { type: 'human_gate:paused'; nodeId: string; @@ -279,9 +292,9 @@ These three (and `run:paused` / `human_gate:paused`) are **non-terminal** — th `node:failed.error.code` and `run:failed.error.code` are a closed **`ErrorCode`** enum (not a free string), so surfaces can branch on cause and `retryable` is unambiguous: -`validation` · `provider_auth` · `provider_rate_limit` · `provider_unavailable` · `tool_denied` · `tool_failed` · `budget_exceeded` · `run_timeout` · `turn_limit` · `cancelled` · `sandbox_error` · `internal` +`validation` · `content_filter` · `provider_auth` · `provider_rate_limit` · `provider_unavailable` · `tool_denied` · `tool_failed` · `budget_exceeded` · `run_timeout` · `turn_limit` · `cancelled` · `sandbox_error` · `internal` -The retryable/fatal mapping is owned by [error-handling.md](../../standards/error-handling.md) (e.g. `provider_rate_limit`/`provider_unavailable` retryable; `provider_auth`/`validation`/`tool_denied`/`turn_limit`/`cancelled` fatal). `turn_limit` is the limit-family code for a **hard** agent/session turn/round cap (the exact knob is settled with `AgentSession`, 1.V) — distinct from `run_timeout`/`budget_exceeded` so a capped conversation surfaces its own cause rather than a silent stop; continuing past it is an explicit user action, never a retry. It is **not** the `[chat].max_messages` knob, which is a session-history **trim** threshold ([config-spec.md](config-spec.md)) — trimming continues the session and emits no error. Messages remain user-safe and secret-free. +The retryable/fatal mapping is owned by [error-handling.md](../../standards/error-handling.md) (e.g. `provider_rate_limit`/`provider_unavailable` retryable; `provider_auth`/`validation`/`content_filter`/`tool_denied`/`turn_limit`/`cancelled` fatal). `content_filter` is a provider content-policy rejection (text or media generation) — a fatal cause distinct from `validation` (an authoring/shape error), so a surface shows the right reason; the `content_filter` `LlmErrorKind` maps here (1.AG, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §6). `turn_limit` is the limit-family code for a **hard** agent/session turn/round cap (the exact knob is settled with `AgentSession`, 1.V) — distinct from `run_timeout`/`budget_exceeded` so a capped conversation surfaces its own cause rather than a silent stop; continuing past it is an explicit user action, never a retry. It is **not** the `[chat].max_messages` knob, which is a session-history **trim** threshold ([config-spec.md](config-spec.md)) — trimming continues the session and emits no error. Messages remain user-safe and secret-free. ## Forward-compatibility diff --git a/docs/reference/contracts/workflow-yaml-spec.md b/docs/reference/contracts/workflow-yaml-spec.md index a5834e68..fba624a9 100644 --- a/docs/reference/contracts/workflow-yaml-spec.md +++ b/docs/reference/contracts/workflow-yaml-spec.md @@ -210,6 +210,8 @@ Each node has an `id` (kebab-case, unique within the workflow) and a `type`. The required: [score] properties: { score: { type: number } } output_modalities: [text, image] # optional (1.AF): non-text output to request; OUTPUT_MODALITIES vocab + count: 1 # optional (1.AG Section C): images per call — generative model only + duration_seconds: 8 # optional (1.AG Section C): audio/video length — generative model only timeout_ms: 60000 retry: { max: 3, backoff: linear } # linear | exponential ``` @@ -223,7 +225,7 @@ Each node has an `id` (kebab-case, unique within the workflow) and a `type`. The > and VS Code completion ([ADR-0023](../../decisions/0023-strict-authored-yaml-validation.md)). A > `condition` node selects a branch rather than producing shaped data, so it carries no `output_schema`. -> **`output_modalities` (1.AF, [ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md)).** An optional **non-empty** array of the non-text output a turn requests, lowered to `LlmRequest.outputModalities`. The member vocabulary is `OUTPUT_MODALITIES` = `text` | `image` | `audio` | `video`; `document` is **rejected** (PDF is input-only, never a chat-turn output), and an **empty array is rejected** (omit the field for the default, text-only). Beyond the vocabulary check at parse, the requested set is validated at **load time** against the resolved model's `media.outputCombinations` **membership** — an incapable model or a wire-invalid combination is a load-time error, never a silent runtime drop (the catalog-aware engine-loader pass, since the YAML schema has no model catalog). Generated bytes ride the seam as a **handle**; the surface writes them only via an output node's `save_to` (below). +> **`output_modalities` (1.AF, [ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md)).** An optional **non-empty** array of the non-text output a turn requests, lowered to `LlmRequest.outputModalities`. The member vocabulary is `OUTPUT_MODALITIES` = `text` | `image` | `audio` | `video`; `document` is **rejected** (PDF is input-only, never a chat-turn output), and an **empty array is rejected** (omit the field for the default, text-only). Beyond the vocabulary check at parse, the requested set is validated at **load time** against the resolved model's `media.outputCombinations` **membership** — an incapable model or a wire-invalid combination is a load-time error, never a silent runtime drop (the catalog-aware engine-loader pass, since the YAML schema has no model catalog). Generated bytes ride the seam as a **handle**; the surface writes them only via an output node's `save_to` (below). **Inline delivery (1.AG Section B, [ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) §1).** When `output_modalities` requests media for a `media_surface: 'chat'` model, the turn is a single-shot `generate()` with **no tool access** — see [ADR-0046 §1](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) for the de-inlining of the in-turn media to handles. `output_schema` is **not applied** (the media is the artifact, not JSON-validated text), and a turn that requests media but **produces none** fails `validation` (never a silent text completion). **Generative routing (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §1).** The inline path above is for a `media_surface: 'chat'` model. A model the catalog marks **`media_surface: 'generative'`** (gpt-image-1, Imagen, OpenAI TTS) routes the agent node to the separate-endpoint **`generateMedia`** instead — `output_modalities` must then declare **exactly one** media modality (no text), and the optional **`count`** (images per call) / **`duration_seconds`** (audio/video length) knobs (inline on the agent node) set the requested volume → priced pre-egress + folded once at the realized cost. The node output is pure media (`{ media: [...] }`, no chat text); the engine de-inlines it to a handle exactly as the inline path. ### `human_gate` node diff --git a/docs/reference/desktop/database-schema.md b/docs/reference/desktop/database-schema.md index 7434217f..761adaa4 100644 --- a/docs/reference/desktop/database-schema.md +++ b/docs/reference/desktop/database-schema.md @@ -98,6 +98,7 @@ Models offered by each provider, including pricing used for local cost tracking. | `media_image_cost_microcents` | INTEGER | NULL — µ¢ per output image (1.AF/D17) | | `media_audio_cost_microcents` | INTEGER | NULL — µ¢ per output audio-second | | `media_video_cost_microcents` | INTEGER | NULL — µ¢ per output video-second | +| `media_surface` | TEXT | NOT NULL DEFAULT `'chat'` — `'chat'` \| `'generative'`; routes an agent node to the normal turn vs `generateMedia()` (1.AG/ADR-0045 §1) | | `supports_tool_calling` | INTEGER (bool) | NOT NULL DEFAULT 0 | | `supports_vision` | INTEGER (bool) | NOT NULL DEFAULT 0 | | `supports_streaming` | INTEGER (bool) | NOT NULL DEFAULT 1 | diff --git a/docs/reference/shared-core/agent-runner.md b/docs/reference/shared-core/agent-runner.md index 5b36ad1a..747b725b 100644 --- a/docs/reference/shared-core/agent-runner.md +++ b/docs/reference/shared-core/agent-runner.md @@ -20,6 +20,7 @@ The **AgentRunner** is the single dispatching `NodeExecutor` ([ADR-0036](../../d The host injects only platform capabilities; the credential is threaded **opaquely** and is never stored, inspected, logged, persisted, or sent to the frontend by `@relavium/core` ([ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md), rule 6). - `resolveProvider(providerId): LlmProvider | undefined` — the one genuinely-new capability: the authored `Agent.provider` / `fallback_chain[].provider` are provider-**id** strings, but a `FallbackPlanEntry.provider` is a concrete adapter instance, which the engine cannot construct (vendor SDK + `@types/node` break engine purity). `undefined` ⇒ a host-wiring gap → a `NodeFailure{ code: 'internal' }`. +- `resolveMediaSurface?(model): MediaSurface | undefined` — the catalog projection of `model_catalog.media_surface` (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §1) that selects the **inline-vs-generative** dispatch. `'generative'` routes the node to the separate-endpoint `generateMedia`; `'chat'` — **the default, and the value when this dep is absent or returns `undefined`** — uses the normal turn. The engine is platform-pure (no DB), so the host injects the lookup; the production catalog wiring is deferred to 1.AH (until then every model is `'chat'` — no generative model is runtime-reachable). - `registry` + `tools` — the shared [`ToolRegistry`](tool-registry.md) (for dispatch) and its `ToolDef`s (the source of the LLM-visible schema + descriptions for the granted tools). - `keyFor` / `sleep` / `now?` / `onAuthError?` — **forwarded** into the per-node `FallbackChain` (the existing `FallbackChainOptions` seam — **not** re-declared as a parallel credential surface). `onAuthError` (the single out-of-band credential refresh) is host-owned. - `resolverCapabilities?` (the `read_file` filter for a prompt), `fsScope?` (default `'sandboxed'`), `limits?`, `preEgress?`. @@ -30,6 +31,7 @@ The runner owns the **cost path** itself — one `CostTracker` per node executio 1. **Resolve the agent.** An absent `resolvedAgent` ([run-plan.md §AgentPlanConfig](run-plan.md)) ⇒ `NodeFailure{ code: 'validation' }` naming the `agent_ref` (an authoring error — distinct from an unresolved provider id, which is `internal`). Never a raw throw. 2. **Build the fallback plan.** Primary `{ provider, model: node.model ?? agent.model, maxAttempts: (node.retry ?? agent.retry)?.max ?? 1, backoff: (node.retry ?? agent.retry)?.backoff }` (node-retry overrides the agent default) + each `fallback_chain` entry. **One `FallbackChain` per node execution**, reused across the tool loop so per-provider cooldown and the [ADR-0039](../../decisions/0039-same-provider-reasoning-replay.md) strip-latch survive. + - **Generative fork (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §1/§6).** After the plan + the resolved prompt, the primary model's surface is read via `resolveMediaSurface?.(primary.model) ?? 'chat'`. A **`'generative'`** result dispatches to the separate-endpoint `generateMedia` (**one** provider, **no** chain failover — a generative call is provider-bound) and returns early: an empty-prompt / multi-modality node fails `validation`; a pre-egress budget gate runs first (gate-only — the [ADR-0028](../../decisions/0028-workflow-resource-governance.md) pre-egress governor; the token estimate is pinned to zero for a token-free generative call, see the code comment); the SYNC `{ media }` becomes the `{ text: '', media: [part] }` node output (de-inlined to a `media://` handle like the inline path); exactly **one** realized `cost:updated` is emitted (request volume × per-model media rate, degrade-to-0 on a missing rate); a `jobId` (async LRO) is Section D. A **`'chat'`** model continues to the turn core below. 3. **Narrow the tool grant.** `node.tools` must be a **subset** of `agent.tools` — a widening attempt ⇒ `validation` ([ADR-0029](../../decisions/0029-tool-policy-hardening.md)). 4. **Assemble messages.** `system` = **authored text ONLY** (`agent.system_prompt` + `node.system_prompt_append`), concatenated **verbatim — 1.O does not interpolate the system role**, so an untrusted `{{ run.outputs }}` / `read_file` reference can never resolve into `system` (it ships as literal authored text). The parser still *collects* a `{{ … }}` reference in `system_prompt_append` for the parse-time **secret-taint** scan (catching a stray `{{ secrets.* }}`), but does not promise dispatch-time resolution of system fields. Only the resolved `prompt_template` — which may draw on untrusted `run.outputs` / `read_file` — lands in a **`user`** position, never `system` ([security-review.md §Prompt-injection](../../standards/security-review.md#prompt-injection-posture), the structural placement guarantee — no value-level taint carrier needed for an agent node, which cannot launder a secret into `run.outputs`). *(A future parse-time gate that admits trusted `{{ inputs }}` / `{{ ctx }}` in system fields while rejecting untrusted sources is a recorded follow-up.)* 5. **`output_schema` (node override wins over the agent default).** Lowered to `LlmRequest.responseFormat` (a **request-side hint**), **and** validated **node-side**: the seam's `responseFormat` does not guarantee a schema-conformant response (DeepSeek degrades to bare `json_object`), so the runner parses the output and a non-JSON result ⇒ `validation` ([ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md), [error-handling.md](../../standards/error-handling.md)). *Phase-1 scope: parse-as-JSON; deep JSON-Schema conformance is a recorded follow-up (needs a validator dependency/ADR).* @@ -48,7 +50,7 @@ The error mapping to the closed `ErrorCode` ([error-handling.md](../../standards | `UnknownToolError` / `ToolArgsInvalidError` | (model-correctable) | — | converted to an `isError` tool result fed back, within a bounded correction budget; after it ⇒ `tool_failed` | | `ToolExecutionError` | `tool_failed` | true | | | absent host capability | `internal` | false | | -| chain-exhausted `LlmError` | `provider_auth` / `provider_rate_limit` / `provider_unavailable` / `validation` (content_filter, bad_request) / `internal` (unknown) | per `LlmError.retryable` | classified from `error.kind`, never `error.message` | +| chain-exhausted `LlmError` | `provider_auth` / `provider_rate_limit` / `provider_unavailable` / `content_filter` (content_filter, 1.AG/ADR-0045 §6) / `validation` (bad_request) / `internal` (unknown) | per `LlmError.retryable` | classified from `error.kind`, never `error.message` | | max-tool-turns hit | `turn_limit` | false | | ## Events diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index a0af02bd..d5696d06 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -160,10 +160,12 @@ interface LlmProvider { generate(req: LlmRequest, key: string): Promise; stream(req: LlmRequest, key: string): AsyncIterable; readonly supports: CapabilityFlags; // { tools, streaming, parallelToolCalls, vision, promptCache, reasoning, media } — vision is the derived alias of media.input.image (ADR-0031) - // ADR-0031 decision #6 — separate-endpoint media generation, RESERVED at 1.AD (A5): the methods - // are optional shape only; behavior + the async poll/checkpoint loop's own ADR land at 1.AG. + // ADR-0031 decision #6 — separate-endpoint media generation. The A5 ADR ([ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)) + // is landed and the SHAPE is final (the additive pollMediaJob `signal` param, 1.AG Section A); the BEHAVIOR + // is WIRED — `generateMedia` SYNC de-inline (1.AG Section C) + the engine-owned async poll/checkpoint/ + // resume/cancel loop (1.AG Section D). The Sora/Veo/Imagen/TTS ADAPTER impls are 1.AH host-wiring. generateMedia?(req: MediaGenRequest, key: string): Promise; // sync → { media }; async → { jobId } (Relavium-opaque — never a vendor operation name) - pollMediaJob?(jobId: string, key: string): Promise; // pending(progress?) | done(media) | failed(LlmError) + pollMediaJob?(jobId: string, key: string, signal?: AbortSignalLike): Promise; // pending(progress?) | done(media) | failed(LlmError); signal aborts the in-flight poll (1.AG/ADR-0045 §4) } // The per-modality capability matrix (ADR-0031 decision #3). Input composability is unconstrained @@ -179,6 +181,7 @@ interface CapabilityFlags { media: { input: { image: boolean; audio: boolean; video: boolean; document: boolean }; outputCombinations: OutputModality[][]; + surface?: 'chat' | 'generative'; // media-output surface (1.AG/ADR-0045 §1); absent ⇒ 'chat'; the seam projection of model_catalog.media_surface }; } ``` @@ -361,24 +364,37 @@ managed mode) are recorded in the ADR — this section is the dry shape referenc managed-mode metering record (counts-not-content, ADR-0015). - **`LlmRequest.outputModalities`** — request non-text output on the inline path (default `['text']`), the symmetric mechanism to ADR-0030's `responseFormat`. Lowering - is per-adapter (Gemini `responseModalities`; OpenAI audio via Chat `modalities`). - **The OpenAI image-out exception:** inline image generation is the Responses - `image_generation` **built-in tool** — it routes through the `providerExecuted` - `tool_result` arm, never through `outputModalities`. + is per-adapter (Gemini `responseModalities`; OpenAI audio via Chat `modalities`+`audio`). + **Delivery path (wired at 1.AG Section B, [ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md)):** + a media-output turn issues a single-shot **`generate()`** (the chain's existing + non-streaming path) whose `LlmResult.content` carries the in-flight base64 `media` + part — the engine de-inlines it to a `media://` handle at `#emitDurable`. The + **streaming** triad stays host-deferred (ADR-0046 §4). **The OpenAI image-out + exception:** inline image generation is the Responses `image_generation` **built-in + tool** — it routes through the `providerExecuted` `tool_result` arm, never through + `outputModalities` (the Responses-API wire is deferred; the shape is defined). - **`tool_result.media: DurableMediaPart[]`** (content part **and** stream arm) — typed, handle-only media attachments. **Raw media bytes inside the opaque `result` are forbidden** so the typed guard reaches provider-executed image-gen results; `result` carries at most a descriptor. - **Optional `generateMedia?` / `pollMediaJob?` on `LlmProvider`** (decision #6; - **reserved**, A5) with `MediaGenRequest` / `MediaGenResult` / `MediaJobStatus`: a sync - generator resolves `{ media }`; an async one (Sora, Veo) resolves a **Relavium-opaque** - `jobId` (no vendor operation name crosses the seam); `failed` carries the existing - classified `LlmError` (content-policy → `content_filter`). The engine-owned - poll/checkpoint/resume/cancel loop gets its own ADR at 1.AG. No Phase-1 adapter - implements either method. + [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)) with + `MediaGenRequest` / `MediaGenResult` / `MediaJobStatus`: a sync generator resolves `{ media }`; + an async one (Sora, Veo) resolves a **Relavium-opaque** `jobId` (no vendor operation name + crosses the seam); `failed` carries the existing classified `LlmError` (content-policy → + `content_filter`). **Wired (1.AG Sections C/D):** `generateMedia` SYNC — the OpenAI adapter + implements gpt-image-1 image generation (`images.generate` → base64 `media`); a `media_surface: + 'generative'` agent node routes here instead of the inline `generate()`/`stream()` (the engine + resolves the per-model surface). The ASYNC `pollMediaJob` poll/checkpoint/resume/cancel loop is + WIRED in the engine (Section D — `media_job:submitted` park, the derived `pendingMediaJobs` slot, + re-attach-on-resume, a host-timer poll cadence, deadline→retryable-timeout, cancel→abort→terminal + sweep; failed→`content_filter`), proven against a conforming stub async provider. The remaining + work is **1.AH host-wiring**: the Sora/Veo `generateMedia(→jobId)`/`pollMediaJob` adapters, + OpenAI-TTS + Gemini-Imagen sync adapters, the per-model `media_surface` host lookup, and verified + generative pricing rows. ```ts - // RESERVED shape (A5) — deliberately minimal; behavior lands at 1.AG with its own ADR. + // Seam shape (A5; ADR-0045) — behavior WIRED at 1.AG (sync generateMedia Section C, async poll loop Section D). interface MediaGenRequest { model: string; prompt: string; diff --git a/docs/reference/shared-core/node-types.md b/docs/reference/shared-core/node-types.md index fee82bed..160b237c 100644 --- a/docs/reference/shared-core/node-types.md +++ b/docs/reference/shared-core/node-types.md @@ -116,7 +116,7 @@ In the engine's `WorkflowDefinition`, a node carries `id`, `type`, `label`, opti | Engine config block | Present when `type =` | Notable fields | | --- | --- | --- | -| `agent_config` | `agent` | `agent_ref` (resolves to the agent definition), `system_prompt_append`, `prompt_template`, `tools` (**narrows** the agent's grant — never widens, [ADR-0029](../../decisions/0029-tool-policy-hardening.md)), `model`, `temperature`, `max_tokens`, `output_schema` (optional; node override wins over the agent default; validated **node-side** on completion — Phase-1 is **parse-as-JSON only**, deep schema conformance deferred — [ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md), [agent-runner.md](agent-runner.md)), `timeout_ms`, `retry` (fields are inline on the `agent` node — `AgentNodeSchema`; there is no nested `agent_config` object) | +| `agent_config` | `agent` | `agent_ref` (resolves to the agent definition), `system_prompt_append`, `prompt_template`, `tools` (**narrows** the agent's grant — never widens, [ADR-0029](../../decisions/0029-tool-policy-hardening.md)), `model`, `temperature`, `max_tokens`, `output_schema` (optional; node override wins over the agent default; validated **node-side** on completion — Phase-1 is **parse-as-JSON only**, deep schema conformance deferred — [ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md), [agent-runner.md](agent-runner.md)), `timeout_ms`, `retry`, and — when the resolved model is `media_surface: 'generative'` (routed to `generateMedia`, 1.AG/ADR-0045 §1) — the optional media-volume knobs `count` (images per call) / `duration_seconds` (audio/video length), which map to `MediaGenRequest` (wired with the routing in 1.AG Section C) (fields are inline on the `agent` node — `AgentNodeSchema`; there is no nested `agent_config` object) | | `condition_config` | `condition` | `expression_type` (`js` in v1.0; `jmespath`/`jsonlogic` reserved — [ADR-0027](../../decisions/0027-expression-sandbox.md)), `expression` (evaluated once), `branches[]` (each `{ when, target_node }` — `when` is **strictly** matched, `===`, against the result), `default` (taken when no `when` matches) | | `tool_config` | `tool` | `tool_name`, `tool_source` (`builtin`/`mcp`), `mcp_server`, `parameters`, `input_mapping`, `output_mapping` | | `transform_config` | `transform` | `expression_type` (`js` in v1.0; `jmespath`/`jsonlogic` reserved — [ADR-0027](../../decisions/0027-expression-sandbox.md)), `transform` (a **single** `js` expression whose result becomes the node's output — `TransformNodeSchema`), `output_schema` (optional, validated node-side; Phase-1 is parse-as-JSON only, deep conformance deferred) | diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 86502960..58e5bbf5 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -184,7 +184,7 @@ harness is now ✅ Done (2026-06-17), completing **1.m5**; cost-event persistenc > non-breaking; the seam doc carries the full amendment section. **1.AE–1.AH (media > input/engine/output + surfaces) are additive and do NOT gate M2** — the seam lane ran straight to > **1.K** (✅ Done, PR #13), which closed it. **1.AE (media-input adapters + the shared SSRF policy -> primitive) is now ✅ Done (PR #32, 2026-06-18)**; 1.AF–1.AH remain. +> primitive) is now ✅ Done (PR #32, 2026-06-18)**; **1.AF (engine media plumbing) is now ✅ Done (PR #33/#34/#35/#36, 2026-06-20)**; **1.AG (output generation) — Sections A–E landed on `development`, pending PR merge** (ADR-0045/0046; each section dual-reviewed); 1.AH remains. > **Review-pass follow-ups landed (PR #12, merged 2026-06-11).** The 2026-06-10 engine/tooling > review pass landed as docs/decisions only — no Phase-1 workstream changed: **MCP client scheduling** @@ -217,22 +217,22 @@ harness is now ✅ Done (2026-06-17), completing **1.m5**; cost-event persistenc > **media-input adapters + the shared SSRF policy primitive (1.AE) are ✅ Done (PR #32, 2026-06-18)** — > after a multi-round + final 8-dimension adversarial review (no SSRF bypass found); the SSRF *mechanism* > half (host DNS-resolve + connect-by-validated-IP), per-modality FallbackChain gating, `mediaUnits`, and -> handle/url media resolution are deferred to **1.AF**. **1.AF (engine media plumbing) is 🔨 in progress: -> P1 + P2 merged (PR #33); P3 + P4/D13 merged (PR #34); the P4 remainder (D12, D11, D15, D16, D17 + the -> byte-delivery review) is landed on `development` (NOT yet merged)** — its three design ADRs -> (0042/0043/0044) are Accepted. +> handle/url media resolution are deferred to **1.AF**. **1.AF (engine media plumbing) is ✅ Done +> (PR #33/#34/#35/#36, 2026-06-20):** P1 + P2 (PR #33); P3 + P4/D13 (PR #34); the P4 remainder (D12, D11, D15, +> D16, D17 + the byte-delivery review) (PR #35); the multi-agent + external review follow-ups (PR #36) — its +> three design ADRs (0042/0043/0044) are Accepted. > **Merged (P1+P2, PR #33):** the `MediaStore` contract impls + `media_objects`/`media_references` tables > (migration 0002), per-modality capability gating + `FallbackChain` skip, `deInlineMedia`, the strict > `output_modalities`/`save_to` node fields, OpenAI `mediaUnits`, and **the I3 keystone — `deInlineMedia` > at the one `#emitDurable` choke point**. -> **Landed on `development` (pending merge):** the binary media-egress + **SSRF mechanism half** (D9 — the +> **Merged (P3 + P4/D13, PR #34):** the binary media-egress + **SSRF mechanism half** (D9 — the > `MediaUrlFetch` re-host hook + `fetchMediaBytes` SSRF-validated host reference: DNS-resolve + > connect-by-validated-IP + per-hop redirect re-validation + streamed size-bound, wired at the choke point); > the `FallbackChain` resolve-before-egress + **byte-free re-materialization sidecar** (D8 + D7); the > **byte-delivery `Range` gate** (D13 — `MediaStore.readRange` + the engine-pure `validateByteRange`); and the > dedicated **P3 egress/SSRF security-review** (independent adversarial — 0 blockers/highs). All green + > Leakwatch-clean. -> **Landed on `development` (P4 remainder, pending merge):** `read_media` + the scope-set authz (D12), the +> **Merged (P4 remainder, PR #35):** `read_media` + the scope-set authz (D12), the > terminal sweep + grace-window GC (D11), the `output_modalities` load-check (D15), the `save_to` write port > (D16 — engine orchestration + the db `createFilesystemMediaWrite` fail-closed jail + the `run.id` > interpolation namespace), and the per-modality media cost governor (D17 — `mediaCost`/`estimateMediaCost` @@ -241,8 +241,14 @@ harness is now ✅ Done (2026-06-17), completing **1.m5**; cost-event persistenc > traversal/symlink-escape/arbitrary-write reachable) and the canonical-home docs (built-in-tools, > config-spec, security-review, database-schema, workflow-yaml-spec). All green + Leakwatch-clean. **Deferred > to 1.AH:** the keychain no-raw-key IPC test (no Phase-1 desktop surface; the gate is recorded per ADR-0044 -> §4). The remaining Phase-1 work is **additive and off the critical path**: only **1.AF–1.AH** (engine media -> plumbing, output, surfaces) remain before Phase 1 closes. **Phase 2 (CLI, milestone M3) is unblocked.** +> §4). +> **Merged (review follow-ups, PR #36):** a **13-dimension, double-verified end-to-end review** of the merged +> 1.AF (no exploitable SSRF bypass; the I3 handle-only invariant holds end-to-end) + an external Sonar/CodeRabbit +> pass — H1 `agent:tool_call.toolInput` byte-redaction, H2 the unified exact-membership `output_modalities` gate +> (+ bidirectional set-equality), the GC grace basis bumped to de-reference time, an egress-mechanism test for +> `nodeMediaEgressDeps`, and the canonical-home doc-drift cluster. All green + Leakwatch-clean. The remaining +> Phase-1 work is **additive and off the critical path**: only **1.AG–1.AH** (output, surfaces) remain before +> Phase 1 closes. **Phase 2 (CLI, milestone M3) is unblocked.** Carry-over hardening is tracked in [deferred-tasks.md](deferred-tasks.md) — pick items up as Phase 1 first touches each file. diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index c4635844..86fcb2f7 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -123,10 +123,88 @@ Severity is the review's verified rating. Check an item off in the PR that resol hook, it must apply these runtime checks. The current `assertHttpsBaseUrl` and `refineInFlightMediaPart` URL validation are construction-time / seam-ingestion-time policy; they catch malformed URLs but cannot catch DNS rebinding or a public hostname resolving to a private IP. **Scope split (resolving the earlier "Phase 2" framing):** the **media** url-carrier mechanism is **pulled into 1.AF** on a new bytes-shaped media-egress capability ([ADR-0043](../decisions/0043-media-egress-failover-rematerialization-ssrf.md)); the **general tool/MCP** `EgressCapability.fetch` enforcement still lands when the desktop/CLI surface implements that fetch hook. *(packages/core/src/tools/types.ts; security-review.md; media → 1.AF/ADR-0043; tool/MCP → surface fetch hook)* -- [ ] **Async media-job ADR (`generateMedia`/`pollMediaJob` behavior, A5)** — the seam shape is reserved - now (1.AD); the engine-owned **poll / checkpoint / resume / cancel loop** for minute-scale LROs - (Sora/Veo) — in the run loop (1.N) + checkpointer (1.R), reusing `LlmError` classification — gets **its - own ADR written at 1.AG (Phase D)**. Highest behavioral complexity in the multimodal design. *(1.AG)* +- [ ] **Async media-job ADR + engine loop (`generateMedia`/`pollMediaJob` behavior, A5) — landed on `development`, pending PR merge (1.AG Sections A–E; the A5 obligation itself discharged in A/C/D).** + [ADR-0045](../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) is written + Accepted, and + the engine-owned **poll / checkpoint / resume / cancel loop** for minute-scale LROs (Sora/Veo) landed in 1.AG + Section D (the run loop 1.N + checkpointer 1.R, reusing `LlmError` classification): `media_job:submitted` park, + the derived `pendingMediaJobs` slot, re-attach-on-resume (MJ-1), a `host.setTimer` poll cadence with exp-backoff, + deadline→retryable-timeout, gate-vs-media resume disambiguation (AG-A-FC-3), and cancel→abort→terminal-sweep — + proven end-to-end with a stub async provider + the manual-timer harness. The **Section-C wiring obligation** + also landed: the generative-vs-chat routing FORK moved up into `AgentRunner.executeAgent` (a `'generative'` + model dispatches to `generateMedia` BEFORE the turn loop), so the `requestsMediaOutput` guard in + [`agent-turn.ts`](../../packages/core/src/engine/agent-turn.ts) only ever sees `'chat'` models — ADR-0046 §1's + `media_surface: 'chat'` conjunct holds STRUCTURALLY (the guard's JSDoc `NOTE` points at the fork). The remaining + VENDOR-adapter + host-wiring work is tracked in the 1.AH entries below. **Tick `[x]` with the merged PR number + when the 1.AG PR lands** (this file's rule: check off in the PR that resolves it). *(1.AG Sections A–E — on development, pending merge)* +- [ ] **Streaming media triad (`media_start`/`media_delta`/`media_end`) — host-deferred ([ADR-0046](../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) §4).** + 1.AG Section B delivers inline media-out through the non-streaming `generate()` path (the in-flight + `media` `ContentPart` is de-inlined at `#emitDurable`). The **streaming** triad stays RESERVED: its Node + de-inline needs a host hook reaching the *pure* adapter (the output twin of `resolveForEgress`, since + `media_end` is handle-only and the adapter has no `MediaStore`) or the desktop Rust CAS ([ADR-0032](../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)). + Acceptable because a media-output turn is typically terminal/single-shot, so token-streaming its short + accompanying text is low value against a frozen-seam change. Wire at **1.AH** (the progressive-preview surface). *(packages/llm/src/types.ts seam; 1.AH)* +- [ ] **OpenAI agentic image-gen via the Responses API — wire deferred ([ADR-0046](../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) §3).** + The normalized delivery **shape** is defined (a `providerExecuted: true` `tool_result` carrying a + normalized `media` part — ADR-0031 §4.3/#7) and 1.AG Section B wires the two Chat-Completions inline + cases (Gemini `responseModalities` → `inlineData`, OpenAI inline audio → `audio`). OpenAI image output is + **not** a Chat-Completions modality (`modalities` is `text`/`audio` only); agentic image-gen is the + Responses API `image_generation` built-in tool — a **separate request surface** the Chat-Completions + adapter does not call. Wire the Responses-API path (request + `image_generation_call` output-item parse → + the already-defined providerExecuted media shape) when a Phase-1 model needs it. *(packages/llm/src/adapters/openai.ts; post-1.AG)* +- [ ] **Per-model `media_surface` lookup — HOST WIRING deferred (1.AG Section C → 1.AH).** Section C added + `AgentRunnerDeps.resolveMediaSurface?(model) → MediaSurface` (the inline-vs-generative routing discriminator, + default `'chat'`; tests inject it). The production wiring — the host reading `model_catalog.media_surface` (the + column landed in Section A) and supplying the lookup — is **1.AH host-wiring**, exactly like the D12/D15/D17 + host-wiring obligations. Until then `resolveMediaSurface` is absent and **every** model routes inline (`'chat'`), + so no generative model is runtime-reachable in Phase 1; the engine mechanism + the OpenAI-image adapter are proven + via injected-surface tests. *(packages/core/src/engine/agent-runner.ts; host catalog read → 1.AH)* +- [ ] **Verified generative model rates + `MODEL_PRICING` rows (1.AG Section C → 1.AH).** The Section C cost + mechanism (pre-egress estimate + the one realized `cost:updated`) reuses `estimateMediaCost`/`mediaCost`, which + **degrade to 0 on a missing rate** (H4). No generative model rows were added to `MODEL_PRICING` — fabricating + billing rates is a mis-bill risk and the existing rows are all "verified". Add the generative rows (gpt-image-1, + Imagen, OpenAI-TTS, Sora, Veo) with **verified** `mediaOutputRates` (image per-image; audio/video per-second) when + the catalog/pricing-page values are confirmed; until then a generative call is correctly gated/folded at 0 cost + (no mis-bill). **Test follow-up:** the realized-cost vertical (`realizedMediaCost` → a non-zero `cost:updated`) + cannot be exercised end-to-end until a generative model carries a rate; the cost MATH is unit-tested via + `mediaCost` against a constructed rate, and a non-zero dispatch assertion lands with these rows. *(packages/llm/src/pricing.ts; verified rates → 1.AH)* +- [ ] **`generateMedia` for OpenAI-TTS audio + Gemini-Imagen — adapter wires deferred (1.AG Section C → 1.AH).** + Section C wires `generateMedia` SYNC for **OpenAI image** (gpt-image-1 `images.generate` → base64), proving the + full engine→adapter→de-inline vertical. The remaining generators are bounded follow-ups: **OpenAI-TTS audio** + (`audio.speech` returns raw bytes → the adapter must base64-encode them + map the requested `response_format` → + MIME) and **Gemini-Imagen** (`generateImages` → `generatedImages[].image.imageBytes`, which needs a + `GeminiTransport.generateImages` extension to keep conformance vendor-free). Neither is runtime-reachable until the + per-model surface lookup is host-wired (above), so they land with that 1.AH wiring. Two bounded image follow-ups + ride along: (a) **multi-image `count > 1`** — the SYNC `MediaGenResult.media` carries a SINGLE part, so the OpenAI + adapter currently rejects `count > 1` (never bill-N-deliver-1); delivering N needs an additive `media: MediaPart[]` + seam amendment (ADR-0031). (b) **image-gen knobs** (`size`/`quality` via `MediaGenRequest.providerOptions`) — the + engine does not yet populate `providerOptions` for a generative call, so the adapter threads only the output format + (`req.mimeType`); a typed per-knob passthrough lands when the engine wires image knobs. *(packages/llm/src/adapters/{openai,gemini}.ts; types.ts MediaGenResult; 1.AH)* +- [ ] **Sora/Veo async-media ADAPTERS (`generateMedia`→`{ jobId }` + `pollMediaJob`) — deferred (1.AG Section D → 1.AH).** + 1.AG Section D landed the ENGINE-owned async-job loop (park on `media_job:submitted`, the `host.setTimer` poll + cadence, exp-backoff to `pollMaxMs`, deadline→retryable timeout, cross-process re-attach (MJ-1), gate-vs-media + resume disambiguation (AG-A-FC-3), cancel-aborts-the-in-flight-poll + terminal sweep) — proven end-to-end with a + STUB async provider + the manual-timer harness. **No Phase-1 adapter implements `pollMediaJob` or returns a + `jobId`** (the only `generateMedia` is OpenAI-image SYNC); the real Sora/Veo `generateMedia`(→jobId)/`pollMediaJob` + adapter implementations are the deferred wiring, alongside the per-model surface lookup + verified rates (1.AH). + **Conformance obligation for that 1.AH PR (review N4):** add a negative assertion that the minted opaque `jobId` + is NOT the recorded vendor operation name (ADR-0045 §7) and that a recorded vendor poll body's `raw` diagnostics + never cross the seam (strip-raw on `MediaJobStatus`) — the seam-contract stub (generative-seam.conformance.test.ts) + proves the SHAPE today, but only a real adapter replaying a vendor body can prove the opacity/strip invariant. *(packages/llm/src/adapters/*; 1.AH)* +- [ ] **Node-retry of a PARKED media job — deferred (1.AG Section D, ADR-0045 §3 "MAY re-dispatch").** + The node-retry budget (1.S) wraps the executor `#dispatch`; an async media job parks AFTER dispatch returns, and + the engine's out-of-band poll loop settles a deadline/poll failure as `node:failed` (retryable for a timeout) directly + — it does NOT re-enter the node-retry wrapper, so a parked media job is not automatically re-dispatched (a fresh paid + submit) on a retryable timeout. ADR-0045 §3 makes this a **MAY** ("the loop itself never silently re-submits"), so the + current behavior is conformant; re-dispatching a parked-then-failed media node through the node-retry budget is an + additive refinement. *(packages/core/src/engine/engine.ts #pollMediaJob; post-1.AG)* +- [ ] **Durable realized cost for a FAILED/CANCELLED paid media job (review M1, Phase 2).** `#emitMediaJobCost` + rides the transient `cost:updated` stream; `node:failed`/`run:failed`/`run:cancelled` carry no + `cumulativeCostMicrocents` (only `node:completed` snapshots it), so a durable-log reader reconstructing cost for a + billed-but-failed/cancelled media job sees the addend only on the LIVE stream, not the persisted log. This is the + PRE-EXISTING behavior of `cost:updated` (always live-only) — NOT 1.AG-introduced, and nothing in 1.AG depends on + durable fail-cost reconstruction (ADR-0045 §5's guarantee is about the live cost report). Snapshotting the + cumulative onto `node:failed`/`run:cancelled` (or a cost field on the terminal) is an additive Phase-2 durability + improvement. *(packages/core/src/engine/engine.ts #emitMediaJobCost; shared/run-event.ts; Phase 2)* - [ ] **Per-modality pre-egress media cost estimate (A6)** — ADR-0028's governor is token-based and cannot price a media-gen call. Add a `[defaults].media_cost_estimate` config default (the media analogue of `max_tokens_estimate`, in [config-spec.md](../reference/contracts/config-spec.md)) **and** a @@ -184,7 +262,8 @@ Severity is the review's verified rating. Check an item off in the PR that resol > The comprehensive 46-agent review confirmed the **host/surface wiring** below is **not yet present** — > so D12/D15/D17 are inert end-to-end until a host (CLI/desktop, 1.AH/Phase-2) wires them. Recorded here > so the roadmap is not read as "live end-to-end." None is a defect in the landed policy; each is the -> deferred mechanism/wiring half. *(matrix row stays ◇ until the PR merges.)* +> deferred mechanism/wiring half. *(1.AF is ✅ Done — all PRs merged #33/#34/#35/#36, 2026-06-20; the items +> below remain, owned by 1.AH.)* - [ ] **`read_media` host `MediaReadAccess` impl + base64 encoder (D12 mechanism)** — there is no host factory that bridges `MediaReferenceStore.describe()` + `MediaStore.readRange()` (which returns diff --git a/docs/roadmap/phases/phase-1-engine-and-llm.md b/docs/roadmap/phases/phase-1-engine-and-llm.md index e98a3b52..d1c9fe69 100644 --- a/docs/roadmap/phases/phase-1-engine-and-llm.md +++ b/docs/roadmap/phases/phase-1-engine-and-llm.md @@ -972,9 +972,10 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. **never returns a raw key from an IPC command** (direct test); and a run reaching a **terminal event (`run:completed|failed|cancelled`) deterministically reclaims its media refs** so a FAILED/CANCELLED run leaves no orphaned partial media (a terminal-state sweep, not refcount-GC alone). - - 🔨 *P1+P2 **merged** (PR #33); P3 + P4/D13 **merged** (PR #34); the **P4 remainder** (D12, D11, D15, D16, - D17 + the byte-delivery review) is landed on `development` (**pending merge**) — see - [ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md)/[0043](../../decisions/0043-media-egress-failover-rematerialization-ssrf.md)/[0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md), the three accepted 1.AF design ADRs.* + - ✅ **Done (PR #33/#34/#35/#36, 2026-06-20).** P1+P2 (PR #33); P3 + P4/D13 (PR #34); the **P4 remainder** + (D12, D11, D15, D16, D17 + the byte-delivery review) (PR #35); the **multi-agent + external review + follow-ups** (PR #36) — see + [ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md)/[0043](../../decisions/0043-media-egress-failover-rematerialization-ssrf.md)/[0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md), the three accepted 1.AF design ADRs. The deferred **host-wiring** half (D12 `MediaReadAccess` + session-scope population, the D15 loader call, the D17/`resolveForEgress` config wiring) and the keychain no-raw-key IPC test are owned by **1.AH** — recorded in [deferred-tasks.md](../deferred-tasks.md) §1.AF.* **Merged (P1 + P2, PR #33):** D5/D6 per-modality `requiredCapabilities()` + `FallbackChain` provider-skip (one shared `mediaSupportReason` predicate); D3 `deInlineMedia` (cycle-safe flight→durable transform + a pure base64 decoder); D1 `MediaStore` contract impls (`FilesystemMediaStore` CAS + `InMemoryMediaStore`); @@ -984,7 +985,7 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. persist-before-deliver preserved; missing-store → loud `run:failed`, terminal stays emit-safe); plus the canonical-home doc updates that landed with the code — database-schema.md §Media tables, workflow-yaml-spec (`output_modalities` / `save_to` / `{{ run.id }}`), and sse-event-schema (`mediaUnits` on `cost:updated`). - **Landed on `development` (P3 + P4/D13, pending merge):** + **Merged (P3 + P4/D13, PR #34):** - **D9** the binary media-egress capability + the SSRF **mechanism** half: `deInlineMedia` re-hosts a `url` source via an injected `MediaUrlFetch` hook (shared); `fetchMediaBytes` SSRF-validated host reference (db — DNS-resolve + connect-by-validated-IP + per-hop redirect re-validation + streamed size-bound, @@ -998,7 +999,7 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. a defensive re-bound) + the engine-pure `validateByteRange` policy (fail-closed; reused by D12 + 1.AH). - The dedicated **P3 egress/SSRF security-review pass** (independent adversarial — **0 blockers/highs**; 4 lower follow-ups fixed). All green (`pnpm turbo` 16/16) + Leakwatch-clean. - **Landed on `development` (P4 remainder, pending merge):** + **Merged (P4 remainder, PR #35):** - **D12 `read_media`** — the 13th built-in tool + the engine-pure scope-set authz (`scopeSetIncludes`) + the `Range` gate + the additive `ToolPolicyDenyReason.media_scope_denied` + the `ToolDispatchContext` `requestingScope`/`mediaRead` fields; the `media_references` row-writing lifecycle (`MediaReferenceStore` @@ -1032,11 +1033,20 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. population, the D15 loader call, the D17 config/`resolveForEgress` wiring) is recorded in [deferred-tasks.md](../deferred-tasks.md) §1.AF P4 — these make D12/D15/D17 inert end-to-end until a surface (1.AH) wires them; the landed engine **policy** is complete + tested. + **Merged (review follow-ups, PR #36):** a **13-dimension, double-verified end-to-end review** of the whole + merged 1.AF (no exploitable SSRF bypass; the I3 handle-only invariant holds end-to-end) plus an external + (Sonar/CodeRabbit) pass — fixes landed: **H1** `agent:tool_call.toolInput` byte-redaction (the symmetric + twin of the result-side `outputSummary` redaction, an I3 event-stream gap); **H2** unify the load-time + + runtime `output_modalities` gates on one exact-membership predicate (`isOutputCombinationSupported`, + + bidirectional set-equality so a duplicate modality cannot false-match); the **GC grace** basis bumped to + de-reference time (ADR-0042 §4); an **egress mechanism test** for the concrete `nodeMediaEgressDeps` + pin/SNI/port wiring; and the canonical-home doc-drift cluster (thirteen built-ins, `media_scope_denied` in + the deny union, `read_media` catalog row, the deferred `cost:updated.mediaUnits`). All green + Leakwatch-clean. **Deferred to 1.AH (no 1.AF surface):** the **keychain-bridge no-raw-key IPC test** — the Tauri keychain IPC command does not exist in Phase 1 (`apps/desktop` is unbuilt); the gate is **owned/recorded** by 1.AF ([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §4 + security-review.md) - and the direct test lands with the bridge in 1.AH. The matrix row stays ◇ until the 1.AF PR(s) merge - (Done-after-merge). + and the direct test lands with the bridge in 1.AH. **All 1.AF PRs (#33/#34/#35/#36) are merged + (2026-06-20) — the matrix row is now ✅ Done.** - **1.AG — Output generation (Phase D).** Inline media-out (Gemini `responseModalities`, OpenAI agentic image-gen via the `providerExecuted`+normalized-`media` arm, OpenAI inline audio); the `generateMedia`/`pollMediaJob` separate-endpoint generators (gpt-image-1, Imagen, TTS sync; Sora/Veo @@ -1046,6 +1056,11 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. node produces a handle; an async video job checkpoints, survives a simulated restart, and resumes to completion; **a CANCEL aborts the in-flight `pollMediaJob` (via `AbortSignal`), emits `run:cancelled`, and triggers the terminal-state media sweep**; a content-policy job failure maps to `content_filter`. + *Status (pending merge):* **Sections A–E landed on `development`** — ADR-0045 (async media-job LRO) + + ADR-0046 (inline media-out via `generate()`), the five engine/seam slices (contracts, inline media-out, + sync `generateMedia` + routing, the async poll/checkpoint/resume/cancel loop, acceptance + generative + conformance), each dual-reviewed (0 blocker/high). Flips to ✅ Done on PR merge; the genuinely-remaining + vendor-adapter + host-wiring work is **1.AH** (deferred-tasks.md). All four acceptance criteria are tested. - **1.AH — Surfaces & managed mode (Phase E, spans Phases 2–6).** Desktop: **[ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)** Rust-side media de-inline on egress + session-scoped `read_media` command + the Rust CAS — **must land before any @@ -1072,7 +1087,7 @@ the latter being the critical-path milestone for the whole product. | 1.m4 ✅ | Agent + non-agent node handlers, gate, checkpoint/resume, retry, tools, **expression sandbox** + pre-egress budget (**all components landed — 1.AC closed it, PR #26, 2026-06-16**) | 1.O, 1.P, 1.Q, 1.R, 1.S, 1.T, **1.AB**, **1.AC** | | **M2 ✅** | **Engine end-to-end from a Node harness (stream + checkpoint + retry + fallback) — CRITICAL-PATH MILESTONE** (**reached — 1.U landed, PR #27, 2026-06-16**) | **1.U** | | **1.m5 ✅** | Agent-first sub-spine: `AgentSession` + session events + persistence + checkpoint/resume + export, proven by its own harness (**additive, parallel — does NOT gate M2**) | 1.V, 1.W, 1.X, 1.Y, 1.Z, **1.AA** | -| 1.m6 | Multimodal I/O: seam amendment (**1.AD ✅ Done, PR #11 — landed before 1.K/1.O so the union members are non-breaking**), then media input/engine/output behavior (**additive — does NOT gate M2**) + surfaces threaded into Phases 2–6 ([ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)) | **1.AD ✅**, 1.AE, 1.AF, 1.AG, 1.AH | +| 1.m6 | Multimodal I/O: seam amendment (**1.AD ✅ Done, PR #11 — landed before 1.K/1.O so the union members are non-breaking**), then media input/engine/output behavior (**additive — does NOT gate M2**) + surfaces threaded into Phases 2–6 ([ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)) | **1.AD ✅, 1.AE ✅, 1.AF ✅**, 1.AG (Sections A–E landed on development, pending merge), 1.AH | ## Sequencing & parallelization @@ -1233,8 +1248,8 @@ flowchart LR | **1.AA** | C | 1.V, 1.W, 1.X, 1.Y, 1.Z | **1.m5** | ✅ — **Done (2026-06-17)** | | 1.AD | D | 1.A (seam types) | **must precede 1.K, 1.O** (non-breaking union members); 1.AE | ⬤ shape-only — ✅ **Done (PR #11)** | | 1.AE | D | 1.AD, 1.G/1.H (adapters) | 1.AF | ✅ — **Done (PR #32, 2026-06-18)** | -| 1.AF | D | 1.AE, 1.K, 1.N, 1.R | 1.AG | ◇ | -| 1.AG | D | 1.AF | 1.AH | ◇ | +| 1.AF | D | 1.AE, 1.K, 1.N, 1.R | 1.AG | ✅ — **Done (PR #33/#34/#35/#36, 2026-06-20)** | +| 1.AG | D | 1.AF | 1.AH | ◇ — Sections A–E landed on `development` (pending PR merge) | | 1.AH | D | 1.AG | Phases 2–6 surfaces | ◇ (spans phases) | > The matrix `Depends on` column is **authoritative** for cross-lane feeder edges (e.g. `1.N → 1.W` diff --git a/docs/standards/error-handling.md b/docs/standards/error-handling.md index 526b728f..3cdac1a9 100644 --- a/docs/standards/error-handling.md +++ b/docs/standards/error-handling.md @@ -79,6 +79,11 @@ We distinguish the two and never leak one as the other: - Errors surfaced through the run-event stream use the canonical `node:failed` and `run:failed` events (see the [SSE event schema](../reference/contracts/sse-event-schema.md)); they carry a user-safe message plus an internal correlation id, not a raw exception. +- **Content-policy rejections are fatal, with their own cause.** A provider content-filter block (a + text turn or a media generation) carries `content_filter` (1.AG, + [ADR-0045](../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)) — **fatal**, + distinct from `validation` (an authoring/shape error) so a surface shows the right cause/remediation; + re-issuing the same blocked content just re-blocks, so it is never retried. - **Resource/limit codes are fatal-without-user-action, never silent.** `budget_exceeded`, `run_timeout`, and `turn_limit` (a **hard** agent/session turn/round cap — distinct from the `[chat].max_messages` history-**trim** threshold of diff --git a/packages/core/src/engine/agent-runner.test.ts b/packages/core/src/engine/agent-runner.test.ts index a67313fe..dd9f24bf 100644 --- a/packages/core/src/engine/agent-runner.test.ts +++ b/packages/core/src/engine/agent-runner.test.ts @@ -1,8 +1,12 @@ -import type { Agent, OutputModality } from '@relavium/shared'; +import type { Agent, ContentPart, OutputModality } from '@relavium/shared'; +import { LlmProviderError, UnsupportedCapabilityError, makeLlmError } from '@relavium/llm'; import type { CapabilityFlags, LlmProvider, LlmRequest, + MediaGenRequest, + MediaGenResult, + MediaJobStatus, ProviderId, StreamChunk, } from '@relavium/llm'; @@ -15,8 +19,10 @@ import { buildMediaUnitsEstimate, createAgentNodeExecutor, DEFAULT_MEDIA_UNIT_ESTIMATE, + generativeUnits, type AgentRunnerDeps, } from './agent-runner.js'; +import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; import type { NodeExecContext, NodeStreamEvent } from './node-executor.js'; const CAPS: CapabilityFlags = { @@ -157,8 +163,20 @@ function reqCapturingProvider(supports: CapabilityFlags = CAPS): { const p: LlmProvider = { id: 'anthropic', supports, - generate: () => { - throw new Error('unused'); + // A media-output turn routes to generate() (1.AG/ADR-0046); a text turn streams. Capture on both + // so the request is observable whichever path the node's output_modalities select. Return a media part + // for a media request so the node COMPLETES (not the no-media → validation failure), keeping these + // request-assembly tests on the success path. + generate: (r) => { + captured = r; + const wantsMedia = r.outputModalities?.some((m) => m !== 'text') ?? false; + return Promise.resolve({ + content: wantsMedia + ? [{ type: 'media', mimeType: 'image/png', source: { kind: 'base64', data: 'aW1n' } }] + : [{ type: 'text', text: 'ok' }], + stopReason: 'stop', + usage: { inputTokens: 1, outputTokens: 1 }, + }); }, stream: (r) => { captured = r; @@ -190,6 +208,73 @@ describe('createAgentNodeExecutor — dispatch', () => { } }); + it('surfaces inline media-out as { text, media } so the engine can de-inline it (1.AG/ADR-0046)', async () => { + const image: ContentPart = { + type: 'media', + mimeType: 'image/png', + source: { kind: 'base64', data: 'aW1nLWJ5dGVz' }, + }; + // A provider whose generate() returns media; its stream THROWS — proving the media node routed to generate(). + const mediaProvider: LlmProvider = { + id: 'gemini', + supports: capsWithOutput([['text', 'image']]), + generate: () => + Promise.resolve({ + content: [{ type: 'text', text: 'here' }, image], + stopReason: 'stop', + usage: { inputTokens: 3, outputTokens: 2 }, + }), + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for an inline media-out turn'); + }, + }; + const exec = createAgentNodeExecutor(deps(mediaProvider)); + const { ctx } = ctxFor( + vertexFor({ + kind: 'agent', + node: agentNode({ output_modalities: ['text', 'image'] }), + resolvedAgent: AGENT, + }), + ); + const outcome = await exec.execute(ctx); + expect(outcome.kind).toBe('completed'); + if (outcome.kind === 'completed') { + // The in-flight base64 media survives to the node output; the engine de-inlines it at #emitDurable. + expect(outcome.output).toEqual({ text: 'here', media: [image] }); + } + }); + + it('FAILS visibly when a media node produced no media — never a silent text completion (Opus-fix, ADR-0046)', async () => { + // A capable model that returns text-only (a refusal, or ignoring the modality) must not pass off the + // incidental text as a successful media turn — the declared-capability pre-skip cannot catch this. + const textOnlyProvider: LlmProvider = { + id: 'gemini', + supports: capsWithOutput([['text', 'image']]), + generate: () => + Promise.resolve({ + content: [{ type: 'text', text: 'I cannot make an image' }], + stopReason: 'stop', + usage: { inputTokens: 3, outputTokens: 2 }, + }), + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a media-out turn'); + }, + }; + const exec = createAgentNodeExecutor(deps(textOnlyProvider)); + const { ctx } = ctxFor( + vertexFor({ + kind: 'agent', + node: agentNode({ output_modalities: ['text', 'image'] }), + resolvedAgent: AGENT, + }), + ); + const outcome = await exec.execute(ctx); + expect(outcome).toMatchObject({ + kind: 'failed', + error: { code: 'validation', retryable: false }, + }); + }); + it('returns a loud typed failure for a non-agent node type (1.P not yet landed)', async () => { const exec = createAgentNodeExecutor(deps(provider([STOP]))); const { ctx } = ctxFor({ ...agentVertex(), type: 'transform' }); @@ -419,7 +504,8 @@ describe('createAgentNodeExecutor — output_schema + grant', () => { resolvedAgent: AGENT, }), ); - await exec.execute(ctx); + const outcome = await exec.execute(ctx); + expect(outcome.kind).toBe('completed'); // the media provider returns a media part → success path expect(req()?.outputModalities).toEqual(['text', 'image']); }); @@ -469,3 +555,434 @@ describe('buildMediaUnitsEstimate (1.AF/D17 — media-cost unit estimate)', () = ]); }); }); + +describe('createAgentNodeExecutor — generative media (1.AG Section C, generateMedia)', () => { + // Typed as the media variant so it satisfies MediaGenResult.media (and a node output) without a cast. + const image: Extract = { + type: 'media', + mimeType: 'image/png', + source: { kind: 'base64', data: 'Z2VuLWltYWdl' }, + }; + + /** A provider flagged media_surface 'generative' whose generateMedia returns (or throws). generate/stream + * THROW — proving a generative node never touches the inline turn path. */ + function generativeProvider(behavior?: { result?: MediaGenResult; throws?: Error }): LlmProvider { + const base = capsWithOutput([['image']]); + return { + id: 'openai', + supports: { ...base, media: { ...base.media, surface: 'generative' } }, + generate: () => { + throw new Error('generate must NOT run for a generative node'); + }, + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a generative node'); + }, + generateMedia: () => + behavior?.throws === undefined + ? Promise.resolve(behavior?.result ?? { media: image, raw: { internal: true } }) + : Promise.reject(behavior.throws), + }; + } + const genDeps = (p: LlmProvider, over: Partial = {}): AgentRunnerDeps => + deps(p, { resolveMediaSurface: () => 'generative', ...over }); + const genVertex = (over: Partial = {}): PlanVertex => + vertexFor({ + kind: 'agent', + node: agentNode({ output_modalities: ['image'], ...over }), + resolvedAgent: AGENT, + }); + + it('routes a generative model to generateMedia and outputs { text:"", media:[part] } + one token-free cost:updated', async () => { + const exec = createAgentNodeExecutor(genDeps(generativeProvider())); + const { ctx, events } = ctxFor(genVertex()); + const outcome = await exec.execute(ctx); + expect(outcome.kind).toBe('completed'); + if (outcome.kind === 'completed') { + expect(outcome.output).toEqual({ text: '', media: [image] }); // raw is NOT placed in the media part + } + const cost = events.filter((e) => e.type === 'cost:updated'); + expect(cost).toHaveLength(1); // exactly one realized addend (ADR-0045 §5) + expect(cost[0]).toMatchObject({ + inputTokens: 0, + outputTokens: 0, + nodeId: 'n1', + model: 'claude-opus-4-8', + }); + }); + + it('a chat model (default surface, no resolveMediaSurface) keeps the normal turn — never generateMedia', async () => { + // No resolveMediaSurface dep → default 'chat'. The provider's generateMedia would throw if reached. + const provider: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('unused'); + }, + stream: () => streamOf([{ type: 'text_delta', text: 'hi' }, STOP]), + generateMedia: () => Promise.reject(new Error('generateMedia must NOT run for a chat node')), + }; + const exec = createAgentNodeExecutor(deps(provider)); // no resolveMediaSurface + const { ctx } = ctxFor(agentVertex()); + const outcome = await exec.execute(ctx); + expect(outcome).toMatchObject({ kind: 'completed', output: 'hi' }); + }); + + it('fails internal when the model is generative but the provider implements no generateMedia (host-wiring gap)', async () => { + const base = capsWithOutput([['image']]); + const provider: LlmProvider = { + id: 'openai', + supports: { ...base, media: { ...base.media, surface: 'generative' } }, + generate: () => { + throw new Error('unused'); + }, + stream: (): AsyncIterable => streamOf([STOP]), + // no generateMedia + }; + const exec = createAgentNodeExecutor(genDeps(provider)); + const outcome = await exec.execute(ctxFor(genVertex()).ctx); + expect(outcome).toMatchObject({ kind: 'failed', error: { code: 'internal' } }); + }); + + it('hands a jobId off as a media_job outcome for the engine poll loop (1.AG Section D)', async () => { + const jobExec = createAgentNodeExecutor( + genDeps(generativeProvider({ result: { jobId: 'job-1', raw: {} } })), + ); + const outcome = await jobExec.execute(ctxFor(genVertex({ count: 2 })).ctx); + expect(outcome).toMatchObject({ + kind: 'media_job', + job: { + jobId: 'job-1', + provider: 'openai', + model: 'claude-opus-4-8', + modality: 'image', + units: 2, // the authored count → the engine's lone realized cost addend at done + }, + }); + }); + + it('fails validation when a generative node does not declare exactly one media modality', async () => { + const exec = createAgentNodeExecutor(genDeps(generativeProvider())); + // text+image (has text + two members) is rejected — a generative model emits pure media. + const outcome = await exec.execute( + ctxFor(genVertex({ output_modalities: ['text', 'image'] })).ctx, + ); + expect(outcome).toMatchObject({ kind: 'failed', error: { code: 'validation' } }); + }); + + it('gates pre-egress with maxTokens:0 + the media estimate → budget_exceeded (no generateMedia egress)', async () => { + let called = false; + let info: { maxTokens?: number; mediaUnitsEstimate?: unknown } | undefined; + const provider = generativeProvider(); + const wrapped: LlmProvider = { + ...provider, + generateMedia: (req, key) => { + called = true; + return provider.generateMedia?.(req, key) ?? Promise.reject(new Error('no')); + }, + }; + const exec = createAgentNodeExecutor( + genDeps(wrapped, { + preEgress: (i) => { + info = i; + return Promise.reject(new BudgetExceededError(100, 50, 120)); + }, + }), + ); + const outcome = await exec.execute(ctxFor(genVertex({ count: 2 })).ctx); + expect(outcome).toMatchObject({ kind: 'failed', error: { code: 'budget_exceeded' } }); + expect(called).toBe(false); // gate fails before any provider egress + // The gate pins the TOKEN estimate to 0 (a generative call emits none) + carries the authored media volume. + expect(info?.maxTokens).toBe(0); + expect(info?.mediaUnitsEstimate).toEqual([{ modality: 'image', units: 2 }]); + }); + + it('maps a generateMedia provider error through the chat taxonomy (content_filter stays content_filter)', async () => { + const exec = createAgentNodeExecutor( + genDeps( + generativeProvider({ + throws: new LlmProviderError( + makeLlmError({ provider: 'openai', kind: 'content_filter', message: 'blocked' }), + ), + }), + ), + ); + const outcome = await exec.execute(ctxFor(genVertex()).ctx); + expect(outcome).toMatchObject({ kind: 'failed', error: { code: 'content_filter' } }); + }); + + it('fails validation for an empty resolved prompt (the seam nonEmptyString contract) — no provider egress', async () => { + let called = false; + const provider = generativeProvider(); + const wrapped: LlmProvider = { + ...provider, + generateMedia: (req, key) => { + called = true; + return provider.generateMedia?.(req, key) ?? Promise.reject(new Error('no')); + }, + }; + // A node with no prompt_template resolves to an empty prompt. + const exec = createAgentNodeExecutor(genDeps(wrapped)); + const ctx = ctxFor( + vertexFor({ + kind: 'agent', + node: { id: 'n1', type: 'agent', agent_ref: 'summarizer', output_modalities: ['image'] }, + resolvedAgent: AGENT, + }), + ).ctx; + expect(await exec.execute(ctx)).toMatchObject({ + kind: 'failed', + error: { code: 'validation' }, + }); + expect(called).toBe(false); + }); + + it('maps the authored count/duration_seconds onto the MediaGenRequest at dispatch', async () => { + let captured: MediaGenRequest | undefined; + const base = capsWithOutput([['image']]); + const provider: LlmProvider = { + id: 'openai', + supports: { ...base, media: { ...base.media, surface: 'generative' } }, + generate: () => { + throw new Error('unused'); + }, + stream: (): AsyncIterable => { + throw new Error('unused'); + }, + generateMedia: (req) => { + captured = req; + return Promise.resolve({ media: image, raw: {} }); + }, + }; + const exec = createAgentNodeExecutor(genDeps(provider)); + await exec.execute(ctxFor(genVertex({ count: 4 })).ctx); + expect(captured?.count).toBe(4); + expect(captured?.durationSeconds).toBeUndefined(); + expect(captured?.modality).toBe('image'); + expect(captured?.prompt).toBe('Summarize: hi'); // the resolved prompt_template + }); + + it('forwards duration_seconds → durationSeconds for an audio generative node', async () => { + let captured: MediaGenRequest | undefined; + const audio: Extract = { + type: 'media', + mimeType: 'audio/wav', + source: { kind: 'base64', data: 'YXVkaW8=' }, + }; + const base = capsWithOutput([['audio']]); + const provider: LlmProvider = { + id: 'openai', + supports: { ...base, media: { ...base.media, surface: 'generative' } }, + generate: () => { + throw new Error('unused'); + }, + stream: (): AsyncIterable => { + throw new Error('unused'); + }, + generateMedia: (req) => { + captured = req; + return Promise.resolve({ media: audio, raw: {} }); + }, + }; + const exec = createAgentNodeExecutor(genDeps(provider)); + const outcome = await exec.execute( + ctxFor(genVertex({ output_modalities: ['audio'], duration_seconds: 8 })).ctx, + ); + expect(outcome.kind).toBe('completed'); + expect(captured?.durationSeconds).toBe(8); + expect(captured?.modality).toBe('audio'); + expect(captured?.count).toBeUndefined(); + }); + + it('fails internal on a neither-media-nor-jobId result (the unreachable-in-practice guard)', async () => { + const exec = createAgentNodeExecutor(genDeps(generativeProvider({ result: { raw: {} } }))); + expect(await exec.execute(ctxFor(genVertex()).ctx)).toMatchObject({ + kind: 'failed', + error: { code: 'internal' }, + }); + }); + + it('fails internal on a BOTH-media-and-jobId result — the XOR is enforced, media is never silently discarded', async () => { + // The seam refine (MediaGenResultSchema) forbids both, but the adapter result is not re-parsed at the + // executor boundary; a hand-built result with both must fail loud (the async jobId branch would otherwise + // win and silently DROP the media), not produce a handle-less media_job nor a discarded image. + const exec = createAgentNodeExecutor( + genDeps(generativeProvider({ result: { media: image, jobId: 'job-x', raw: {} } })), + ); + expect(await exec.execute(ctxFor(genVertex()).ctx)).toMatchObject({ + kind: 'failed', + error: { code: 'internal' }, + }); + }); + + it('classifies a seam UnsupportedCapabilityError as validation (not opaque internal)', async () => { + const exec = createAgentNodeExecutor( + genDeps( + generativeProvider({ + throws: new UnsupportedCapabilityError('openai', 'media', 'no audio'), + }), + ), + ); + expect(await exec.execute(ctxFor(genVertex()).ctx)).toMatchObject({ + kind: 'failed', + error: { code: 'validation' }, + }); + }); + + it('redacts a keyFor throw into a fixed provider_auth failure (no secret leak)', async () => { + const exec = createAgentNodeExecutor( + genDeps(generativeProvider(), { + keyFor: () => { + throw new Error('SECRET-BEARING-KEY-RESOLUTION-DETAIL'); + }, + }), + ); + const outcome = await exec.execute(ctxFor(genVertex()).ctx); + expect(outcome).toMatchObject({ kind: 'failed', error: { code: 'provider_auth' } }); + if (outcome.kind === 'failed') { + expect(outcome.error.message).not.toContain('SECRET-BEARING-KEY-RESOLUTION-DETAIL'); + } + }); + + it('pollMediaJob (the engine delegate): missing pollMediaJob → failed(unknown); a keyFor throw → secret-free failed(auth)', async () => { + const job = { + jobId: 'j1', + provider: 'anthropic' as const, + model: 'm', + modality: 'image' as const, + units: 1, + }; + const signal = { + aborted: false, + addEventListener: () => undefined, + removeEventListener: () => undefined, + }; + // (1) The resolved provider implements no pollMediaJob → a secret-free failed(unknown) MediaJobStatus. + const noPoll: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('unused'); + }, + stream: (): AsyncIterable => streamOf([STOP]), + }; + const exec1 = createAgentNodeExecutor(deps(noPoll)); + const r1: MediaJobStatus | undefined = await exec1.pollMediaJob?.(job, signal); + expect(r1).toMatchObject({ state: 'failed', error: { kind: 'unknown', retryable: false } }); + + // (2) keyFor throws a secret-bearing error → a FIXED secret-free failed(auth), the original dropped. + const withPoll: LlmProvider = { + ...noPoll, + pollMediaJob: () => Promise.resolve({ state: 'pending' }), + }; + const exec2 = createAgentNodeExecutor( + deps(withPoll, { + keyFor: () => { + throw new Error('SECRET-KEY-RESOLUTION-DETAIL'); + }, + }), + ); + const r2: MediaJobStatus | undefined = await exec2.pollMediaJob?.(job, signal); + expect(r2).toMatchObject({ state: 'failed', error: { kind: 'auth' } }); + if (r2?.state === 'failed') { + expect(r2.error.message).not.toContain('SECRET-KEY-RESOLUTION-DETAIL'); + } + }); + + it('fails validation when the produced media modality does not match the request (defense-in-depth)', async () => { + const audio: Extract = { + type: 'media', + mimeType: 'audio/wav', + source: { kind: 'base64', data: 'YXVkaW8=' }, + }; + const exec = createAgentNodeExecutor( + genDeps(generativeProvider({ result: { media: audio, raw: {} } })), + ); + // node requests image, provider returns audio → validation + expect(await exec.execute(ctxFor(genVertex()).ctx)).toMatchObject({ + kind: 'failed', + error: { code: 'validation' }, + }); + }); + + it('a pre-aborted signal fails cancelled with no generateMedia egress', async () => { + let called = false; + const provider = generativeProvider(); + const wrapped: LlmProvider = { + ...provider, + generateMedia: (req, key) => { + called = true; + return provider.generateMedia?.(req, key) ?? Promise.reject(new Error('no')); + }, + }; + const { ctx } = ctxFor(genVertex()); + const aborted: NodeExecContext = { + ...ctx, + signal: { + aborted: true, + addEventListener: () => undefined, + removeEventListener: () => undefined, + }, + }; + expect(await createAgentNodeExecutor(genDeps(wrapped)).execute(aborted)).toMatchObject({ + kind: 'failed', + error: { code: 'cancelled' }, + }); + expect(called).toBe(false); + }); + + it('a BudgetPauseError on the generative pre-egress gate → paused (the human-gate seam, not internal)', async () => { + // The generative path must map a budget PAUSE to the gate seam exactly like the chat path — a future + // refactor that dropped the pause arm would leave a budget-paused generative run non-resumably internal-failed. + const exec = createAgentNodeExecutor( + genDeps(generativeProvider(), { + preEgress: () => Promise.reject(new BudgetPauseError(100, 50, 90)), + }), + ); + expect((await exec.execute(ctxFor(genVertex()).ctx)).kind).toBe('paused'); + }); + + it('a cancel landing DURING generateMedia (the post-await re-check) → cancelled, no stray cost:updated', async () => { + // The pre-egress abort check passes (signal not yet aborted), generateMedia resolves a VALID result while + // the run cancels mid-flight, and the post-await re-check must win — returning `cancelled` BEFORE the cost + // emit (#onOutcome's #settled guard cannot un-emit a fired cost:updated). + const sig = { + aborted: false, + addEventListener: () => undefined, + removeEventListener: () => undefined, + }; + const provider: LlmProvider = { + ...generativeProvider(), + generateMedia: () => { + sig.aborted = true; // the run cancels while generateMedia is in-flight (it still resolves a result) + return Promise.resolve({ media: image, raw: {} }); + }, + }; + const { ctx, events } = ctxFor(genVertex()); + const outcome = await createAgentNodeExecutor(genDeps(provider)).execute({ + ...ctx, + signal: sig, + }); + expect(outcome).toMatchObject({ kind: 'failed', error: { code: 'cancelled' } }); + expect(events.filter((e) => e.type === 'cost:updated')).toHaveLength(0); // the post-await guard suppresses it + }); +}); + +describe('generativeUnits — authored media volume (ADR-0045 §5)', () => { + it('image: count, defaulting to the conservative built-in when unset', () => { + expect(generativeUnits('image', agentNode({ count: 4 }))).toBe(4); + expect(generativeUnits('image', agentNode())).toBe(DEFAULT_MEDIA_UNIT_ESTIMATE.image); + }); + + it('audio/video: duration_seconds, defaulting to the conservative built-in when unset', () => { + expect(generativeUnits('audio', agentNode({ duration_seconds: 12 }))).toBe(12); + expect(generativeUnits('video', agentNode())).toBe(DEFAULT_MEDIA_UNIT_ESTIMATE.video); + }); + + it('audio/video: count × duration_seconds when BOTH are authored (N clips of D seconds)', () => { + expect(generativeUnits('video', agentNode({ duration_seconds: 10, count: 3 }))).toBe(30); + // count alone (no duration) multiplies the conservative duration default — never silently dropped. + expect(generativeUnits('audio', agentNode({ count: 2 }))).toBe( + DEFAULT_MEDIA_UNIT_ESTIMATE.audio * 2, + ); + }); +}); diff --git a/packages/core/src/engine/agent-runner.ts b/packages/core/src/engine/agent-runner.ts index bb33fbbb..b46de24e 100644 --- a/packages/core/src/engine/agent-runner.ts +++ b/packages/core/src/engine/agent-runner.ts @@ -17,19 +17,31 @@ import { MEDIA_BILLED_MODALITIES, + mediaModalityOf, + type AbortSignalLike, type Agent, + type ContentPart, type ErrorCode, type FsScopeTier, type MediaBilledModality, type MediaCostEstimate, + type MediaSurface, type OutputModality, } from '@relavium/shared'; import { + LlmConfigError, + LlmProviderError, ResponseFormatSchema, ToolDefSchema, + cost, + makeLlmError, type FallbackPlanEntry, type LlmMessage, type LlmProvider, + type MediaGenRequest, + type MediaGenResult, + type MediaJobStatus, + type MediaUnitsEntry, type MediaUnitsEstimate, type ProviderId, type ResponseFormat, @@ -43,14 +55,20 @@ import type { ToolDef, ToolDispatchContext, ToolRegistry } from '../tools/types. import { AgentTurnError, DEFAULT_AGENT_TURN_LIMITS, + codeForLlmError, runAgentTurn, type AgentTurnLimits, type AgentTurnResult, type ChainCapabilities, type PreEgressHook, } from './agent-turn.js'; -import { BudgetPauseError } from './budget-governor.js'; -import type { NodeExecContext, NodeExecutor, NodeOutcome } from './node-executor.js'; +import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; +import type { + MediaJobSubmission, + NodeExecContext, + NodeExecutor, + NodeOutcome, +} from './node-executor.js'; type AgentNode = AgentPlanConfig['node']; @@ -63,6 +81,14 @@ type AgentNode = AgentPlanConfig['node']; export interface AgentRunnerDeps { /** Resolve an authored provider id to its concrete adapter instance; `undefined` ⇒ a host-wiring gap. */ readonly resolveProvider: (providerId: ProviderId) => LlmProvider | undefined; + /** + * Resolve a canonical model id to its `media_surface` (1.AG Section C, ADR-0045 §1) — the inline-vs-generative + * routing discriminator, projected from `model_catalog.media_surface`. `'generative'` routes the agent node to + * the separate-endpoint `generateMedia`; `'chat'` (the default, and the value when this dep is absent or returns + * `undefined`) uses the normal turn. The engine is platform-pure (no DB), so the host injects this catalog + * lookup; the production catalog wiring is host-side (1.AH), like the other 1.AF/1.AG host-wiring obligations. + */ + readonly resolveMediaSurface?: (model: string) => MediaSurface | undefined; /** The shared tool registry (1.T) the agent dispatches through (ADR-0037). */ readonly registry: ToolRegistry; /** The registry's tool defs — the source of the LLM-visible schema + descriptions for granted tools. */ @@ -147,7 +173,53 @@ function isBilledModality(modality: OutputModality): modality is MediaBilledModa * `agent` vertex runs the AgentRunner; every other engine type is a loud typed `failed` until 1.P. */ export function createAgentNodeExecutor(deps: AgentRunnerDeps): NodeExecutor { - return { execute: (ctx) => executeNode(ctx, deps) }; + return { + execute: (ctx) => executeNode(ctx, deps), + // The engine owns the async media-job poll loop (1.AG Section D), but provider + credential resolution + // lives here (the AgentRunnerDeps), so the engine delegates the actual poll back through the executor. + pollMediaJob: (job, signal) => pollMediaJobThroughDeps(deps, job, signal), + }; +} + +/** + * Re-resolve the provider + credential for an async media-job poll (1.AG Section D, ADR-0045 §3) and call the + * adapter's `pollMediaJob`. The two resolution faults this owns — a missing adapter / unimplemented + * `pollMediaJob` (host-wiring gap) and a credential failure — become a **secret-free `failed` `MediaJobStatus`** + * the engine settles `node:failed` from (never the original credential error — rule 6). A throw from the + * adapter's OWN `pollMediaJob` (a transient transport fault, or the abort rejection when the run is cancelled) + * PROPAGATES — the engine's poll-loop `try/catch` handles it: an abort / terminal / cleared job returns + * silently, a live fault settles a retryable `node:failed`. + */ +async function pollMediaJobThroughDeps( + deps: AgentRunnerDeps, + job: MediaJobSubmission, + signal: AbortSignalLike, +): Promise { + const provider = deps.resolveProvider(job.provider); + if (provider === undefined || provider.pollMediaJob === undefined) { + return { + state: 'failed', + error: makeLlmError({ + provider: job.provider, + kind: 'unknown', + message: `provider '${job.provider}' implements no pollMediaJob (host-wiring gap)`, + }), + }; + } + let key: string; + try { + key = await deps.keyFor(job.provider); + } catch { + return { + state: 'failed', + error: makeLlmError({ + provider: job.provider, + kind: 'auth', + message: `credential resolution failed for provider ${job.provider}`, + }), + }; + } + return provider.pollMediaJob(job.jobId, key, signal); } // The agent arm's local `failed` factory — the parallel of the canonical one in @@ -213,6 +285,22 @@ async function executeAgent( : await resolvePrompt(node.prompt_template, ctx, deps); if (!prompt.ok) return failed('validation', prompt.message, false); + // Inline-vs-generative routing (1.AG Section C, ADR-0045 §1): a resolved model whose `media_surface` is + // 'generative' dispatches through the separate-endpoint `generateMedia` (one provider, no chain failover — + // §6). A 'chat' model (the default, and the value when the host wires no surface lookup) takes the normal + // turn below. The surface is the host-injected catalog projection (`deps.resolveMediaSurface`). + // NOTE: `buildPlanEntries` above resolved the FULL fallback chain before this fork, so a `'generative'` node + // whose (unused, per §6) `fallback_chain` names an unwired provider fails fast at the plan-build above rather + // than here — intentional: a declared-but-unresolvable fallback is a workflow misconfig regardless of which + // surface the primary uses. The generative dispatch then consumes only `primary` (plan.entries[0]). + const primary = plan.entries[0]; + if ( + primary !== undefined && + (deps.resolveMediaSurface?.(primary.model) ?? 'chat') === 'generative' + ) { + return executeGenerativeMedia(ctx, node, primary, prompt.text, deps); + } + const messages = assembleMessages(agent, node, prompt.text); const llmTools = buildLlmTools(deps.tools, grantedToolIds); const outputSchema = node.output_schema ?? agent.output_schema; @@ -265,26 +353,330 @@ async function executeAgent( return turnOutcomeForError(err); } + return buildChatTurnOutcome(node, result, outputSchema !== undefined); +} + +/** + * Build the node outcome from a completed chat turn (1.O / 1.AG Section B). Order: a turn that produced media + * parts surfaces them as `{ text, media }` so the engine de-inlines the in-flight base64 to `media://` handles + * at #emitDurable (the I3 boundary) — this PRECEDES output_schema (a media turn is not JSON-validated). A turn + * that requested media but produced NONE fails `validation` (ADR-0046's produced-vs-requested check — the + * FallbackChain pre-skip is a declared-capability gate that cannot catch a model returning text anyway). Else, + * `output_schema` (when set) is enforced NODE-SIDE (the seam's responseFormat is a request hint only; an + * adapter never validates, and DeepSeek degrades to bare json_object — ADR-0038/D8); a non-JSON result fails + * `validation`. A plain text turn returns its text verbatim. + */ +function buildChatTurnOutcome( + node: AgentNode, + result: AgentTurnResult, + hasOutputSchema: boolean, +): NodeOutcome { const tokensUsed = { input: result.usage.input, output: result.usage.output, model: result.model, }; + const mediaParts = result.content.filter( + (part): part is Extract => part.type === 'media', + ); + if (mediaParts.length > 0) { + return { kind: 'completed', output: { text: result.text, media: mediaParts }, tokensUsed }; + } + if (node.output_modalities?.some((modality) => modality !== 'text')) { + return failed( + 'validation', + `agent node '${node.id}': output_modalities requested media output but the model returned none`, + false, + ); + } + if (!hasOutputSchema) { + return { kind: 'completed', output: result.text, tokensUsed }; + } + const parsed = tryParseJson(result.text); + if (parsed === PARSE_FAILED) { + return failed( + 'validation', + `agent node '${node.id}': output_schema is set but the model output was not valid JSON`, + false, + ); + } + return { kind: 'completed', output: parsed, tokensUsed }; +} - // output_schema enforcement is NODE-SIDE (the seam's responseFormat is a request hint only; an - // adapter never validates the response, and DeepSeek degrades to bare json_object — ADR-0038/D8). - if (outputSchema !== undefined) { - const parsed = tryParseJson(result.text); - if (parsed === PARSE_FAILED) { - return failed( - 'validation', - `agent node '${node.id}': output_schema is set but the model output was not valid JSON`, - false, - ); +/** + * Dispatch a `media_surface: 'generative'` agent node through the seam's `generateMedia` (1.AG Section C, + * [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §1/§5/§6): + * SYNC one-round-trip generation (gpt-image-1, Imagen, OpenAI TTS) resolving `{ media }`. ONE provider, NO + * cross-provider failover (a generative call is provider-bound — §6). The pre-egress budget gate runs first + * (the authored volume estimate, gate-only — never folded into cumulative); the in-flight media becomes the + * `{ text:'', media:[part] }` node output the engine de-inlines to a handle at #emitDurable (the I3 boundary, + * reusing 1.AF); exactly ONE realized `cost:updated` is emitted (§5). A `jobId` (async LRO) is Section D. + */ +async function executeGenerativeMedia( + ctx: NodeExecContext, + node: AgentNode, + primary: FallbackPlanEntry, + prompt: string, + deps: AgentRunnerDeps, +): Promise { + const modality = singleBilledModality(node.output_modalities, node.id); + if (!modality.ok) return failed('validation', modality.message, false); + + // The seam's `MediaGenRequest.prompt` is `nonEmptyString` — a generative call needs a prompt (unlike the + // chat path, which tolerates an empty prompt by sending no user message). Reject an empty resolved prompt as + // a clean upstream validation failure here, before the gate/egress — never let it surface as a provider + // bad_request (the node's `prompt_template` is optional, so this is reachable). + if (prompt.length === 0) { + return failed( + 'validation', + `agent node '${node.id}': a media_surface 'generative' model requires a non-empty prompt`, + false, + ); + } + + const provider = primary.provider; + if (provider.generateMedia === undefined) { + // The catalog flagged the model 'generative' but its provider implements no generateMedia — a host-wiring + // gap (internal), distinct from an authoring error. Never a raw throw (the engine would flatten it). + return failed( + 'internal', + `agent node '${node.id}': model '${primary.model}' is media_surface 'generative' but provider '${provider.id}' implements no generateMedia (host-wiring gap)`, + false, + ); + } + + // The authored output volume → the per-modality unit count (count for image; duration_seconds for + // audio/video, × count when both are authored — ADR-0045 §5). The SAME number drives the pre-egress estimate + // (gate only) and the realized fold (§5). + const units = generativeUnits(modality.modality, node); + + // Pre-egress budget gate (1.AC): the authored media volume, NEVER added to cumulative (gate only). A + // BudgetExceededError → budget_exceeded; a BudgetPauseError → paused (the human-gate seam) — this mirrors + // the chat path's awaitPreEgress exactly so a generative call is gated identically. `maxTokens: 0` pins the + // TOKEN estimate to zero — a generative call emits no tokens, so once generative pricing rows land the gate + // must not add a spurious token addend on top of the media estimate. `outputModalities` is the validated + // single modality (singleBilledModality), so the budget governor's media addend resolves the same rate. + const preEgress = ctx.preEgress ?? deps.preEgress; + if (preEgress !== undefined) { + try { + await preEgress({ + model: primary.model, + maxTokens: 0, + outputModalities: [modality.modality], + mediaUnitsEstimate: [{ modality: modality.modality, units }], + }); + } catch (err) { + if (err instanceof BudgetExceededError) { + return failed('budget_exceeded', err.message, false); + } + return turnOutcomeForError(err); // BudgetPauseError → paused; anything else re-throws → engine internal } - return { kind: 'completed', output: parsed, tokensUsed }; } - return { kind: 'completed', output: result.text, tokensUsed }; + // A cancel landing inside the (async) budget check costs no egress: re-check before engaging the provider. + if (ctx.signal.aborted) { + return failed( + 'cancelled', + `agent node '${node.id}': run cancelled before media generation`, + false, + ); + } + + const req: MediaGenRequest = { + model: primary.model, + prompt, + modality: modality.modality, + ...(node.count === undefined ? {} : { count: node.count }), + ...(node.duration_seconds === undefined ? {} : { durationSeconds: node.duration_seconds }), + signal: ctx.signal, + }; + + let key: string; + try { + key = await deps.keyFor(provider.id); + } catch { + // A host credential-resolution failure must NEVER surface its (possibly secret-bearing) message — replace + // it with a fixed, secret-free `provider_auth` failure, exactly as the FallbackChain's `#resolveKey` does + // on the chat path (rule 6). The original is dropped, not carried as a cause a sink could serialize. + return failed( + 'provider_auth', + `agent node '${node.id}': credential resolution failed for provider ${provider.id}`, + false, + ); + } + + let result: MediaGenResult; + try { + result = await provider.generateMedia(req, key); + } catch (err) { + return mapGenerateMediaError(err); + } + + // A cancel that landed WHILE generateMedia was in-flight (a non-cooperative adapter that ignored the signal, + // or one that resolved just as the run cancelled) must win: skip BOTH the async park / sync media outcome AND + // the stray cost:updated below — mirroring the post-turn abort re-check the inline/stream chat paths run. + // (#onOutcome drops a post-cancel completed anyway; returning `cancelled` here also suppresses the cost emit.) + if (ctx.signal.aborted) { + return failed( + 'cancelled', + `agent node '${node.id}': run cancelled during media generation`, + false, + ); + } + + return buildGenerativeOutcome(ctx, node, primary, modality.modality, units, result); +} + +/** + * Map a `generateMedia` throw through the SAME taxonomy as the chat path: an `LlmProviderError` → + * `codeForLlmError` (content-policy → `content_filter`, cancel → `cancelled`, …); an `LlmConfigError` (e.g. + * `UnsupportedCapabilityError` for a non-image modality / DeepSeek) → `validation` with its secret-free + * message (never let the engine catch-all flatten it to opaque `internal`); anything else → `turnOutcomeForError`. + */ +function mapGenerateMediaError(err: unknown): NodeOutcome { + if (err instanceof LlmProviderError) { + return failed(codeForLlmError(err.llmError), err.llmError.message, err.llmError.retryable); + } + if (err instanceof LlmConfigError) { + return failed('validation', err.message, false); + } + return turnOutcomeForError(err); +} + +/** + * Validate a resolved `generateMedia` result against the seam contract and build the node outcome. The + * adapter result is NOT re-parsed at this boundary, so the `MediaGenResult` exactly-one-of refine is enforced + * explicitly: BOTH present would let the async `jobId` branch silently DISCARD `media`, NEITHER would leave no + * output — both are a misbehaving/hand-built adapter result (internal), not an authoring error. An async + * `jobId` → the engine's media-job handoff (Section D); a sync `media` part → the de-inlined `{ text:'', media }` + * output plus the lone realized `cost:updated` (ADR-0045 §5). + */ +function buildGenerativeOutcome( + ctx: NodeExecContext, + node: AgentNode, + primary: FallbackPlanEntry, + modality: MediaBilledModality, + units: number, + result: MediaGenResult, +): NodeOutcome { + if (result.jobId !== undefined && result.media !== undefined) { + return failed( + 'internal', + `agent node '${node.id}': generateMedia returned BOTH media and jobId (exactly one required)`, + false, + ); + } + if (result.jobId !== undefined) { + // ASYNC LRO (Sora/Veo, 1.AG Section D, ADR-0045): the provider accepted a minute-scale job and returned an + // opaque jobId. Hand it to the ENGINE (poll/checkpoint/resume/cancel) — `units` is the authored volume for + // the lone realized cost addend the engine emits at `done` (ADR-0045 §5), NOT emitted here. + return { + kind: 'media_job', + job: { + jobId: result.jobId, + provider: primary.provider.id, + model: primary.model, + modality, + units, + }, + }; + } + if (result.media === undefined) { + return failed( + 'internal', + `agent node '${node.id}': generateMedia resolved neither media nor jobId`, + false, + ); + } + // Defense-in-depth: the produced media's modality must match what the node requested (and was budgeted for) — + // a misbehaving adapter returning an audio part for an image request is a validation failure, not a silent + // mis-billed handle. mediaModalityOf derives the modality from the bare MIME. + if (mediaModalityOf(result.media.mimeType) !== modality) { + return failed( + 'validation', + `agent node '${node.id}': generative model returned ${result.media.mimeType} but the node requested '${modality}'`, + false, + ); + } + // Exactly ONE realized cost:updated (ADR-0045 §5) — derived from the request volume × the per-model media + // rate (best-effort: an unknown/unrated model degrades to 0, H4). The engine folds it into the run cumulative. + ctx.emit({ + type: 'cost:updated', + nodeId: node.id, + model: primary.model, + inputTokens: 0, + outputTokens: 0, + costMicrocents: realizedMediaCost(primary.model, modality, units), + cumulativeCostMicrocents: 0, // placeholder — the engine overwrites with the authoritative run-wide total + }); + // The pure-media node output ({ text:'', media:[part] }) de-inlines at #emitDurable exactly like the inline + // path. `MediaGenResult.raw` is provider-internal and is never part of the media part (strip-on-sink, §7). + return { + kind: 'completed', + output: { text: '', media: [result.media] }, + tokensUsed: { input: 0, output: 0, model: primary.model }, + }; +} + +/** + * Derive the SINGLE billed output modality a generative node produces. A generative model emits pure media, + * so `output_modalities` must declare exactly one of `image`/`audio`/`video` — no text, no second modality. + */ +function singleBilledModality( + outputModalities: readonly OutputModality[] | undefined, + nodeId: string, +): { ok: true; modality: MediaBilledModality } | { ok: false; message: string } { + const all = outputModalities ?? []; + const billed = all.filter(isBilledModality); + const [only] = billed; + if (only === undefined || billed.length !== 1 || all.length !== 1) { + return { + ok: false, + message: `agent node '${nodeId}': a media_surface 'generative' model requires output_modalities to declare exactly one media modality (image | audio | video), with no text`, + }; + } + return { ok: true, modality: only }; +} + +/** + * The authored generation volume for a modality: `count` (image) or `duration_seconds` (audio/video). For an + * audio/video generator that takes BOTH (N clips of D seconds each), the volume is `count × duration_seconds` + * (ADR-0045 §5 — "count × durationSeconds for a generator that takes both"); a `duration`-only node multiplies + * by an implicit `count` of 1, so it is unchanged. Each missing field falls back to the conservative built-in + * default ({@link DEFAULT_MEDIA_UNIT_ESTIMATE}). + * + * NOTE: unlike the chat path's {@link buildMediaUnitsEstimate} (which reads `[defaults].media_cost_estimate`), + * the generative path uses the AUTHOR-DECLARED volume (`count`/`duration_seconds`) directly — the realized fold + * must reflect the actual requested volume, and the same number gates the pre-egress estimate (no host default). + */ +export function generativeUnits(modality: MediaBilledModality, node: AgentNode): number { + if (modality === 'image') { + return node.count ?? DEFAULT_MEDIA_UNIT_ESTIMATE.image; + } + const duration = node.duration_seconds ?? DEFAULT_MEDIA_UNIT_ESTIMATE[modality]; + return duration * (node.count ?? 1); +} + +/** + * Best-effort realized media cost for a generative call (ADR-0045 §5): the request volume × the per-model + * media rate, via the shared `cost()` fold (token counts are 0). An unknown/unrated model degrades to 0 (H4) + * — never a hard fail on a missing rate, exactly as the chat path's best-effort cost does. + */ +export function realizedMediaCost( + model: string, + modality: MediaBilledModality, + units: number, +): number { + const mediaUnits: MediaUnitsEntry[] = [ + { modality, direction: 'output', units, unit: modality === 'image' ? 'count' : 'second' }, + ]; + try { + return cost(model, { inputTokens: 0, outputTokens: 0, mediaUnits }); + } catch { + // Best-effort (H4): an unknown/unrated model — OR any pricing-layer fault — must NEVER turn a successful, + // already-paid generation into a failed node. Degrade to 0, matching the chain's best-effort cost fold + // (FallbackChain #emitSuccess swallows a pricing throw the same way). + return 0; + } } /** Build the ordered fallback plan: primary (node-over-agent model) + each authored fallback entry. */ diff --git a/packages/core/src/engine/agent-turn.test.ts b/packages/core/src/engine/agent-turn.test.ts index ed19ac2c..d38d5bb3 100644 --- a/packages/core/src/engine/agent-turn.test.ts +++ b/packages/core/src/engine/agent-turn.test.ts @@ -1,4 +1,12 @@ -import type { CapabilityFlags, LlmProvider, ProviderId, StreamChunk } from '@relavium/llm'; +import { LlmProviderError, makeLlmError } from '@relavium/llm'; +import type { + CapabilityFlags, + LlmProvider, + LlmResult, + ProviderId, + StreamChunk, +} from '@relavium/llm'; +import type { ContentPart } from '@relavium/shared'; import { describe, expect, it } from 'vitest'; import { @@ -68,6 +76,29 @@ const STOP = (reason: 'stop' | 'tool_use' = 'stop'): StreamChunk => ({ usage: { inputTokens: 10, outputTokens: 5 }, }); +/** Caps advertising inline image output (so the chain's per-attempt capability pre-skip keeps the model). */ +const MEDIA_CAPS: CapabilityFlags = { + ...CAPS, + media: { + input: { image: false, audio: false, video: false, document: false }, + outputCombinations: [['image'], ['text', 'image']], + surface: 'chat', + }, +}; + +/** A provider whose non-streaming `generate` returns a scripted `LlmResult`; its `stream` THROWS — so a + * test that completes proves the media-output turn routed through `generate()`, never `stream()` (1.AG). */ +function mediaGenerateProvider(id: ProviderId, result: LlmResult): LlmProvider { + return { + id, + supports: MEDIA_CAPS, + generate: () => Promise.resolve(result), + stream: (): AsyncIterable => { + throw new Error('stream must NOT be used for an inline media-out turn'); + }, + }; +} + const CAPABILITIES: ChainCapabilities = { keyFor: () => 'test-key', sleep: () => Promise.resolve(), @@ -158,6 +189,174 @@ describe('runAgentTurn — streaming + cost', () => { }); }); +describe('runAgentTurn — inline media-out (1.AG/ADR-0046)', () => { + const image: ContentPart = { + type: 'media', + mimeType: 'image/png', + source: { kind: 'base64', data: 'aW1nLWJ5dGVz' }, + }; + + it('routes a node requesting non-text output to generate() (not stream()) and returns the media content', async () => { + const provider = mediaGenerateProvider('gemini', { + content: [{ type: 'text', text: 'here is your image' }, image], + stopReason: 'stop', + usage: { inputTokens: 10, outputTokens: 5 }, + }); + const params = baseParams(provider, { + planEntries: [{ provider, model: 'gemini-2.5-flash', maxAttempts: 1 }], + outputModalities: ['text', 'image'], + }); + + const result = await runAgentTurn(params); + + expect(result.stopReason).toBe('stop'); + expect(result.content).toContainEqual(image); // the in-flight base64 media survives the turn (engine de-inlines it) + expect(result.text).toBe('here is your image'); + + // The generate() path streams no tokens, but still settles one cost:updated for the attempt, with the + // accurate per-attempt model + attemptNumber (the onAttempt plumbing is shared with the stream path). + const events = eventsOf(params); + expect(events.some((e) => e.type === 'agent:token')).toBe(false); + const cost = events.find((e) => e.type === 'cost:updated'); + expect(cost?.type === 'cost:updated' && cost.model).toBe('gemini-2.5-flash'); + expect(cost?.type === 'cost:updated' && cost.attemptNumber).toBe(1); + }); + + it('fails over on the generate() path: a retryable primary error advances to a capable secondary', async () => { + const primary: LlmProvider = { + id: 'gemini', + supports: MEDIA_CAPS, + generate: () => + Promise.reject( + new LlmProviderError( + makeLlmError({ provider: 'gemini', kind: 'overloaded', message: 'busy' }), + ), + ), + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a media-out turn'); + }, + }; + const secondary = mediaGenerateProvider('openai', { + content: [{ type: 'text', text: 'made it' }, image], + stopReason: 'stop', + usage: { inputTokens: 4, outputTokens: 2 }, + }); + const params = baseParams(primary, { + planEntries: [ + { provider: primary, model: 'gemini-2.5-flash', maxAttempts: 1 }, + { provider: secondary, model: 'gpt-image-1', maxAttempts: 1 }, + ], + outputModalities: ['text', 'image'], + }); + const result = await runAgentTurn(params); + expect(result.content).toContainEqual(image); + expect(result.model).toBe('gpt-image-1'); // attributed to the succeeding (failed-over) model + }); + + it('maps a generate() budget-exceeded cause to budget_exceeded (the throwMappedChainError cause unwrap)', async () => { + const provider: LlmProvider = { + id: 'gemini', + supports: MEDIA_CAPS, + generate: () => + Promise.reject( + new LlmProviderError( + makeLlmError({ + provider: 'gemini', + kind: 'unknown', + message: 'budget', + cause: new BudgetExceededError(120, 50, 130), + }), + ), + ), + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a media-out turn'); + }, + }; + const params = baseParams(provider, { + planEntries: [{ provider, model: 'gemini-2.5-flash', maxAttempts: 1 }], + outputModalities: ['image'], + }); + await expect(runAgentTurn(params)).rejects.toMatchObject({ code: 'budget_exceeded' }); + }); + + it('a pre-aborted signal on the media path fails cancelled with zero generate() egress', async () => { + let called = false; + const provider: LlmProvider = { + id: 'gemini', + supports: MEDIA_CAPS, + generate: () => { + called = true; + return Promise.resolve({ + content: [image], + stopReason: 'stop' as const, + usage: { inputTokens: 1, outputTokens: 1 }, + }); + }, + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a media-out turn'); + }, + }; + const aborted = { + aborted: true, + addEventListener: () => undefined, + removeEventListener: () => undefined, + }; + const params = baseParams(provider, { + planEntries: [{ provider, model: 'gemini-2.5-flash', maxAttempts: 1 }], + outputModalities: ['image'], + signal: aborted, + }); + await expect(runAgentTurn(params)).rejects.toMatchObject({ code: 'cancelled' }); + expect(called).toBe(false); // a cancel before egress engages no provider + }); + + it('keeps a text-only node on the streaming path (no generate())', async () => { + // A provider whose generate throws — a text turn must never reach it. + const provider: LlmProvider = { + id: 'anthropic', + supports: CAPS, + generate: () => { + throw new Error('generate must NOT be used for a text turn'); + }, + stream: () => streamOf([{ type: 'text_delta', text: 'hi' }, STOP()]), + }; + const result = await runAgentTurn(baseParams(provider, { outputModalities: ['text'] })); + expect(result.text).toBe('hi'); + }); + + it('fails loud (provider_unavailable) when a media-output turn returns a tool_use stop — never silently drops it', async () => { + // ADR-0046: a media turn is single-shot/terminal and is offered NO tools, so a tool_use stop is a provider + // protocol anomaly (a tool call we never offered, unrunnable). It must fail loud, not complete with empty + // output, and uses provider_unavailable — the same code the stream path's tool_use-anomaly guard uses. + const provider = mediaGenerateProvider('gemini', { + content: [{ type: 'tool_call', id: 't1', name: 'echo', args: {} }], + stopReason: 'tool_use', + usage: { inputTokens: 10, outputTokens: 5 }, + }); + const params = baseParams(provider, { + planEntries: [{ provider, model: 'gemini-2.5-flash', maxAttempts: 1 }], + outputModalities: ['text', 'image'], + }); + await expect(runAgentTurn(params)).rejects.toMatchObject({ code: 'provider_unavailable' }); + }); + + it('maps a generate() chain failure into the turn error taxonomy (symmetric with the stream path)', async () => { + const provider: LlmProvider = { + id: 'gemini', + supports: MEDIA_CAPS, + generate: () => Promise.reject(new Error('boom')), + stream: (): AsyncIterable => { + throw new Error('stream must NOT be used for a media-out turn'); + }, + }; + const params = baseParams(provider, { + planEntries: [{ provider, model: 'gemini-2.5-flash', maxAttempts: 1 }], + outputModalities: ['image'], + }); + await expect(runAgentTurn(params)).rejects.toBeInstanceOf(AgentTurnError); + }); +}); + describe('runAgentTurn — tool loop', () => { // A tool-use turn (a tool_call to `echo`, then a tool_use stop) — shared by the tool-loop scenarios. const toolUseTurn = (id: string): StreamChunk[] => [ @@ -496,6 +695,26 @@ describe('runAgentTurn — failover + cancel + reasoning', () => { }); }); + it('maps a content_filter LlmError to the fatal content_filter ErrorCode (not validation) — 1.AG/ADR-0045 §6', async () => { + const provider = scriptedProvider('anthropic', [ + [ + { + type: 'error', + error: { + kind: 'content_filter', + retryable: false, + provider: 'anthropic', + message: 'content policy block', + }, + }, + ], + ]); + await expect(runAgentTurn(baseParams(provider))).rejects.toMatchObject({ + code: 'content_filter', + retryable: false, + }); + }); + it('maps an aborted signal to cancelled (cancel wins)', async () => { const provider = scriptedProvider('anthropic', [[{ type: 'text_delta', text: 'x' }, STOP()]]); const aborted = { diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index 39e7fd75..98d1e2b7 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -36,6 +36,7 @@ import type { import { CostTracker, FallbackChain, + LlmProviderError, type AttemptRecord, type FallbackChainOptions, type FallbackPlanEntry, @@ -173,8 +174,10 @@ export class AgentTurnError extends Error { } } -/** Map a classified `LlmError` (chain-exhausted) to the node `ErrorCode` taxonomy (never `error.message`). */ -function codeForLlmError(error: LlmError): ErrorCode { +/** Map a classified `LlmError` (chain-exhausted) to the node `ErrorCode` taxonomy (never `error.message`). + * Exported so the generative `generateMedia` dispatch (1.AG Section C, agent-runner) maps its provider + * errors through the SAME taxonomy as the chat path — one classification, never a second vocabulary. */ +export function codeForLlmError(error: LlmError): ErrorCode { switch (error.kind) { case 'auth': return 'provider_auth'; @@ -187,6 +190,7 @@ function codeForLlmError(error: LlmError): ErrorCode { case 'cancelled': return 'cancelled'; case 'content_filter': + return 'content_filter'; // a provider content-policy block — its own fatal cause (1.AG/ADR-0045 §6), not `validation` case 'bad_request': return 'validation'; case 'unknown': @@ -279,7 +283,12 @@ function buildRequest(messages: readonly LlmMessage[], params: AgentTurnParams): model: params.planEntries[0]?.model ?? '', ...(params.system === undefined ? {} : { system: params.system }), messages: [...messages], - ...(params.tools === undefined ? {} : { tools: [...params.tools] }), + // A media-output turn is single-shot/terminal (1.AG/ADR-0046): it runs one `generate()` with no tool + // loop, so offering tools is meaningless and would invite an unrunnable `tool_use` stop. Omit them — a + // text turn (the only other `buildRequest` caller, via `streamOneTurn`) keeps its tool grant. + ...(params.tools === undefined || requestsMediaOutput(params) + ? {} + : { tools: [...params.tools] }), ...(params.responseFormat === undefined ? {} : { responseFormat: params.responseFormat }), ...(params.temperature === undefined ? {} : { temperature: params.temperature }), ...(params.maxTokens === undefined ? {} : { maxTokens: params.maxTokens }), @@ -310,28 +319,68 @@ async function streamOneTurn( for await (const chunk of chain.stream(buildRequest(messages, params))) { foldChunk(chunk, acc, params, getModel); if (chunk.type === 'error') { - // A pre-egress budget hook may throw its own AgentTurnError or Budget*Error; preserve it - // rather than remapping the wrapped LlmError to a generic internal code. - if (chunk.error.cause instanceof AgentTurnError) { - throw chunk.error.cause; - } - if (chunk.error.cause instanceof BudgetExceededError) { - throw new AgentTurnError('budget_exceeded', chunk.error.cause.message, false); - } - if (chunk.error.cause instanceof BudgetPauseError) { - throw chunk.error.cause; - } - throw new AgentTurnError( - codeForLlmError(chunk.error), - chunk.error.message, - chunk.error.retryable, - ); + throwMappedChainError(chunk.error); } if (chunk.type === 'stop') stopReason = chunk.stopReason; } return { content: accumulatorToContent(acc), stopReason }; } +/** + * Run one **inline media-out** turn via the chain's existing non-streaming `generate()` (1.AG/[ADR-0046]): + * a `media_surface: 'chat'` model emitting media IN the turn (Gemini `responseModalities`, OpenAI inline + * audio). The settled `LlmResult.content` carries in-flight base64 `media` parts; the engine de-inlines + * them to `media://` handles at `node:completed.output` (the 1.AF `#emitDurable` choke point). It is + * **terminal/single-shot** — no token streaming, no further tool round (the tool loop is built around + * `stream()`). Errors map identically to the streamed path: `generate()` throws an `LlmProviderError` + * whose `.llmError.cause` preserves a budget/turn error symmetrically with `stream()`'s error chunk. + */ +async function generateOneTurn( + chain: FallbackChain, + messages: readonly LlmMessage[], + params: AgentTurnParams, +): Promise<{ content: ContentPart[]; stopReason: StopReason }> { + try { + const result = await chain.generate(buildRequest(messages, params)); + return { content: result.content, stopReason: result.stopReason }; + } catch (err) { + if (err instanceof LlmProviderError) { + throwMappedChainError(err.llmError); + } + throw err; + } +} + +/** Map a chain failure — a streamed `error` chunk or a thrown `generate()` error — into the turn taxonomy. */ +function throwMappedChainError(error: LlmError): never { + // A pre-egress budget hook may throw its own AgentTurnError or Budget*Error; preserve it rather than + // remapping the wrapped LlmError to a generic internal code. + if (error.cause instanceof AgentTurnError) { + throw error.cause; + } + if (error.cause instanceof BudgetExceededError) { + throw new AgentTurnError('budget_exceeded', error.cause.message, false); + } + if (error.cause instanceof BudgetPauseError) { + throw error.cause; + } + throw new AgentTurnError(codeForLlmError(error), error.message, error.retryable); +} + +/** + * True when the node authored a non-text output modality — the inline media-out routing signal (1.AG). + * + * ADR-0046 §1's full condition is `media_surface: 'chat'` **and** a non-text `output_modalities`. The + * `'chat'` conjunct is satisfied STRUCTURALLY, not here: the AgentRunner forks a `'generative'` model to + * `generateMedia` (Section C, ADR-0045 §1) BEFORE it ever calls `runAgentTurn`, so this turn-core predicate + * only ever runs for a `'chat'` model — a `'generative'` model never reaches the inline `generate()` path. + * (The turn core is correlation-agnostic and holds no `CapabilityFlags`, so the surface check rightly lives + * at the routing layer that resolves the provider, not in this predicate.) + */ +function requestsMediaOutput(params: AgentTurnParams): boolean { + return params.outputModalities?.some((m) => m !== 'text') ?? false; +} + /** Fold a single stream chunk into the accumulator, emitting `agent:token` for visible text deltas. */ function foldChunk( chunk: StreamChunk, @@ -617,6 +666,36 @@ export async function runAgentTurn(params: AgentTurnParams): Promise { expect(state?.pendingGates).toEqual([{ gateId: 'g1', nodeId: 'gate', isBudgetGate: false }]); }); + const mediaJob = (seq: number, nodeId: string, jobId: string): RunEvent => ({ + type: 'media_job:submitted', + ...base(seq), + nodeId, + jobId, + provider: 'openai', + model: 'sora', + modality: 'video', + startedAt: TS, + deadlineAt: '2026-01-01T00:30:00.000Z', + }); + + it('parks an in-flight async media job (paused node + pendingMediaJobs entry) — ADR-0045 §2', () => { + const state = reconstructCheckpointState([started, mediaJob(1, 'gen', 'job-1')]); + expect(state?.nodeStates.get('gen')).toEqual({ status: 'paused' }); // suspended, NOT absent → re-attach not re-run + expect(state?.pendingMediaJobs).toEqual([ + { + nodeId: 'gen', + jobId: 'job-1', + provider: 'openai', + model: 'sora', + modality: 'video', + startedAt: TS, + deadlineAt: '2026-01-01T00:30:00.000Z', + }, + ]); + }); + + it("clears the node's media-job entry on its terminal node:completed (nothing to re-attach) — ADR-0045 §2", () => { + const state = reconstructCheckpointState([ + started, + mediaJob(1, 'gen', 'job-1'), + completed(2, 'gen', { handle: 'media://sha256-x' }), + ]); + expect(state?.pendingMediaJobs).toEqual([]); + expect(state?.nodeStates.get('gen')?.status).toBe('completed'); + }); + + it('latest media_job:submitted wins for a node (a node-retry re-dispatch replaces the entry) — ADR-0045 §2', () => { + const state = reconstructCheckpointState([ + started, + mediaJob(1, 'gen', 'job-old'), + mediaJob(2, 'gen', 'job-new'), + ]); + expect(state?.pendingMediaJobs).toHaveLength(1); + expect(state?.pendingMediaJobs[0]?.jobId).toBe('job-new'); + }); + + it('clears a media-job entry on its terminal node:failed (the failed job leaves nothing to re-attach)', () => { + const state = reconstructCheckpointState([ + started, + mediaJob(1, 'gen', 'job-1'), + { + type: 'node:failed', + ...base(2), + nodeId: 'gen', + error: { code: 'content_filter', message: 'blocked', retryable: false }, + }, + ]); + expect(state?.pendingMediaJobs).toEqual([]); + expect(state?.nodeStates.get('gen')?.status).toBe('failed'); + }); + + it('clears a media-job entry on the defensive node:skipped path too (ADR-0045 §2 clear-set)', () => { + const state = reconstructCheckpointState([ + started, + mediaJob(1, 'gen', 'job-1'), + { type: 'node:skipped', ...base(2), nodeId: 'gen', reason: 'upstream_unreachable' }, + ]); + expect(state?.pendingMediaJobs).toEqual([]); + expect(state?.nodeStates.get('gen')?.status).toBe('skipped'); + }); + + it('a stale/out-of-order media_job:submitted never resurrects an already-settled node (defensive — MJ-2)', () => { + const state = reconstructCheckpointState([ + started, + mediaJob(1, 'gen', 'job-1'), + completed(2, 'gen', { handle: 'media://sha256-x' }), + mediaJob(3, 'gen', 'job-stale'), // an out-of-order/duplicate submit AFTER the terminal — must be ignored + ]); + expect(state?.pendingMediaJobs).toEqual([]); // not re-added + expect(state?.nodeStates.get('gen')?.status).toBe('completed'); // not re-parked + }); + + it('folds in-flight media jobs across MULTIPLE distinct nodes independently', () => { + const state = reconstructCheckpointState([ + started, + mediaJob(1, 'genA', 'job-a'), + mediaJob(2, 'genB', 'job-b'), + completed(3, 'genA', { handle: 'media://sha256-a' }), // genA settles; genB stays parked + ]); + expect(state?.pendingMediaJobs.map((j) => j.nodeId)).toEqual(['genB']); + expect(state?.nodeStates.get('genA')?.status).toBe('completed'); + expect(state?.nodeStates.get('genB')?.status).toBe('paused'); + }); + it('keeps isBudgetGate=true across the budget:paused → human_gate:paused pair + restores the cumulative cost — H1/H2', () => { // The engine emits budget:paused THEN human_gate:paused with the SAME gateId for a budget gate. The fold // must not let the later human_gate:paused downgrade it to a plain human gate — else a resumed REJECTED diff --git a/packages/core/src/engine/checkpoint.ts b/packages/core/src/engine/checkpoint.ts index 7a61d2ed..204bf824 100644 --- a/packages/core/src/engine/checkpoint.ts +++ b/packages/core/src/engine/checkpoint.ts @@ -15,7 +15,7 @@ * wrongly skip-propagated; the dimmed branches are restored from `node:skipped`. */ -import type { RunEvent, RunStatus } from '@relavium/shared'; +import type { LlmProviderId, MediaBilledModality, RunEvent, RunStatus } from '@relavium/shared'; import type { NodeFailure } from './node-executor.js'; @@ -41,6 +41,23 @@ export interface CheckpointPendingGate { readonly isBudgetGate: boolean; } +/** + * An async media-generation job still in flight at the checkpoint (1.AG, ADR-0045 §2-3). The run resumes by + * **re-polling** the persisted opaque `jobId` (re-attach), NEVER re-submitting. Keyed by `nodeId` in the + * fold — one in-flight job per node, latest `media_job:submitted` wins. Carries everything a re-attach needs. + */ +export interface CheckpointPendingMediaJob { + readonly nodeId: string; + /** The Relavium-opaque job id to re-poll (never the vendor operation-name). */ + readonly jobId: string; + readonly provider: LlmProviderId; + readonly model: string; + readonly modality: MediaBilledModality; + /** ISO-8601 submit time and absolute deadline; on resume `now > deadlineAt` short-circuits a doomed re-poll. */ + readonly startedAt: string; + readonly deadlineAt: string; +} + /** The derived state a rehydrating run is rebuilt from — never a persisted blob (reconstructed from rows). */ export interface CheckpointState { readonly schemaVersion: number; @@ -56,6 +73,8 @@ export interface CheckpointState { readonly completedNodeIds: readonly string[]; /** Gates still pending a decision — the run is resumable via `engine.resume(runId, gateId, decision)`. */ readonly pendingGates: readonly CheckpointPendingGate[]; + /** Async media jobs in flight at the checkpoint — the run resumes by re-polling each (re-attach, ADR-0045 §3). */ + readonly pendingMediaJobs: readonly CheckpointPendingMediaJob[]; /** Gate ids ALREADY resolved (a `human_gate:resumed` was persisted) — so re-delivering a decision after a * reconnect is an idempotent no-op rather than advancing the run twice (execution-model.md §gate). */ readonly resolvedGateIds: readonly string[]; @@ -92,6 +111,8 @@ interface ReconAccumulator { cumulativeCostMicrocents: number; readonly nodeStates: Map; readonly pendingGates: Map; + /** Keyed by `nodeId` — one in-flight media job per node; a re-submit replaces the entry (latest wins). */ + readonly pendingMediaJobs: Map; readonly resolvedGateIds: Set; } @@ -119,6 +140,15 @@ function applyRunEvent(acc: ReconAccumulator, event: RunEvent): void { /** Node-level settlements: completed (+ branch selection, token tally), failed, skipped. */ function applyNodeEvent(acc: ReconAccumulator, event: RunEvent): void { + // A node's terminal clears any in-flight media job it had parked (ADR-0045 §2): the artifact (or failure) + // has settled, so there is nothing to re-attach to on resume. + if ( + event.type === 'node:completed' || + event.type === 'node:failed' || + event.type === 'node:skipped' + ) { + acc.pendingMediaJobs.delete(event.nodeId); + } switch (event.type) { case 'node:completed': acc.nodeStates.set(event.nodeId, { @@ -159,6 +189,35 @@ function applyNodeEvent(acc: ReconAccumulator, event: RunEvent): void { } } +/** + * Async media-job lifecycle (ADR-0045 §2-3): a `media_job:submitted` PARKS the node (a non-terminal + * suspension) and records the job keyed by `nodeId` (latest submit replaces — a node-retry re-dispatch + * mints a fresh job for the same node). The terminal clear lives in {@link applyNodeEvent}. + */ +function applyMediaJobEvent(acc: ReconAccumulator, event: RunEvent): void { + if (event.type !== 'media_job:submitted') { + return; + } + // Defensive: never resurrect a node that has already SETTLED. The emit-order invariant (a submit precedes + // its node's terminal) holds, so this is belt-and-suspenders — but it makes the fold order-independent, so + // a stale/duplicate submit folded after a node:completed/failed/skipped cannot re-park the node or re-add a + // cleared entry. + const settled = acc.nodeStates.get(event.nodeId)?.status; + if (settled === 'completed' || settled === 'failed' || settled === 'skipped') { + return; + } + acc.nodeStates.set(event.nodeId, { status: 'paused' }); + acc.pendingMediaJobs.set(event.nodeId, { + nodeId: event.nodeId, + jobId: event.jobId, + provider: event.provider, + model: event.model, + modality: event.modality, + startedAt: event.startedAt, + deadlineAt: event.deadlineAt, + }); +} + /** Human-gate / budget-gate lifecycle: park a pending gate, or resolve it (decision becomes the gate vertex output). */ function applyGateEvent(acc: ReconAccumulator, event: RunEvent): void { if (event.type === 'human_gate:paused' || event.type === 'budget:paused') { @@ -220,6 +279,7 @@ export function reconstructCheckpointState( cumulativeCostMicrocents: 0, nodeStates: new Map(), pendingGates: new Map(), + pendingMediaJobs: new Map(), resolvedGateIds: new Set(), }; @@ -230,6 +290,7 @@ export function reconstructCheckpointState( } applyRunEvent(acc, event); applyNodeEvent(acc, event); + applyMediaJobEvent(acc, event); applyGateEvent(acc, event); } @@ -251,6 +312,7 @@ export function reconstructCheckpointState( nodeId: entry.nodeId, isBudgetGate: entry.isBudgetGate, })), + pendingMediaJobs: [...acc.pendingMediaJobs.values()], resolvedGateIds: [...acc.resolvedGateIds], lastSequenceNumber: acc.lastSequenceNumber, totalInputTokens: acc.totalInputTokens, diff --git a/packages/core/src/engine/engine.test.ts b/packages/core/src/engine/engine.test.ts index 4183b793..ce1c6f72 100644 --- a/packages/core/src/engine/engine.test.ts +++ b/packages/core/src/engine/engine.test.ts @@ -141,6 +141,21 @@ function seedStarted(store: RunStore, runId: string, lastType?: RunEvent['type'] message: 'approve?', }); } + if (lastType === 'media_job:submitted') { + events.push({ + type: 'media_job:submitted', + runId, + timestamp: startedAt, + sequenceNumber: 1, + nodeId: 'gen', + jobId: 'vendor-op-1', + provider: 'openai', + model: 'sora-2', + modality: 'video', + startedAt, + deadlineAt: '2026-06-13T00:30:00.000Z', + }); + } return Promise.all(events.map((e) => store.persistEvent(e))).then(() => undefined); } @@ -2338,6 +2353,17 @@ describe('WorkflowEngine — crash reconciliation', () => { expect(await engine.reconcile()).toHaveLength(0); }); + it('leaves a media-job-parked run (crash in the submit→pause window) for resume, not reconciliation (1.AG/ADR-0045 §2-3)', async () => { + // A crash AFTER media_job:submitted persisted but BEFORE run:paused leaves media_job:submitted as the + // durable last event. reconcile() must NOT fail it — the paid, still-generating provider LRO is + // re-attachable via resumeFromCheckpoint (re-poll the opaque jobId, never re-submit). Failing it here + // would orphan the job (billed, output discarded) and permanently close the run at the terminal gate. + const store = new InMemoryRunStore(); + await seedStarted(store, 'media-parked-1', 'media_job:submitted'); + const engine = engineWith(undefined, createInMemoryHost({ store })); + expect(await engine.reconcile()).toHaveLength(0); + }); + it('reclaims a crashed run’s media references at reconciliation (1.AF/D11 — no orphaned partial media)', async () => { // A crashed non-resumable run never ran its in-process terminal sweep; reconcile() must reclaim its // `run`-kind refs, else the partial media stays refcount>0 forever and is never GC-eligible (ADR-0042 §4). diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index 95c236fe..3b0d87c7 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -31,6 +31,7 @@ import { DEFAULT_MAX_MEDIA_DOWNLOAD_BYTES, GateDecisionSchema, + MEDIA_JOB_POLL_DEFAULTS, RETRYABLE_ERROR_CODES, RunEventSchema, collectDurableMediaHandles, @@ -39,7 +40,9 @@ import { type DurableMediaMeta, type ExecutionMode, type GateDecision, + type LlmProviderId, type MaskedSecret, + type MediaBilledModality, type MediaUrlFetch, type NodeSkippedReason, type Retry, @@ -47,6 +50,7 @@ import { type RunStatus, type TokensUsed, } from '@relavium/shared'; +import type { MediaJobStatus } from '@relavium/llm'; import { buildRunPlan, type BuildRunPlanOptions } from '../dag.js'; import { InterpolationError } from '../errors.js'; @@ -62,12 +66,15 @@ import type { CheckpointState } from './checkpoint.js'; import type { AbortControllerLike, ExecutionHost } from './execution-host.js'; import type { GateRequest, + MediaJobSubmission, NodeExecContext, NodeExecutor, NodeFailure, NodeOutcome, NodeStreamEvent, } from './node-executor.js'; +import { codeForLlmError } from './agent-turn.js'; +import { DEFAULT_MEDIA_UNIT_ESTIMATE, generativeUnits, realizedMediaCost } from './agent-runner.js'; import { createClosedRunHandle, createRunHandle, type RunHandle } from './run-handle.js'; /** A vertex's live status in one run. `paused` (at a gate) and `running` are not yet *settled*. */ @@ -80,6 +87,29 @@ interface VertexState { selectedTargets?: ReadonlySet; } +/** + * A parked async media job (1.AG Section D, [ADR-0045](../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)) — + * the in-memory record the engine polls. Keyed by `nodeId` (the AG-A-FC-3 disambiguator vs `#pendingGates`, + * which is keyed by gateId). The credential is NEVER stored here (re-resolved per poll via the executor); + * `units` is the authored volume for the lone realized cost addend (recomputed from the node config on + * cross-process resume); `backoffMs` grows per poll (exp, capped at `pollMaxMs`). + */ +interface ParkedMediaJob { + readonly jobId: string; + readonly provider: LlmProviderId; + readonly model: string; + readonly modality: MediaBilledModality; + readonly units: number; + /** ISO-8601 absolute deadline; a poll where `now > deadlineAt` abandons the job as a retryable timeout. */ + readonly deadlineAt: string; + /** Elapsed-ms-since-run-start at submission — the basis for the completed node's full LRO `durationMs` + * (submit→done wall-clock, not the ~0 of the synchronous settle). Recoverable across a resume from the + * persisted `startedAt` (M2). */ + readonly submittedAtMs: number; + /** The current poll interval (ms), grown exponentially per `pending` poll, capped at `pollMaxMs`. */ + backoffMs: number; +} + /** A vertex status counts as *settled* (its dependents can evaluate) when it is one of these. */ const SETTLED: ReadonlySet = new Set([ 'completed', @@ -157,9 +187,37 @@ export interface ResumeFromCheckpointInput { /** MUST match the run's original mode (not yet checkpoint-derived — see the interface note). */ readonly executionMode?: ExecutionMode; readonly planOptions?: BuildRunPlanOptions; - /** The gate to resolve + the decision to apply (the run was suspended at this gate). */ - readonly gateId: string; - readonly decision: GateDecision; + /** + * The gate to resolve + the decision to apply, when the run was suspended at a HUMAN GATE. **Both omitted** + * for a run suspended ONLY on an async media job (1.AG Section D, ADR-0045 §3) — that resume re-attaches + + * re-polls the persisted job(s) with no external decision. When a run is parked on BOTH a gate and a media + * job (AG-A-FC-3), pass the gate's `gateId`/`decision`: the gate decision advances while `#seedFromCheckpoint` + * independently re-attaches the parked media job(s). + */ + readonly gateId?: string; + readonly decision?: GateDecision; +} + +/** + * Validate a {@link ResumeFromCheckpointInput}'s gate fields (1.R / 1.AG Section D). A gate resume supplies + * BOTH `gateId` + `decision`; a media-only resume supplies NEITHER — a half-supplied pair is a caller misuse, + * and a supplied `decision` must parse. Throws a typed {@link EngineStateError} (`invalid_decision`); pulled + * out of `resumeFromCheckpoint` so the entry stays within the cognitive-complexity budget. + */ +function assertValidResumeInput(input: ResumeFromCheckpointInput): void { + if ((input.gateId === undefined) !== (input.decision === undefined)) { + throw new EngineStateError( + 'invalid_decision', + 'resumeFromCheckpoint needs BOTH gateId + decision (a gate resume) or NEITHER (a media-job resume)', + { runId: input.runId, ...(input.gateId === undefined ? {} : { gateId: input.gateId }) }, + ); + } + if (input.decision !== undefined && !GateDecisionSchema.safeParse(input.decision).success) { + throw new EngineStateError('invalid_decision', 'the gate decision failed validation', { + runId: input.runId, + ...(input.gateId === undefined ? {} : { gateId: input.gateId }), + }); + } } /** Construction dependencies for the engine — the injected host and node-executor seams. */ @@ -233,6 +291,10 @@ class RunExecution { readonly #resolvedGates = new Set(); /** Disarm callbacks for armed gate-timeout timers, by gateId — disarmed on resume / settle (1.Q). */ readonly #gateTimers = new Map void>(); + /** Parked async media jobs by nodeId (1.AG Section D) — the engine-owned poll/checkpoint/resume/cancel loop. */ + readonly #pendingMediaJobs = new Map(); + /** Disarm callbacks for armed media-job poll timers, by nodeId — disarmed on settle/cancel (ADR-0045 §4). */ + readonly #mediaJobTimers = new Map void>(); /** The run-level wall-clock timeout timer, when a `timeout_ms` is configured (ADR-0028). */ #runTimeoutDisarm: (() => void) | undefined; /** The pre-egress budget governor, when a workflow `budget` is configured (ADR-0028, 1.AC). */ @@ -461,6 +523,50 @@ class RunExecution { isBudgetGate: gate.isBudgetGate, }); } + // Re-attach each parked async media job (MJ-1, ADR-0045 §3): re-register it + RE-ARM a poll of the + // persisted opaque jobId. NEVER re-call generateMedia — the node is `'paused'` (applyMediaJobEvent set it), + // not absent, so it is not re-run via the `'pending'` path; this overrides the checkpoint + // running-at-crash-re-runs default for the async-media node specifically. `units` is NOT persisted in the + // slot — recompute it from the node config (count/duration_seconds), which IS persisted in the workflow. + // Unlike a gate (whose decision arrives externally), a media job has no external trigger — only the + // engine's own re-poll advances it, so the re-arm is unconditional here. A past-deadline job is + // short-circuited to a timeout by the first `#pollMediaJob` (which checks `now > deadlineAt`). + for (const job of cp.pendingMediaJobs) { + const vertex = plan.vertices.get(job.nodeId); + // The agent branch is the only one ever taken in practice — a media job is ALWAYS sourced from an agent + // vertex (executeGenerativeMedia), so `generativeUnits` (which honors the authored count/duration_seconds, + // matching the submit-side compute exactly) is the real path. The else is a defensive last resort for a + // vertex that is missing or non-agent (an invariant violation, e.g. a workflow edited between processes); + // there is no AgentNode to read, so the conservative per-modality DEFAULT is the only available estimate — + // it never executes on a well-formed resume. + const units = + vertex?.config.kind === 'agent' + ? generativeUnits(job.modality, vertex.config.node) + : DEFAULT_MEDIA_UNIT_ESTIMATE[job.modality]; + this.#pendingMediaJobs.set(job.nodeId, { + jobId: job.jobId, + provider: job.provider, + model: job.model, + modality: job.modality, + units, + deadlineAt: job.deadlineAt, + // Recompute elapsed-at-submit from the persisted `startedAt` against the ORIGINAL run start so a + // completed re-attached job reports its full submit→done wall-clock `durationMs` (M2). `cp.startedAtMs` + // is the same value seeded into `#startEpochMs` below. + submittedAtMs: Date.parse(job.startedAt) - cp.startedAtMs, + backoffMs: MEDIA_JOB_POLL_DEFAULTS.pollInitialMs, + }); + this.#armMediaPoll(job.nodeId); + } + // Suppress a DUPLICATE `run:paused` on resume ONLY when the prior process actually announced one — i.e. the + // checkpoint's runStatus is already `'paused'` (L2). In the CRASH-IN-WINDOW case (`media_job:submitted` + // persisted but `run:paused` not — RUN_STATUS_BY_EVENT has no entry for `media_job:submitted`, so runStatus + // stays `'running'`) the prior process never emitted `run:paused`, so the resumed process MUST emit it + // (gap-free stream, ADR-0036) — leaving `#pauseEpisode` false here. A gate resume independently resets this + // in `resume()`; `#clearMediaJob` resets it when a job settles — so a genuinely later pause still emits. + if (cp.pendingMediaJobs.length > 0 && cp.runStatus === 'paused') { + this.#pauseEpisode = true; + } for (const gateId of cp.resolvedGateIds) { this.#resolvedGates.add(gateId); } @@ -512,6 +618,68 @@ class RunExecution { } } + /** + * Resume a run suspended ONLY on async media job(s) (1.AG Section D, ADR-0045 §3) — no gate decision. The + * media jobs were already re-attached + re-armed by `#seedFromCheckpoint` (MJ-1); this just re-resolves the + * workflow context (not checkpointed) and kicks the loop, which the armed poll timers then advance. A run + * with no pending media job AND no pending gate is a misuse (`run_not_paused`). + */ + async beginResumeMediaJobs(): Promise { + this.#armRunTimeout(); + if (this.#pendingMediaJobs.size === 0) { + // A media-only resume REQUIRES a parked media job. A run parked ONLY on a human gate must be resumed via + // the gate form (gateId + decision) — resuming it here would silently re-park it forever (no decision). + throw new EngineStateError( + 'run_not_paused', + 'the run has no pending media job to re-attach (a gate-parked run resumes via gateId + decision)', + { runId: this.runId }, + ); + } + if (this.#pendingGates.size > 0) { + // The run is parked on BOTH a media job AND a human gate (AG-A-FC-3) but the caller supplied no gate + // decision (the media-only resume form). Re-attaching the media job alone would leave the gate + // unresolved: after the job settles the run would silently re-park on the gate with no caller signal. + // Reject the misuse eagerly — the caller must pass the gate's gateId + decision (which advances the gate + // while `#seedFromCheckpoint` independently re-attaches the media job). + throw new EngineStateError( + 'pending_gate_requires_decision', + "the run is also parked on a human gate — resume with that gate's gateId + decision (a media-only resume cannot resolve it)", + { runId: this.runId }, + ); + } + if (!(await this.#resolveContextOrFail())) { + await this.#settle(this.#cancelling ? 'run:cancelled' : 'run:failed'); + return; + } + this.#schedule(); + } + + /** + * Tear down a half-initialized execution that is being REJECTED before it ever ran for the caller — an + * invalid `resumeFromCheckpoint` form whose validation guard threw AFTER the constructor's + * `#seedFromCheckpoint` armed the parked jobs' media-poll timers (and after `beginResume*` armed the + * run-timeout). Disarm every armed timer + abort so NO orphaned poll later hits the provider for a run the + * caller saw rejected (which would also let a natural retry double-attach the same opaque jobId). Emits + * NOTHING and runs no terminal — it is an abandon, not a settle; the façade drops the execution from `#runs`. + */ + abandon(): void { + if (this.#settled) { + return; // already torn down (a real settle ran) — idempotent + } + this.#settled = true; // any straggler timer callback now short-circuits on the #settled guard + this.#abort.abort(); + for (const disarm of this.#gateTimers.values()) { + disarm(); + } + this.#gateTimers.clear(); + for (const disarm of this.#mediaJobTimers.values()) { + disarm(); + } + this.#mediaJobTimers.clear(); + this.#pendingMediaJobs.clear(); + this.#disarmRunTimeout(); + } + requestCancel(): void { if (this.#settled) { throw new EngineStateError('run_already_terminal', 'the run has already terminated', { @@ -760,7 +928,11 @@ class RunExecution { if (running > 0) { return; // still executing — wait for the next settlement to re-evaluate } - if (this.#pendingGates.size > 0) { + if (this.#pendingGates.size > 0 || this.#pendingMediaJobs.size > 0) { + // Parked on a human gate AND/OR an async media job (1.AG Section D, MJ-1) — the run is PAUSED, not + // stalled. A media-parked node is invisible to #claimReady (it claims only 'pending'), so without this + // the run would fall through to the loud stall-fail below; the poll timer (live) or the + // #seedFromCheckpoint re-attach (resume) advances it. Reuses the human-gate pause emit (ADR-0045 §2). await this.#emitPausedOnce(); return; } @@ -1020,6 +1192,9 @@ class RunExecution { case 'paused': await this.#settlePaused(vertex, outcome.gate); break; + case 'media_job': + await this.#settleMediaJobParked(vertex, outcome.job); + break; } } catch { // Backstop while settling a node: a bus/Zod stamp failure on a malformed event, OR — by design — a @@ -1155,6 +1330,260 @@ class RunExecution { } } + // --- Async media-job loop (1.AG Section D, ADR-0045) ----------------------------------------- + + /** + * A `media_job` outcome (ADR-0045 §2/§3): park the node for the engine-owned poll loop. Set status + * `'paused'` (slot-free, like a gate — keyed by nodeId, the AG-A-FC-3 disambiguator vs `#pendingGates`), + * record the job, emit the durable `media_job:submitted`, and arm the first poll. The realized cost is + * emitted by the poll loop at `done`, NEVER here (§5). + */ + async #settleMediaJobParked(vertex: PlanVertex, job: MediaJobSubmission): Promise { + const state = this.#states.get(vertex.id); + if (state !== undefined) { + state.status = 'paused'; + } + const startedAt = this.#host.clock.now(); + const deadlineAt = new Date( + Date.parse(startedAt) + (job.deadlineMs ?? MEDIA_JOB_POLL_DEFAULTS.deadlineMs), + ).toISOString(); + this.#pendingMediaJobs.set(vertex.id, { + jobId: job.jobId, + provider: job.provider, + model: job.model, + modality: job.modality, + units: job.units, + deadlineAt, + // Derive from the SAME `startedAt` that is persisted on `media_job:submitted`, so the resume-side + // recompute (from the checkpoint slot) yields an identical value (M2). + submittedAtMs: Date.parse(startedAt) - this.#startEpochMs, + backoffMs: MEDIA_JOB_POLL_DEFAULTS.pollInitialMs, + }); + await this.#emitDurable({ + type: 'media_job:submitted', + runId: this.runId, + nodeId: vertex.id, + jobId: job.jobId, + provider: job.provider, + model: job.model, + modality: job.modality, + startedAt, + deadlineAt, + }); + this.#armMediaPoll(vertex.id); + } + + /** Arm (or re-arm) the one-shot poll timer for a parked media job via the INJECTED host timer (never an + * ambient `setTimeout` — engine purity). Disarm-then-arm so a re-arm never leaks a prior timer. */ + #armMediaPoll(nodeId: string): void { + const job = this.#pendingMediaJobs.get(nodeId); + if (job === undefined) { + return; + } + this.#disarmMediaTimer(nodeId); + const disarm = this.#host.setTimer(job.backoffMs, () => { + void this.#pollMediaJob(nodeId); + }); + this.#mediaJobTimers.set(nodeId, disarm); + } + + /** Disarm and forget a media-job poll timer (idempotent). */ + #disarmMediaTimer(nodeId: string): void { + const disarm = this.#mediaJobTimers.get(nodeId); + if (disarm !== undefined) { + this.#mediaJobTimers.delete(nodeId); + disarm(); + } + } + + /** Remove a parked media job + disarm its timer (on done/failed). Resetting `#pauseEpisode` mirrors the gate + * `resume()` so a LATER pause (a downstream gate / a second media job parking in a fresh idle cycle) re-emits + * an accurate aggregate `run:paused` instead of being suppressed. */ + #clearMediaJob(nodeId: string): void { + this.#pendingMediaJobs.delete(nodeId); + this.#disarmMediaTimer(nodeId); + this.#pauseEpisode = false; + } + + /** Emit the lone realized media-cost addend for a job (ADR-0045 §5) — folded into the run cumulative + + * streamed by `#nodeEmit`. Emitted exactly once per job: at `done`, or — for a paid job abandoned by a + * fail/deadline/cancel — at that settle (the provider bills regardless, a cost-integrity requirement). */ + #emitMediaJobCost(nodeId: string, job: ParkedMediaJob): void { + this.#nodeEmit({ + type: 'cost:updated', + nodeId, + model: job.model, + inputTokens: 0, + outputTokens: 0, + costMicrocents: realizedMediaCost(job.model, job.modality, job.units), + cumulativeCostMicrocents: 0, // #nodeEmit overwrites with the authoritative run-wide total + }); + } + + /** + * One poll of a parked media job (ADR-0045 §3). Idempotent against a settled run / cleared job. Past the + * deadline → abandon as a retryable `provider_unavailable` timeout (a node-retry MAY re-submit; the loop + * itself never silently re-submits). Otherwise delegate the poll to the executor (which owns provider + + * credential resolution), passing the run abort signal so a cancel aborts the in-flight poll. + */ + async #pollMediaJob(nodeId: string): Promise { + this.#disarmMediaTimer(nodeId); + const job = this.#pendingMediaJobs.get(nodeId); + if (this.#settled || job === undefined) { + return; // run terminal, or the job was already cleared (nothing to clean up) + } + const vertex = this.#plan.vertices.get(nodeId); + if (vertex === undefined) { + // The parked node no longer exists in the plan — only reachable via same-slug workflow CONTENT drift on + // resume (the identity guard checks the surrogate workflow id, not content). A silent return would strand + // the run paused forever on a job that can never re-attach. The provider billed the submitted job + // regardless of the drift, so emit its lone realized cost addend BEFORE clearing (ADR-0045 §5: exactly + // one addend on EVERY terminal path — there is no vertex to settle node:failed against, but the cost is + // still owed). Then clear + drive the loop so the now-jobless idle settles the run instead of hanging. + this.#emitMediaJobCost(nodeId, job); + this.#clearMediaJob(nodeId); + this.#schedule(); + return; + } + // The whole settle path is wrapped: a synchronous bus/Zod throw (or a #nodeEmit fault) must NOT escape the + // fire-and-forget `void #pollMediaJob` as an unhandled rejection — route it to a single run:failed instead + // (mirroring the #onOutcome backstop), keeping the run total for faults. + try { + if (Date.parse(this.#host.clock.now()) > Date.parse(job.deadlineAt)) { + await this.#settleMediaJobFailed(vertex, job, { + code: 'provider_unavailable', + message: `media job '${job.jobId}' exceeded its deadline (${job.deadlineAt})`, + retryable: true, + }); + return; + } + if (this.#executor.pollMediaJob === undefined) { + await this.#settleMediaJobFailed(vertex, job, { + code: 'internal', + message: 'the executor implements no pollMediaJob (host-wiring gap)', + retryable: false, + }); + return; + } + const submission: MediaJobSubmission = { + jobId: job.jobId, + provider: job.provider, + model: job.model, + modality: job.modality, + units: job.units, + }; + let status: MediaJobStatus; + try { + status = await this.#executor.pollMediaJob(submission, this.#abort.signal); + } catch { + // A cancel (the abort surfaced as a throw) / terminal / cleared job → return silently; the #settle + // path emits run:cancelled. Only a genuine poll fault on a live job settles node:failed. + if (this.#settled || this.#abort.signal.aborted || !this.#pendingMediaJobs.has(nodeId)) { + return; + } + // A raw throw escaping the executor poll on a LIVE job (the missing-adapter + credential cases are + // already mapped to a `failed` status upstream) is a transient provider/transport fault — classify it + // like the deadline path: a retryable `provider_unavailable`, so when async-node retry wiring lands + // (1.AH) a transient poll fault re-submits a fresh job rather than being a dead-end `internal` (N2). + await this.#settleMediaJobFailed(vertex, job, { + code: 'provider_unavailable', + message: `media job '${job.jobId}' poll failed`, + retryable: true, + }); + return; + } + if (this.#settled || this.#abort.signal.aborted || !this.#pendingMediaJobs.has(nodeId)) { + return; // a cancel / terminal raced the poll await — let #settle close the run + } + await this.#applyMediaJobStatus(vertex, job, status); + } catch { + if (!this.#settled) { + this.#clearMediaJob(nodeId); + this.#failNodeInternal(vertex, 'the media job poll loop failed while settling the node'); + // Drive the loop so `#step` observes `#failure` and settles `run:failed`. Unlike `#onOutcome` (whose + // backstop is followed by an unconditional `#schedule()`), this poll is fired out-of-band from a timer + // — nothing else re-enters the loop, so without this the run would hang at `run:paused` forever (M1). + this.#schedule(); + } + } + } + + /** Route one `MediaJobStatus` to: re-arm (pending) / complete (done) / fail (failed). */ + async #applyMediaJobStatus( + vertex: PlanVertex, + job: ParkedMediaJob, + status: MediaJobStatus, + ): Promise { + switch (status.state) { + case 'pending': { + // Exponential backoff (no jitter) capped at pollMaxMs, then re-arm. Progress is TRANSIENT (ADR-0045 + // §2) — never persisted; no run event today. + job.backoffMs = Math.min(job.backoffMs * 2, MEDIA_JOB_POLL_DEFAULTS.pollMaxMs); + this.#armMediaPoll(vertex.id); + return; + } + case 'done': + await this.#settleMediaJobDone(vertex, job, status.media); + return; + case 'failed': + await this.#settleMediaJobFailed(vertex, job, { + code: codeForLlmError(status.error), + message: status.error.message, + retryable: status.error.retryable, + }); + return; + default: + // Defense-in-depth: an out-of-union job state (a future seam value / a non-conforming adapter) would + // otherwise fall through, leaving the node parked with no re-arm and no terminal — a silent hang. Fail + // loudly with a terminal node:failed instead (the closed-union switch above type-checks without this). + await this.#settleMediaJobFailed(vertex, job, { + code: 'internal', + message: 'media poll returned an unrecognized job state', + retryable: false, + }); + } + } + + /** A media job resolved `done`: emit the lone realized cost addend, then drive the node to `completed` + * through `#onOutcome` (which de-inlines the media at `#emitDurable` + re-schedules — the out-of-band poll + * is not on the `#onOutcome→#schedule` path, so completion must re-enter it). */ + async #settleMediaJobDone( + vertex: PlanVertex, + job: ParkedMediaJob, + media: Extract['media'], + ): Promise { + this.#clearMediaJob(vertex.id); + this.#emitMediaJobCost(vertex.id, job); // the lone realized cost:updated (ADR-0045 §5) + // The pure-media node output ({ text:'', media }) matches the SYNC generative shape exactly (so a downstream + // {{ outputs.x.text }} resolves to '' regardless of sync-vs-LRO) and de-inlines to a media:// handle at + // #emitDurable (the I3 boundary); `media` is the seam MediaPart (base64 or a re-hostable url). + await this.#onOutcome( + vertex, + { + kind: 'completed', + output: { text: '', media: [media] }, + tokensUsed: { input: 0, output: 0, model: job.model }, + }, + // The job's submit time (not `now`) — so `node:completed.durationMs` is the full async wall-clock + // (submit→done), parallel to how `#dispatch` captures `startedAtMs` before a synchronous node's await (M2). + job.submittedAtMs, + ); + } + + /** A media job failed / timed out: emit the paid job's lone cost addend (§5 — the provider billed even though + * it failed), clear it, and drive the node to `failed` through `#onOutcome`. */ + async #settleMediaJobFailed( + vertex: PlanVertex, + job: ParkedMediaJob, + error: NodeFailure, + ): Promise { + this.#clearMediaJob(vertex.id); + this.#emitMediaJobCost(vertex.id, job); + // `startedAtMs` is unused by the `failed` settle path (no `durationMs`), but pass the submit time for + // symmetry with the `done` path (M2). + await this.#onOutcome(vertex, { kind: 'failed', error }, job.submittedAtMs); + } + /** * A gate's timeout elapsed with no decision (1.Q). Idempotent: a no-op once the gate resolved (a human * beat the timer — resume disarmed it, but a fired-and-queued callback still guards here) or the run @@ -1197,7 +1626,9 @@ class RunExecution { /** Mark a vertex failed and fail the run (unless already cancelling/failing) — the internal backstop. */ #failNodeInternal(vertex: PlanVertex, message: string): void { const state = this.#states.get(vertex.id); - if (state?.status === 'running') { + // A media-parked node is `'paused'`, not `'running'`, when its poll loop's settle-path backstop fires — + // transition it to `'failed'` too so the in-memory state matches the run's terminal outcome (L1). + if (state?.status === 'running' || state?.status === 'paused') { state.status = 'failed'; } if (!this.#settled && this.#failure === undefined && !this.#cancelling) { @@ -1212,11 +1643,13 @@ class RunExecution { } this.#pauseEpisode = true; const gateIds = [...this.#pendingGates.keys()]; + const mediaJobNodeIds = [...this.#pendingMediaJobs.keys()]; await this.#emitDurable({ type: 'run:paused', runId: this.runId, pendingGateCount: gateIds.length, gateIds, + ...(mediaJobNodeIds.length === 0 ? {} : { pendingMediaJobNodeIds: mediaJobNodeIds }), }); } @@ -1226,11 +1659,26 @@ class RunExecution { } this.#settled = true; this.#abort.abort(); // make sure any straggler executor sees cancellation - // The run is closing — no gate timer may fire afterwards (1.Q). Disarm each, then clear in one shot. + // The run is closing — no gate or media-poll timer may fire afterwards (1.Q / ADR-0045 §4). Disarm each, + // then clear in one shot. The #abort.abort() above also aborts any in-flight pollMediaJob (the signal is + // threaded into the executor poll), so a cancelled job's open provider request is dropped, not just its + // next schedule. for (const disarm of this.#gateTimers.values()) { disarm(); } this.#gateTimers.clear(); + // A paid media job still pending at the terminal (a cancel, or a sibling's failure abandoning it) was + // billed by the provider even though its output is discarded — emit its lone cost addend before clearing + // (ADR-0045 §5, the local-only-cancel cost-integrity caveat). run:completed never reaches here with a + // pending job (each completes + clears at its own `done`). Emit BEFORE the terminal so the run total folds it. + for (const [nodeId, job] of this.#pendingMediaJobs) { + this.#emitMediaJobCost(nodeId, job); + } + for (const disarm of this.#mediaJobTimers.values()) { + disarm(); + } + this.#mediaJobTimers.clear(); + this.#pendingMediaJobs.clear(); this.#budgetApprovedVertices.clear(); // drop any unconsumed budget-approval (a sibling failure/cancel // can settle the run between resume() arming it and the re-dispatch — no stale entry on the retained run) this.#disarmRunTimeout(); @@ -1786,13 +2234,9 @@ export class WorkflowEngine { * closed by a Phase-2 store-level uniqueness constraint, not the in-memory reference (checkpoint.ts). */ async resumeFromCheckpoint(input: ResumeFromCheckpointInput): Promise { - const parsed = GateDecisionSchema.safeParse(input.decision); - if (!parsed.success) { - throw new EngineStateError('invalid_decision', 'the gate decision failed validation', { - runId: input.runId, - gateId: input.gateId, - }); - } + // A gate resume supplies gateId + decision; a media-ONLY resume (1.AG Section D) supplies neither. + const isGateResume = input.gateId !== undefined && input.decision !== undefined; + assertValidResumeInput(input); // a half-supplied pair, or a malformed decision, is a caller misuse if (this.#runs.has(input.runId)) { throw new EngineStateError( 'run_already_active', @@ -1848,16 +2292,25 @@ export class WorkflowEngine { this.#runs.set(input.runId, execution); try { // beginResume re-resolves the workflow context (not checkpointed) then drives: kick if the gate was - // already resolved in the prior process (no re-apply), else apply the decision. The events buffer on - // the returned handle for the consumer. - await execution.beginResume( - input.gateId, - parsed.data, - checkpoint.resolvedGateIds.includes(input.gateId), - ); + // already resolved in the prior process (no re-apply), else apply the decision. A media-ONLY resume + // (no gate) re-attaches + re-polls the parked job(s). The events buffer on the returned handle. + if (isGateResume && input.gateId !== undefined && input.decision !== undefined) { + await execution.beginResume( + input.gateId, + input.decision, + checkpoint.resolvedGateIds.includes(input.gateId), + ); + } else { + await execution.beginResumeMediaJobs(); + } } catch (error) { - // resume() validates the gate AFTER rehydration; an unknown_gate / run_not_paused throw must not - // strand the half-initialized execution in #runs (a retry would then wrongly hit run_already_active). + // resume() validates the gate AFTER rehydration; an unknown_gate / run_not_paused / + // pending_gate_requires_decision throw must not strand the half-initialized execution in #runs (a retry + // would then wrongly hit run_already_active) NOR leave its armed timers firing — the constructor's + // #seedFromCheckpoint already armed a media-poll timer per parked job and beginResume* armed the + // run-timeout. Abandon (disarm + abort) BEFORE dropping it, else an orphan poll would later hit the + // provider for a run the caller saw rejected (and a natural retry could double-attach the same jobId). + execution.abandon(); this.#runs.delete(input.runId); throw error; } diff --git a/packages/core/src/engine/errors.ts b/packages/core/src/engine/errors.ts index 97f04c2a..e21a32ff 100644 --- a/packages/core/src/engine/errors.ts +++ b/packages/core/src/engine/errors.ts @@ -21,6 +21,7 @@ export type EngineStateErrorCode = | 'run_not_paused' // `resume` was called while the run has no pending gate to resolve | 'unknown_gate' // the `gateId` does not match any gate currently pending on the run | 'invalid_decision' // the supplied `GateDecision` failed schema validation at the boundary + | 'pending_gate_requires_decision' // a media-only `resumeFromCheckpoint` hit a run also parked on a gate (pass gateId + decision) | 'workflow_mismatch'; // `resumeFromCheckpoint` was handed a workflow that is not the one the run started on /** diff --git a/packages/core/src/engine/execution-host.test.ts b/packages/core/src/engine/execution-host.test.ts index 496adba7..38832d15 100644 --- a/packages/core/src/engine/execution-host.test.ts +++ b/packages/core/src/engine/execution-host.test.ts @@ -149,6 +149,22 @@ describe('InMemoryRunStore', () => { spentMicrocents: 100, limitMicrocents: 50, }, + { + // An async media-job park whose `run:paused` never persisted (crash in the submit→pause window) is + // STILL resumable — the run re-attaches the parked job via the derived pendingMediaJobs slot (1.AG, + // ADR-0045 §2-3); reconciling it to run:failed would orphan a paid provider LRO. + type: 'media_job:submitted', + runId: 'r1', + timestamp: at, + sequenceNumber: 1, + nodeId: 'gen', + jobId: 'vendor-op-1', + provider: 'openai', + model: 'sora-2', + modality: 'video', + startedAt: at, + deadlineAt: '2026-06-13T00:30:00.000Z', + }, ]; for (const last of lastEvents) { const store = new InMemoryRunStore(); diff --git a/packages/core/src/engine/execution-host.ts b/packages/core/src/engine/execution-host.ts index f850a216..a4c7d5e4 100644 --- a/packages/core/src/engine/execution-host.ts +++ b/packages/core/src/engine/execution-host.ts @@ -210,6 +210,12 @@ const RESUMABLE_LAST_TYPES: ReadonlySet = new Set([ 'human_gate:paused', 'run:paused', 'budget:paused', + // An async media-job park (1.AG Section D, ADR-0045 §2-3): `media_job:submitted` is persisted in its own + // turn BEFORE the later `run:paused`, so a crash in that window leaves it as the durable last event. The + // run is re-attachable — the checkpoint fold derives a `pendingMediaJobs` slot from it and + // `resumeFromCheckpoint` re-polls the opaque jobId (never re-submits). Reconciling it to `run:failed` would + // orphan a paid, still-generating provider LRO, so it must be left for the resume path — like a gate park. + 'media_job:submitted', ]); /** Format a counter into a syntactically-valid (RFC-4122-shaped) UUID — deterministic for tests. */ diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 89280b7d..8c76b0da 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -26,8 +26,21 @@ * not on the curated public surface). */ -import type { CapabilityFlags, LlmProvider, ProviderId, StreamChunk } from '@relavium/llm'; -import { RunEventSchema, type RunEvent } from '@relavium/shared'; +import type { + CapabilityFlags, + LlmProvider, + MediaJobStatus, + ProviderId, + StreamChunk, +} from '@relavium/llm'; +import { + RunEventSchema, + type AbortSignalLike, + type ContentPart, + type MediaReferencePort, + type MediaStore, + type RunEvent, +} from '@relavium/shared'; import { beforeAll, describe, expect, it } from 'vitest'; import { createExpressionSandbox, type ExpressionSandbox } from '../expression/sandbox.js'; @@ -112,6 +125,185 @@ const retryableError = (providerId: ProviderId): StreamChunk => ({ error: { kind: 'overloaded', retryable: true, provider: providerId, message: 'busy' }, }); +/** Caps advertising inline image output (chat surface) — so the chain keeps a media-output model. */ +const MEDIA_CAPS: CapabilityFlags = { + ...CAPS, + media: { + input: { image: false, audio: false, video: false, document: false }, + outputCombinations: [['text', 'image']], + surface: 'chat', + }, +}; + +/** A 5-byte in-flight base64 image part — the media-out fixture ("hello"). Typed as the media variant so it + * satisfies both a `ContentPart[]` (inline turn) and `MediaGenResult.media` (generative) without a cast. */ +const IMAGE_PART: Extract = { + type: 'media', + mimeType: 'image/png', + source: { kind: 'base64', data: 'aGVsbG8=' }, +}; + +/** A provider whose non-streaming generate() returns inline media; its stream THROWS (media routes to generate). */ +function mediaProvider(id: ProviderId = 'gemini'): LlmProvider { + return { + id, + supports: MEDIA_CAPS, + generate: () => + Promise.resolve({ + content: [{ type: 'text', text: 'here is your image' }, IMAGE_PART], + stopReason: 'stop', + usage, + }), + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for an inline media-out turn'); + }, + }; +} + +/** A provider flagged media_surface 'generative' whose generateMedia returns a SYNC image (1.AG Section C); + * generate/stream THROW — proving a generative node routes to the separate endpoint, never the inline turn. */ +function generativeMediaProvider(id: ProviderId = 'openai'): LlmProvider { + return { + id, + supports: { ...MEDIA_CAPS, media: { ...MEDIA_CAPS.media, surface: 'generative' } }, + generate: () => { + throw new Error('generate must NOT run for a generative node'); + }, + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a generative node'); + }, + generateMedia: () => Promise.resolve({ media: IMAGE_PART, raw: { internal: true } }), + }; +} + +/** A provider whose generateMedia SUBMITS an async job ({ jobId }) and whose pollMediaJob replays a scripted + * MediaJobStatus sequence (call N → script[N]; the last entry repeats). generate/stream throw. Records the + * generateMedia + pollMediaJob call counts (the re-attach test asserts generateMedia is called ONCE). */ +function asyncMediaProvider( + script: readonly MediaJobStatus[], + id: ProviderId = 'openai', +): { provider: LlmProvider; generateCalls: () => number; pollCalls: () => number } { + let generateCalls = 0; + let pollCalls = 0; + const provider: LlmProvider = { + id, + supports: { ...MEDIA_CAPS, media: { ...MEDIA_CAPS.media, surface: 'generative' } }, + generate: () => { + throw new Error('generate must NOT run for a generative node'); + }, + stream: (): AsyncIterable => { + throw new Error('stream must NOT run for a generative node'); + }, + generateMedia: () => { + generateCalls += 1; + return Promise.resolve({ jobId: 'job-1', raw: { internal: true } }); + }, + pollMediaJob: (jobId: string, _key: string, signal?: AbortSignalLike) => { + if (signal?.aborted === true) { + return Promise.reject(new Error('poll aborted')); // a run cancel aborts the in-flight poll (ADR-0045 §4) + } + if (jobId !== 'job-1') { + // The engine must re-poll the PERSISTED opaque jobId generateMedia returned — especially across a + // cross-process resume RE-ATTACH (MJ-1). A mismatch means a wrong/regenerated id is being polled. + return Promise.reject( + new Error( + `pollMediaJob got unexpected jobId '${jobId}' (expected the persisted 'job-1')`, + ), + ); + } + const status = script[Math.min(pollCalls, script.length - 1)]; + pollCalls += 1; + return Promise.resolve(status ?? { state: 'pending' }); + }, + }; + return { provider, generateCalls: () => generateCalls, pollCalls: () => pollCalls }; +} + +/** + * Drive a run that parks on an async media job to its terminal. A `pending` poll re-arms a timer but emits NO + * event (progress is transient — ADR-0045 §2), so the run cannot be driven purely off the event stream: consume + * events in the background and FIRE armed poll timers until the run settles (each fire = one poll). `onPaused` + * (e.g. cross-process resume / cancel) runs once when the run first parks. + */ +async function driveMediaRun( + handle: RunHandle, + host: Host, + onPaused?: () => Promise | void, +): Promise { + const events: RunEvent[] = []; + let settled = false; + let paused = false; + const consume = (async (): Promise => { + for await (const event of handle.events) { + events.push(event); + if ( + event.type === 'run:completed' || + event.type === 'run:failed' || + event.type === 'run:cancelled' + ) { + settled = true; + } + } + })(); + let guard = 0; + while (!settled) { + await Promise.resolve(); + if (!paused && events.some((e) => e.type === 'run:paused')) { + paused = true; + if (onPaused !== undefined) await onPaused(); + } + if (host.armedCount() > 0) { + host.fireTimers(); + } + if ((guard += 1) > 100_000) { + throw new Error('driveMediaRun did not settle'); + } + } + await consume; + return events; +} + +const isRecord = (v: unknown): v is Record => typeof v === 'object' && v !== null; + +/** Extract the first media handle ref from a `{ media: [{ source: { ref } }] }` node output via a runtime + * guard (no unsafe cast) — the de-inlined durable form both media-out e2e tests assert. */ +function firstMediaRef(out: unknown): string | undefined { + if (!isRecord(out) || !Array.isArray(out['media'])) return undefined; + const first: unknown = out['media'][0]; + if (!isRecord(first) || !isRecord(first['source'])) return undefined; + const ref = first['source']['ref']; + return typeof ref === 'string' ? ref : undefined; +} + +/** A pure fake-digest in-memory MediaStore (no crypto) — content-addressed enough for the e2e. */ +function stubMediaStore(): MediaStore { + const puts: { handle: string; bytes: Uint8Array }[] = []; + const digest = (bytes: Uint8Array): string => { + let hex = ''; + for (let seed = 0; seed < 8; seed += 1) { + let h = (2166136261 ^ (seed * 0x9e3779b1)) >>> 0; + for (const b of bytes) h = Math.imul(h ^ b, 16777619) >>> 0; + hex += h.toString(16).padStart(8, '0'); + } + return hex; + }; + return { + put: (bytes) => { + const handle = `media://sha256-${digest(bytes)}`; + puts.push({ handle, bytes }); + return Promise.resolve(handle); + }, + get: (handle) => { + const found = puts.find((p) => p.handle === handle); + return found === undefined + ? Promise.reject(new Error('no bytes')) + : Promise.resolve(found.bytes); + }, + resolveForEgress: () => Promise.reject(new Error('unused by this test')), + readRange: () => Promise.reject(new Error('unused by this test')), + }; +} + // --- Tool stubs: a sanitized echo registry + its LLM-visible def (mirror agent-runner.e2e.test.ts) ---- const echoRegistry: ToolRegistry = { @@ -193,6 +385,123 @@ workflow: `, ); +/** Inline media-out — an agent node requesting image output (1.AG Section B, ADR-0046). */ +const MEDIA_OUT = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-media-out + inputs: + - { name: topic, type: string } + agents: + - id: painter + model: gemini-2.5-flash + provider: gemini + system_prompt: You make images. + nodes: + - { id: in, type: input } + - { id: work, type: agent, agent_ref: painter, prompt_template: 'Draw: {{inputs.topic}}', output_modalities: [text, image] } + - { id: out, type: output } + edges: + - { from: in, to: work } + - { from: work, to: out } +`, +); + +/** Generative media-out — an agent node on a media_surface:'generative' model (1.AG Section C, ADR-0045). */ +const GENERATIVE_OUT = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-generative-out + inputs: + - { name: topic, type: string } + agents: + - id: painter + model: gpt-image-1 + provider: openai + system_prompt: You make images. + nodes: + - { id: in, type: input } + - { id: work, type: agent, agent_ref: painter, prompt_template: 'Draw: {{inputs.topic}}', output_modalities: [image], count: 1 } + - { id: out, type: output } + edges: + - { from: in, to: work } + - { from: work, to: out } +`, +); + +/** GENERATIVE_OUT after same-slug CONTENT drift — identical `workflow.id` (so the resume identity guard, which + * compares only the surrogate id, passes) but the `work` media node is GONE (in→out direct). Resuming a + * checkpoint whose pendingMediaJobs references `work` against this drifted plan exercises the orphaned-vertex + * path in #pollMediaJob (vertex === undefined). */ +const GENERATIVE_OUT_DRIFTED = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-generative-out + inputs: + - { name: topic, type: string } + nodes: + - { id: in, type: input } + - { id: out, type: output } + edges: + - { from: in, to: out } +`, +); + +/** A human gate AND a generative media node parked CONCURRENTLY (1.AG Section D, AG-A-FC-3) — two parallel + * branches from `in`, joined at `out`. The resume applies the gate decision while the media job re-attaches. */ +const GATE_AND_MEDIA = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-gate-and-media + inputs: + - { name: topic, type: string } + agents: + - id: painter + model: gpt-image-1 + provider: openai + system_prompt: You make images. + nodes: + - { id: in, type: input } + - { id: gen, type: agent, agent_ref: painter, prompt_template: 'Draw: {{inputs.topic}}', output_modalities: [image], count: 1 } + - { id: g, type: human_gate, gate_type: approval } + - { id: out, type: output } + edges: + - { from: in, to: gen } + - { from: in, to: g } + - { from: gen, to: out } + - { from: g, to: out } +`, +); + +// GATE_AND_MEDIA plus a whole-run `timeout_ms` — used ONLY by the abandon() guard test, which rejects a resume +// WITHOUT ever firing timers (so the inert 1h timeout never trips the manual controller, unlike the timer-firing +// driveMediaRun tests). It exists so the rejected-resume armedCount()===0 assertion also covers abandon()'s +// run-timeout disarm leg (#runTimeoutDisarm), not just the media-poll leg. +const GATE_AND_MEDIA_TIMED = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-gate-and-media-timed + timeout_ms: 3600000 + inputs: + - { name: topic, type: string } + agents: + - id: painter + model: gpt-image-1 + provider: openai + system_prompt: You make images. + nodes: + - { id: in, type: input } + - { id: gen, type: agent, agent_ref: painter, prompt_template: 'Draw: {{inputs.topic}}', output_modalities: [image], count: 1 } + - { id: g, type: human_gate, gate_type: approval } + - { id: out, type: output } + edges: + - { from: in, to: gen } + - { from: in, to: g } + - { from: gen, to: out } + - { from: g, to: out } +`, +); + const INPUTS = { topic: 'the report' } as const; // --- The reusable driver --------------------------------------------------------------------------- @@ -202,6 +511,7 @@ type Host = ReturnType; function buildEngine( host: Host, resolveProvider: (id: ProviderId) => LlmProvider | undefined, + resolveMediaSurface?: (model: string) => 'chat' | 'generative' | undefined, ): WorkflowEngine { return new WorkflowEngine({ host, @@ -209,6 +519,7 @@ function buildEngine( sandbox, agent: { resolveProvider, + ...(resolveMediaSurface === undefined ? {} : { resolveMediaSurface }), registry: echoRegistry, tools: [echoToolDef], keyFor: () => 'k', @@ -323,6 +634,547 @@ describe('M2 — end-to-end Node harness (1.U)', () => { assertCanonicalSchema(events); }); + it('inline media-out: an agent node requesting image output routes to generate() and de-inlines to a handle (1.AG/ADR-0046)', async () => { + // The end-to-end proof of the "previously-missing integration link": a real agent node (output_modalities + // [text, image]) → generate() (its stream throws) → { text, media } node output → the engine de-inlines + // the in-flight base64 to a media:// handle at #emitDurable, gap-free, with NO base64 on the durable stream. + const host = createInMemoryHost({ mediaStore: stubMediaStore() }); + const engine = buildEngine(host, () => mediaProvider('gemini')); + const { events } = await drive(engine.start({ workflow: MEDIA_OUT, inputs: INPUTS }), host); + + expect(events.at(-1)?.type).toBe('run:completed'); + const out = nodeOutput(events, 'work'); + expect(out).toMatchObject({ + text: 'here is your image', + media: [{ type: 'media', mimeType: 'image/png', source: { kind: 'handle' } }], + }); + // The de-inlined source is a canonical media:// sha256 handle (the durable form — never a raw byte carrier). + expect(firstMediaRef(out)).toMatch(/^media:\/\/sha256-[0-9a-f]{64}$/); + // I3 — the in-flight base64 ("hello") never appears on the delivered (or persisted) run-event stream. + expect(JSON.stringify(events)).not.toContain('aGVsbG8='); + + assertGapFreeSeq(events); + assertCanonicalSchema(events); + }); + + it('generative media-out: an agent node on a generative model routes to generateMedia and de-inlines to a handle (1.AG Section C/ADR-0045)', async () => { + // The end-to-end proof of the generative SYNC link: a real agent node on a media_surface:'generative' model + // → generateMedia (its generate/stream throw) → { text:'', media } node output → the engine de-inlines the + // in-flight base64 to a media:// handle at #emitDurable, gap-free, with NO base64 on the durable stream. + const host = createInMemoryHost({ mediaStore: stubMediaStore() }); + const engine = buildEngine( + host, + () => generativeMediaProvider('openai'), + () => 'generative', + ); + const { events } = await drive( + engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:completed'); + const out = nodeOutput(events, 'work'); + expect(out).toMatchObject({ + text: '', // a generative node is PURE media — no accompanying chat text + media: [{ type: 'media', mimeType: 'image/png', source: { kind: 'handle' } }], + }); + expect(firstMediaRef(out)).toMatch(/^media:\/\/sha256-[0-9a-f]{64}$/); + // I3 — no base64 on the delivered (or persisted) stream; and MediaGenResult.raw (the provider-internal + // diagnostic) is structurally excluded from the node output (strip-on-sink, ADR-0045 §7) — the stub's + // raw:{ internal:true } must never appear. Exactly one realized cost:updated for the node. + expect(JSON.stringify(events)).not.toContain('aGVsbG8='); + expect(JSON.stringify(events)).not.toContain('"internal":true'); + expect(costsOf(events).filter((c) => c.nodeId === 'work')).toHaveLength(1); + + assertGapFreeSeq(events); + assertCanonicalSchema(events); + }); + + it('async media job: parks, polls (pending then done), de-inlines to a handle, completes (1.AG Section D/ADR-0045)', async () => { + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + const job = asyncMediaProvider([{ state: 'pending' }, { state: 'done', media: IMAGE_PART }]); + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + ); + const events = await driveMediaRun( + engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:completed'); + expect(events.some((e) => e.type === 'media_job:submitted')).toBe(true); + expect(events.some((e) => e.type === 'run:paused')).toBe(true); // PARKED, not a stall-fail (MJ-1) + const out = nodeOutput(events, 'work'); + expect(out).toMatchObject({ + media: [{ type: 'media', mimeType: 'image/png', source: { kind: 'handle' } }], + }); + expect(firstMediaRef(out)).toMatch(/^media:\/\/sha256-[0-9a-f]{64}$/); + expect(JSON.stringify(events)).not.toContain('aGVsbG8='); // I3 — the in-flight bytes never go durable + expect(job.generateCalls()).toBe(1); // submitted once + expect(job.pollCalls()).toBeGreaterThanOrEqual(2); // pending then done + expect(costsOf(events).filter((c) => c.nodeId === 'work')).toHaveLength(1); // the lone realized addend (§5) + assertGapFreeSeq(events); + assertCanonicalSchema(events); + }); + + it('async media job: a content-policy poll failure settles node:failed (content_filter) → run:failed', async () => { + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + const job = asyncMediaProvider([ + { + state: 'failed', + error: { provider: 'openai', kind: 'content_filter', retryable: false, message: 'blocked' }, + }, + ]); + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + ); + const events = await driveMediaRun( + engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host, + ); + expect(events.at(-1)?.type).toBe('run:failed'); + const failedEvent = events.find((e) => e.type === 'node:failed'); + expect(failedEvent?.type === 'node:failed' && failedEvent.error.code).toBe('content_filter'); + // The FAILED path still emits exactly one realized cost addend (the provider billed the paid job even + // though it was content-filtered — ADR-0045 §5); pin it so a dropped #emitMediaJobCost on the failed arm + // is a loud regression, not silent (L1). + expect(costsOf(events).filter((c) => c.nodeId === 'work')).toHaveLength(1); + }); + + it('async media job: a cross-process resume RE-ATTACHES (re-polls) the persisted jobId — never re-submits (MJ-1)', async () => { + const store = new InMemoryRunStore(); + // Process 1: submit + park, then "crash" (break at run:paused; the poll timer is never fired). + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + ); + const { events: events1 } = await drive( + engine1.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + expect(events1.some((e) => e.type === 'media_job:submitted')).toBe(true); + expect(job1.generateCalls()).toBe(1); + const runId = events1[0]?.runId ?? ''; + + // Process 2: a fresh engine resumes purely from the store — re-attaches + re-polls to done. The opaque + // jobId is re-polled; generateMedia is NEVER called again (ADR-0045 §3, the no-double-submit rule). + const job2 = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const host2 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine2 = buildEngine( + host2, + () => job2.provider, + () => 'generative', + ); + const events2 = await driveMediaRun( + await engine2.resumeFromCheckpoint({ runId, workflow: GENERATIVE_OUT, inputs: INPUTS }), + host2, + ); + expect(events2.at(-1)?.type).toBe('run:completed'); + expect(job2.generateCalls()).toBe(0); // RE-ATTACH, never re-submit + expect(job2.pollCalls()).toBeGreaterThanOrEqual(1); + expect(firstMediaRef(nodeOutput(events2, 'work'))).toMatch(/^media:\/\/sha256-[0-9a-f]{64}$/); + // The prior process already emitted run:paused for this park; the resume must NOT re-announce it (L2). + expect(events2.filter((e) => e.type === 'run:paused')).toHaveLength(0); + // The lone realized cost addend fires on the resume→done path too (exactly one — ADR-0045 §5). + expect(costsOf(events2).filter((c) => c.nodeId === 'work')).toHaveLength(1); + }); + + it('async media job: a run cancel aborts the in-flight poll → run:cancelled (ADR-0045 §4)', async () => { + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + const job = asyncMediaProvider([{ state: 'pending' }]); // never resolves on its own + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + ); + const handle = engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }); + // Cancel once the run parks on the (never-resolving) media job: the run reaches run:cancelled through the + // real engine cancel path. This stub detects the abort at poll-call entry, so it proves the cancel SETTLES + // the run — the dedicated mid-await abort-observation proof is the deterministic test below. + const events = await driveMediaRun(handle, host, () => handle.cancel()); + expect(events.at(-1)?.type).toBe('run:cancelled'); + }); + + it('async media job: a cancel of a parked job fires the terminal media sweep (#reclaimRunMedia, ADR-0045 §4)', async () => { + const reclaims: string[] = []; + const mediaReferences: MediaReferencePort = { + recordRunMedia: () => undefined, + reclaimRun: (runId) => { + reclaims.push(runId); + }, + }; + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + mediaReferences, + }); + const job = asyncMediaProvider([{ state: 'pending' }]); // never resolves on its own + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + ); + const handle = engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }); + const events = await driveMediaRun(handle, host, () => handle.cancel()); + expect(events.at(-1)?.type).toBe('run:cancelled'); + const runId = events[0]?.runId ?? ''; + // The terminal sweep must reclaim the run's media references on the CANCEL terminal too (not only on a + // happy-path run:completed) — a parked job's run-scoped refs (incl. any done→cancel-race produced byte) + // are GC-eligible the instant the run settles. Exactly one sweep, for this run. + expect(reclaims).toEqual([runId]); + // The #settle sweep also emits the paid job's lone realized cost addend before clearing it (the provider + // billed the cancelled-but-still-generating job — ADR-0045 §5); pin exactly one so a dropped cancel-path + // #emitMediaJobCost is a loud regression (L1). + expect(costsOf(events).filter((c) => c.nodeId === 'work')).toHaveLength(1); + }); + + it('AG-A-FC-3: a run parked on BOTH a human gate AND a media job resumes both — gate decision + media re-attach', async () => { + const store = new InMemoryRunStore(); + // Process 1: both branches park (the gate + the media job); break at the first run:paused. + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + ); + const { events: events1, gateId } = await drive( + engine1.start({ workflow: GATE_AND_MEDIA, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + expect(events1.some((e) => e.type === 'media_job:submitted')).toBe(true); + expect(events1.some((e) => e.type === 'human_gate:paused')).toBe(true); + const paused = events1.find((e) => e.type === 'run:paused'); + // run:paused carries BOTH reasons (the AG-A-FC-3 disambiguator). + expect( + paused?.type === 'run:paused' && + paused.gateIds.length === 1 && + paused.pendingMediaJobNodeIds?.includes('gen'), + ).toBe(true); + const runId = events1[0]?.runId ?? ''; + expect(gateId).toBeDefined(); + + // Process 2: resume WITH the gate decision — the gate advances (decision) AND the media job re-attaches + // (re-poll, no decision). Both reach terminal; exactly one run terminal. + const job2 = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const host2 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine2 = buildEngine( + host2, + () => job2.provider, + () => 'generative', + ); + const events2 = await driveMediaRun( + await engine2.resumeFromCheckpoint({ + runId, + workflow: GATE_AND_MEDIA, + inputs: INPUTS, + gateId: gateId ?? '', + decision: { decision: 'approved', decidedBy: 'human' }, + }), + host2, + ); + expect(events2.at(-1)?.type).toBe('run:completed'); + expect(events2.filter((e) => e.type === 'run:completed')).toHaveLength(1); // exactly one terminal + expect(job2.generateCalls()).toBe(0); // the media job re-attached, never re-submitted + expect(firstMediaRef(nodeOutput(events2, 'gen'))).toMatch(/^media:\/\/sha256-[0-9a-f]{64}$/); + }); + + it('AG-A-FC-3 guard: a media-only resume (no gate decision) of a both-parked run is rejected (pending_gate_requires_decision)', async () => { + const store = new InMemoryRunStore(); + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + ); + const { events: events1 } = await drive( + engine1.start({ workflow: GATE_AND_MEDIA_TIMED, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + const runId = events1[0]?.runId ?? ''; + + const host2 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine2 = buildEngine( + host2, + () => asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]).provider, + () => 'generative', + ); + // Neither gateId nor decision: the media-only resume form cannot resolve the still-parked gate. The engine + // must reject the misuse EAGERLY rather than re-attach the media job and silently re-park on the gate after + // it settles (L3). The half-initialized execution is dropped from #runs (a retry can re-resume correctly). + await expect( + engine2.resumeFromCheckpoint({ runId, workflow: GATE_AND_MEDIA_TIMED, inputs: INPUTS }), + ).rejects.toMatchObject({ code: 'pending_gate_requires_decision' }); + // The rejected resume must leave NO armed timer behind: before the guard threw, the constructor's + // #seedFromCheckpoint armed a media-poll timer for the parked job AND beginResumeMediaJobs armed the + // whole-run timeout (GATE_AND_MEDIA_TIMED declares timeout_ms) — the catch must abandon (disarm) BOTH legs, else + // an orphan timer would later poll the provider for a run the caller saw rejected (and a retry could + // double-attach the same jobId). armedCount()===0 pins both the #mediaJobTimers and #runTimeoutDisarm + // disarm legs of abandon() (H2). + expect(host2.armedCount()).toBe(0); + }); + + it('async media job: a resume past the deadline short-circuits to a retryable timeout, never re-polls (ADR-0045 §3)', async () => { + const store = new InMemoryRunStore(); + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); // clock ~2026-01-01 + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + ); + const { events: events1 } = await drive( + engine1.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + const runId = events1[0]?.runId ?? ''; + + // Process 2: a clock FAR past the 30-min default deadline → the re-attach's first poll short-circuits to a + // retryable timeout BEFORE polling (the provider has surely dropped the job). The done-script never runs. + const job2 = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const host2 = createInMemoryHost({ + store, + mediaStore: stubMediaStore(), + baseEpochMs: Date.parse('2026-02-01T00:00:00.000Z'), + }); + const engine2 = buildEngine( + host2, + () => job2.provider, + () => 'generative', + ); + const events2 = await driveMediaRun( + await engine2.resumeFromCheckpoint({ runId, workflow: GENERATIVE_OUT, inputs: INPUTS }), + host2, + ); + expect(events2.at(-1)?.type).toBe('run:failed'); + const failed = events2.find((e) => e.type === 'node:failed'); + expect(failed?.type === 'node:failed' && failed.error.code).toBe('provider_unavailable'); + expect(failed?.type === 'node:failed' && failed.error.retryable).toBe(true); + expect(job2.pollCalls()).toBe(0); // short-circuited before polling + expect(job2.generateCalls()).toBe(0); // never re-submitted + // The deadline-abandoned job was still provider-billed → exactly one realized cost addend (ADR-0045 §5). + expect(costsOf(events2).filter((c) => c.nodeId === 'work')).toHaveLength(1); + }); + + it('async media job: an orphaned-vertex resume (content drift) still emits the paid job cost addend (H1, ADR-0045 §5)', async () => { + const store = new InMemoryRunStore(); + // Process 1: park the 'work' media job (then "crash" at run:paused). + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + ); + const { events: events1 } = await drive( + engine1.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + const runId = events1[0]?.runId ?? ''; + + // Process 2: resume against the DRIFTED workflow (same id, no 'work' node). #seedFromCheckpoint re-attaches + // the parked 'work' job, but the poll finds no vertex → the orphaned-vertex branch must emit the paid job's + // lone cost addend before clearing it (not silently drop it, the H1 bug). generateMedia/pollMediaJob are + // never (re-)invoked for 'work' — the branch short-circuits before the executor poll. + const job2 = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const host2 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine2 = buildEngine( + host2, + () => job2.provider, + () => 'generative', + ); + const events2 = await driveMediaRun( + await engine2.resumeFromCheckpoint({ + runId, + workflow: GENERATIVE_OUT_DRIFTED, + inputs: INPUTS, + }), + host2, + ); + expect(events2.at(-1)?.type).toBe('run:completed'); // the drifted plan (in→out) finishes; 'work' is orphaned + expect(costsOf(events2).filter((c) => c.nodeId === 'work')).toHaveLength(1); // the paid orphan is still billed + expect(job2.generateCalls()).toBe(0); // never re-submitted + expect(job2.pollCalls()).toBe(0); // the orphaned-vertex branch short-circuits before the executor poll + }); + + it('async media job: a crash-in-window resume (no persisted run:paused) emits run:paused on resume (H2, ADR-0036)', async () => { + const store = new InMemoryRunStore(); + const job1 = asyncMediaProvider([{ state: 'pending' }]); + const host1 = createInMemoryHost({ store, mediaStore: stubMediaStore() }); + const engine1 = buildEngine( + host1, + () => job1.provider, + () => 'generative', + ); + const { events: events1 } = await drive( + engine1.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host1, + { breakOnPause: true }, + ); + const runId = events1[0]?.runId ?? ''; + + // Simulate the CRASH-IN-WINDOW: the prior process persisted media_job:submitted but crashed BEFORE + // run:paused. Copy every event EXCEPT run:paused into a fresh store, so the checkpoint folds runStatus + // 'running' (RUN_STATUS_BY_EVENT has no media_job:submitted entry) — the resumed process MUST emit run:paused. + const crashStore = new InMemoryRunStore(); + for (const e of store.eventsFor(runId)) { + if (e.type !== 'run:paused') { + await crashStore.persistEvent(e); + } + } + const job2 = asyncMediaProvider([{ state: 'done', media: IMAGE_PART }]); + const host2 = createInMemoryHost({ store: crashStore, mediaStore: stubMediaStore() }); + const engine2 = buildEngine( + host2, + () => job2.provider, + () => 'generative', + ); + const events2 = await driveMediaRun( + await engine2.resumeFromCheckpoint({ runId, workflow: GENERATIVE_OUT, inputs: INPUTS }), + host2, + ); + expect(events2.at(-1)?.type).toBe('run:completed'); + // The prior process never announced run:paused, so the resumed process must — exactly once, before terminal. + const pausedIdx = events2.findIndex((e) => e.type === 'run:paused'); + expect(events2.filter((e) => e.type === 'run:paused')).toHaveLength(1); + expect(pausedIdx).toBeGreaterThanOrEqual(0); + expect(pausedIdx).toBeLessThan(events2.length - 1); // before the terminal + expect(job2.generateCalls()).toBe(0); // re-attach, not re-submit + }); + + it('async media job: an IN-FLIGHT poll is aborted by a run cancel (deterministic) → run:cancelled (ADR-0045 §4)', async () => { + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + let observedAbort = false; + const provider: LlmProvider = { + id: 'openai', + supports: { ...MEDIA_CAPS, media: { ...MEDIA_CAPS.media, surface: 'generative' } }, + generate: () => { + throw new Error('generate must NOT run'); + }, + stream: (): AsyncIterable => { + throw new Error('stream must NOT run'); + }, + generateMedia: () => Promise.resolve({ jobId: 'job-1', raw: {} }), + // A poll that stays IN-FLIGHT until the run abort signal fires (then rejects) — proving the cancel + // reaches the open provider request, not just the next schedule. + pollMediaJob: (_jobId: string, _key: string, signal?: AbortSignalLike) => + new Promise((_resolve, reject) => { + if (signal?.aborted === true) { + reject(new Error('aborted')); + return; + } + signal?.addEventListener('abort', () => { + observedAbort = true; + reject(new Error('aborted')); + }); + }), + }; + const engine = buildEngine( + host, + () => provider, + () => 'generative', + ); + const handle = engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }); + const events = await driveMediaRun(handle, host, () => handle.cancel()); + expect(events.at(-1)?.type).toBe('run:cancelled'); + expect(observedAbort).toBe(true); // the in-flight poll observed the abort + }); + + it('async media job: an UNRECOGNIZED poll state fails the node loud (no silent park-forever hang) — the default arm (N2)', async () => { + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + const provider: LlmProvider = { + id: 'openai', + supports: { ...MEDIA_CAPS, media: { ...MEDIA_CAPS.media, surface: 'generative' } }, + generate: () => { + throw new Error('generate must NOT run'); + }, + stream: (): AsyncIterable => { + throw new Error('stream must NOT run'); + }, + generateMedia: () => Promise.resolve({ jobId: 'job-1', raw: {} }), + // An out-of-union job state (a future seam value / a non-conforming adapter) forces the closed-switch's + // default arm, which MUST fail the node terminally rather than leave it parked with no re-arm (silent hang). + pollMediaJob: () => + // @ts-expect-error — 'frozen' is intentionally outside the MediaJobStatus union (see above). + Promise.resolve({ state: 'frozen' }), + }; + const engine = buildEngine( + host, + () => provider, + () => 'generative', + ); + const events = await driveMediaRun( + engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host, + ); + expect(events.at(-1)?.type).toBe('run:failed'); // settled, not hung + const failed = events.find((e) => e.type === 'node:failed'); + expect(failed?.type === 'node:failed' && failed.error.code).toBe('internal'); + expect(failed?.type === 'node:failed' && failed.error.message).toContain( + 'unrecognized job state', + ); + }); + + it('async media job: survives repeated pending polls (backoff re-arm) before done', async () => { + const host = createInMemoryHost({ + store: new InMemoryRunStore(), + mediaStore: stubMediaStore(), + }); + const job = asyncMediaProvider([ + { state: 'pending' }, + { state: 'pending' }, + { state: 'pending' }, + { state: 'done', media: IMAGE_PART }, + ]); + const engine = buildEngine( + host, + () => job.provider, + () => 'generative', + ); + const events = await driveMediaRun( + engine.start({ workflow: GENERATIVE_OUT, inputs: INPUTS }), + host, + ); + expect(events.at(-1)?.type).toBe('run:completed'); + expect(job.pollCalls()).toBeGreaterThanOrEqual(4); // 3 pending + done + expect(events.filter((e) => e.type === 'node:completed' && e.nodeId === 'work')).toHaveLength( + 1, + ); + expect(costsOf(events).filter((c) => c.nodeId === 'work')).toHaveLength(1); // still exactly one addend + // The completed node reports the full submit→done wall-clock, not the ~0 of the synchronous settle (M2): + // each poll advanced the harness clock, so the elapsed across the 3 pending polls is strictly positive. + const done = events.find((e) => e.type === 'node:completed' && e.nodeId === 'work'); + expect(done?.type === 'node:completed' && done.durationMs > 0).toBe(true); + }); + it('flagship: one run — retry then failover, pause at the gate, cross-process resume reproduces the final output', async () => { // The primary (anthropic) ALWAYS errors retryably pre-content; the fallback (openai) fails the first // dispatch then succeeds — so dispatch 1 exhausts the chain (→ node retry), dispatch 2 fails over to the diff --git a/packages/core/src/engine/node-executor.ts b/packages/core/src/engine/node-executor.ts index 8e5eef02..3734fb59 100644 --- a/packages/core/src/engine/node-executor.ts +++ b/packages/core/src/engine/node-executor.ts @@ -21,10 +21,14 @@ import type { AbortSignalLike, ErrorCode, HumanGatePausedEvent, + LlmProviderId, + MediaBilledModality, TokensUsed, ToolPolicy, } from '@relavium/shared'; +import type { MediaJobStatus } from '@relavium/llm'; + import type { RunEventDraft } from './event-bus.js'; import type { PlanVertex } from '../run-plan.js'; @@ -91,6 +95,27 @@ export interface GateRequest { readonly isBudgetGate?: boolean; } +/** + * An async media-generation job the executor submitted but cannot await synchronously (Sora/Veo, + * minute-scale LROs — 1.AG Section D, [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * The executor resolved the provider + submitted the job (`generateMedia` → `{ jobId }`); the ENGINE then + * owns the poll/checkpoint/resume/cancel loop. The opaque `jobId` is re-polled (never re-submitted), so the + * record carries everything the loop needs to poll + price the result without the executor: `provider` + + * `model` re-resolve the adapter + credential (the key is NEVER persisted — re-resolved on resume), `modality` + * de-inlines + prices the `done` media, and `units` is the authored volume for the lone realized cost addend + * (ADR-0045 §5). `units` is recomputed from the node config on cross-process resume (it is not persisted). + */ +export interface MediaJobSubmission { + readonly jobId: string; + readonly provider: LlmProviderId; + readonly model: string; + readonly modality: MediaBilledModality; + /** The authored output volume (count for image; duration_seconds for audio/video) → the realized cost addend. */ + readonly units: number; + /** Optional override of `MEDIA_JOB_POLL_DEFAULTS.deadlineMs`; the engine computes `deadlineAt` against its clock. */ + readonly deadlineMs?: number; +} + /** * The result of executing one vertex. Discriminated on `kind`: * - `completed` — the node produced an `output` (and, for an LLM node, `tokensUsed`). @@ -100,6 +125,8 @@ export interface GateRequest { * live; every other dependent of the condition is skip-propagated by the loop. * - `paused` — a `human_in_the_loop` node suspends the run; the loop emits `human_gate:paused`, parks * the gate, and continues other branches until it resumes via `engine.resume`. + * - `media_job` — a generative node submitted an async media job; the loop emits `media_job:submitted`, + * parks the node (reusing the gate suspend machinery), and drives the poll loop to terminal (1.AG Section D). */ export type NodeOutcome = | { readonly kind: 'completed'; readonly output: unknown; readonly tokensUsed?: TokensUsed } @@ -110,7 +137,8 @@ export type NodeOutcome = readonly selected: readonly string[]; readonly tokensUsed?: TokensUsed; } - | { readonly kind: 'paused'; readonly gate: GateRequest }; + | { readonly kind: 'paused'; readonly gate: GateRequest } + | { readonly kind: 'media_job'; readonly job: MediaJobSubmission }; /** The context handed to a node executor for one dispatch of one vertex. */ export interface NodeExecContext { @@ -160,4 +188,13 @@ export interface NodeExecContext { /** The injected per-vertex executor. 1.O (`AgentRunner`) and 1.P (node handlers) implement it. */ export interface NodeExecutor { execute(ctx: NodeExecContext): Promise; + /** + * Poll an async media job the executor previously submitted (1.AG Section D, ADR-0045 §3). The ENGINE owns + * the poll/checkpoint/resume/cancel loop, but provider + credential resolution lives in the executor (the + * `resolveProvider`/`keyFor` capabilities), so the engine delegates the actual poll here. The executor + * re-resolves the adapter + key from `job.provider` and calls `provider.pollMediaJob(job.jobId, key, signal)`. + * `signal` aborts the in-flight poll on a run cancel. Optional — an executor with no generative providers + * (or before Section D is wired) omits it; the engine treats its absence as a host-wiring gap (`internal`). + */ + pollMediaJob?(job: MediaJobSubmission, signal: AbortSignalLike): Promise; } diff --git a/packages/core/src/engine/node-handlers/dispatcher.ts b/packages/core/src/engine/node-handlers/dispatcher.ts index 5f215144..01777dbf 100644 --- a/packages/core/src/engine/node-handlers/dispatcher.ts +++ b/packages/core/src/engine/node-handlers/dispatcher.ts @@ -25,6 +25,12 @@ export type NodeExecutorMap = Partial>; /** Compose a per-type handler map into the one `NodeExecutor` the engine dispatches every vertex through. */ export function createDispatchingNodeExecutor(handlers: NodeExecutorMap): NodeExecutor { + // A media job always originates from an `agent` vertex (executeGenerativeMedia, 1.AG Section D), so the + // engine's poll delegates to the agent handler. Forward `pollMediaJob` only when an agent handler with the + // capability is wired. The `.bind(agent)` is a no-op for behaviour today (the handler's `pollMediaJob` is an + // arrow that closes over its `deps` and never reads `this`); it is kept as the lint-sanctioned, future-proof + // form should that property ever become a `this`-referencing method. + const agent = handlers.agent; return { execute(ctx) { const handler = handlers[ctx.vertex.type]; @@ -35,6 +41,7 @@ export function createDispatchingNodeExecutor(handlers: NodeExecutorMap): NodeEx } return handler.execute(ctx); }, + ...(agent?.pollMediaJob === undefined ? {} : { pollMediaJob: agent.pollMediaJob.bind(agent) }), }; } diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 4c297a96..f7d6dc56 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -120,6 +120,7 @@ export type { CheckpointState, CheckpointNodeState, CheckpointPendingGate, + CheckpointPendingMediaJob, } from './engine/checkpoint.js'; export type { ExecutionHost, diff --git a/packages/core/src/validate-catalog.test.ts b/packages/core/src/validate-catalog.test.ts index 39f2f1a8..4914c11b 100644 --- a/packages/core/src/validate-catalog.test.ts +++ b/packages/core/src/validate-catalog.test.ts @@ -36,6 +36,32 @@ describe('validateWorkflowWithCatalog (1.AF/D15 — output_modalities load-check expect(() => validateWorkflowWithCatalog(wf, catalog)).not.toThrow(); }); + it('SKIPS the load-check for a generative-surface model (output is the generateMedia modality, not outputCombinations) — 1.AG Section C', () => { + // A generative model's outputCombinations is empty (chat-surface only); without the surface skip this + // would WRONGLY reject a valid generative node (output_modalities: [image]) once D15 is host-wired. + const wf = agentWorkflow(", model: gpt-image-1, output_modalities: ['image']"); + const generativeCaps: CapabilityFlags = { + ...caps([]), + media: { ...caps([]).media, surface: 'generative' }, + }; + const catalog: WorkflowModelCatalog = () => generativeCaps; + expect(() => validateWorkflowWithCatalog(wf, catalog)).not.toThrow(); + }); + + it('THROWS for a generative-surface model with an invalid output_modalities shape (text mixed / two media) — 1.AG Section C', () => { + // A generative model produces pure SINGLE-modality media (the runtime singleBilledModality rule): text + // mixed in, or two media modalities, must fail fast at LOAD — not only at runtime dispatch. + const generativeCaps: CapabilityFlags = { + ...caps([]), + media: { ...caps([]).media, surface: 'generative' }, + }; + const catalog: WorkflowModelCatalog = () => generativeCaps; + const textMixed = agentWorkflow(", model: gpt-image-1, output_modalities: ['text', 'image']"); + expect(() => validateWorkflowWithCatalog(textMixed, catalog)).toThrow(WorkflowValidationError); + const twoMedia = agentWorkflow(", model: gpt-image-1, output_modalities: ['image', 'audio']"); + expect(() => validateWorkflowWithCatalog(twoMedia, catalog)).toThrow(WorkflowValidationError); + }); + it('throws a field-named WorkflowValidationError when the model cannot output the combination', () => { const wf = agentWorkflow(", model: m1, output_modalities: ['text', 'image']"); const catalog: WorkflowModelCatalog = () => caps([['text']]); // text-only model diff --git a/packages/core/src/validate-catalog.ts b/packages/core/src/validate-catalog.ts index 69ece354..985c688a 100644 --- a/packages/core/src/validate-catalog.ts +++ b/packages/core/src/validate-catalog.ts @@ -1,8 +1,19 @@ import { isOutputCombinationSupported, type CapabilityFlags } from '@relavium/llm'; +import { + MEDIA_BILLED_MODALITIES, + type MediaBilledModality, + type OutputModality, +} from '@relavium/shared'; import { WorkflowValidationError, type WorkflowIssue } from './errors.js'; import type { WorkflowDefinition } from './parser.js'; +/** Type guard: a media-billed output modality (image | audio | video) — no `text`. Avoids an `as` cast when + * narrowing a node's `output_modalities` to the billed subset (the generative one-modality load-check). */ +function isBilledModality(modality: OutputModality): modality is MediaBilledModality { + return MEDIA_BILLED_MODALITIES.some((billed) => billed === modality); +} + /** * A host-provided model → {@link CapabilityFlags} lookup (1.AF/D15, ADR-0044 §2) — sourced from the DB * `model_catalog`. Returns `undefined` for a model the host cannot resolve, in which case the load-check @@ -33,6 +44,22 @@ export function validateWorkflowWithCatalog( if (caps === undefined) { continue; // unresolvable model — defer to the runtime FallbackChain pre-skip (never a silent drop) } + if (caps.media.surface === 'generative') { + // A `media_surface: 'generative'` model (gpt-image-1, Imagen, TTS) routes to `generateMedia` (1.AG + // Section C, ADR-0045 §1); its producible output is defined by the generateMedia modality, NOT by the + // inline `outputCombinations` (which is empty / chat-surface only). The inline membership check does not + // apply — but the SAME one-media-modality rule the runtime dispatch enforces (`singleBilledModality`: + // exactly one of image|audio|video, no text) IS checked here, so a malformed generative node fails fast + // at load rather than only at runtime. + const billed = node.output_modalities.filter(isBilledModality); + if (node.output_modalities.length !== 1 || billed.length !== 1) { + issues.push({ + field: `node \`${node.id}\`.output_modalities`, + message: `a media_surface 'generative' model requires output_modalities to declare exactly one media modality (image | audio | video) with no text, got [${node.output_modalities.join(', ')}]`, + }); + } + continue; // the inline outputCombinations load-check does not apply to a generative model + } if (!isOutputCombinationSupported(caps.media.outputCombinations, node.output_modalities)) { issues.push({ field: `node \`${node.id}\`.output_modalities`, diff --git a/packages/db/drizzle/0004_numerous_ultimatum.sql b/packages/db/drizzle/0004_numerous_ultimatum.sql new file mode 100644 index 00000000..12d8491d --- /dev/null +++ b/packages/db/drizzle/0004_numerous_ultimatum.sql @@ -0,0 +1 @@ +ALTER TABLE `model_catalog` ADD `media_surface` text DEFAULT 'chat' NOT NULL; \ No newline at end of file diff --git a/packages/db/drizzle/meta/0004_snapshot.json b/packages/db/drizzle/meta/0004_snapshot.json new file mode 100644 index 00000000..640c4964 --- /dev/null +++ b/packages/db/drizzle/meta/0004_snapshot.json @@ -0,0 +1,2017 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "22f9aac7-a71c-499d-b523-f213fe371acb", + "prevId": "0dd3b0a4-a3e3-40c2-b471-19a2a4151130", + "tables": { + "agent_sessions": { + "name": "agent_sessions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_slug": { + "name": "agent_slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "agent_snapshot": { + "name": "agent_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "working_dir": { + "name": "working_dir", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "git_ref": { + "name": "git_ref", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fs_scope_tier": { + "name": "fs_scope_tier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'sandboxed'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'active'" + }, + "context_json": { + "name": "context_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "total_input_tokens": { + "name": "total_input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_output_tokens": { + "name": "total_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_cost_microcents": { + "name": "total_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "exported_workflow_path": { + "name": "exported_workflow_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_agent_sessions_status": { + "name": "idx_agent_sessions_status", + "columns": [ + "status", + "\"updated_at\" desc" + ], + "isUnique": false, + "where": "\"agent_sessions\".\"deleted_at\" is null" + }, + "idx_agent_sessions_agent": { + "name": "idx_agent_sessions_agent", + "columns": [ + "agent_id", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"agent_sessions\".\"agent_id\" is not null" + } + }, + "foreignKeys": { + "agent_sessions_agent_id_agents_id_fk": { + "name": "agent_sessions_agent_id_agents_id_fk", + "tableFrom": "agent_sessions", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "agent_sessions_model_id_model_catalog_id_fk": { + "name": "agent_sessions_model_id_model_catalog_id_fk", + "tableFrom": "agent_sessions", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "agent_sessions_fs_scope_tier_check": { + "name": "agent_sessions_fs_scope_tier_check", + "value": "\"agent_sessions\".\"fs_scope_tier\" in ('sandboxed', 'project', 'full')" + }, + "agent_sessions_status_check": { + "name": "agent_sessions_status_check", + "value": "\"agent_sessions\".\"status\" in ('active', 'idle', 'exported', 'ended')" + } + } + }, + "agents": { + "name": "agents", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "system_prompt": { + "name": "system_prompt", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "tools": { + "name": "tools", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "input_schema": { + "name": "input_schema", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "output_schema": { + "name": "output_schema", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "source_path": { + "name": "source_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_agents_slug": { + "name": "idx_agents_slug", + "columns": [ + "slug" + ], + "isUnique": true, + "where": "\"agents\".\"deleted_at\" is null" + }, + "idx_agents_model": { + "name": "idx_agents_model", + "columns": [ + "model_id" + ], + "isUnique": false + }, + "idx_agents_active": { + "name": "idx_agents_active", + "columns": [ + "is_active", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"agents\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "agents_model_id_model_catalog_id_fk": { + "name": "agents_model_id_model_catalog_id_fk", + "tableFrom": "agents", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "llm_providers": { + "name": "llm_providers", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "base_url": { + "name": "base_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "api_key_keychain_ref": { + "name": "api_key_keychain_ref", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "default_headers": { + "name": "default_headers", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_llm_providers_name": { + "name": "idx_llm_providers_name", + "columns": [ + "name" + ], + "isUnique": true, + "where": "\"llm_providers\".\"deleted_at\" is null" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "media_objects": { + "name": "media_objects", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "handle": { + "name": "handle", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "mime_type": { + "name": "mime_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "modality": { + "name": "modality", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "byte_length": { + "name": "byte_length", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "duration_ms": { + "name": "duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_referenced_at": { + "name": "last_referenced_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "media_objects_handle_unique": { + "name": "media_objects_handle_unique", + "columns": [ + "handle" + ], + "isUnique": true + }, + "idx_media_objects_gc": { + "name": "idx_media_objects_gc", + "columns": [ + "last_referenced_at" + ], + "isUnique": false, + "where": "\"media_objects\".\"deleted_at\" is null" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "media_objects_modality_check": { + "name": "media_objects_modality_check", + "value": "\"media_objects\".\"modality\" in ('image', 'audio', 'video', 'document')" + } + } + }, + "media_references": { + "name": "media_references", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "handle": { + "name": "handle", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "scope_kind": { + "name": "scope_kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "scope_id": { + "name": "scope_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_media_references_unique": { + "name": "idx_media_references_unique", + "columns": [ + "handle", + "scope_kind", + "scope_id" + ], + "isUnique": true + }, + "idx_media_references_scope": { + "name": "idx_media_references_scope", + "columns": [ + "scope_kind", + "scope_id" + ], + "isUnique": false + }, + "idx_media_references_handle": { + "name": "idx_media_references_handle", + "columns": [ + "handle" + ], + "isUnique": false + } + }, + "foreignKeys": { + "media_references_handle_media_objects_handle_fk": { + "name": "media_references_handle_media_objects_handle_fk", + "tableFrom": "media_references", + "tableTo": "media_objects", + "columnsFrom": [ + "handle" + ], + "columnsTo": [ + "handle" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "media_references_scope_kind_check": { + "name": "media_references_scope_kind_check", + "value": "\"media_references\".\"scope_kind\" in ('run', 'node', 'session', 'workspace')" + } + } + }, + "messages": { + "name": "messages", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "step_execution_id": { + "name": "step_execution_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "sequence_number": { + "name": "sequence_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "content_parts": { + "name": "content_parts", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_calls": { + "name": "tool_calls", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_call_id": { + "name": "tool_call_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finish_reason": { + "name": "finish_reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_messages_step": { + "name": "idx_messages_step", + "columns": [ + "step_execution_id", + "sequence_number" + ], + "isUnique": false + }, + "idx_messages_run": { + "name": "idx_messages_run", + "columns": [ + "run_id", + "created_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "messages_step_execution_id_step_executions_id_fk": { + "name": "messages_step_execution_id_step_executions_id_fk", + "tableFrom": "messages", + "tableTo": "step_executions", + "columnsFrom": [ + "step_execution_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "model_catalog": { + "name": "model_catalog", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "context_window_tokens": { + "name": "context_window_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "max_output_tokens": { + "name": "max_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "input_cost_per_mtok_microcents": { + "name": "input_cost_per_mtok_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_cost_per_mtok_microcents": { + "name": "output_cost_per_mtok_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cached_input_cost_per_mtok_microcents": { + "name": "cached_input_cost_per_mtok_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "media_image_cost_microcents": { + "name": "media_image_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "media_audio_cost_microcents": { + "name": "media_audio_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "media_video_cost_microcents": { + "name": "media_video_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "media_surface": { + "name": "media_surface", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'chat'" + }, + "supports_tool_calling": { + "name": "supports_tool_calling", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "supports_vision": { + "name": "supports_vision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "supports_streaming": { + "name": "supports_streaming", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "supports_json_mode": { + "name": "supports_json_mode", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "capabilities": { + "name": "capabilities", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "deprecation_date": { + "name": "deprecation_date", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_model_catalog_provider_model": { + "name": "idx_model_catalog_provider_model", + "columns": [ + "provider_id", + "model_id" + ], + "isUnique": true, + "where": "\"model_catalog\".\"deleted_at\" is null" + }, + "idx_model_catalog_provider": { + "name": "idx_model_catalog_provider", + "columns": [ + "provider_id" + ], + "isUnique": false + }, + "idx_model_catalog_active": { + "name": "idx_model_catalog_active", + "columns": [ + "is_active" + ], + "isUnique": false, + "where": "\"model_catalog\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "model_catalog_provider_id_llm_providers_id_fk": { + "name": "model_catalog_provider_id_llm_providers_id_fk", + "tableFrom": "model_catalog", + "tableTo": "llm_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "run_costs": { + "name": "run_costs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "node_id": { + "name": "node_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cost_microcents": { + "name": "cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_run_costs_run": { + "name": "idx_run_costs_run", + "columns": [ + "run_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "run_costs_run_id_runs_id_fk": { + "name": "run_costs_run_id_runs_id_fk", + "tableFrom": "run_costs", + "tableTo": "runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "run_costs_model_id_model_catalog_id_fk": { + "name": "run_costs_model_id_model_catalog_id_fk", + "tableFrom": "run_costs", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "run_events": { + "name": "run_events", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "step_execution_id": { + "name": "step_execution_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "seq": { + "name": "seq", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "event_type": { + "name": "event_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "level": { + "name": "level", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'info'" + }, + "node_id": { + "name": "node_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "payload_json": { + "name": "payload_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "ts": { + "name": "ts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_run_events_run_seq": { + "name": "idx_run_events_run_seq", + "columns": [ + "run_id", + "seq" + ], + "isUnique": true + }, + "idx_run_events_step": { + "name": "idx_run_events_step", + "columns": [ + "step_execution_id", + "ts" + ], + "isUnique": false, + "where": "\"run_events\".\"step_execution_id\" is not null" + }, + "idx_run_events_run_type": { + "name": "idx_run_events_run_type", + "columns": [ + "run_id", + "event_type", + "ts" + ], + "isUnique": false + } + }, + "foreignKeys": { + "run_events_run_id_runs_id_fk": { + "name": "run_events_run_id_runs_id_fk", + "tableFrom": "run_events", + "tableTo": "runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "runs": { + "name": "runs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "workflow_id": { + "name": "workflow_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "workflow_path": { + "name": "workflow_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "project_root": { + "name": "project_root", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "workflow_definition_snapshot": { + "name": "workflow_definition_snapshot", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'pending'" + }, + "execution_mode": { + "name": "execution_mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'local'" + }, + "trigger_type": { + "name": "trigger_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'manual'" + }, + "trigger_metadata": { + "name": "trigger_metadata", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "input_json": { + "name": "input_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "output_json": { + "name": "output_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_json": { + "name": "error_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "total_input_tokens": { + "name": "total_input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_output_tokens": { + "name": "total_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_cost_microcents": { + "name": "total_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_runs_workflow": { + "name": "idx_runs_workflow", + "columns": [ + "workflow_id", + "\"created_at\" desc" + ], + "isUnique": false + }, + "idx_runs_status": { + "name": "idx_runs_status", + "columns": [ + "status", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"runs\".\"deleted_at\" is null" + }, + "idx_runs_cost": { + "name": "idx_runs_cost", + "columns": [ + "workflow_id", + "created_at", + "total_cost_microcents" + ], + "isUnique": false, + "where": "\"runs\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "runs_workflow_id_workflows_id_fk": { + "name": "runs_workflow_id_workflows_id_fk", + "tableFrom": "runs", + "tableTo": "workflows", + "columnsFrom": [ + "workflow_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "runs_status_check": { + "name": "runs_status_check", + "value": "\"runs\".\"status\" in ('pending', 'running', 'paused', 'completed', 'failed', 'cancelled')" + }, + "runs_execution_mode_check": { + "name": "runs_execution_mode_check", + "value": "\"runs\".\"execution_mode\" in ('local', 'cloud', 'managed')" + } + } + }, + "session_messages": { + "name": "session_messages", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "sequence_number": { + "name": "sequence_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "content_parts": { + "name": "content_parts", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_calls": { + "name": "tool_calls", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_call_id": { + "name": "tool_call_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finish_reason": { + "name": "finish_reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cost_microcents": { + "name": "cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_session_messages_seq": { + "name": "idx_session_messages_seq", + "columns": [ + "session_id", + "sequence_number" + ], + "isUnique": true + }, + "idx_session_messages_session": { + "name": "idx_session_messages_session", + "columns": [ + "session_id", + "created_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "session_messages_session_id_agent_sessions_id_fk": { + "name": "session_messages_session_id_agent_sessions_id_fk", + "tableFrom": "session_messages", + "tableTo": "agent_sessions", + "columnsFrom": [ + "session_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "session_messages_model_id_model_catalog_id_fk": { + "name": "session_messages_model_id_model_catalog_id_fk", + "tableFrom": "session_messages", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "step_executions": { + "name": "step_executions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "node_id": { + "name": "node_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "node_type": { + "name": "node_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_snapshot": { + "name": "agent_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "attempt_number": { + "name": "attempt_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 1 + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'pending'" + }, + "input_json": { + "name": "input_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "output_json": { + "name": "output_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_json": { + "name": "error_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "duration_ms": { + "name": "duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cached_tokens": { + "name": "cached_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cost_microcents": { + "name": "cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_step_exec_run": { + "name": "idx_step_exec_run", + "columns": [ + "run_id", + "created_at" + ], + "isUnique": false + }, + "idx_step_exec_run_node": { + "name": "idx_step_exec_run_node", + "columns": [ + "run_id", + "node_id", + "attempt_number" + ], + "isUnique": false + }, + "idx_step_exec_agent": { + "name": "idx_step_exec_agent", + "columns": [ + "agent_id", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"step_executions\".\"agent_id\" is not null" + }, + "idx_step_exec_model": { + "name": "idx_step_exec_model", + "columns": [ + "model_id", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"step_executions\".\"model_id\" is not null" + }, + "idx_step_exec_cost": { + "name": "idx_step_exec_cost", + "columns": [ + "model_id", + "created_at", + "cost_microcents" + ], + "isUnique": false, + "where": "\"step_executions\".\"model_id\" is not null" + } + }, + "foreignKeys": { + "step_executions_run_id_runs_id_fk": { + "name": "step_executions_run_id_runs_id_fk", + "tableFrom": "step_executions", + "tableTo": "runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "step_executions_agent_id_agents_id_fk": { + "name": "step_executions_agent_id_agents_id_fk", + "tableFrom": "step_executions", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "step_executions_model_id_model_catalog_id_fk": { + "name": "step_executions_model_id_model_catalog_id_fk", + "tableFrom": "step_executions", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "step_executions_status_check": { + "name": "step_executions_status_check", + "value": "\"step_executions\".\"status\" in ('pending', 'running', 'completed', 'failed', 'skipped')" + } + } + }, + "workflows": { + "name": "workflows", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "definition": { + "name": "definition", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "input_schema": { + "name": "input_schema", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "source_path": { + "name": "source_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_workflows_slug": { + "name": "idx_workflows_slug", + "columns": [ + "slug" + ], + "isUnique": true, + "where": "\"workflows\".\"deleted_at\" is null" + }, + "idx_workflows_active": { + "name": "idx_workflows_active", + "columns": [ + "is_active", + "\"updated_at\" desc" + ], + "isUnique": false, + "where": "\"workflows\".\"deleted_at\" is null" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": { + "idx_agent_sessions_status": { + "columns": { + "\"updated_at\" desc": { + "isExpression": true + } + } + }, + "idx_agent_sessions_agent": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_agents_active": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_runs_workflow": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_runs_status": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_step_exec_agent": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_step_exec_model": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_workflows_active": { + "columns": { + "\"updated_at\" desc": { + "isExpression": true + } + } + } + } + } +} \ No newline at end of file diff --git a/packages/db/drizzle/meta/_journal.json b/packages/db/drizzle/meta/_journal.json index 9d607f39..ff1216f7 100644 --- a/packages/db/drizzle/meta/_journal.json +++ b/packages/db/drizzle/meta/_journal.json @@ -29,6 +29,13 @@ "when": 1781879145602, "tag": "0003_cloudy_wind_dancer", "breakpoints": true + }, + { + "idx": 4, + "version": "6", + "when": 1781909685036, + "tag": "0004_numerous_ultimatum", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema.ts b/packages/db/src/schema.ts index 60f2e191..c9d67c20 100644 --- a/packages/db/src/schema.ts +++ b/packages/db/src/schema.ts @@ -9,6 +9,7 @@ import { type FsScopeTier, type MediaModality, type MediaScopeKind, + type MediaSurface, type RunStatus, type SessionStatus, } from '@relavium/shared'; @@ -111,6 +112,14 @@ export const modelCatalog = sqliteTable( mediaImageCostMicrocents: integer('media_image_cost_microcents'), mediaAudioCostMicrocents: integer('media_audio_cost_microcents'), mediaVideoCostMicrocents: integer('media_video_cost_microcents'), + // Inline-vs-generative media-output routing (1.AG/ADR-0045 §1): a 'generative' model routes an agent + // node to generateMedia() (separate-endpoint sync / async-LRO); 'chat' (default) uses the normal turn + // with output_modalities. Projects onto CapabilityFlags.media.surface; data-driven (no hardcoded ids). + // Value-set enforced by the `MediaSurface` type + the closed `MEDIA_SURFACES` set (and, at the seam, + // the optional `CapabilityFlags.media.surface = z.enum(MEDIA_SURFACES)`). A DB CHECK is deferred: + // SQLite does not support adding a CHECK via `ALTER TABLE ADD COLUMN` — enforcing it would need a full + // table-rebuild migration (the sibling enum checks live in the initial CREATE; this column is an ALTER ADD). + mediaSurface: text('media_surface').$type().notNull().default('chat'), supportsToolCalling: boolFlag('supports_tool_calling', false), supportsVision: boolFlag('supports_vision', false), supportsStreaming: boolFlag('supports_streaming', true), diff --git a/packages/llm/src/adapters/anthropic.test.ts b/packages/llm/src/adapters/anthropic.test.ts index ec9afc38..0dcb424e 100644 --- a/packages/llm/src/adapters/anthropic.test.ts +++ b/packages/llm/src/adapters/anthropic.test.ts @@ -78,6 +78,7 @@ describe('AnthropicAdapter', () => { // seam ceiling, so advertising it would be "advertised-but-unsendable" (ADR-0031). input: { image: true, audio: false, video: false, document: false }, outputCombinations: [], + surface: 'chat', }, }); }); diff --git a/packages/llm/src/adapters/anthropic.ts b/packages/llm/src/adapters/anthropic.ts index 4aaaab9e..c88c2a51 100644 --- a/packages/llm/src/adapters/anthropic.ts +++ b/packages/llm/src/adapters/anthropic.ts @@ -57,6 +57,7 @@ const SUPPORTS: CapabilityFlags = { // "advertised-but-unsendable" — the gate would admit a PDF the mapper then rejects (ADR-0031). input: { image: true, audio: false, video: false, document: false }, outputCombinations: [], + surface: 'chat', // no media generation — 1.AG/ADR-0045 §1 }, }; diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index 267f9ef0..7a4de3e9 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -66,6 +66,7 @@ describe('Gemini adapter', () => { // blocked by the seam ceiling, so advertising them would be "advertised-but-unsendable" (ADR-0031). input: { image: true, audio: true, video: false, document: false }, outputCombinations: [['text'], ['text', 'image'], ['text', 'audio']], + surface: 'chat', }); }); @@ -181,6 +182,15 @@ describe('Gemini adapter', () => { } }); + it('rejects a supported media output on the STREAM path — media-out is generate()-only (1.AG/ADR-0046)', () => { + // Even a model-supported combination (text+image) is rejected on stream(): the streaming media triad is + // host-deferred (ADR-0046 §4) and the streaming fold drops media, so streaming media output would be a + // silent loss. generate() is the only media-out path. + const adapter = createGeminiAdapter({ transport: fakeTransport({ candidates: [] }) }); + const req: LlmRequest = { ...REQ, outputModalities: ['text', 'image'] }; + expect(() => adapter.stream(req, 'k')).toThrowError(UnsupportedCapabilityError); + }); + it('maps finish reasons (STOP+tools → tool_use; SAFETY → content_filter; MALFORMED → error)', () => { expect(mapStopReason('STOP', false)).toBe('stop'); expect(mapStopReason('STOP', true)).toBe('tool_use'); @@ -230,6 +240,49 @@ describe('Gemini adapter', () => { expect(parts[2].id.length).toBeGreaterThan(0); // synthesized } }); + + it('mapContent surfaces inline media-out (inlineData) as an in-flight base64 media part (1.AG/ADR-0046)', () => { + const response: GeminiResponse = { + candidates: [ + { + content: { + parts: [ + { text: 'here is your image' }, + { inlineData: { mimeType: 'image/png', data: 'aW1nLWJ5dGVz' } }, + ], + }, + }, + ], + }; + const parts = mapContent(response, new GeminiToolCallIds()); + expect(parts[0]).toEqual({ type: 'text', text: 'here is your image' }); + expect(parts[1]).toEqual({ + type: 'media', + mimeType: 'image/png', + source: { kind: 'base64', data: 'aW1nLWJ5dGVz' }, + }); + }); + + it('mapContent skips an empty inlineData part (no data)', () => { + const response: GeminiResponse = { + candidates: [ + { + content: { parts: [{ inlineData: { mimeType: 'image/png', data: '' } }, { text: 'x' }] }, + }, + ], + }; + const parts = mapContent(response, new GeminiToolCallIds()); + expect(parts).toEqual([{ type: 'text', text: 'x' }]); + }); + + it('mapContent skips a mimeType-less inlineData part rather than emitting a doomed octet-stream (Opus-fix)', () => { + // A mimeType-less media part would HARD-FAIL the engine de-inline (mediaModalityOf undefined → run:failed), + // so it is dropped symmetric with the empty-data skip — never defaulted to application/octet-stream. + const response: GeminiResponse = { + candidates: [{ content: { parts: [{ inlineData: { data: 'aW1n' } }, { text: 'x' }] } }], + }; + expect(mapContent(response, new GeminiToolCallIds())).toEqual([{ type: 'text', text: 'x' }]); + }); }); describe('geminiErrorToLlmError — classification', () => { @@ -284,6 +337,16 @@ describe('Gemini adapter — request building (buildGeminiRequest)', () => { }); }); + it('lowers non-text output_modalities to responseModalities (inline media-out, 1.AG/ADR-0046)', () => { + const request = buildGeminiRequest({ ...REQ, outputModalities: ['text', 'image'] }); + expect(request.config['responseModalities']).toEqual(['TEXT', 'IMAGE']); + }); + + it('omits responseModalities for a text-only output (default behavior unchanged)', () => { + const request = buildGeminiRequest({ ...REQ, outputModalities: ['text'] }); + expect('responseModalities' in request.config).toBe(false); + }); + it('threads a real AbortSignal into config.abortSignal', () => { const controller = new AbortController(); const request = buildGeminiRequest({ ...REQ, signal: controller.signal }); diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index 31844214..163d87eb 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -1,7 +1,7 @@ import { GoogleGenAI } from '@google/genai'; import { mediaModalityOf } from '@relavium/shared'; -import type { ContentPart, StopReason } from '@relavium/shared'; +import type { ContentPart, OutputModality, StopReason } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; @@ -19,7 +19,12 @@ import type { Usage, } from '../types.js'; -import { REASONING_ID, assertMediaCapabilities, isAbortSignal } from './shared.js'; +import { + REASONING_ID, + assertMediaCapabilities, + assertNoStreamingMediaOutput, + isAbortSignal, +} from './shared.js'; /** * The Gemini adapter (1.H) over `@google/genai` — the riskiest adapter: a restricted tool schema and @@ -53,6 +58,7 @@ const GEMINI_SUPPORTS: CapabilityFlags = { // "advertised-but-unsendable" — the gate would admit a part the mapper then rejects (ADR-0031). input: { image: true, audio: true, video: false, document: false }, outputCombinations: [['text'], ['text', 'image'], ['text', 'audio']], + surface: 'chat', // inline media-out (responseModalities) is a chat turn; generative endpoints are 1.AG Section C }, }; @@ -97,6 +103,9 @@ interface GeminiPart { thought?: boolean; thoughtSignature?: string; functionCall?: GeminiFunctionCall; + // Inline media-out (responseModalities image/audio): a generated artifact returned IN the chat turn + // as base64 (1.AG/ADR-0046). The engine de-inlines it to a handle at the #emitDurable choke point (1.AF). + inlineData?: { mimeType?: string; data?: string }; } /** The subset of a `GenerateContentResponse` the fold reads (the real SDK type satisfies this). */ @@ -195,34 +204,64 @@ export function mapUsage(usage: { /** Fold a non-streaming Gemini response into canonical content parts (text + synthesized tool_call). */ export function mapContent(response: GeminiResponse, ids: GeminiToolCallIds): ContentPart[] { const parts: ContentPart[] = []; - // A response `inlineData` part (generated image/audio output) is intentionally NOT surfaced here — - // output media de-inlining is deferred to 1.AF/1.AG; only text / reasoning / functionCall are mapped today. for (const part of response.candidates?.[0]?.content?.parts ?? []) { - if (part.functionCall !== undefined) { - const name = part.functionCall.name ?? ''; - parts.push( - normalizeToolCall(PROVIDER, { - id: ids.synthesize(name), // Gemini has no native id — mint a stable one (1.E) - name, - args: part.functionCall.args ?? {}, - }), - ); - } else if (part.text !== undefined && part.text.length > 0) { - if (part.thought === true) { - // Reasoning (ADR-0030); thoughtSignature is the ephemeral same-provider continuity token. - parts.push( - part.thoughtSignature !== undefined && part.thoughtSignature.length > 0 - ? { type: 'reasoning', text: part.text, signature: part.thoughtSignature } - : { type: 'reasoning', text: part.text }, - ); - } else { - parts.push({ type: 'text', text: part.text }); - } + const mapped = mapGeminiPart(part, ids); + if (mapped !== undefined) { + parts.push(mapped); } } return parts; } +/** + * Map ONE Gemini response part to its canonical {@link ContentPart}, or `undefined` to skip it. Mirrors the + * wire's mutual exclusivity in priority order — a `functionCall` part, else an inline-media part + * (responseModalities), else a text/reasoning part — and the **first matching KIND consumes the part**: a + * matched-but-unusable inline-media part (empty data/MIME) is SKIPPED, never re-interpreted as text (the early + * returns preserve the original `if/else if` consumption semantics). Pulled out of the `mapContent` loop so + * the per-part branching is not multiplied by the loop nesting (cognitive-complexity budget). + */ +function mapGeminiPart(part: GeminiPart, ids: GeminiToolCallIds): ContentPart | undefined { + if (part.functionCall !== undefined) { + const name = part.functionCall.name ?? ''; + return normalizeToolCall(PROVIDER, { + id: ids.synthesize(name), // Gemini has no native id — mint a stable one (1.E) + name, + args: part.functionCall.args ?? {}, + }); + } + const inline = part.inlineData; + if ( + inline?.data !== undefined && + inline.data.length > 0 && + inline.mimeType !== undefined && + inline.mimeType.length > 0 + ) { + // Inline media-out (responseModalities): a generated image/audio part, base64, IN the chat turn + // (1.AG/ADR-0046). Emit an IN-FLIGHT media part; the engine de-inlines it to a handle at #emitDurable + // (1.AF). No vendor shape escapes (I1) — only the normalized media ContentPart. Gemini AUDIO output carries + // a PARAMETERIZED mime (e.g. `audio/L16;codec=pcm;rate=24000`), but the seam's MediaMimeTypeSchema admits + // only a BARE type/subtype — strip parameters to the bare prefix (the modality derives from it; the durable + // media part cannot carry parameters anyway). A pathological `;…`-only value strips to empty and is SKIPPED + // (returns undefined — still consumed, never re-interpreted as text), symmetric with the empty-data skip, + // never emitted as a doomed `application/octet-stream` that would HARD-FAIL the de-inline. + const bareMime = inline.mimeType.split(';')[0]?.trim() ?? ''; + return bareMime.length > 0 + ? { type: 'media', mimeType: bareMime, source: { kind: 'base64', data: inline.data } } + : undefined; + } + if (part.text === undefined || part.text.length === 0) { + return undefined; + } + if (part.thought !== true) { + return { type: 'text', text: part.text }; + } + // Reasoning (ADR-0030); thoughtSignature is the ephemeral same-provider continuity token. + return part.thoughtSignature !== undefined && part.thoughtSignature.length > 0 + ? { type: 'reasoning', text: part.text, signature: part.thoughtSignature } + : { type: 'reasoning', text: part.text }; +} + /** Classify any transport/SDK throwable into a normalized `LlmError` — no vendor shape escapes. */ export function geminiErrorToLlmError(err: unknown): LlmError { if (err instanceof Error && err.name === 'AbortError') { @@ -353,6 +392,18 @@ function toResponseObject(result: unknown): Record { return isRecord(result) ? result : { result }; } +/** Map a Relavium output modality to its Gemini `responseModalities` enum value (inline media-out, 1.AG). + * `video` is present only to satisfy `Record` exhaustiveness: Gemini's chat-surface + * `responseModalities` accepts TEXT/IMAGE/AUDIO only, and no advertised `outputCombinations` includes a + * video set, so the exact-membership capability gate (`assertMediaCapabilities`) rejects a video request + * before this map is read — the entry is unreachable, kept solely for type completeness. */ +const GEMINI_RESPONSE_MODALITY: Record = { + text: 'TEXT', + image: 'IMAGE', + audio: 'AUDIO', + video: 'VIDEO', +}; + /** Lower a canonical request into the Gemini request shape (system → `systemInstruction`, etc.). */ export function buildGeminiRequest(req: LlmRequest): GeminiRequest { const config: Record = {}; @@ -376,6 +427,12 @@ export function buildGeminiRequest(req: LlmRequest): GeminiRequest { if (req.maxTokens !== undefined) { config['maxOutputTokens'] = req.maxTokens; } + if (req.outputModalities !== undefined && req.outputModalities.some((m) => m !== 'text')) { + // Lower the node's non-text output_modalities to Gemini `responseModalities` (inline media-out, + // 1.AG/ADR-0046). The per-modality capability gate (assertMediaCapabilities) has already rejected an + // unsupported combination, so every member maps to a Gemini modality enum. + config['responseModalities'] = req.outputModalities.map((m) => GEMINI_RESPONSE_MODALITY[m]); + } if (req.stopSequences !== undefined) { config['stopSequences'] = req.stopSequences; } @@ -597,6 +654,7 @@ export function createGeminiAdapter(deps: GeminiAdapterDeps = {}): LlmProvider { assertSupported(PROVIDER, GEMINI_SUPPORTS, req); // fail fast on an unsupported feature assertStreamable(PROVIDER, GEMINI_SUPPORTS); assertMediaCapabilities(PROVIDER, GEMINI_SUPPORTS, req); // per-modality input/output gate (ADR-0031, 1.AE) + assertNoStreamingMediaOutput(PROVIDER, req); // media-out is generate()-only; streaming triad deferred (ADR-0046 §4) return streamChunks(transport, buildGeminiRequest(req), key); }, }; diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 93a3fea4..bab310b4 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -3,7 +3,13 @@ import { describe, expect, it } from 'vitest'; import { InvalidBaseUrlError, UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; -import type { StreamChunk } from '../types.js'; +import type { + LlmProvider, + LlmRequest, + MediaGenRequest, + MediaGenResult, + StreamChunk, +} from '../types.js'; import { createOpenAiAdapter, deepseekAdapter, @@ -12,8 +18,22 @@ import { mapUsage, openaiAdapter, openaiErrorToLlmError, + outputAudioMime, } from './openai.js'; +/** Call the adapter's optional `generateMedia` via `?.()` — a call (binds `this`), never an extraction, so the + * unbound-method lint stays happy; the `??` branch asserts the method is implemented. */ +function genMedia( + adapter: LlmProvider, + req: MediaGenRequest, + key: string, +): Promise { + return ( + adapter.generateMedia?.(req, key) ?? + Promise.reject(new Error('adapter implements no generateMedia')) + ); +} + const isRecord = (value: unknown): value is Record => typeof value === 'object' && value !== null && !Array.isArray(value); @@ -86,6 +106,7 @@ describe('OpenAI-compatible adapter', () => { expect(openaiAdapter.supports.media).toEqual({ input: { image: true, audio: true, video: false, document: false }, outputCombinations: [['text'], ['text', 'audio']], + surface: 'chat', }); expect(openaiAdapter.supports.reasoning).toBe(false); expect(deepseekAdapter.supports.reasoning).toBe(true); @@ -191,6 +212,186 @@ describe('OpenAI-compatible adapter', () => { }); }); + it('round-trips inline audio-out: lowers output_modalities → modalities+audio and parses the response (1.AG/ADR-0046)', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response( + completion({ + role: 'assistant', + content: null, + refusal: null, + audio: { + id: 'a1', + data: 'YXVkaW8tYnl0ZXM=', + transcript: 'spoken words', + expires_at: 0, + }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ), + ); + }, + }); + const result = await adapter.generate( + { + model: 'gpt-4o-audio-preview', + outputModalities: ['text', 'audio'], + providerOptions: { audio: { voice: 'verse', format: 'mp3' } }, + messages: [{ role: 'user', content: [{ type: 'text', text: 'say hi' }] }], + }, + 'k', + ); + // Request side: the node's audio output_modality lowers to modalities + the merged voice/format. + expect(sent['modalities']).toEqual(['text', 'audio']); + expect(sent['audio']).toEqual({ voice: 'verse', format: 'mp3' }); + // Response side: transcript surfaces as text PLUS the audio as an in-flight base64 media part (audio/mpeg). + expect(result.content).toEqual([ + { type: 'text', text: 'spoken words' }, + { + type: 'media', + mimeType: 'audio/mpeg', + source: { kind: 'base64', data: 'YXVkaW8tYnl0ZXM=' }, + }, + ]); + }); + + it('defaults the audio voice/format when providerOptions omits them', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + await adapter.generate( + { + model: 'gpt-4o-audio-preview', + outputModalities: ['text', 'audio'], + messages: [{ role: 'user', content: [{ type: 'text', text: 'say hi' }] }], + }, + 'k', + ); + expect(sent['modalities']).toEqual(['text', 'audio']); + expect(sent['audio']).toEqual({ voice: 'alloy', format: 'wav' }); + }); + + it('rejects a non-text outputModalities on the STREAM path — media-out is generate()-only (1.AG/ADR-0046)', () => { + // The streaming media triad is host-deferred (ADR-0046 §4); the streaming fold drops media, so a stream() + // requesting media output would silently lose it. The guard fails loud instead (never reaching egress). + const adapter = createOpenAiAdapter({ + fetch: () => Promise.reject(new Error('must fail fast before any egress')), + }); + const req: LlmRequest = { + model: 'gpt-4o-audio-preview', + outputModalities: ['text', 'audio'], + messages: [{ role: 'user', content: [{ type: 'text', text: 'speak' }] }], + }; + expect(() => adapter.stream(req, 'k')).toThrowError(UnsupportedCapabilityError); + }); + + it('generateMedia (image) returns a base64 PNG media part from images.generate (1.AG Section C/ADR-0045)', async () => { + const adapter = createOpenAiAdapter({ + fetch: () => + Promise.resolve( + new Response(JSON.stringify({ created: 0, data: [{ b64_json: 'Z2VuLWltYWdl' }] }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ), + }); + const result = await genMedia( + adapter, + { model: 'gpt-image-1', prompt: 'a cat', modality: 'image' }, + 'k', + ); + expect(result.jobId).toBeUndefined(); // SYNC arm + expect(result.media).toEqual({ + type: 'media', + mimeType: 'image/png', + source: { kind: 'base64', data: 'Z2VuLWltYWdl' }, + }); + }); + + it('generateMedia rejects a non-image modality + DeepSeek with a typed capability error', async () => { + const oai = createOpenAiAdapter({ + fetch: () => Promise.reject(new Error('must fail fast before any egress')), + }); + await expect( + genMedia(oai, { model: 'gpt-4o-mini-tts', prompt: 'x', modality: 'audio' }, 'k'), + ).rejects.toThrowError(UnsupportedCapabilityError); + const ds = createOpenAiAdapter({ + providerId: 'deepseek', + fetch: () => Promise.reject(new Error('must fail fast before any egress')), + }); + await expect( + genMedia(ds, { model: 'm', prompt: 'x', modality: 'image' }, 'k'), + ).rejects.toThrowError(UnsupportedCapabilityError); + }); + + it('generateMedia maps a no-data image response to a typed bad_request LlmProviderError', async () => { + const adapter = createOpenAiAdapter({ + fetch: () => + Promise.resolve( + new Response(JSON.stringify({ created: 0, data: [] }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ), + }); + await expect( + genMedia(adapter, { model: 'gpt-image-1', prompt: 'x', modality: 'image' }, 'k'), + ).rejects.toThrowError(LlmProviderError); + }); + + it('generateMedia rejects count > 1 (single-artifact SYNC seam) with a typed bad_request before any egress', async () => { + const adapter = createOpenAiAdapter({ + fetch: () => Promise.reject(new Error('must fail fast before any egress')), + }); + await expect( + genMedia(adapter, { model: 'gpt-image-1', prompt: 'x', modality: 'image', count: 3 }, 'k'), + ).rejects.toMatchObject({ llmError: { kind: 'bad_request', retryable: false } }); + }); + + it('generateMedia honors a requested output format (req.mimeType → output_format + result MIME)', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response(JSON.stringify({ created: 0, data: [{ b64_json: 'aW1n' }] }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ); + }, + }); + const result = await genMedia( + adapter, + { model: 'gpt-image-1', prompt: 'x', modality: 'image', mimeType: 'image/webp' }, + 'k', + ); + expect(sent['output_format']).toBe('webp'); + expect(result.media?.mimeType).toBe('image/webp'); + }); + + it('generateMedia maps an image content-policy refusal to content_filter (the documented taxonomy)', async () => { + const adapter = createOpenAiAdapter({ + fetch: () => + Promise.resolve( + new Response( + JSON.stringify({ error: { message: 'blocked', code: 'content_policy_violation' } }), + { status: 400, headers: { 'content-type': 'application/json' } }, + ), + ), + }); + await expect( + genMedia(adapter, { model: 'gpt-image-1', prompt: 'x', modality: 'image' }, 'k'), + ).rejects.toMatchObject({ llmError: { kind: 'content_filter' } }); + }); + it('lowers media on the STREAM path too (shared buildCommonBody — the §1.AE both-paths requirement)', async () => { let sent: Record = {}; const adapter = createOpenAiAdapter({ @@ -399,6 +600,57 @@ describe('OpenAI-compatible adapter', () => { ); expect(parts).toEqual([{ type: 'tool_call', id: 't1', name: 'f', args: {} }]); }); + + it('mapContent surfaces inline audio-out as a transcript text part PLUS a base64 media part (1.AG/ADR-0046)', () => { + const parts = mapContent( + { content: null, audio: { data: 'YXVkaW8tYnl0ZXM=', transcript: 'hello there' } }, + 'openai', + 'audio/mpeg', + ); + expect(parts).toEqual([ + { type: 'text', text: 'hello there' }, + { + type: 'media', + mimeType: 'audio/mpeg', + source: { kind: 'base64', data: 'YXVkaW8tYnl0ZXM=' }, + }, + ]); + }); + + it('mapContent emits the audio media part even when the transcript is empty', () => { + const parts = mapContent( + { content: null, audio: { data: 'YXVkaW8=', transcript: '' } }, + 'openai', + ); + expect(parts).toEqual([ + { type: 'media', mimeType: 'audio/wav', source: { kind: 'base64', data: 'YXVkaW8=' } }, + ]); + }); + + it('mapContent ignores a null/empty audio field', () => { + expect(mapContent({ content: 'x', audio: null }, 'openai')).toEqual([ + { type: 'text', text: 'x' }, + ]); + expect(mapContent({ content: 'x', audio: { data: '', transcript: 't' } }, 'openai')).toEqual([ + { type: 'text', text: 'x' }, + ]); + }); + + it('outputAudioMime maps the requested providerOptions.audio.format (default wav)', () => { + const mk = (format?: string): LlmRequest => ({ + model: 'gpt-4o-audio-preview', + messages: [{ role: 'user', content: [{ type: 'text', text: 'speak' }] }], + ...(format === undefined ? {} : { providerOptions: { audio: { format } } }), + }); + expect(outputAudioMime(mk('mp3'))).toBe('audio/mpeg'); + expect(outputAudioMime(mk('opus'))).toBe('audio/opus'); + expect(outputAudioMime(mk('flac'))).toBe('audio/flac'); + expect(outputAudioMime(mk('aac'))).toBe('audio/aac'); + expect(outputAudioMime(mk('pcm16'))).toBe('audio/L16'); + expect(outputAudioMime(mk('wav'))).toBe('audio/wav'); + expect(outputAudioMime(mk())).toBe('audio/wav'); // no providerOptions → default + expect(outputAudioMime(mk('something-odd'))).toBe('audio/wav'); // unknown → default + }); }); describe('openaiErrorToLlmError — classification', () => { @@ -429,6 +681,15 @@ describe('openaiErrorToLlmError — classification', () => { ).toMatchObject({ kind: 'unknown', retryable: false }); }); + it('classifies a content-policy / moderation code as content_filter regardless of HTTP status (1.AG §6)', () => { + const policy = new APIError(400, undefined, 'blocked', undefined); + Object.assign(policy, { code: 'content_policy_violation' }); + expect(openaiErrorToLlmError(policy, 'openai')).toMatchObject({ kind: 'content_filter' }); + const moderation = new APIError(400, undefined, 'blocked', undefined); + Object.assign(moderation, { code: 'moderation_blocked' }); + expect(openaiErrorToLlmError(moderation, 'openai')).toMatchObject({ kind: 'content_filter' }); + }); + it('falls back to unknown for a non-Error throwable', () => { expect(openaiErrorToLlmError('boom', 'openai')).toMatchObject({ kind: 'unknown', diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index 5b0f0b48..460e1708 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -14,16 +14,19 @@ import { } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; -import { InvalidBaseUrlError } from '../errors.js'; +import { InvalidBaseUrlError, UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; import { normalizeToolCall, toWire } from '../tool-normalizer.js'; import type { CapabilityFlags, LlmError, + LlmErrorKind, LlmMessage, LlmProvider, LlmRequest, LlmResult, + MediaGenRequest, + MediaGenResult, MediaUnitsEntry, ProviderId, StreamChunk, @@ -32,7 +35,12 @@ import type { Usage, } from '../types.js'; -import { REASONING_ID, assertMediaCapabilities, isAbortSignal } from './shared.js'; +import { + REASONING_ID, + assertMediaCapabilities, + assertNoStreamingMediaOutput, + isAbortSignal, +} from './shared.js'; /** * The shared OpenAI-compatible adapter (1.G) — one implementation over the `openai` SDK serving both @@ -64,6 +72,7 @@ const OPENAI_SUPPORTS: CapabilityFlags = { // adapter, so advertising document:true would be "advertised-but-unsendable" (ADR-0031). input: { image: true, audio: true, video: false, document: false }, outputCombinations: [['text'], ['text', 'audio']], + surface: 'chat', // inline audio is a chat turn; gpt-image-1/Sora generative endpoints are 1.AG Section C/D }, }; @@ -78,6 +87,7 @@ const DEEPSEEK_SUPPORTS: CapabilityFlags = { media: { input: { image: false, audio: false, video: false, document: false }, outputCombinations: [], + surface: 'chat', // DeepSeek: text-only, no media generation — 1.AG/ADR-0045 §1 }, }; @@ -162,7 +172,51 @@ function parseToolArgs(raw: string): unknown { } } -/** Fold a non-streaming assistant message into canonical content parts (text + tool_call). */ +const isRecord = (value: unknown): value is Record => + typeof value === 'object' && value !== null && !Array.isArray(value); + +/** OpenAI's accepted inline-audio output formats (a closed union on the SDK param) — used to narrow a + * caller-supplied `providerOptions.audio.format` to a valid value (default `wav`). */ +const OPENAI_AUDIO_FORMATS = ['wav', 'aac', 'mp3', 'flac', 'opus', 'pcm16'] as const; + +/** Map a requested image-out MIME (`MediaGenRequest.mimeType`) to OpenAI's image `output_format` enum, or + * `undefined` to leave the default (gpt-image-1 → PNG). Only the gpt-image-1-supported formats are honored. */ +function imageOutputFormat(mimeType: string | undefined): 'png' | 'jpeg' | 'webp' | undefined { + switch (mimeType) { + case 'image/png': + return 'png'; + case 'image/jpeg': + case 'image/jpg': // the common non-canonical alias — map it rather than silently falling back to PNG + return 'jpeg'; + case 'image/webp': + return 'webp'; + default: + return undefined; + } +} + +/** Map a requested output-audio `format` (providerOptions.audio.format) to its MIME — OpenAI's response + * echoes no format, so the requested one types the media part. Defaults to `audio/wav` (OpenAI's default). */ +export function outputAudioMime(req: LlmRequest): string { + const opts = req.providerOptions; + const audio = isRecord(opts) ? opts['audio'] : undefined; + switch (isRecord(audio) ? audio['format'] : undefined) { + case 'mp3': + return 'audio/mpeg'; + case 'opus': + return 'audio/opus'; + case 'flac': + return 'audio/flac'; + case 'aac': + return 'audio/aac'; + case 'pcm16': + return 'audio/L16'; + default: + return 'audio/wav'; + } +} + +/** Fold a non-streaming assistant message into canonical content parts (text + tool_call + inline audio). */ export function mapContent( message: { content: string | null; @@ -172,8 +226,11 @@ export function mapContent( tool_calls?: | ReadonlyArray<{ id: string; function?: { name: string; arguments: string } }> | undefined; + // Inline audio-out (modalities ['text','audio']): base64 audio + its transcript (1.AG/ADR-0046). + audio?: { data: string; transcript?: string | null } | null; }, provider: ProviderId, + audioMime = 'audio/wav', ): ContentPart[] { const parts: ContentPart[] = []; if ( @@ -186,6 +243,21 @@ export function mapContent( if (message.content !== null && message.content.length > 0) { parts.push({ type: 'text', text: message.content }); } + // Inline audio-out (1.AG/ADR-0046): surface the spoken transcript as a text part PLUS the audio as an + // in-flight base64 media part; the engine de-inlines the media to a handle at #emitDurable (1.AF). + // OpenAI sets `message.content` to null when audio output is requested, so the transcript text and the + // `content` text above are mutually exclusive in practice — the transcript is not a duplicate of `content`. + if (message.audio !== null && message.audio !== undefined && message.audio.data.length > 0) { + const transcript = message.audio.transcript ?? ''; + if (transcript.length > 0) { + parts.push({ type: 'text', text: transcript }); + } + parts.push({ + type: 'media', + mimeType: audioMime, + source: { kind: 'base64', data: message.audio.data }, + }); + } for (const call of message.tool_calls ?? []) { if (call.function === undefined) { continue; // custom (non-function) tool calls are off the common path @@ -211,6 +283,12 @@ function firstNonEmptyString(...values: readonly unknown[]): string | undefined return undefined; } +/** An OpenAI content-policy / moderation rejection code (image-gen + chat). Its own fatal cause, not a + * generic bad_request — so `codeForLlmError` yields `content_filter` (1.AG/ADR-0045 §6). */ +function isContentPolicyCode(code: string | undefined): boolean { + return code === 'content_policy_violation' || code === 'moderation_blocked'; +} + /** Normalize an SDK `APIError` into an `LlmError`, typed by the structural subset it reads. */ function mapOpenAiApiError( err: { status?: unknown; code?: unknown; type?: unknown; message: string }, @@ -218,7 +296,16 @@ function mapOpenAiApiError( ): LlmError { const status = typeof err.status === 'number' ? err.status : undefined; const code = firstNonEmptyString(err.code, err.type); - const kind = status === undefined ? 'unknown' : kindFromHttpStatus(status); + // A content-policy block normalizes to content_filter regardless of HTTP status (a moderation 400 would + // otherwise map to bad_request) — the wired image-gen path then delivers the documented taxonomy. + let kind: LlmErrorKind; + if (isContentPolicyCode(code)) { + kind = 'content_filter'; + } else if (status === undefined) { + kind = 'unknown'; + } else { + kind = kindFromHttpStatus(status); + } return makeLlmError({ provider, kind, @@ -460,21 +547,7 @@ function buildCommonBody( body.tool_choice = toOpenAiToolChoice(req.toolChoice); } if (req.responseFormat?.type === 'json') { - if (provider === 'deepseek') { - // DeepSeek only supports json_object; json_schema returns 400 (ADR-0030). - // Note: DeepSeek json_object also requires the word "json" to appear in the prompt. - body.response_format = { type: 'json_object' }; - } else { - // Native structured output for OpenAI (ADR-0030). The canonical JSON-Schema bridges here. - body.response_format = { - type: 'json_schema', - json_schema: { - name: toJsonSchemaName(req.responseFormat.name), - schema: req.responseFormat.schema as Record, - strict: req.responseFormat.strict ?? true, - }, - }; - } + body.response_format = toOpenAiResponseFormat(req.responseFormat, provider); } if (req.temperature !== undefined) { body.temperature = req.temperature; @@ -485,6 +558,12 @@ function buildCommonBody( if (req.stopSequences !== undefined) { body.stop = req.stopSequences; } + if (req.outputModalities?.includes('audio')) { + // Inline audio-out (1.AG/ADR-0046): request the text+audio combination. `body` is spread last below, so + // these win over a raw providerOptions echo. + body.modalities = ['text', 'audio']; + body.audio = resolveOpenAiAudio(req.providerOptions); + } if (req.providerOptions === undefined) { return body; } @@ -492,6 +571,40 @@ function buildCommonBody( return { ...req.providerOptions, ...body }; } +/** Lower a canonical `responseFormat: json` to OpenAI's `response_format`: DeepSeek supports only + * `json_object` (json_schema 400s — ADR-0030; it also needs the word "json" in the prompt); OpenAI uses + * native `json_schema` (the canonical JSON-Schema bridges here). */ +function toOpenAiResponseFormat( + responseFormat: Extract, { type: 'json' }>, + provider: ProviderId, +): NonNullable { + if (provider === 'deepseek') { + return { type: 'json_object' }; + } + return { + type: 'json_schema', + json_schema: { + name: toJsonSchemaName(responseFormat.name), + schema: responseFormat.schema as Record, + strict: responseFormat.strict ?? true, + }, + }; +} + +/** Resolve OpenAI inline-audio `voice` (any string) + `format` (a closed union) from + * `providerOptions.audio` when supplied, else OpenAI's defaults (1.AG/ADR-0046). */ +function resolveOpenAiAudio(providerOptions: LlmRequest['providerOptions']): { + voice: string; + format: (typeof OPENAI_AUDIO_FORMATS)[number]; +} { + const audioOpts = isRecord(providerOptions) ? providerOptions['audio'] : undefined; + const voice = + isRecord(audioOpts) && typeof audioOpts['voice'] === 'string' ? audioOpts['voice'] : 'alloy'; + const rawFormat = isRecord(audioOpts) ? audioOpts['format'] : undefined; + const format = OPENAI_AUDIO_FORMATS.find((f) => f === rawFormat) ?? 'wav'; + return { voice, format }; +} + function buildRequestOptions(req: LlmRequest): { signal?: AbortSignal } { return isAbortSignal(req.signal) ? { signal: req.signal } : {}; } @@ -732,9 +845,14 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { // A non-null refusal is a safety decline — normalize to content_filter, not a clean stop. const refused = typeof choice?.message.refusal === 'string' && choice.message.refusal.length > 0; + // Compute the inline-audio output MIME only when audio was actually requested (it is read solely by + // mapContent's audio branch); a text turn keeps the default and pays no providerOptions scan. + const audioMime = req.outputModalities?.includes('audio') + ? outputAudioMime(req) + : 'audio/wav'; return { // An empty `choices` array is a complete-but-empty 200 — a clean empty stop, not an error. - content: choice === undefined ? [] : mapContent(choice.message, providerId), + content: choice === undefined ? [] : mapContent(choice.message, providerId, audioMime), stopReason: refused ? 'content_filter' : mapStopReason(choice?.finish_reason), usage: completion.usage ? mapUsage(completion.usage) : ZERO_USAGE, raw: completion, @@ -747,8 +865,69 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { assertSupported(providerId, supports, req); // fail fast on an unsupported feature or no streaming assertStreamable(providerId, supports); assertMediaCapabilities(providerId, supports, req); // per-modality input/output gate (ADR-0031, 1.AE) + assertNoStreamingMediaOutput(providerId, req); // media-out is generate()-only; streaming triad deferred (ADR-0046 §4) return streamChunks(createClient(key), req, providerId); }, + /** + * Separate-endpoint media generation (1.AG Section C, [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * SYNC image generation via gpt-image-1 (`client.images.generate` → base64). Audio (TTS via + * `audio.speech`) and video are NOT wired here yet — they fail loud with a typed capability error, never a + * silent drop (deferred — deferred-tasks.md). DeepSeek generates no media. No vendor type crosses the seam: + * the result is a normalized `MediaGenResult` whose `raw` is strip-discarded by sinks. + */ + async generateMedia(req: MediaGenRequest, key: string): Promise { + if (providerId !== 'openai' || req.modality !== 'image') { + throw new UnsupportedCapabilityError( + providerId, + 'media', + `${providerId} generateMedia supports only OpenAI image generation, not '${req.modality}' (audio/video deferred)`, + ); + } + if (req.count !== undefined && req.count > 1) { + // The SYNC seam carries a SINGLE MediaPart. Rather than generate (and have the engine bill) N images + // and silently drop N-1, reject count > 1 loudly until a multi-part media-array result lands (a future + // additive ADR-0031 seam amendment, deferred-tasks.md) — never deliver fewer artifacts than billed. + throw new LlmProviderError( + makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: `OpenAI image generateMedia delivers a single image; count ${String(req.count)} > 1 is not supported on the SYNC seam`, + }), + ); + } + // Honor a requested output format (req.mimeType → png/jpeg/webp); the result MIME reflects what we asked + // for (or gpt-image-1's PNG default). (providerOptions image knobs — size/quality — are not set by the + // engine in Section C; threading them is a bounded follow-up with the image-knob work, deferred-tasks.md.) + const outputFormat = imageOutputFormat(req.mimeType); + let response: OpenAI.ImagesResponse; + try { + response = await createClient(key).images.generate( + { + model: req.model, + prompt: req.prompt, + ...(outputFormat === undefined ? {} : { output_format: outputFormat }), + }, + isAbortSignal(req.signal) ? { signal: req.signal } : {}, + ); + } catch (err) { + throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); + } + const b64 = response.data?.[0]?.b64_json; + if (b64 === undefined || b64.length === 0) { + throw new LlmProviderError( + makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: 'OpenAI image generation returned no base64 image data', + }), + ); + } + const mimeType = outputFormat === undefined ? 'image/png' : `image/${outputFormat}`; + return { + media: { type: 'media', mimeType, source: { kind: 'base64', data: b64 } }, + raw: response, + }; + }, }; } diff --git a/packages/llm/src/adapters/shared.ts b/packages/llm/src/adapters/shared.ts index d6507d3a..2c39eb59 100644 --- a/packages/llm/src/adapters/shared.ts +++ b/packages/llm/src/adapters/shared.ts @@ -43,6 +43,24 @@ export function assertMediaCapabilities( } } +/** + * Reject a non-text `outputModalities` on the STREAMING path (1.AG/[ADR-0046] §4). Inline media-out is + * delivered ONLY through `generate()` (the in-flight base64 rides `LlmResult.content`); the streaming media + * triad (`media_start`/`media_delta`/`media_end`) is host-deferred to 1.AH, and the streaming folds drop a + * provider's media parts — so a `stream()` that requested media output would SILENTLY lose it. Fail loud + * here instead. The engine routes a media-output turn to `generate()`, so this never fires on the run path; + * it is a defensive guard for a direct seam consumer. Call it at `stream()` entry, after the media gate. + */ +export function assertNoStreamingMediaOutput(provider: ProviderId, req: LlmRequest): void { + if (req.outputModalities?.some((modality) => modality !== 'text') === true) { + throw new UnsupportedCapabilityError( + provider, + 'media', + 'streaming media output is unsupported — inline media-out is delivered via generate() (ADR-0046 §4)', + ); + } +} + /** * The single-track reasoning-channel id used by the OpenAI/DeepSeek and Gemini streaming folds — those * providers emit one reasoning block per response (no concurrent tracks). A provider that interleaves diff --git a/packages/llm/src/budget-estimator.ts b/packages/llm/src/budget-estimator.ts index e50d90be..14bde116 100644 --- a/packages/llm/src/budget-estimator.ts +++ b/packages/llm/src/budget-estimator.ts @@ -44,7 +44,9 @@ export function estimateMediaCost( for (const { modality, units } of estimate) { const rate = p.mediaOutputRates?.[modality]; if (rate !== undefined && units > 0) { - total += units * rate; + // Round per entry, exactly as the realized `mediaCost` fold does (cost-tracker.ts), so the pre-egress + // gate estimate and the realized addend agree to the micro-cent on a fractional duration (N3). + total += Math.round(units * rate); } } return total; diff --git a/packages/llm/src/conformance/fixtures/openai.ts b/packages/llm/src/conformance/fixtures/openai.ts index 0ff4c21f..13771cf4 100644 --- a/packages/llm/src/conformance/fixtures/openai.ts +++ b/packages/llm/src/conformance/fixtures/openai.ts @@ -148,6 +148,14 @@ const structuredOutput = JSON.stringify({ usage: { prompt_tokens: 8, completion_tokens: 4, total_tokens: 12 }, }); +// A recorded images.generate (gpt-image-1) reply — a single base64 image, the SYNC generative seam path +// (1.AG Section C). The OpenAI SDK's images.generate goes through the same `fetch` override, so this drives +// the real parser + our normalization offline (b64_json → a base64 `media` part, mimeType image/png default). +const imageGenerate = JSON.stringify({ + created: 0, + data: [{ b64_json: 'aGVsbG8tY29uZm9ybWFuY2UtaW1hZ2U=' }], +}); + // No reasoningStream fixture: OpenAI chat.completions emits no reasoning output (the conformance // reasoning scenario is skipped for this provider). export const OPENAI_FIXTURES: ConformanceFixtures = { @@ -158,6 +166,7 @@ export const OPENAI_FIXTURES: ConformanceFixtures = { rateLimit: { status: 429, body: rateLimitError }, streamError: { status: 503, body: streamError }, structuredOutput: { status: 200, body: structuredOutput }, + mediaGenerate: { status: 200, body: imageGenerate }, toolLoop: { turn1: { status: 200, body: toolMessage }, turn2: { status: 200, body: textMessage }, @@ -170,5 +179,6 @@ export const OPENAI_FIXTURES: ConformanceFixtures = { toolStream: { toolName: 'get_weather', stopReason: 'tool_use' }, streamErrorKind: 'overloaded', structuredOutput: { text: '{"ok":true}' }, + mediaGenerate: { mimeType: 'image/png', data: 'aGVsbG8tY29uZm9ybWFuY2UtaW1hZ2U=' }, }, }; diff --git a/packages/llm/src/conformance/generative-seam.conformance.test.ts b/packages/llm/src/conformance/generative-seam.conformance.test.ts new file mode 100644 index 00000000..4c83ca6a --- /dev/null +++ b/packages/llm/src/conformance/generative-seam.conformance.test.ts @@ -0,0 +1,151 @@ +import { describe, expect, it } from 'vitest'; + +import { makeLlmError } from '../llm-error.js'; +import { MediaGenResultSchema, MediaJobStatusSchema } from '../types.js'; +import type { CapabilityFlags, LlmProvider, MediaGenRequest, MediaJobStatus } from '../types.js'; + +/** + * The GENERATIVE-surface seam-contract conformance suite (1.AG Section E, A5 / [ADR-0045]). The chat + * conformance suite replays recorded vendor responses through each real adapter; the ASYNC generative arm + * (a `generateMedia` that returns an opaque `jobId`, then `pollMediaJob` driving pending→done/failed) has + * NO production adapter yet — the Sora/Veo adapters are deferred to 1.AH. So this suite asserts the SEAM + * CONTRACT itself against a CONFORMING STUB provider: the exact shape any future async generative adapter + * must satisfy — the `MediaGenResult` exactly-one-of refine, an opaque non-empty `jobId`, the discriminated + * `MediaJobStatus` states, and a content-policy refusal classified as the `content_filter` LlmError kind. + * (The SYNC arm is covered against the real OpenAI adapter by the per-provider suite's `mediaGenerate` + * scenario; the engine-side poll/checkpoint/resume/cancel loop is covered in `@relavium/core`.) + */ + +const KEY = 'generative-seam-test-key'; + +const GENERATIVE_CAPS: CapabilityFlags = { + tools: false, + streaming: false, + parallelToolCalls: false, + vision: false, + promptCache: false, + reasoning: false, + media: { + input: { image: false, audio: false, video: false, document: false }, + outputCombinations: [], + surface: 'generative', + }, +}; + +const VIDEO_REQUEST: MediaGenRequest = { + model: 'conformance-video-model', + prompt: 'a wave breaking on a beach', + modality: 'video', + durationSeconds: 4, +}; + +/** A stub whose `generateMedia`/`pollMediaJob` are non-optional, so call sites need no narrowing. */ +interface GenerativeProvider extends LlmProvider { + generateMedia: NonNullable; + pollMediaJob: NonNullable; +} + +/** A conforming async generative provider: `generateMedia` mints an opaque jobId; `pollMediaJob` walks the + * scripted status sequence (clamping on the last entry once exhausted). */ +function stubAsyncGenerativeProvider(opts: { + readonly jobId?: string; + readonly polls: readonly MediaJobStatus[]; +}): GenerativeProvider { + let call = 0; + return { + id: 'openai', // any ProviderId — the async seam is provider-agnostic; the adapter holds the vendor↔opaque map + supports: GENERATIVE_CAPS, + generate: () => { + throw new Error('a generative provider exposes no chat generate'); + }, + stream: () => { + throw new Error('a generative provider exposes no chat stream'); + }, + generateMedia: () => Promise.resolve({ jobId: opts.jobId ?? 'vendor-op-7f3a', raw: {} }), + pollMediaJob: () => { + const status = opts.polls[Math.min(call, opts.polls.length - 1)]; + call += 1; + if (status === undefined) { + return Promise.reject(new Error('no poll status scripted')); + } + return Promise.resolve(status); + }, + }; +} + +describe('generative seam — async media job (conformance, A5)', () => { + it('generateMedia (async): resolves an OPAQUE non-empty jobId and NO media (exactly-one-of)', async () => { + const provider = stubAsyncGenerativeProvider({ + jobId: 'vendor-op-7f3a', + polls: [{ state: 'pending' }], + }); + const result = await provider.generateMedia(VIDEO_REQUEST, KEY); + expect(MediaGenResultSchema.safeParse(result).success).toBe(true); + expect(result.media).toBeUndefined(); + expect(typeof result.jobId).toBe('string'); + expect((result.jobId ?? '').length).toBeGreaterThan(0); + }); + + it('MediaGenResult: rejects carrying BOTH media and jobId, or NEITHER (the refine is the seam guard)', () => { + const media = { + type: 'media', + mimeType: 'video/mp4', + source: { kind: 'base64', data: 'AAAA' }, + }; + expect(MediaGenResultSchema.safeParse({ media, jobId: 'x', raw: {} }).success).toBe(false); + expect(MediaGenResultSchema.safeParse({ raw: {} }).success).toBe(false); + }); + + it('pollMediaJob: pending / done / failed each conform to the discriminated MediaJobStatus', async () => { + const media: MediaJobStatus = { + state: 'done', + media: { type: 'media', mimeType: 'video/mp4', source: { kind: 'base64', data: 'AAAA' } }, + }; + const provider = stubAsyncGenerativeProvider({ + polls: [ + { state: 'pending', progress: 0.5 }, + media, + { + state: 'failed', + error: makeLlmError({ + provider: 'openai', + kind: 'content_filter', + message: 'blocked by policy', + }), + }, + ], + }); + const first = await provider.pollMediaJob('job', KEY); + const second = await provider.pollMediaJob('job', KEY); + const third = await provider.pollMediaJob('job', KEY); + for (const status of [first, second, third]) { + expect(MediaJobStatusSchema.safeParse(status).success).toBe(true); + } + expect(first.state).toBe('pending'); + expect(second.state).toBe('done'); + expect(third.state).toBe('failed'); + // A content-policy refusal reuses the one failure vocabulary: the classified `content_filter` LlmError + // (fatal — the engine's codeForLlmError maps it to the fatal `content_filter` ErrorCode, ADR-0045 §6). + if (third.state === 'failed') { + expect(third.error.kind).toBe('content_filter'); + expect(third.error.retryable).toBe(false); + } + }); + + it('pollMediaJob: forwards the caller-owned abort signal (a cancel reaches the in-flight poll, A5)', async () => { + const controller = new AbortController(); + let forwardedCallerSignal = false; + const provider: GenerativeProvider = { + ...stubAsyncGenerativeProvider({ polls: [{ state: 'pending' }] }), + pollMediaJob: (_jobId, _key, signal) => { + // IDENTITY, not mere presence: the engine must thread the RUN's abort signal through, not a fresh + // controller of the adapter's own — else a run cancel would never reach the in-flight poll (A5 §4). + forwardedCallerSignal = signal === controller.signal; + return Promise.resolve({ state: 'pending' }); + }, + }; + const status = await provider.pollMediaJob('job', KEY, controller.signal); + expect(MediaJobStatusSchema.safeParse(status).success).toBe(true); + expect(forwardedCallerSignal).toBe(true); + }); +}); diff --git a/packages/llm/src/conformance/spec.ts b/packages/llm/src/conformance/spec.ts index 03f821fd..b508304a 100644 --- a/packages/llm/src/conformance/spec.ts +++ b/packages/llm/src/conformance/spec.ts @@ -1,10 +1,16 @@ import { describe, expect, it } from 'vitest'; -import type { StopReason } from '@relavium/shared'; +import { MEDIA_SURFACES, type StopReason } from '@relavium/shared'; import { LlmProviderError } from '../llm-error.js'; -import { LlmResultSchema, StreamChunkSchema } from '../types.js'; -import type { LlmErrorKind, LlmProvider, LlmRequest, StreamChunk } from '../types.js'; +import { LlmResultSchema, MediaGenResultSchema, StreamChunkSchema } from '../types.js'; +import type { + LlmErrorKind, + LlmProvider, + LlmRequest, + MediaGenRequest, + StreamChunk, +} from '../types.js'; import type { RecordedResponse } from './replay.js'; /** @@ -38,6 +44,10 @@ export interface ConformanceExpectations { readonly reasoningStream?: { text: string; reasoningTokens?: number }; /** The JSON text a model returns under `responseFormat: json` (ADR-0030) — providers that support it. */ readonly structuredOutput?: { text: string }; + /** The canonical values a SYNC `generateMedia` image normalizes to (1.AG Section C, A5) — providers that + * implement `generateMedia` supply this. `data` is the exact base64 the recorded reply carries, so a + * bytes-mangling normalization regression fails the suite (parity with the text/structured scenarios). */ + readonly mediaGenerate?: { mimeType: string; data: string }; } /** The recorded provider responses a conformance run needs — one per canonical scenario. */ @@ -58,6 +68,10 @@ export interface ConformanceFixtures { readonly reasoningStream?: RecordedResponse; /** A non-streaming reply produced under `responseFormat: json` (ADR-0030) — omit if unsupported. */ readonly structuredOutput?: RecordedResponse; + /** A recorded SYNC media-generation response (e.g. an image-generation reply carrying base64) — omit for + * a provider with no `generateMedia` (`media_surface: 'chat'`-only). Drives the generative seam-contract + * scenario (1.AG Section C, A5). */ + readonly mediaGenerate?: RecordedResponse; /** * A multi-turn tool loop (the path every agent node exercises): `turn1` is a tool-call reply; `turn2` is * the continuation the provider returns AFTER the caller appends the tool result. The conformance test @@ -116,6 +130,12 @@ const JSON_REQUEST: LlmRequest = { }, }; +const MEDIA_GEN_REQUEST: MediaGenRequest = { + model: 'conformance-image-model', + prompt: 'a red circle on a white background', + modality: 'image', +}; + async function collect(stream: AsyncIterable): Promise { const chunks: StreamChunk[] = []; for await (const chunk of stream) { @@ -327,5 +347,53 @@ export function defineConformanceSuite( expect(r2.content.every((part) => part.type !== 'tool_call')).toBe(true); // a text continuation, not another call }, ); + + it('capabilities: media.surface projects to a valid MediaSurface (1.AG, A5)', () => { + // Every adapter's capability projection must resolve `media.surface` to a member of the closed + // MediaSurface union (the data-driven routing seam ADR-0045 §1 keys on) — an adapter that emits a + // garbage/absent surface would silently mis-route. Phase-1 adapters all project 'chat'; the per-adapter + // capability tests pin the exact value, this asserts the cross-adapter shape contract. + const surface = makeReplayAdapter(fixtures.textGenerate).supports.media.surface; + expect(surface).toBeDefined(); + if (surface !== undefined) { + expect(MEDIA_SURFACES).toContain(surface); + } + }); + + it.skipIf(fixtures.mediaGenerate === undefined)( + 'generateMedia (sync): normalizes a generated image to a base64 media part, no jobId (1.AG Section C, A5)', + async () => { + const recorded = fixtures.mediaGenerate; + if (recorded === undefined) { + return; // narrow for skipIf + } + const adapter = makeReplayAdapter(recorded); + if (adapter.generateMedia === undefined) { + throw new Error( + 'a mediaGenerate fixture requires the adapter to implement generateMedia', + ); + } + const result = await adapter.generateMedia(MEDIA_GEN_REQUEST, KEY); + // Defense-in-depth: the whole result must satisfy the canonical MediaGenResult schema (incl. the + // exactly-one-of media|jobId refine). + expect(MediaGenResultSchema.safeParse(result).success).toBe(true); + // SYNC: an in-flight `media` part the engine de-inlines to a handle; NEVER an async `jobId`. + expect(result.jobId).toBeUndefined(); + expect(result.media?.type).toBe('media'); + if (result.media !== undefined) { + expect(result.media.source.kind).toBe('base64'); // no bytes-less url leaks across the seam here + if (expected.mediaGenerate !== undefined) { + expect(result.media.mimeType).toBe(expected.mediaGenerate.mimeType); + // Exact-value parity with the text/structured scenarios: the recorded base64 must round-trip + // byte-for-byte, so a wrong-but-nonempty normalization regression fails here too. + expect(result.media.source).toEqual({ + kind: 'base64', + data: expected.mediaGenerate.data, + }); + } + } + expect(result.raw).toBeDefined(); + }, + ); }); } diff --git a/packages/llm/src/cost-tracker.test.ts b/packages/llm/src/cost-tracker.test.ts index fbe3739f..63b640cc 100644 --- a/packages/llm/src/cost-tracker.test.ts +++ b/packages/llm/src/cost-tracker.test.ts @@ -92,6 +92,16 @@ describe('mediaCost (1.AF/D17 — per-modality media output addend)', () => { ).toBe(0); }); + it('rounds a fractional addend to an integer micro-cent (a fractional durationSeconds × rate)', () => { + // 2.5s × 201/s = 502.5 → ROUNDED to 503; the cost path must never emit a non-integer micro-cent (1.AG §5). + const oddAudio: ModelPricing = { ...PRICED_MEDIA, mediaOutputRates: { audio: 201 } }; + const total = mediaCost(oddAudio, [ + { modality: 'audio', direction: 'output', units: 2.5, unit: 'second' }, + ]); + expect(total).toBe(503); + expect(Number.isInteger(total)).toBe(true); + }); + it('skips a modality the model does not rate (degrade-to-0, never hard-fail)', () => { const onlyImage: ModelPricing = { ...PRICED_MEDIA, mediaOutputRates: { image: 1_000 } }; expect( diff --git a/packages/llm/src/cost-tracker.ts b/packages/llm/src/cost-tracker.ts index ebac9dab..f774a765 100644 --- a/packages/llm/src/cost-tracker.ts +++ b/packages/llm/src/cost-tracker.ts @@ -85,7 +85,9 @@ export function mediaCost( if (rate === undefined || !unitMatchesBilledModality(entry.modality, entry.unit)) { continue; // unpriced modality, or a unit that does not match the model's billed unit → observability-only } - total += entry.units * rate; + // Round once to an INTEGER micro-cent (a fractional `durationSeconds` × per-second rate would otherwise + // produce a non-integer addend) — matching the token path's per-class rounding so every cost stays integer. + total += Math.round(entry.units * rate); } return total; } diff --git a/packages/llm/src/fallback-chain.ts b/packages/llm/src/fallback-chain.ts index d3607abe..4e718988 100644 --- a/packages/llm/src/fallback-chain.ts +++ b/packages/llm/src/fallback-chain.ts @@ -604,7 +604,7 @@ export class FallbackChain { provider: ProviderId, resolve: (handle: string, provider: ProviderId) => Promise, ): Promise { - const key = `${provider}${handle}`; + const key = `${provider}\u0000${handle}`; // NUL joiner: cannot occur in a provider id or a handle const cached = this.#egressSidecar.get(key); if (cached !== undefined) { return cached; diff --git a/packages/llm/src/types.ts b/packages/llm/src/types.ts index 00067a97..05be1e14 100644 --- a/packages/llm/src/types.ts +++ b/packages/llm/src/types.ts @@ -7,6 +7,7 @@ import { DurableMediaPartSchema, LLM_PROVIDERS, MEDIA_BILLED_MODALITIES, + MEDIA_SURFACES, MEDIA_HANDLE_PATTERN, MEDIA_MESSAGE_CAPS, MediaMimeTypeSchema, @@ -196,6 +197,14 @@ export const MediaCapabilitiesSchema = z.object({ document: z.boolean(), // application/pdf (A2) — distinct token/cost profile from image }), outputCombinations: z.array(ModalitySetSchema), + // The model's media-output SURFACE (1.AG/ADR-0045 §1): `'chat'` routes an agent node to the normal + // turn with `output_modalities`; `'generative'` routes it to the separate-endpoint `generateMedia()` + // (sync or async LRO). The seam projection of `model_catalog.media_surface`. **Absent ⇒ `'chat'`** (the + // column's NOT NULL default + the read-site `?? 'chat'`); optional so a CapabilityFlags literal that + // predates routing stays valid. `surface` is a per-MODEL catalog property, not an adapter capability — the + // OpenAI adapter implements sync `generateMedia` (gpt-image-1 image, 1.AG Section C), yet its capability + // surface here is still `'chat'`; which models route generative is catalog state, not an adapter flag. + surface: z.enum(MEDIA_SURFACES).optional(), }); export type MediaCapabilities = z.infer; @@ -401,10 +410,11 @@ export type StreamChunk = z.infer; /** * The request for a separate-endpoint media generation (`media_surface: 'generative'` models — - * gpt-image-1, Imagen, TTS, Sora, Veo). **RESERVED shape at 1.AD (A5)** — `generateMedia` behavior, - * the async poll/checkpoint/resume/cancel loop, and that loop's own ADR land at Phase D (1.AG). - * Deliberately minimal: provider-specific generation knobs (voice, aspect ratio, fps, …) ride - * `providerOptions`; conditioning-media inputs can be ADDED later (an optional field is additive). + * gpt-image-1, Imagen, TTS, Sora, Veo). Seam shape defined at 1.AD (A5); the BEHAVIOR is WIRED at + * 1.AG — sync `generateMedia` de-inline (Section C) + the engine-owned async poll/checkpoint/resume/ + * cancel loop (ADR-0045, Section D). Deliberately minimal: provider-specific generation knobs (voice, + * aspect ratio, fps, …) ride `providerOptions`; conditioning-media inputs can be ADDED later (an + * optional field is additive). */ export const MediaGenRequestSchema = z.object({ model: nonEmptyString, // canonical model id, mapped per adapter @@ -440,10 +450,11 @@ export const MediaGenResultSchema = z export type MediaGenResult = z.infer; /** - * One poll of an async media job (**reserved**, A5). `failed` carries the existing classified - * `LlmError` — a content-policy rejection maps to `content_filter`, a deadline to the retryable - * `timeout` — so the job path reuses the one failure vocabulary instead of inventing a second. - * The job path never uses StopReason. + * One poll of an async media job (A5; the engine-owned poll/checkpoint/resume/cancel loop is wired at + * 1.AG Section D, [ADR-0045](../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * `failed` carries the existing classified `LlmError` — a content-policy rejection maps to `content_filter`, + * a deadline to the retryable `timeout` — so the job path reuses the one failure vocabulary instead of + * inventing a second. The job path never uses StopReason. */ export const MediaJobStatusSchema = z.discriminatedUnion('state', [ z.object({ state: z.literal('pending'), progress: z.number().min(0).max(1).optional() }), @@ -466,11 +477,16 @@ export interface LlmProvider { /** * Separate-endpoint media generation (`media_surface: 'generative'`), OPTIONAL on the one seam — * deliberately not a sibling `GenerativeMediaProvider` (that would duplicate the - * id/key/capability/error/fallback registry). **RESERVED at 1.AD; wired at Phase D (1.AG) with - * its own ADR for the engine-owned poll/checkpoint/resume/cancel loop (A5).** No Phase-1 adapter - * implements it. + * id/key/capability/error/fallback registry). Shape at 1.AD; WIRED at 1.AG — the OpenAI adapter + * implements SYNC image generation (Section C) and the engine owns the async poll/checkpoint/resume/ + * cancel loop (Section D, A5). The Sora/Veo/Imagen/TTS adapters are 1.AH host-wiring. */ generateMedia?(req: MediaGenRequest, key: string): Promise; - /** Poll an async media job by its Relavium-opaque id (**reserved**, A5 — see `generateMedia`). */ - pollMediaJob?(jobId: string, key: string): Promise; + /** + * Poll an async media job by its Relavium-opaque id (A5, [ADR-0045](../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * `signal` aborts the IN-FLIGHT poll so a run cancel reaches the open provider request, not just the + * next schedule. The engine drives this loop (1.AG Section D); no Phase-1 vendor adapter implements it + * yet (the async Sora/Veo adapters are 1.AH). + */ + pollMediaJob?(jobId: string, key: string, signal?: AbortSignalLike): Promise; } diff --git a/packages/shared/src/config.test.ts b/packages/shared/src/config.test.ts index eb901066..866e3cb0 100644 --- a/packages/shared/src/config.test.ts +++ b/packages/shared/src/config.test.ts @@ -45,6 +45,35 @@ describe('config schemas', () => { expect(ProjectConfigSchema.safeParse({}).success).toBe(true); }); + it('accepts the media_job poll/deadline defaults; rejects non-positive (1.AG/ADR-0045 §7)', () => { + expect( + ProjectConfigSchema.safeParse({ + defaults: { + media_job_poll_initial_ms: 5000, + media_job_poll_max_ms: 30000, + media_job_deadline_ms: 1_800_000, + }, + }).success, + ).toBe(true); + expect( + ProjectConfigSchema.safeParse({ defaults: { media_job_poll_initial_ms: 0 } }).success, + ).toBe(false); // positiveInt rejects 0 + expect(ProjectConfigSchema.safeParse({ defaults: { media_job_deadline_ms: -1 } }).success).toBe( + false, + ); + expect( + ProjectConfigSchema.safeParse({ defaults: { media_job_poll_max_ms: 1.5 } }).success, + ).toBe(false); // fractional rejected + }); + + it('rejects media_job_poll_max_ms < media_job_poll_initial_ms (cross-field refine)', () => { + expect( + ProjectConfigSchema.safeParse({ + defaults: { media_job_poll_initial_ms: 30000, media_job_poll_max_ms: 5000 }, + }).success, + ).toBe(false); + }); + it('rejects an unknown/typo config key (strict — ADR-0023 parity)', () => { expect(GlobalConfigSchema.safeParse({ updatechannel: 'stable' }).success).toBe(false); // top-level typo expect( diff --git a/packages/shared/src/config.ts b/packages/shared/src/config.ts index 0074fb0e..d4685d10 100644 --- a/packages/shared/src/config.ts +++ b/packages/shared/src/config.ts @@ -130,8 +130,24 @@ export const ProjectConfigSchema = z max_tokens_estimate: positiveInt.optional(), // Per-modality media-output unit-count default for the pre-egress media cost estimate (1.AF/D17). media_cost_estimate: MediaCostEstimateSchema.optional(), + // Async media-job (generateMedia LRO) poll cadence + deadline (1.AG/ADR-0045 §7). The engine + // polls at `poll_initial_ms`, exponential-backs-off (no jitter) capped at `poll_max_ms`, and + // abandons a job past `deadline_ms` (from submit) as a retryable timeout. Defaults: 5s / 30s / 30min. + media_job_poll_initial_ms: positiveInt.optional(), + media_job_poll_max_ms: positiveInt.optional(), + media_job_deadline_ms: positiveInt.optional(), }) .strict() + .refine( + (d) => + d.media_job_poll_max_ms === undefined || + d.media_job_poll_initial_ms === undefined || + d.media_job_poll_max_ms >= d.media_job_poll_initial_ms, + { + message: 'media_job_poll_max_ms must be >= media_job_poll_initial_ms', + path: ['media_job_poll_max_ms'], + }, + ) .optional(), variables: z.record(z.string(), z.string()).optional(), chat: ChatConfigSchema, diff --git a/packages/shared/src/constants.test.ts b/packages/shared/src/constants.test.ts new file mode 100644 index 00000000..0b54b830 --- /dev/null +++ b/packages/shared/src/constants.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; + +import { + ERROR_CODES, + MEDIA_SURFACES, + RETRYABLE_ERROR_CODES, + RUN_EVENT_TYPES, +} from './constants.js'; + +describe('error-code classification', () => { + it('content_filter is a member of the closed ErrorCode union (1.AG/ADR-0045 §6)', () => { + expect(ERROR_CODES).toContain('content_filter'); + }); + + it('content_filter is FATAL — never in RETRYABLE_ERROR_CODES (a re-issue just re-blocks)', () => { + // Pins the classification so a future edit cannot silently make a content-policy block retryable. + expect(RETRYABLE_ERROR_CODES).not.toContain('content_filter'); + }); + + it('every RETRYABLE code is a valid ErrorCode (the retryable set is a subset)', () => { + for (const code of RETRYABLE_ERROR_CODES) { + expect(ERROR_CODES).toContain(code); + } + }); +}); + +describe('media constants', () => { + it('media_job:submitted is a canonical run-event type (1.AG/ADR-0045 §2)', () => { + expect(RUN_EVENT_TYPES).toContain('media_job:submitted'); + }); + + it('MEDIA_SURFACES is the closed chat|generative routing set (1.AG/ADR-0045 §1)', () => { + expect([...MEDIA_SURFACES]).toEqual(['chat', 'generative']); + }); +}); diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index ddbb7422..7554aeaf 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -32,6 +32,7 @@ export const RUN_EVENT_TYPES = [ 'node:failed', 'node:skipped', 'node:retrying', + 'media_job:submitted', 'human_gate:paused', 'human_gate:resumed', 'run:completed', @@ -73,6 +74,10 @@ export type SessionEventType = (typeof SESSION_EVENT_TYPES)[number]; */ export const ERROR_CODES = [ 'validation', + // A provider content-policy rejection (text or media-generation) — a FATAL cause distinct from + // `validation` (an authoring/shape error), so a surface shows the right reason/remediation. The + // `content_filter` LlmErrorKind maps here (1.AG/ADR-0045 §6); not in RETRYABLE_ERROR_CODES. + 'content_filter', 'provider_auth', 'provider_rate_limit', 'provider_unavailable', @@ -143,6 +148,29 @@ export type OutputModality = (typeof OUTPUT_MODALITIES)[number]; export const MEDIA_BILLED_MODALITIES = ['image', 'audio', 'video'] as const; export type MediaBilledModality = (typeof MEDIA_BILLED_MODALITIES)[number]; +/** + * A model's **media-output surface** (`model_catalog.media_surface`, 1.AG/ADR-0045 §1). `'chat'` + * (default) routes an agent node to the normal turn with `output_modalities`; `'generative'` routes + * it to the separate-endpoint `generateMedia()` (sync or async LRO). Data-drives the inline-vs-generative + * dispatch (no hardcoded model ids). Projects onto `CapabilityFlags.media.surface` in `@relavium/llm`. + */ +export const MEDIA_SURFACES = ['chat', 'generative'] as const; +export type MediaSurface = (typeof MEDIA_SURFACES)[number]; + +/** + * The async media-job (generateMedia LRO) poll cadence + deadline DEFAULTS (1.AG/ADR-0045 §7). The single + * source of these magic numbers — the engine poll loop (Section D) uses them directly: poll at `pollInitialMs`, + * exponential-back-off (no jitter) capped at `pollMaxMs`, abandon a job past `deadlineMs` (from submit) as a + * retryable timeout. The `[defaults].media_job_poll_initial_ms` / `_max_ms` / `_deadline_ms` config OVERRIDES + * exist + validate (config.ts), but the engine does NOT yet read them — wiring the host-resolved overrides into + * the run loop (the `max_tokens_estimate` pattern) is 1.AH host-wiring, like the other `[defaults].*` reads. + */ +export const MEDIA_JOB_POLL_DEFAULTS = { + pollInitialMs: 5_000, + pollMaxMs: 30_000, + deadlineMs: 1_800_000, // 30 min +} as const; + /** * The **media reference scope kinds** persisted in the `media_references` junction (ADR-0042 §3). * A deliberate **superset** with two roles: `run` / `node` references are refcount + terminal-sweep diff --git a/packages/shared/src/node.test.ts b/packages/shared/src/node.test.ts index fdaa6493..263df671 100644 --- a/packages/shared/src/node.test.ts +++ b/packages/shared/src/node.test.ts @@ -88,6 +88,38 @@ describe('NodeSchema', () => { ).toBe(false); }); + it('accepts the generative media-volume knobs count/duration_seconds on an agent node (1.AG/ADR-0045 §1)', () => { + expect( + NodeSchema.safeParse({ + id: 'a', + type: 'agent', + agent_ref: 'x', + output_modalities: ['image'], + count: 4, + }).success, + ).toBe(true); + expect( + NodeSchema.safeParse({ + id: 'a', + type: 'agent', + agent_ref: 'x', + output_modalities: ['audio'], + duration_seconds: 12.5, + }).success, + ).toBe(true); + // non-positive / non-integer count is rejected + expect(NodeSchema.safeParse({ id: 'a', type: 'agent', agent_ref: 'x', count: 0 }).success).toBe( + false, + ); + expect( + NodeSchema.safeParse({ id: 'a', type: 'agent', agent_ref: 'x', count: 1.5 }).success, + ).toBe(false); + expect( + NodeSchema.safeParse({ id: 'a', type: 'agent', agent_ref: 'x', duration_seconds: -1 }) + .success, + ).toBe(false); + }); + it('accepts a relative save_to on an output node, rejects absolute/traversal/unknown-key (1.AF, A9)', () => { expect( NodeSchema.safeParse({ id: 'o', type: 'output', save_to: 'out/{{ run.id }}/image.png' }) diff --git a/packages/shared/src/node.ts b/packages/shared/src/node.ts index aa0c75db..e264be76 100644 --- a/packages/shared/src/node.ts +++ b/packages/shared/src/node.ts @@ -102,6 +102,12 @@ export const AgentNodeSchema = z max_tokens: positiveInt.optional(), output_schema: OutputSchemaSchema.optional(), output_modalities: OutputModalitiesSchema.optional(), // non-text output request (1.AF, ADR-0031/0044) + // Media-volume knobs for a `media_surface: 'generative'` model (1.AG Section C, ADR-0045 §1): the author's + // requested output volume, mapped to `MediaGenRequest.count`/`durationSeconds` at dispatch. `count` = + // images per call (image generators); `duration_seconds` = target length (audio/video generators). Inline + // on the agent node (no nested `agent_config` object); ignored by a `'chat'` model. node-types.md is SSOT. + count: positiveInt.optional(), + duration_seconds: z.number().positive().optional(), timeout_ms: positiveInt.optional(), retry: RetrySchema.optional(), }) diff --git a/packages/shared/src/run-event.test.ts b/packages/shared/src/run-event.test.ts index 1a5cd165..cb79604d 100644 --- a/packages/shared/src/run-event.test.ts +++ b/packages/shared/src/run-event.test.ts @@ -88,6 +88,17 @@ const valid: Record> = { error: { code: 'tool_failed', message: 'boom', retryable: true }, delayMs: 1000, }, + 'media_job:submitted': { + type: 'media_job:submitted', + ...env, + nodeId: 'n', + jobId: 'job-1', + provider: 'openai', + model: 'sora', + modality: 'video', + startedAt: '2026-06-20T00:00:00.000Z', + deadlineAt: '2026-06-20T00:30:00.000Z', + }, 'human_gate:paused': { type: 'human_gate:paused', ...env, @@ -218,6 +229,37 @@ const reject: Record> = { attemptNumber: 1, error: { code: 'tool_failed', message: 'x', retryable: true }, }, + 'media_job:submitted (bad provider)': { + ...valid['media_job:submitted'], + provider: 'cohere', // not in LLM_PROVIDERS + }, + 'media_job:submitted (bad modality)': { + ...valid['media_job:submitted'], + modality: 'document', // billed modalities are image|audio|video only + }, + 'media_job:submitted (non-datetime deadlineAt)': { + ...valid['media_job:submitted'], + deadlineAt: 'soon', + }, + 'media_job:submitted (non-datetime startedAt)': { + ...valid['media_job:submitted'], + startedAt: 'tomorrow', + }, + // deadlineAt = startedAt + media_job_deadline_ms by construction; an earlier deadlineAt is malformed and + // would invert the resume `now > deadlineAt` short-circuit (union-level superRefine, Date.parse-compared). + 'media_job:submitted (deadlineAt before startedAt)': { + ...valid['media_job:submitted'], + startedAt: '2026-06-20T00:30:00.000Z', + deadlineAt: '2026-06-20T00:00:00.000Z', + }, + 'media_job:submitted (empty jobId)': { + ...valid['media_job:submitted'], + jobId: '', + }, + 'media_job:submitted (missing jobId)': { + ...valid['media_job:submitted'], + jobId: undefined, + }, 'human_gate:paused (bad gateType)': { type: 'human_gate:paused', ...env, @@ -253,8 +295,25 @@ const reject: Record> = { error: { code: 'internal', message: 'm', retryable: false }, // compliant — isolate the missing partialOutputs }, 'run:cancelled (negative sequenceNumber)': { type: 'run:cancelled', ...env, sequenceNumber: -1 }, - 'run:paused (empty gateIds)': { ...valid['run:paused'], gateIds: [] }, - 'run:paused (pendingGateCount 0)': { ...valid['run:paused'], pendingGateCount: 0 }, + // A media-job park may carry empty gateIds / pendingGateCount 0 (1.AG Section D), so those are no longer + // rejected; an empty pendingMediaJobNodeIds (min 1) still is, and a ZERO-reason pause (no gate, no media) is + // rejected by the union-level superRefine. + 'run:paused (empty pendingMediaJobNodeIds)': { + ...valid['run:paused'], + pendingMediaJobNodeIds: [], + }, + 'run:paused (no suspension reason)': { + ...valid['run:paused'], + pendingGateCount: 0, + gateIds: [], + }, + // pendingGateCount is the aggregate of gateIds — a divergent pair (count 5, two ids) is malformed and the + // union-level superRefine rejects it (the count/list relaxation must not let them drift, 1.AG Section D). + 'run:paused (pendingGateCount/gateIds mismatch)': { + ...valid['run:paused'], + pendingGateCount: 5, + gateIds: ['g1', 'g2'], + }, 'run:timeout (negative elapsedMs)': { ...valid['run:timeout'], elapsedMs: -1 }, 'budget:warning (thresholdPct > 100)': { ...valid['budget:warning'], thresholdPct: 101 }, 'budget:warning (negative thresholdPct)': { ...valid['budget:warning'], thresholdPct: -1 }, @@ -274,7 +333,19 @@ describe('RunEvent union — every variant', () => { expect(RunEventSchema.safeParse(reject[name]).success).toBe(false); }); - it('covers exactly the 20 canonical colon-namespaced names, pinned to a literal list', () => { + it('accepts a media-only run:paused park: empty gateIds + pendingGateCount 0 + media node ids (1.AG Section D)', () => { + expect( + RunEventSchema.safeParse({ + type: 'run:paused', + ...env, + pendingGateCount: 0, + gateIds: [], + pendingMediaJobNodeIds: ['work'], + }).success, + ).toBe(true); + }); + + it('covers exactly the 21 canonical colon-namespaced names, pinned to a literal list', () => { // A hardcoded contract list — independent of RUN_EVENT_TYPES — so the union and the // constant cannot silently drift together. const CONTRACT_NAMES = [ @@ -289,6 +360,7 @@ describe('RunEvent union — every variant', () => { 'node:failed', 'node:skipped', 'node:retrying', + 'media_job:submitted', 'human_gate:paused', 'human_gate:resumed', 'run:completed', @@ -305,7 +377,7 @@ describe('RunEvent union — every variant', () => { // RunEventSchema wraps the union in the correlation-key refinement; reach the raw union. expect(RunEventSchema.innerType().options).toHaveLength(CONTRACT_NAMES.length); expect(new Set(RUN_EVENT_TYPES)).toEqual(new Set(CONTRACT_NAMES)); - expect(Object.keys(valid)).toEqual(CONTRACT_NAMES); // the matrix covers all 20 + expect(Object.keys(valid)).toEqual(CONTRACT_NAMES); // the matrix covers all 21 }); it('pins the RunEvent discriminant to RunEventType (type-level)', () => { @@ -522,6 +594,13 @@ describe('event envelope + ErrorCode + attemptNumber invariants', () => { error: { code: 'sandbox_error', message: 'x', retryable: false }, }).success, ).toBe(true); + // The 1.AG content_filter ErrorCode parses on the event path (a content-policy block surfaces here). + expect( + RunEventSchema.safeParse({ + ...valid['node:failed'], + error: { code: 'content_filter', message: 'content policy block', retryable: false }, + }).success, + ).toBe(true); }); it('accepts an optional 1-based attemptNumber on every carrier event', () => { diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index 4e57a044..393097b8 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -6,6 +6,8 @@ import { ERROR_CODES, EXECUTION_MODES, FS_SCOPE_TIERS, + LLM_PROVIDERS, + MEDIA_BILLED_MODALITIES, STOP_REASONS, } from './constants.js'; import { GateTypeSchema, TimeoutActionSchema } from './node.js'; @@ -258,6 +260,29 @@ export const NodeRetryingEventSchema = z.object({ delayMs: nonNegativeInt, }); +/** + * An async media-generation job was submitted to a provider; the engine now owns its + * poll/checkpoint/resume/cancel loop (1.AG, ADR-0045 §2). **Durable** so a crash-resume RE-ATTACHES + * (re-polls the persisted opaque `jobId`) rather than re-submitting (which would double-bill + orphan + * the vendor job). The node parks — a NON-terminal suspension — until its terminal + * `node:completed | node:failed | node:skipped`. Per-poll progress is transient (never persisted). + * `jobId` is Relavium-opaque, never the vendor operation-name (ADR-0011 I1). + */ +export const MediaJobSubmittedEventSchema = z.object({ + type: z.literal('media_job:submitted'), + ...runBase, + nodeId: nonEmptyString, + jobId: nonEmptyString, // the Relavium-opaque job id the engine polls (never the vendor op-name) + provider: z.enum(LLM_PROVIDERS), + model: nonEmptyString, // canonical model id + modality: z.enum(MEDIA_BILLED_MODALITIES), // image | audio | video + startedAt: z.string().datetime({ offset: true }), // ISO-8601 job-SUBMIT time (when generateMedia returned the jobId; the deadlineAt anchor), not the node-start time + // deadlineAt = startedAt + [defaults].media_job_deadline_ms; on resume `now > deadlineAt` short-circuits a + // doomed re-poll. An offset is allowed, so a consumer MUST compare via Date.parse, never lexicographically. + deadlineAt: z.string().datetime({ offset: true }), +}); +export type MediaJobSubmittedEvent = z.infer; + /** Why a node was skipped — `branch_not_taken` (a `condition` routed away) or `upstream_unreachable`. */ export const NodeSkippedReasonSchema = z.enum(['branch_not_taken', 'upstream_unreachable']); export type NodeSkippedReason = z.infer; @@ -329,9 +354,14 @@ export const RunCancelledEventSchema = z.object({ export const RunPausedEventSchema = z.object({ type: z.literal('run:paused'), ...runBase, - // The multi-gate aggregate — emitted while ≥1 gate is pending (parallel branches each gate). - pendingGateCount: positiveInt, - gateIds: z.array(nonEmptyString).min(1), + // The multi-gate aggregate — gates pending while the run parks (parallel branches each gate). `0`/empty + // when the run parks ONLY on an async media job (1.AG Section D); the engine emits `run:paused` only while + // genuinely parked (a gate OR a media job OR BOTH — AG-A-FC-3), so ≥1 reason holds by construction. + pendingGateCount: z.number().int().min(0), + gateIds: z.array(nonEmptyString), + // The async media-job park (1.AG Section D, [ADR-0045](../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §2): + // node ids parked on an engine-owned `pollMediaJob` loop, reusing the gate-suspend machinery. + pendingMediaJobNodeIds: z.array(nonEmptyString).min(1).optional(), }); export const RunTimeoutEventSchema = z.object({ @@ -371,6 +401,7 @@ const RunEventUnionSchema = z.discriminatedUnion('type', [ NodeFailedEventSchema, NodeSkippedEventSchema, NodeRetryingEventSchema, + MediaJobSubmittedEventSchema, HumanGatePausedEventSchema, HumanGateResumedEventSchema, RunCompletedEventSchema, @@ -413,6 +444,45 @@ export const RunEventSchema = RunEventUnionSchema.superRefine((event, ctx) => { path: ['timeoutAction'], }); } + // A pause always has a reason: ≥1 gate OR ≥1 media job (1.AG Section D). The member's field constraints were + // relaxed (a media-only park has 0 gates), so the disjunction is enforced here at the union level (a + // discriminatedUnion member can't carry its own cross-field refinement). The engine never emits a zero-reason + // run:paused; this rejects a malformed one. + if ( + event.type === 'run:paused' && + event.gateIds.length === 0 && + (event.pendingMediaJobNodeIds?.length ?? 0) === 0 + ) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: 'run:paused must carry at least one suspension reason (a gate or a media job)', + path: ['gateIds'], + }); + } + // `pendingGateCount` is the aggregate count of `gateIds` — relaxing both fields to `min(0)` (for a media-only + // park) dropped the structural guarantee that they agree, so re-assert it here. A consumer that reads + // `pendingGateCount` as the authoritative gate count must not diverge from the `gateIds` list it pairs with. + if (event.type === 'run:paused' && event.pendingGateCount !== event.gateIds.length) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: 'run:paused pendingGateCount must equal gateIds.length', + path: ['pendingGateCount'], + }); + } + // `deadlineAt = startedAt + media_job_deadline_ms` by construction, so deadlineAt < startedAt is a malformed + // durable event that would invert the resume `now > deadlineAt` short-circuit. Compare via Date.parse (an + // offset is allowed, so never lexically). Enforced at the union level — a member-level `.refine()` would make + // a ZodEffects and break the discriminatedUnion (same reason the run:paused cross-field checks live here). + if ( + event.type === 'media_job:submitted' && + Date.parse(event.deadlineAt) < Date.parse(event.startedAt) + ) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: 'media_job:submitted deadlineAt must be >= startedAt', + path: ['deadlineAt'], + }); + } }); export type RunEvent = z.infer;