diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/README.md b/docs/design/agent-workflows/projects/agent-multi-modality/README.md new file mode 100644 index 0000000000..c60f4c8923 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/README.md @@ -0,0 +1,110 @@ +# Agent multi-modality + +This project lets a person share files with an agent and have two things happen at once: +the model reads the file, and the agent can work on the file with its tools. Today neither +happens. A person can attach an image or a document in the agent chat box, but the file is +thrown away just before it reaches the model, so the model never sees it and the agent never +gets it. + +A second goal sits beside the first. Every file a person shares must stay easy to find later, +even after the agent has changed things. If you gave the agent a spreadsheet an hour ago, you +should be able to open exactly that spreadsheet again, unchanged, without hunting through the +agent's scratch files. + +This folder holds the research, the design, and the plan for closing that gap. The work is +staged so that the first stages are small and safe and later stages add the harder modalities: +audio, where the first release ships browser dictation only and recordings stay workspace-only, +and documents, which wait on adapter work. + +## Glossary + +These terms appear across every file here. Each one names a real thing in our system and says +where it runs. + +- **Agent.** A coding assistant that can read and write files and call tools, driven by a + large language model. In our product it runs inside a sandbox. +- **Model.** The model is the large language model itself (for example Claude or a GPT model). The + model is what perceives an image or a document when the bytes are delivered to it in the right way. +- **Harness.** The program that drives the model through one agent turn (for example Claude + Code or Pi). The harness owns the model conversation. We do not own the harness code. +- **Runner.** Our own Node.js service that starts a sandbox, launches the harness inside it, + and passes each turn's message to the harness. The runner is where the file is dropped + today. It lives at `services/runner/`. +- **Sandbox.** The isolated Linux environment the agent runs in. The agent's files and shell + commands all happen inside the sandbox. +- **Session.** One ongoing conversation with an agent. A session has a stable id that the + front end creates before the first message is sent. +- **Turn.** One request-and-reply within a session. The person sends a message, the agent + works and answers. That is one turn. +- **cwd (working directory).** The folder the agent runs in inside the sandbox. It survives + across turns of the same session. The agent can create, change, and delete anything in it. +- **Mount.** Our file storage unit. A mount is a named area in an object store (S3) with its + own keys and its own access credentials. The `cwd` is backed by a mount. Mounts live in the + API at `api/oss/src/core/mounts/`. +- **geesefs / FUSE mount.** The technology that makes a storage mount appear as a normal + folder inside the sandbox. When a mount is "mounted into the sandbox," the agent sees it as + a directory it can read and write. +- **ACP (Agent Client Protocol).** The message format the runner uses to talk to the harness. + It is an external standard published by Zed Industries. We cannot change it. It defines the + content types a turn can carry (text, image, audio, and so on). +- **Content block.** One piece of a message in ACP: a text block, an image block, an audio + block, or a resource block. A turn is a list of content blocks. +- **Materialize.** The runner materializes a file when it writes a copy of the stored original into + the agent's working directory so the agent's tools can open it. +- **Records.** Records are the API's durable, per-session log of every event in a conversation. The + runner writes to it after every event. +- **Drawer / drive / Files drawer.** The front-end panel that lists the files in a session. It + is a view over one or more mounts. In the code it is the `DriveExplorer`. A file uploaded here + writes directly to the mount at the chosen path; it is not an attachment (decision D13). +- **Attachment resource / attachment_id.** The record the API returns when a file is uploaded. It + holds a server-issued `attachment_id`, the filename, the media type the server verified, and the + size. The API owns the storage location behind the id, so a client never sees it. The + `attachment_id` is the opaque token that travels on the wire, in the saved history, and in the + traces. +- **Reference (attachment reference).** The `attachment_id` plus a little display metadata, sent in + a message in place of the file's bytes. It carries no bytes and no raw storage coordinates; it + points at the attachment resource above, which the runner resolves through the API. +- **Front end / composer.** The front end is the browser app. The composer is its message editor: + the box where a person types, attaches files, and sends a turn. +- **API.** Our FastAPI backend service. It owns storage, permissions, sessions, and records, and it + is the only component that talks to the object store with full rights. +- **SDK.** Our Python library that sits between the API and the runner. It parses incoming messages + into content blocks and writes the wire payload the runner reads. +- **Adapter.** The translation layer inside the harness that converts ACP content blocks into the + model provider's own API calls. We run two: `claude-agent-acp` (maintained by Zed) for Claude, and + `pi-acp` (from the Pi project) for Pi. +- **Object store.** The S3-compatible storage service where mount bytes live. +- **Wire.** The serialized request that travels from the front end through the API to the runner for + one turn. +- **Native modality.** A kind of content (an image, audio, a document) that a model perceives + directly through its own encoder when the bytes are delivered inside the message content. + Everything else is a plain file that only tools can open. +- **Capability.** A yes-or-no fact about what a layer supports, advertised or configured, used to + decide whether a modality can reach the model. +- **Warm session / cold start.** A session is warm when its harness process is still running and can + continue the conversation. A cold start is when that process is gone and the runner must rebuild + the conversation before running the new turn. +- **Cold replay.** The runner's reconstruction of the conversation during a cold start, today by + flattening past messages to text. + +## Read in this order + +1. **[context.md](context.md)**: What a person experiences today, why the file is lost, and + what "done" looks like. Start here if you want the plain story with no design. +2. **[research.md](research.md)**: The findings the design rests on: what the mounts API can + do and who may call it, how different kinds of files are handled by models, how ACP carries + content and what each harness does with it, how other tools (Zed, opencode) handle + attachments, and the state of records and capability flags in our code. Every claim links to + a file and line or to a source. +3. **[design.md](design.md)**: The design itself, written as options and decisions. For each + major choice it gives the alternatives, what breaks under each, the decision, and why. It + keeps every interaction diagram, each with the flow explained in words first. +4. **[plan.md](plan.md)**: The staged implementation plan: what changes in each layer, who + owns each part, how each stage is tested, and how the plan holds up against the review + lenses (responsibilities, engineering practice, tradeoffs, scale, and fit with the rest of + the architecture). +5. **[scope.md](scope.md)**: What is in, what is out, and the follow-up work with a sentence + each on why it waits. +6. **[decisions.md](decisions.md)**: The decision log and the open questions, each with what + is unknown, why it matters, and how to settle it. +7. **[status.md](status.md)**: Where the project stands and what happens next. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/context.md b/docs/design/agent-workflows/projects/agent-multi-modality/context.md new file mode 100644 index 0000000000..d6fc22774c --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/context.md @@ -0,0 +1,99 @@ +# Context: what happens today and what we want + +This file tells the story in plain language. It has no design in it. If you want the evidence +behind each claim, read [research.md](research.md). If you want the design, read +[design.md](design.md). + +## What a person experiences today + +A person opens the agent chat, attaches a photo, and types "what is this?" The chat box +accepts the photo. It shows a small preview. The message looks like it was sent with the +image. Then the agent answers as if no image were there. It saw only the words "what is this?" +and nothing else. + +If the person attaches only the photo and no text at all, the turn fails outright with an +error that says there is no message to send. + +Nothing warns the person that this will happen. The attachment looks accepted and then quietly +does nothing. That silent gap is the core problem. + +There is a second, separate product we already ship: the prompt playground, which is not an +agent. That one does handle images and documents correctly. So agents are behind the rest of +the product, and the difference is invisible until you notice the agent ignoring your file. + +## Why the file is lost + +The file travels correctly almost the whole way. The chat box reads the file into memory. The +request carries it. Our SDK carries it. The runner, our service that drives the agent, receives +it intact. Then, at the very last step, the runner builds the message it hands to the harness +using only the text. It writes a single text block and drops everything else. The runner discards +every content block that is not text at that one call. + +That last step was written when agents were text-only. It was never updated when attachments +were wired up through the rest of the path. So the path is complete except for the final step. + +There was a related habit that made this worse. The front end used to resend the whole +conversation on every new turn, including every earlier attachment, in full, as raw bytes +encoded as text. That habit is gone: the front end now sends only the newest message, and the +runner keeps the conversation history itself, in durable records it can rebuild the +conversation from. But the fix does not cover attachments. In the turn where a file is +attached, its bytes still travel in that one message as encoded text, and a single five +megabyte image becomes about seven megabytes of it. The same bytes are still saved in the +browser's local storage and still land in our tracing data. And the durable records keep only +the words, so a conversation rebuilt from them has lost the file. None of this is needed once +a file is sent by reference instead of by value. + +## What we want to achieve + +We want three outcomes, and all three at the same time. + +**The model reads the file.** When a person shares an image, the model should see the image. +When they share audio, the model should hear it. When they share a document, the model should +read it. This is the part that is broken at the last runner step today. + +**The agent can work on the file.** Beyond the model reading it, the file should be present on +the agent's own working directory so the agent's tools can open it, convert it, edit it, or run +a program over it. Whether the agent changes the file is the person's call in each +conversation, not a fixed rule. + +**The file stays findable.** A person must always be able to return to exactly what they +shared, unchanged, and download it or see it again, even if the agent changes or deletes files in +its working directory. The thing you gave the agent is a durable record, not a temporary +scratch file that the agent might overwrite or delete. + +## What "done" looks like + +A person attaches a spreadsheet and asks the agent to chart the third column. The model reads +the spreadsheet. The agent opens the same spreadsheet with its tools, computes the chart, and +writes a new chart file. The person can still open the original spreadsheet, untouched, from the +files panel, and can also see the new chart the agent produced. If the person attaches a kind of +file the current model cannot handle, the chat box says so before the message is sent, instead +of accepting it and then ignoring it. + +## The three hard constraints the design must respect + +**The protocol to the harness is external.** The runner talks to the harness in ACP, a +standard we do not own. In ACP, for the model to actually perceive an image or audio, the bytes +must be delivered inline in the turn. There is no way to hand the model a link and trust it to +fetch the file. Storing the file in our object store does not remove this rule. It only removes +the need to carry the bytes in the saved conversation records. At the moment the runner hands +the turn to the harness, the bytes must be present. Details and sources are in +[research.md](research.md). + +**We build on mounts, our existing file storage.** We already have a working file storage +system with per-session storage, access controls, upload, download, and listing. The design +reuses it rather than inventing new storage. What a mount is, what it can do, and who may call +it are laid out in [research.md](research.md). + +**The harness adapters we run today deliver images, but not audio or documents.** The runner talks +to the model through two adapter packages (one for Claude, one for Pi). Reading their code shows that +both deliver an image to the model natively, but neither delivers native audio, and both fail to +deliver a document (one drops it, the other turns it into a byte count). So images can be shipped +now, while native audio and native document delivery wait on adapter work. The first release still +ships speech input: the composer's dictation mode turns live speech into text in the browser, while +a recorded clip stays a workspace-only attachment with a visible notice and server-side +transcription is deferred ([design.md](design.md), decision D14). The evidence is in +[research.md](research.md), section 4. + +The full list of verified failure modes, with the exact files and lines, lives in +[research.md](research.md) under "Current state of the code." diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/decisions.md b/docs/design/agent-workflows/projects/agent-multi-modality/decisions.md new file mode 100644 index 0000000000..01fd98dd78 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/decisions.md @@ -0,0 +1,82 @@ +# Decisions and open questions + +The reasoning behind each decision is in [design.md](design.md). This file is the compact log so it +can be updated without editing the design narrative. + +## Decision log + +| # | Question | Options | Decision | Reason | +| --- | --- | --- | --- | --- | +| D1 | How does the file reach the model? | inline content blocks; file on disk plus a path; both | Both | Inline is the only guaranteed perception path: deterministic at the prompt-build seam, uniform across harnesses, and it survives a replay. The disk copy lets the agent's tools work on the file, and on some harnesses the agent's own read tool adds a second, agent-mediated route to perception (research.md, section 4). The two serve different goals. | +| D2 | Does storing the file in the object store remove the need to send bytes to the model? | yes, fully; no, only from the wire and the saved history | No, only from the wire and the saved history | Storage keeps bytes out of the durable records the runner replays on a cold start, but the model boundary still needs the bytes inline at prompt time, rebuilt per turn by the runner from the stored original. | +| D3 | Where does the unchanging original live? | a subfolder in cwd; the agent-files mount; a dedicated session mount kept out of the sandbox; a future project drive | A dedicated session-scoped attachments mount, kept out of the sandbox | The working directory is last-writer-wins, so an original there could be deleted or overwritten by the agent. Findability needs the original out of the agent's reach, and the mount technology exposes whole prefixes, so out-of-reach means its own mount. | +| D4 | One copy or two? | one copy (perceive and edit the same object); two copies | Two copies: an unchanging original and a disposable working copy | One copy cannot be both safe-to-find and freely-editable. Two copies make both goals true and turn an agent edit into a new, visible output rather than a destroyed original. | +| D5 | How does the system know whether a modality will reach the model? | a single capability flag; the intersection of three layers (protocol transport, adapter fidelity, model modalities) | The three-layer intersection, gated at the composer (courtesy, from a pre-send approximation) and the runner (final authority) | A flag can be advertised while the delivery is lossy or dropped, so one flag is not enough. Tool use over the working copy is separate and works regardless of all three layers. | +| D6 | What happens when the model cannot perceive an attached kind? | refuse the attachment; drop it silently; fail the turn; attach it as a workspace-only file with a visible notice | Attach it as a workspace-only file with a visible notice; the runner fails the turn only on a contract violation (asked to deliver a native block the harness cannot accept) | Attach means two things: show it to the model, and put it in the workspace. The workspace half always works, so refusing or failing would block legitimate tool use. Silent dropping is the current trap. A visible notice keeps it honest. | +| D7 | How is the original kept immutable? | keep it out of the sandbox; also write only through a create-only upload route and never sign the mount; also a read-only credential scope | Keep it out of the sandbox, and write only through the create-only upload route with no signed credentials for the mount; read-only scope is a hardening follow-up | Keeping it out of the sandbox stops the agent, the main threat. Originals enter only through the create-only upload route, and the API refuses overwrite and delete of an original as an explicit check, because `write_file` overwrites silently today. No signed credentials are issued for the mount, because any signed credential is read-write today. A read-only scope is the strongest form and needs a signing change, so it waits. | +| D8 | Is audio in scope? | defer; include | Include as a product goal | Audio stays a goal, and there is no disk-read fallback for it. Native delivery is infeasible today at two independent layers: neither pinned adapter (`claude-agent-acp`, `pi-acp`) advertises ACP audio, and the Anthropic Messages API has no audio input at all (research.md, section 4). The releases are staged: browser dictation now (D14), server-side transcription later (surveyed and deferred, research.md, section 4), the native ACP audio block last. | +| D9 | Can the inline-only version grow into the full one? | it is a different architecture; it grows cleanly with no rework; the model-facing seam is preserved but the rest is system-wide | The model-facing seam is preserved, but adopting durable references later is still a system-wide change | Shipping the inline-only version first avoids a rewrite of the one prompt-builder seam. It does not avoid the rest: front-end persistence, API storage ownership, SDK wire types, runner resolution, the record schema, authorization, and cleanup. So the benefit is narrow and honest, not a free migration. | +| D10 | What does the reference on the wire contain? | raw storage coordinates (mount id, path, client media type); an opaque server-issued id | An opaque server-issued attachment id, with the API owning the storage location and the verified metadata | Raw coordinates let a client forge a reference to another session's file and make the client's media type authoritative even though a client can lie. An opaque id keeps storage private to the API, lets the server verify the media type on upload, and makes the session-binding check natural to express. | +| D11 | What does attaching a file promise in the first release? | image perception only (strict limits, no durability); durable agent input (immutable original, workspace copy, findability, records) | Durable agent input from day one, with workspace-always plus native-when-possible semantics | The product owner decided on 2026-07-31 to build the full reference-based design from the first release; no inline-only version will be built. Durable agent input matches the three outcomes in context.md and avoids shipping a storage-less version that would need rework (D9). | +| D12 | What does a cold replay deliver for historical attachments? | re-deliver inline within a count and byte budget; a textual mention with the real filename and working-copy path, a cold-start sweep that restores missing working copies, inline delivery only for the running turn | Mention-first: the rebuilt conversation carries the mention, the cold start restores every referenced working copy, and inline delivery is reserved for the running turn's references | The harnesses' read tools verifiably deliver images from disk on all three harnesses and PDFs on Claude (research.md, section 4), so a mentioned file is one tool call away. Nothing inline grows with the history, which removes the recurring re-delivery cost and dissolves the budget problem; this decision replaces the former open question on the cold-replay budget numbers. Accepted cost: historical-image perception is agent-discretionary instead of guaranteed. On Pi, documents lose nothing, because inline PDFs never worked there. Accepted corner: only the cold start sweeps, so a warm prose-only turn after the agent deleted a working copy still finds it missing; revisit with evidence. | +| D13 | Do drawer uploads and composer attachments share one pipeline? | route every upload through the attachment pipeline; split by surface | Split by surface, deliberately. A file dragged into the Files drawer (or its folders) uploads directly to that mount at that path, the existing drive-upload flow. The attachment pipeline (attachment resource, immutable original, opaque reference) applies only to files shared through the chat composer, whether typed, pasted, dropped, or picked there. | The two surfaces express two different intents: "put this file there" versus "share this with the agent in the conversation." Routing a drawer drop through attachments would give a plain folder write the wrong semantics: immutability and per-session attachment scoping where the person asked for a file at a path they chose. The drive path already exists and works (PR #5459, merged behind `NEXT_PUBLIC_AGENT_FILE_UPLOADS`). | +| D14 | What does the first release do with speech and audio files? | browser dictation only; transcription through the person's stored keys via litellm; transcription on an Agenta platform key; whisper in the services layer | Browser dictation only. The composer's dictation mode (Web Speech API, `web/oss/src/components/AgentChatSlice/hooks/useVoiceInput.ts`, merged dark in PR #5458) turns live speech into text in the composer while the person talks; no audio file is involved and nothing new is built. A recorded voice message or an uploaded audio file gets no transcription: it follows the D6 workspace-only path (immutable original, working copy, and the visible notice that the model will not hear it), and the agent's tools can still process the file. Server-side transcription is deferred out of the first release: no transcription endpoint, no key resolver, no platform key, no metering. Native ACP audio delivery is the last upgrade. | The recorded-audio limit is structural: the Web Speech API transcribes only a live microphone stream, so the browser cannot transcribe a recorded blob (research.md, section 4). Crossing that line means a server-side service with key resolution and metering, and the decision is to add none of that machinery now. The survey behind the deferred follow-up (litellm call path, prices, who has a usable key) is in research.md, section 4. | + +## Settled by evidence + +These questions are answered by evidence in the code or by constraints already in place, not by a +product choice, and are recorded here so they are not reopened by accident. + +- **Does a document reach the model natively?** Answered by reading the adapter code, not by a live + test. Neither pinned adapter delivers a document natively today: the Claude adapter drops a blob + resource entirely, and the Pi adapter renders it as a byte count (see [research.md](research.md), + section 4). So documents do not arrive natively today. This is no longer an open question; it is a + Stage 2 blocker (see [plan.md](plan.md), Stage 2). +- **What path convention does the working copy use?** Decided: the id-namespaced visible path + `cwd/attachments//`, which is discoverable by the agent and collision-proof + across same-name files ([design.md](design.md), The working-copy path and edited copies). +- **Does the runner read the original through the API or directly from the object store?** Decided: + through the API download route for the first release, with the session-binding check kept in one + place. Reading the object store directly with a read-only credential scope is the Stage 3 hardening + ([design.md](design.md), decision D7). +- **What is the media-type, validation, and limits matrix?** Settled 2026-07-31 as engineering + defaults; the full matrix is in [design.md](design.md), "The media-type, validation, and limits + matrix". In brief: the server classifies by inspected magic bytes and stores the verified type + (declared type never trusted, per D10); rejection happens only for size and unrecognizable bytes, + never for kind (D6); the per-kind caps mirror the shipped client caps (100 files per turn, 10 MB + images, 15 MB audio, 10 MB documents, `attachments.ts`, verified 2026-07-31); the compose gateway + `client_max_body_size` rises from 10 MB to 32 MB + (`hosting/docker-compose/oss/nginx/nginx.conf`; railway already allows 32 MB) with the upload + route's per-kind caps as the enforced truth; and an original that exceeds a provider's inline cap + (for example Anthropic's 10 MB base64 per image, about 7.5 MB raw) is delivered workspace-only + with the D6 notice, never resized and never a failed turn. +- **What are the retention rules when a session is archived or deleted?** Settled 2026-07-31: + attachment originals share their session's lifecycle. They are deleted when the session is + deleted, and kept as long as the session exists, including while it is archived. The attachments + mount is a session mount, so the existing session-mount teardown already implements this: + `delete_session_mounts` hard-deletes the mount rows and their object-store prefixes, and + `archive_session_mounts` soft-archives reversibly without touching the bytes + (`api/oss/src/core/mounts/service.py`, verified 2026-07-31). An independent expiry while the + session lives would break the design's own findability promise, because the conversation renders + by resolving references against originals. The implementation cost is one Stage 3 test asserting + the attachments mount is included in the existing session teardown. Per-workspace retention + policies and user-initiated purge of a single original are deferred follow-ups + ([scope.md](scope.md)). + +## Open questions + +All the product decisions are taken (D1 through D14). One implementation-time question remains. It +says what is unknown, why it matters, how to settle it, and what it blocks. + +1. **How are unused uploads cleaned up (the refinement)?** + - We do not yet know when to move from a time-to-live sweep to reference counting against the + conversation records. + - This matters because a file uploaded but never sent leaves a stored object, and the sweep needs + a marker telling it which uploads are still unused. Reference counting against the records is + more precise than a marker plus an age cutoff. + - The starting answer is that a time-to-live sweep ships first, in Stage 1, over uploads that were + never claimed. Nothing is marked by being downloaded: the runner calls the claim operation once + it has validated the current turn, and only a claimed upload is durable. Reference counting is + added only after records reliably carry references, because counting needs the record schema to + hold the reference. + - This blocks the cleanup refinement in Stage 3. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/design.md b/docs/design/agent-workflows/projects/agent-multi-modality/design.md new file mode 100644 index 0000000000..a8b8bfcce1 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/design.md @@ -0,0 +1,1015 @@ +# Design + +This file presents the design as a set of decisions. It explains the major choices in full: for each +it lists the options, says what breaks under each, gives the decision, and explains why. The +complete compact log D1 through D14 lives in [decisions.md](decisions.md); D2 and D8 are small enough +that they are stated inline where they arise. This file keeps every interaction diagram, and each +diagram is explained in words before it is shown. Read [research.md](research.md) first for the facts +these decisions rest on. Each decision carries the same D-number here and in the compact log in +[decisions.md](decisions.md), so the two files never disagree about which decision is which. + +## The idea in one paragraph + +Stop sending file bytes on the wire. When a person attaches a file, the front end uploads it once +through the API's session-mount upload route. The API stores the bytes and returns an **attachment +resource**: a small record it owns, with a server-issued `attachment_id`, the filename, the +media type the server verified, and the size. The message then carries only that `attachment_id` +plus a little display metadata, never the bytes and never raw storage coordinates. Behind that id +the system keeps two copies. One is the **original**, which never changes and which the agent +cannot reach. It is the source of truth for what the model reads, what the files panel lists, and +what a download returns. The other is a **working copy** that the runner writes into the agent's +working directory so the agent's tools can open, convert, or edit it. At the moment the runner +builds a turn, it resolves the `attachment_id` through the API, reads the original, turns it into +the right inline content block, and hands that to the harness in place of the single text block +that is hard-coded today. + +## Where the original and the working copy live + +The design puts the original in a storage mount that is deliberately not made visible to the agent, +and the working copy in the agent's normal working directory. The reference on the wire points at +the original. + +Read the diagram this way. The `attachment_id` travels on the wire, in the saved history, and in +the traces. It names the attachment resource the API owns; only the API knows which object in the +store that resource points at. From the original, three things happen: the runner reads it to build +the model's content block, the runner copies it into the working directory for the agent's tools, +and the files panel lists and downloads it. The working copy is separate and disposable. Note that +the storage key shown as `mounts///...` uses `` only as +the tenant partition of the object key. It does not mean the original lives in a "project mount." +There is no project-scoped mount kind; the attachments mount is scoped to the session. + +The reference on the wire is an opaque, server-issued id, not the file's storage coordinates. This +is deliberate. If the wire carried the raw storage location and the client's own media type, then a +client could forge a reference to another session's file by naming its coordinates, and the client's +media type would be authoritative even though a client can lie about it. An opaque id avoids both +problems. The storage location stays private to the API, the server verifies the media type on +upload, and the id makes one authorization check natural: does this attachment belong to the +session being run. The full options are in decision D10. + +```mermaid +flowchart LR + subgraph store["Object store (S3)"] + orig["Original
attachments mount (session-scoped)
never changes · agent cannot reach it
mounts/<project_id>/<attachments_mount>/…"] + end + subgraph sandbox["Agent sandbox"] + cwd["Working copy
cwd/attachments/<attachment_id>/<filename>
agent may read, edit, or delete"] + end + ref["Attachment reference
{attachment_id, filename, media_type, size}
opaque id on the wire, in history, in traces"] + + ref -.addresses.-> orig + orig ==>|runner reads → base64| acp["ACP image / audio / document block
→ model perceives it"] + orig ==>|runner copies once| cwd + cwd ==>|Read / Bash / edit| tools["agent tools operate on it"] + orig ==>|list + download| ui["conversation + files panel"] + + classDef immut fill:#e8f0fe,stroke:#356,stroke-width:2px; + classDef mut fill:#fdf1e3,stroke:#a63,stroke-width:1px; + class orig immut + class cwd mut +``` + +--- + +## Decision D1: how the file reaches the model + +**The question.** How does the file's content actually reach the model so the model perceives it? + +**Options.** + +- **A. Inline content blocks.** The runner reads the file and puts its bytes into an ACP image, + audio, or document block in the turn. +- **B. File on disk plus a path in the prompt.** The runner writes the file into the working + directory and the prompt text mentions the path, trusting the harness to read it. +- **C. Both, for different purposes.** Deliver native modalities inline for perception, and also + place the file on disk for tool use. + +**What breaks under each.** Option B leaves perception to the agent's discretion and makes it +uneven across harnesses. The harnesses' own read tools can deliver a file natively when called +(Claude Code reads images and PDFs as real vision and document input; Pi and Codex read images; +none reads audio; see [research.md](research.md), sections 4 and 5), but ACP has no content type +that hands the model a link and guarantees the model reads it, so nothing forces that call to +happen, and a turn rebuilt from records never repeats it. So with B alone, a person who shares an +image and asks "what is this?" can still get an answer that ignores the image. Option A alone leaves the agent unable to run a tool over the file, +because the file never lands on disk. For audio there is no disk-read fallback at all, so B cannot +serve audio. + +**Decision.** Option C. Deliver native modalities inline so the model perceives them, and also +materialize a working copy on disk so the agent's tools can work on the file. The two are not +redundant; they serve the two separate goals of perception and tool use. + +**Why inline is unavoidable for perception (decision D2 in the log).** In ACP the bytes for an +image, audio clip, or embedded document are required and inline. Storing the file in our object +store keeps the bytes out of the durable records the runner replays on a cold start, but it does +not remove the requirement that the bytes be present at the moment the runner builds the turn. The +runner reconstructs the inline block per turn from the stored original. + +--- + +## The one runner call that changes + +The whole model-facing change is one place in the runner: the call that hands a turn to the harness. + +Today that call sends a single text block. The proposed version resolves the message's references, +reads the originals, materializes working copies, and sends a list of real content blocks. Read the +before-and-after this way: on the left, the runner flattens the message to its text and sends only +that, so the model sees text and a replayed image shows up as the string "[image]"; on the right, +the runner turns each reference into the matching content block and sends the whole list, so the +model perceives the image and the agent can also open the file. + +```mermaid +flowchart TB + subgraph now["TODAY: the file is dropped here"] + n1["wire: content = [text, image(base64)]"] --> n2["messageText() keeps only type == text"] + n2 --> n3["turnText = 'what is this?'"] + n3 --> n4["session.prompt([{type:'text', text: turnText}])"] + n4 --> n5(["model sees text only
· replay shows '[image]'
· image with no text → run rejected"]) + end + subgraph next["PROPOSED"] + p1["wire: content = [text, attachment reference]"] --> p2["resolveContentBlocks(request)"] + p2 --> p3["read original(s) → base64
copy working copy → cwd"] + p3 --> p4["session.prompt([ImageContent(data), TextContent])"] + p4 --> p5(["model perceives image + text
agent can also open the file"]) + end + now -.replace.-> next + + style n5 fill:#fde,stroke:#c33 + style p5 fill:#efe,stroke:#3a3 +``` + +--- + +## Decision D3: where the original lives + +**The question.** Where do we store the unchanging original of a shared file? + +**Options.** + +- **A. A folder inside the working-directory mount, for example `cwd/_uploads/`.** +- **B. The agent-files mount** (the durable, cross-session agent mount). +- **C. A dedicated session-scoped attachments mount that is not made visible to the agent.** +- **D. A future project-level drive.** + +**What breaks under each.** The working directory is last-writer-wins and fully under the agent's +control (see [research.md](research.md), section 2). Concretely, under option A the following breaks +the findability goal: a person shares `report.pdf`, then asks the agent to "clean up the workspace," +and the agent runs `rm -rf _uploads` or a cleanup script, or simply overwrites `report.pdf` with a +derived version. Now the original the person shared is gone, and "always findable" is false. Even +without malice, an agent that reorganizes its files can move or replace the original. Option B avoids +the deletion problem only if agent-files is also kept out of the agent's reach, but agent-files +exists precisely to be visible and writable by the agent across sessions, so using it would either +change its meaning or expose the original to the same last-writer-wins risk. Its cross-session +lifecycle is also wrong for a per-session attachment. Option D does not exist yet and would couple +this feature to unshipped work; a project drive is also the wrong scope, since an attachment belongs +to one conversation. + +**Decision.** Option C. A dedicated attachments mount, scoped to the session, that is created with +the existing `get_or_create_session_mount(session_id, name="attachments")` machinery but is +deliberately never added to the set of mounts made visible in the sandbox. It is also protected +against our own generic mount routes: it appears in no mount listing, cannot be signed, edited, or +archived on its own, and is removed only when the session is torn down (decision D7). The runner +reads an original through the session-bound API download route, never with signed credentials: +decision D7 rules that no signed credentials are ever issued for the attachments mount, because any +signed credential is read-write today. Reading the object store directly with a read-only credential +scope is the deferred Stage 3 hardening (decision D7 and [plan.md](plan.md), Stage 3). + +**Why its own mount rather than a subfolder.** The technology that makes a mount visible to the +agent (geesefs) exposes the whole mount prefix as a writable folder. There is no way to expose part +of a mount and hide the rest. So "the agent cannot reach the original" forces the original into a +prefix that is not exposed at all, which means its own mount. The attachments mount reuses the +mount machinery (create, upload, download, list) and differs from the other mounts in two ways: it +is left off the sandbox's visible set, and it is never signed (decision D7). + +**Why lifecycle alone does not decide the location.** On lifecycle alone, an attachment is session-scoped, exactly like +the working directory, so a subfolder inside `cwd` would have been the simplest choice. The +findability requirement, not the lifecycle, is what pushes the original out of the agent's reach and +therefore into its own mount. + +--- + +## Decision D4: one copy or two + +**The question.** Should there be a single copy of the file that the agent both perceives and edits, +or two copies with different rules? + +**Options.** + +- **A. One copy.** The file lives in one place. The agent reads and edits it there. Findability and + editing share the same object. +- **B. Two copies.** An unchanging original plus a disposable working copy. + +**What breaks under A.** If there is one copy and the agent edits it, the person can no longer find +what they originally shared. If there is one copy and it is read-only, the agent cannot edit it, +which breaks the "agent can work on it" goal. A single copy cannot satisfy both goals at once, +because the two goals want opposite things from the same object. The agent is not even the only +writer in the working directory: the harness itself can mutate a file there (Claude Code's Read +tool has an open report of overwriting the original image it read, +[claude-code #77729](https://github.com/anthropics/claude-code/issues/77729)), which independently +supports keeping the original out of the sandbox. + +**Decision.** Option B, two copies. The original never changes and backs findability, download, and +model perception. The working copy is what the agent touches. When the agent edits its working copy, +the result is a new file the agent produces, which shows up under the agent's own origin in the files +panel, while the original stays under "Shared by you." The person then sees both the input they gave +and the output the agent made, which is more useful than having the original silently replaced. + +**Why not make it a policy switch.** A read-only-versus-read-write policy on a single copy cannot +satisfy both requirements at once, because findability wants the file safe and tool use wants it +editable. Two copies dissolve the question. There is no global read-only-versus-read-write policy. +The original is always safe; the agent can always do anything it likes to its copy; and whether the +agent changes the file at all is just what the conversation calls for, decided per conversation +rather than by a platform setting. + +### The working-copy path and edited copies + +The working copy lives at `cwd/attachments//`. The `attachment_id` segment +is what keeps two files from colliding. Two people can each share a file named `report.pdf` in the +same session, and because each one sits under its own id-named folder, neither overwrites the other. + +Re-materialization has one rule: the runner restores a working copy only when the file is missing. +It never overwrites a working copy that is already there, because the agent may have edited it and +that edit is real work. So if a later turn arrives and the working copy is present, the runner +leaves it alone. If the working copy was deleted, the runner writes it again from the original. On +a cold start, the runner additionally sweeps the session's records and restores every referenced +working copy that is missing, so every path the rebuilt conversation mentions exists (decision +D12); the same never-overwrite rule applies to the sweep. + +This creates a divergence that the design accepts on purpose. When a later turn references the same +attachment natively, the runner always reads the **original** to build the model's content block, +while the agent's tools continue to see the **edited** working copy. So the model perceives the file +as it was shared, and the tools operate on the file as the agent has changed it. That is the +intended behavior: the original is the immutable source of truth for perception, and the working +copy is the mutable scratch object for tools. + +--- + +## The successful upload and delivery flow, end to end + +Here is the full path for the common case: a person attaches a photo and asks a question, on a warm +turn. Read it as: the session already has an id, so there is no "create the session first" step; the +front end uploads the photo once through the API's attachment create route, and the API stores the +bytes and hands back an attachment resource; the front end sends a message that carries the +`attachment_id` instead of the bytes; the runner claims the attachment for the turn so the cleanup +sweep can no longer take it, reads the original through the API download route, writes a working copy +into the agent's directory, builds the real content blocks, and calls the harness; the model +perceives the image and the agent can also open the file; and the front end renders the photo inline +by resolving the reference, not from a giant inline blob. + +The front end never touches the object store directly and never holds write credentials for the +attachments mount. The upload route is the only way bytes enter, which is what decision D7 relies on. + +```mermaid +sequenceDiagram + autonumber + participant U as User + participant FE as Chat box (front end) + participant API as API (mounts + sessions) + participant S3 as Object store + participant RUN as Runner + participant SBX as Sandbox (cwd) + participant ACP as Harness (ACP) + participant M as Model + + Note over FE: session_id already exists (front end created it) + U->>FE: attach photo.png + type "what is this?" + FE->>API: POST create attachment (multipart file + idempotency key) + API->>API: bounded read, classify bytes, get_or_create attachments mount + API->>API: insert pending row (one per session + idempotency key) + API->>S3: write original bytes + API->>API: mark the row ready (only a ready row is readable) + API-->>FE: attachment resource {attachment_id, filename, media_type, size} + FE->>API: run turn, message carries the attachment_id (no base64) + API->>RUN: /run (wire: message.content = [text, attachment reference]) + RUN->>API: claim the attachment for this turn + API->>API: check the attachment belongs to this session, set referenced_at + API-->>RUN: claimed (the sweep can no longer take it) + RUN->>API: download the original for this turn + API->>API: check the attachment belongs to this session + API->>S3: GET original + S3-->>API: bytes + API-->>RUN: bytes + RUN->>SBX: write working copy to cwd/attachments/attachment_id/photo.png + RUN->>RUN: build ACP blocks [ImageContent(data=base64), TextContent] + RUN->>ACP: session.prompt([...blocks]) + ACP->>M: image + text + M-->>ACP: answer (may open the working copy with a tool) + ACP-->>RUN: stream + RUN-->>FE: stream, rendered + Note over FE: the photo renders inline by resolving the reference through a download, not from base64 +``` + +The one change that makes the model perceive the image is at the "build ACP blocks" and +"session.prompt" steps. The working-copy write happens once per file per session, because the working +directory survives across turns, so later turns skip it. + +--- + +## The lifecycle of one attachment + +An attachment moves through a few clear states. Read the state machine as: the person picks a file; +it is rejected up front only when it is too large or an invalid file, and an unsupported kind is not +rejected but accepted as a workspace-only attachment with a notice; an accepted file is uploaded +once, after which the original is stored and unchanging; when a turn is sent, the `attachment_id` +goes with it; the runner materializes the working copy when it is missing (it never overwrites one +that is already there, because the agent may have edited it), and delivers the bytes to the model +only when the capability intersection allows; and from there the attachment is a durable, findable record +whose findability does not depend on what the agent does to its working copy. + +```mermaid +stateDiagram-v2 + [*] --> Selected: user picks / pastes / drops + Selected --> Accepted: passes size and format-validation checks + Selected --> Rejected: too large or invalid file (told before send) + Rejected --> [*] + Accepted --> Uploading: upload once through the API + Uploading --> Stored: original persisted (never changes) + Uploading --> UploadFailed: retry or surface error + UploadFailed --> Uploading: retry + UploadFailed --> [*]: user removes + + Stored --> Referenced: turn sent with the attachment_id + Referenced --> Materialized: runner materializes the working copy when missing (never overwrites) + Referenced --> Perceived: native delivery when the capability intersection allows + Referenced --> WorkspaceOnly: intersection does not allow, delivered as a workspace-only attachment with a notice + + Perceived --> Findable + WorkspaceOnly --> Findable + Materialized --> AgentOperates: read / convert / edit / delete + AgentOperates --> Findable + + state Findable { + [*] --> Listed + Listed --> Listed: renders inline · listed in panel · downloadable + } + note right of Findable + The original backs findability. + The agent deleting its working copy + does not change this. + end note + Findable --> [*]: session deleted (originals share the session lifecycle) +``` + +--- + +## The media-type, validation, and limits matrix + +This section settles the former open question on media types, validation, and limits. The rules are +engineering defaults derived from the shipped client limits, the gateway ceiling, and the providers' +inline caps, so they are recorded under "Settled by evidence" in [decisions.md](decisions.md) rather +than as a product decision. The upload route enforces the matrix server-side; the shipped client +limits (`DEFAULT_ATTACHMENT_LIMITS` in +`web/oss/src/components/AgentChatSlice/assets/attachments.ts`, lines 49 to 59, verified 2026-07-31) +stay as they are and become the courtesy layer a client can bypass but the server does not. + +### How the server classifies a file + +The server inspects the file's leading bytes (magic bytes) and stores the type it verified, per +decision D10. The declared type is display metadata at most; it never decides anything. + +- Bytes that match a known signature are stored under the inspected type. When the inspected type + differs from the declared one, the inspected type wins silently: the upload succeeds and the + attachment resource carries the verified type. The declared type is never trusted. +- Bytes with no known signature that decode as UTF-8 text are stored as text. The declared type is + kept when it is a `text/*` subtype or `application/json`, because bytes cannot distinguish + Markdown from CSV; otherwise the stored type is `text/plain`. +- Bytes with no known signature that are not valid text, and empty files, are rejected at upload + with a structured error. This is the "invalid file" rejection in the lifecycle above, and it is + the only content-based rejection. A recognized type that is not a native kind is never rejected + (decision D6); it is stored and attached workspace-only. + +### The matrix + +Size caps apply to the raw file at upload; base64 inflation matters only at delivery, because the +upload is a multipart request, not base64. The count cap applies across all kinds, and the runner +enforces it over the current user turn, before it downloads anything: each upload is its own request, +so the create route cannot count a turn. The number is runner configuration, not API configuration, +and the composer's shipped cap is the courtesy layer in front of it. The per-kind caps mirror the +shipped client caps (`attachments.ts`, lines 49 to 59). + +| Kind (by inspected type) | Accepted formats | Per-file cap | Per-turn count | At upload | At delivery | +| --- | --- | --- | --- | --- | --- | +| image | PNG, JPEG, GIF, WebP | 10 MB | 100 across all kinds | over the cap: rejected with a structured error | native when the D5 intersection allows; workspace-only with the D6 notice when the original exceeds a provider inline cap (rule below) | +| audio | recognized audio containers (WAV, MP3, OGG, WebM, FLAC, M4A) | 15 MB | shared 100 | same | always workspace-only in the first release (decision D14); native ACP audio is the last upgrade | +| document | PDF; text (`text/*`); JSON | 10 MB | shared 100 | same | workspace-only today (native documents are the Stage 2 adapter blocker); small text can inline as text | +| other (recognized, not a native kind) | any other recognized type (archives, office formats, video, and so on) | 10 MB | shared 100 | same | always workspace-only with the D6 notice | +| unrecognizable | no known signature and not valid text | none stored | none | rejected as an invalid file | never stored | + +The per-turn count is not the only bound. Each session also has a quota the create route enforces: +1,000 stored attachments, 256 MB of stored originals, and at most 20 uploads in flight at once. A +per-turn count bounds a polite client and not a hostile one, and a cold start restores every +referenced working copy (decision D12), so an unbounded session costs stored bytes nobody asked for +and turns the runner into an amplifier on every cold start. The session, not the single turn, is +where the real bound belongs; the per-turn count only shapes one prompt. + +One shipped behavior changes to match this matrix: the composer today rejects an unknown kind +outright ("isn't a supported file type", `attachments.ts`, line 131). Under decision D6 an +unsupported kind is accepted workspace-only with a notice, so Stage 1 replaces that client-side +rejection with the notice. The recognized-format list for the "other" row is whatever the +server's magic-bytes library recognizes; widening it is cheap and needs no design change. + +### The gateway ceiling + +Stage 1 raises the compose gateway request-body cap from 10 MB to 32 MB: `client_max_body_size 10M` +becomes `32m` in `hosting/docker-compose/oss/nginx/nginx.conf` (line 35, verified 2026-07-31). That +file is the only nginx config in the compose stack: the OSS gh compose files mount it behind the +`with-nginx` profile (`docker-compose.gh.yml`, `docker-compose.gh.local.yml`), and the dev stacks +and the EE compose files run no nginx of their own. The railway gateway already allows 32 MB +(`client_max_body_size 32m` in `hosting/railway/oss/gateway/nginx.conf`, line 23, verified +2026-07-31). + +The ceiling is infrastructure, not policy: the upload route's per-kind caps are the enforced truth. +Each upload carries one file, so the ceiling binds per file, and every per-kind cap (15 MB at most) +clears 32 MB with room for multipart overhead. The old 10 MB ceiling is what made the shipped 15 MB +audio cap impossible to use; raising the ceiling resolves that in favor of the per-kind caps. + +The ceiling is also not everywhere, which is why the route cannot lean on it. It exists only behind +the `with-nginx` profile, and an EE deployment has no body cap in front of the API at all. So the +create route bounds its own read: it consumes the request body in chunks, stops at the largest +per-kind cap plus one byte, and refuses what is over, all before it classifies anything. Reading the +whole body first and checking its size afterwards would make an arbitrary upload an API memory cost +on every deployment that runs no gateway; with the bounded read, a missing ceiling costs one byte of +over-read. + +### Provider inline caps at delivery + +The runner rebuilds inline blocks from stored originals (decision D2), so an accepted upload can +still exceed what a provider takes inline. The current numbers, all verified 2026-07-31: + +- **Anthropic Messages API, images.** 10 MB per image, measured on the base64-encoded data, on the + Claude API directly; 5 MB base64 on Amazon Bedrock and Google Cloud. Maximum dimensions 8000x8000 + px, with a stricter limit near 2000 px per side when a request carries more than 20 images. 100 + images per request on 200k-context models, 600 otherwise. Formats: JPEG, PNG, GIF, WebP. Whole + request capped at 32 MB. + [Anthropic vision docs](https://platform.claude.com/docs/en/build-with-claude/vision). +- **Anthropic Messages API, PDF.** 32 MB maximum request size; 600 pages per request, dropping to + 100 pages when the request's context window is under 1M tokens; standard unencrypted PDF. + [Anthropic PDF support docs](https://platform.claude.com/docs/en/build-with-claude/pdf-support). +- **OpenAI vision.** 512 MB total payload per request and 1500 image inputs per request, with no + documented per-image cap. Formats: PNG, JPEG, WebP, non-animated GIF. + [OpenAI images and vision guide](https://developers.openai.com/api/docs/guides/images-vision). +- **Gemini.** Inline data shares a 20 MB total request budget (text plus bytes); larger files go + through the Files API. Up to 3,600 images per request. Formats: PNG, JPEG, WebP, HEIC, HEIF. + [Gemini image understanding](https://ai.google.dev/gemini-api/docs/image-understanding). + +The 5-per-turn count sits far under every provider count cap, so count never binds at delivery. + +### The delivery rule for oversized originals + +When a stored original passes the upload caps but exceeds a constraint of the provider actually +selected (per-image bytes after base64 inflation, the request budget, or dimensions), the runner +does not fail the turn and does not resize. It attaches the file workspace-only with the D6 notice, +exactly as it does for a kind the model cannot perceive. Concretely on Claude: base64 inflates by +about a third, so an accepted image over roughly 7.5 MB raw exceeds the 10 MB base64 per-image cap +and goes workspace-only; the agent can still read it from disk (and on Claude the Read tool +delivers it as real vision input, [research.md](research.md), section 4). Runner-side downscaling +to fit a provider cap is a possible later refinement, listed in [scope.md](scope.md) follow-ups; it +is machinery this release does not build. + +### Sanity check + +Every per-file cap sits under the 32 MB gateway ceiling. The shipped caps stay as shipped (10 MB +images, 15 MB audio, 10 MB documents): no provider cap forces them down, because the delivery +boundary is handled by the workspace-only rule above, not by shrinking uploads. + +--- + +## Decision D5: the layered capability model + +**The question.** How does the system know whether a given modality will actually reach the model? + +**The layered answer.** Whether a modality is perceived natively is the intersection of three +layers. All three must allow it, and the effective capability is the weakest of the three: + +1. **What the ACP transport supports.** The prompt capability flag (`image`, `audio`, + `embeddedContext`) that the adapter advertises. Without the flag the content type cannot even be + carried across the protocol. +2. **What the harness adapter actually delivers natively.** Adapter fidelity. A flag can be + advertised while the delivery is lossy or dropped. This is where the two adapters we run diverge + sharply (see [research.md](research.md), section 4): both pass an image through as a real image, + but the Claude adapter drops a blob resource entirely and the Pi adapter renders it as a byte + count. So a document can clear the transport layer and still never reach the model. +3. **What the selected model perceives.** Model modalities, carried on the run request by the + resolved connection, which is where the model and its provider are already settled. A non-vision + model does not see an image even when the transport and the adapter would carry it. This layer + has a defined default: when the request carries no modalities, or none we recognize, the layer + reads unknown, and unknown means workspace-only with the D6 notice. Silence never means vision, + because an assumed capability fails the way the current bug fails, silently. + +Tool use over the file is a separate, fourth consideration, not part of this intersection. It needs +only the working copy on disk and works regardless of all three layers above. An agent whose model +cannot see an image can still run a program over the image bytes. This is why the design places the +file on disk for every attachment, independent of native perception. + +**Decision on gating (D5).** Compute the intersection and gate in two places. The composer gates for +the person's benefit, so the person learns before sending whether a file will be perceived. The +runner gates as the final authority, because the composer's view can be stale. Both are needed; the +composer gate is a courtesy and the runner gate is the truth. + +Here is the layered capability, per modality, as it stands today. + +| Modality | ACP transport (prompt capability) | Adapter fidelity today | Model modalities | Tool use over the working copy | +| --- | --- | --- | --- | --- | +| Image | `image`, advertised by both adapters | native image on both adapters | needs a vision model | always available | +| Audio | `audio`, advertised by neither adapter | no native audio path on either adapter | needs an audio model | always available | +| Document (PDF and similar) | `embeddedContext`, off by default on Pi | Claude drops the blob, Pi renders a byte count | depends on the model and the format | always available | + +The perception columns describe the guaranteed inline path only; on Claude, the working copy alone +already yields image and PDF perception through the harness's Read tool when the agent chooses to +read it (see [research.md](research.md), section 4). For audio, the first release sidesteps the +native path entirely: the composer's browser dictation turns live speech into text, and a recorded +clip is a workspace-only attachment with the D6 notice (decision D14). + +### Two integration gaps to close + +Two facts about the current code mean the layered model needs plumbing that does not exist yet, and +these are stated as work, not as afterthoughts. + +- **Capabilities are surfaced only after a run.** Today the runner returns its capabilities in the + run response, after a run has happened (`protocol.ts`, near line 539). The composer has no way to + discover them before the person sends anything. So the composer needs a pre-send discovery + surface. A static approximation derived from the model and harness catalog is acceptable for the + first release, with the runner remaining the final authority at prompt-build time. +- **The delivery outcome has nothing to travel on.** Today the run error field is a plain string + (`error?: string`, `protocol.ts`, near line 557), and a per-attachment outcome has no carrier at + all. So the runner emits an `attachment_delivery` event for every attachment in the turn, carrying + the attachment id, the outcome (native, workspace-only, or failed), a stable reason code, and the + working-copy path. The event is persisted with the session's records, parsed by the SDK in + `wire.py` beside the run result, and carried to the browser on the Vercel stream as a data part, so + the front end renders the D6 notice from what actually happened and the notice comes back on a + reload. A single error code on the terminal result would not do the job: it says nothing when the + turn succeeded and one attachment was flattened, and it cannot name which of three files that was. + +### Renaming capability fields without a simultaneous deploy + +The internal capability flags need new names (`images`, `audio`, `documents`) and the old ones +(`fileAttachments`, `file_attachments`) need to retire. Do **not** do this as a rename that must land +in every component at the same time. The front end, the API, the SDK, and the runner deploy +independently, so a rename that lands +in one component before another breaks the versions in between. Instead, introduce the new fields +alongside the old ones, keep the old names accepted as aliases through the rollout, and remove the +old names later once every component speaks the new ones. The removal timing follows the alias +rollout's settle condition in [plan.md](plan.md), Stage 2. + +--- + +## Decision D6: what does attaching a file promise, and what happens on an unsupported kind + +**The question.** When a person attaches a file the model cannot perceive, what should happen? +Failing the turn on every unsupported kind would contradict the tool-use goal: an agent can still +process a file its model cannot perceive natively. The resolution is to separate the two meanings of +attach. + +**The resolution: separate the two meanings of attach.** "Attach" means two different things, and +untangling them removes the contradiction: + +- **Show this to the model.** Deliver the file as a native content block the model perceives. +- **Put this file in the agent's workspace.** Materialize it on disk so the agent's tools can open + it. + +**The design's answer.** Attaching a file **always** puts it in the workspace (it is materialized +for tools). Native delivery to the model happens **when the capability intersection from D5 allows +it**. When the intersection does not allow it, the composer says so up front, and the turn still +proceeds with the file as a workspace-only attachment. That is a visible notice to the person, not a +silent drop and not a failed turn. The runner fails the turn only when it is explicitly asked to +deliver a native block the harness cannot accept, which is a contract violation rather than an +ordinary unsupported kind (for example, a stale front end that asks for native audio the adapter +never advertised). So a silent drop never happens, and a hard failure happens only on a real +protocol contract violation. + +**The fuller model, and the first-release simplification.** The richer version of this would attach +a per-attachment intent to each file: *perception required* (fail if the model cannot perceive it), +*perception preferred* (deliver natively when possible, otherwise workspace-only), or *workspace-only* +(never try native delivery). The first release does not build that per-attachment control. It treats +every native kind as *preferred* and offers no per-attachment toggle in the UI. This is the simpler +model that still avoids both the silent drop and the surprise failure. + +**What the first release promises is decided.** The first release promises durable agent input: +the workspace copy, findability, and records, not image perception alone. That is decision D11, +taken by the product owner on 2026-07-31 and laid out below. + +Read the gating diagram as: the harness advertises capabilities at start-up; the runner maps them +and also knows the adapter fidelity; a pre-send discovery surface gives the composer an +approximation; the composer attaches every file, delivering natively where the intersection allows +and attaching workspace-only with a visible notice where it does not; and the runner, as final +authority, materializes the working copy when it is missing (never overwriting an edited one) and +fails the turn only when asked to deliver a native block the harness cannot accept. + +```mermaid +flowchart TD + A["Harness start-up (ACP)"] --> B["prompt capability flags
{ image, audio, embeddedContext }"] + B --> C["runner maps to { images, audio, documents }
and knows the adapter fidelity"] + C --> D["pre-send discovery surface
(static catalog approximation today,
runner is final authority)"] + + subgraph fe["Composer (front end)"] + D --> E{native intersection allows this kind?} + E -->|yes| F["attach and mark for native delivery"] + E -->|no| G["attach as workspace-only
with a visible notice
(not refused, not silently dropped)"] + end + + subgraph run["Runner (final authority)"] + F --> H["materialize the working copy when missing"] + G --> H + H --> I{asked to deliver a native block
the harness cannot accept?} + I -->|no| J["deliver native where allowed,
workspace-only otherwise"] + I -->|yes, contract violation| K["fail the turn with a
structured error code"] + end + + style G fill:#fef6e0,stroke:#a83 + style K fill:#fde,stroke:#c33 + style J fill:#efe,stroke:#3a3 +``` + +Note the three distinct capabilities: an image maps to `image`, audio maps to `audio`, but a PDF or +document maps to `embeddedContext`, a different flag. Do not treat "supports images" as "supports +documents." + +--- + +## Decision D11: what does attaching a file promise in the first release? + +**The question.** What does the first release promise a person who attaches a file? The engineering +design supports either answer; the choice is about what to promise first. + +**Options.** + +- **A. Image perception only.** The first release lets the model see images, with strict size and + count limits and no durability promise. The file is not guaranteed to remain findable after the + turn. This is the smallest honest fix for the original bug. +- **B. Durable agent input.** The first release delivers the full model here: an original that + never changes, rendered inline in the conversation and downloadable at any time (the + Files-drawer listing follows in Stage 3); a working copy for the agent's tools; and a reference + stored in the records. Attaching a file always puts it in the workspace and delivers it natively + when the capability intersection allows. + +**What breaks under A.** The three outcomes in [context.md](context.md) do not hold: the file is +not findable and the agent's tools cannot rely on it. And the durability that A skips is not a free +deferral: adopting durable references later is a system-wide change (decision D9), so A ships a +version that needs rework. + +**Decision.** Option B, durable agent input, from day one, with the workspace-always plus +native-when-possible semantics described in D6. The product owner chose the full reference-based +design for the first release on 2026-07-31; no inline-only version will be built. + +--- + +## Decision D13: two upload surfaces, deliberately different pipelines + +**The question.** The product has two places a person can hand over a file: the Files drawer (the +drive view over the session's mounts) and the chat composer. Do both go through the attachment +pipeline? + +**Options.** + +- **A. Route everything through attachments.** Every upload, from the drawer or the composer, + creates an attachment resource with an immutable original and an opaque reference. +- **B. Split by surface.** A file dragged into the Files drawer (or one of its folders) uploads + directly to that mount at that path, the existing drive-upload flow. The attachment pipeline + applies only to files shared through the chat composer, whether typed, pasted, dropped, or picked + there. + +**What breaks under A.** A drawer drop is the person saying "put this file there." Routing it +through attachments gives that plain folder write the wrong semantics: the file would become an +immutable, session-scoped original addressed by an opaque id, when the person asked for an ordinary +file at a path they chose, editable and deletable like its neighbors. It would also replace a flow +that already exists and works: the drive-upload surfaces (the upload button, drop-to-upload in the +Files drawer, drop-to-stage on a recents peek) merged in PR #5459 behind +`NEXT_PUBLIC_AGENT_FILE_UPLOADS`. + +**Decision.** Option B. The two surfaces express two different intents: "put this file there" +versus "share this with the agent in the conversation." The drawer keeps the direct drive-upload +flow; the attachment pipeline (attachment resource, immutable original, opaque reference) covers +only files shared through the composer. + +--- + +## Decision D7: enforcing that the original does not change + +**The question.** The original must not change. What actually enforces that? + +**Options.** + +- **A. Keep the mount out of the sandbox.** The agent never sees it, so the agent cannot change it. +- **B. Also treat the mount as protected: every generic mount operation behaves as though it does + not exist, no signed credentials are ever issued for it, and the create route is the only writer.** +- **C. Also add a read-only credential scope so the runner could read the object store directly and + even a leaked credential could not write.** + +**What each gives.** Option A stops the agent, which is the main threat, but it does not stop a bug +elsewhere in our own code that holds credentials. Option B closes the credential path. Recall from +[research.md](research.md), section 2, that any signed credential is read-write today: the signing +call has no read-only variant, so signing a mount at all hands out write access. So immutability +cannot rest on signed credentials. Recall too that `write_file` overwrites silently, so create-only +is not something the storage layer gives for free. Option B is therefore broader than one refusal. +The attachments mount is classified as protected, and every generic mount operation behaves as though +it does not exist: query, fetch, list, read, download, export, edit, archive, unarchive, write, +folder-create, upload, delete, and sign all answer the way they answer for a mount that is not there. +The classification is a server-owned `purpose` field on the mount, set when the create route makes it +and absent from the edit model, so it is a property of the row rather than a naming convention +someone can imitate or rename. The single writer is the create route, which reaches storage through +one narrow operation that opts out of the policy. Session teardown keeps seeing the mount, because it +owns the lifecycle (see [decisions.md](decisions.md), "Settled by evidence"). Guarding only the +writes would leave the reads open: a project member could otherwise list the mount, read an original +by path, or export it, and so bypass the session-binding check that the download route exists to +make. The refusal to overwrite stays an explicit API-level check, not an assumption, precisely +because the storage layer would otherwise overwrite without complaint. Option C is the strongest and is also the path that would later let the runner read the object store +directly, but it needs a read-only variant added to the signing call, which does not exist today. + +**Decision.** Start with A and B together. Originals enter only through the create-only upload route, +which is the only writer that the protected-mount policy lets through. Every other mount operation, +by id or by listing, answers as though the mount is not there, and the classification lives in the +mount's server-owned `purpose` field. The API refuses overwrite and delete for attachment originals, +enforced as an explicit check because `write_file` overwrites silently today. No signed credentials +are ever issued for the attachments mount, because today any signed credential is read-write. A +read-only credential scope is the later hardening that would let the runner read the object store +directly; it is noted in +[plan.md](plan.md). Together A and B give real, enforced immutability for the case that matters, +without blocking on the signing change. + +--- + +## Modality to content block mapping + +This shows how each kind of attachment becomes a content block on each side. Read it as: the person's +file has a media type; our front end and SDK turn it into an Agenta content block; and the runner +turns that into the matching ACP block that the harness understands. Small text does not need to be a +resource at all; it can be inlined as text, which every harness supports. Audio stays a product goal +(decision D8 in the log), and its target remains the `audio` block on our side mapped to the ACP +`AudioContent` block, but that native path is the last upgrade: in the first release the only +audio-to-text is the composer's browser dictation, a recorded clip travels as a normal attachment on +the D6 workspace-only path, and server-side transcription is the deferred middle step (decision D14, +below). The diagram shows the target shape. Neither pinned adapter +delivers native audio or native documents today (see [research.md](research.md), section 4), so the +audio and document rows describe where these paths lead once the adapters support them, not what +works right now. + +```mermaid +flowchart LR + subgraph in["User attaches"] + img["image/*"] + aud["audio/*"] + pdf["application/pdf"] + txt["text/* · application/json"] + end + subgraph agenta["Agenta content block (add 'audio')"] + aimg["image"] + aaud["audio ⟵ NEW"] + ares["resource"] + end + subgraph acp["ACP content block (external)"] + cimg["ImageContent(data)"] + caud["AudioContent(data)"] + cemb["EmbeddedResource(blob)"] + ctxt["TextContent"] + end + + img --> aimg --> cimg + aud --> aaud --> caud + pdf --> ares --> cemb + txt -->|inline as text| ctxt + + cimg --> gate1["gated on prompt capability image"] + caud --> gate2["gated on prompt capability audio"] + cemb --> gate3["gated on prompt capability embeddedContext"] +``` + +A hard caution carried from [research.md](research.md), section 4: the document path in this diagram +does not work today. The Claude adapter drops a blob resource entirely, and the Pi adapter renders +it as a byte count, so a PDF sent as an embedded resource reaches neither model as a document. The +diagram shows the intended shape, and the document stage is blocked on adapter work (either +adapter-native document handling, or a decision to deliver documents as extracted text). Images map +cleanly and work end to end. Audio has no native path on either adapter yet, so its row is likewise +a target, not a working path; what the first release does for audio is decision D14, below. + +--- + +## Decision D14: first-release audio is browser dictation only + +**The question.** Audio is a product goal (D8) and the voice-capture UI is already built and merged +dark (PR #5458). But native audio delivery is infeasible today at two independent layers: no adapter +advertises the ACP `audio` capability, and the Anthropic Messages API has no audio input block at +all ([research.md](research.md), section 4). What does the first release do with speech and audio +files? + +**Options.** + +- **A. Browser dictation only.** The composer's dictation mode (the Web Speech API in + `web/oss/src/components/AgentChatSlice/hooks/useVoiceInput.ts`, merged dark in PR #5458) turns + live speech into text in the composer while the person talks. No audio file is involved. Recorded + voice messages and uploaded audio files get no transcription and follow the D6 workspace-only + path. +- **B. Transcription through the person's stored provider keys.** The API transcribes the recording + through litellm's `/audio/transcriptions` support with a key the person already stored, and the + transcript reaches the model inline as text. +- **C. Transcription on an Agenta platform key.** The same call path as B, but Agenta supplies the + key and meters the usage, so it works for people with no stored key. +- **D. Transcription in the services layer.** A whisper model Agenta runs (for example + faster-whisper) transcribes the working copy, with no external provider involved. + +**Decision.** Option A, and that is all for the first release. Live dictation is the only +audio-to-text: speech becomes text in the composer as the person talks, and nothing new is built. +A recorded voice message or an uploaded audio file uploads as a normal attachment through the whole +D11 pipeline (immutable original, workspace copy, referenced in the records) and follows the D6 +workspace-only path: the composer shows the visible notice that the model will not hear it, and the +agent's tools can still process the file on disk. Options B, C, and D are deferred together, out of +scope for the first release: no transcription endpoint, no key resolver, no platform key, no +metering. The survey behind that deferred decision, including why B and C share one litellm call +path Agenta already runs, is recorded in [research.md](research.md), section 4 ("Server-side +transcription, surveyed and deferred"). Native ACP audio delivery remains the last upgrade; the +modality mapping above keeps `AudioContent` as its target. + +**Why.** The recorded-audio limit is structural, not a postponement of easy work: the Web Speech +API transcribes only a live microphone stream, so the browser cannot transcribe a recorded blob +(Chrome 139 can replay a blob through recognition, but only in Chrome, only at real-time speed, and +it produces no server-side transcript; [research.md](research.md), section 4). Crossing that line +means a server-side service with key resolution and metering, and the decision is to add none of +that machinery now. Dictation covers everyday speech-to-text with zero new build, and the D6 notice +keeps recorded audio honest instead of silently unheard. The accepted cost is that a recorded clip +reaches the model neither as sound nor as words until server-side transcription ships; the agent's +tools remain the only way to work on it. + +--- + +## Where the person finds a shared file + +Findability shows up in three places, all backed by the unchanging original. Read the diagram as: the +original backs a reference saved in the conversation record; that record renders the file inline where +it was sent; the same original is listed in the files panel under a new "Shared by you" origin and can +be downloaded as exact bytes; and separately, when the agent edits its working copy, that produces a +new file under the agent's own origin, so the person sees both what they gave and what the agent made. +Of these surfaces, the inline render in the conversation and the download are part of the first +release; the Files-drawer listing under "Shared by you" is Stage 3 work. + +```mermaid +flowchart TD + orig["Original (never changes)
attachments mount"] --> ref["reference saved in the record"] + ref --> S1 + orig --> S2 + orig --> S3 + + subgraph surfaces["Where the person finds it"] + S1["Inline in the conversation
renders where it was sent
(reference resolves to the original, not base64)"] + S2["Files panel, 'Shared by you' origin
a new origin beside
'session' and 'agent-files'"] + S3["Download
exact bytes, any time
(download endpoint)"] + end + + ao["Agent edits its working copy"] --> art["New file in cwd
shows under the agent's origin"] + art -.different from.-> S2 + note["The person sees both:
what they gave (Shared by you)
and what the agent made (agent origin)"] + S2 -.-> note + art -.-> note +``` + +--- + +## Why the first upload can safely create the mount + +A person might attach a file before the very first message, so there is a fair question about whether +the attachments mount exists yet. It does, because the front end owns the session id before the first +message and the upload route's get-or-create is idempotent. Read the diagram as: the front end +already holds a stable session id when the pane opens; the first upload targets the attachments +mount, and the upload route creates that mount if it is not there yet (an idempotent get-or-create +keyed on the session id); so the mount is ready on the first upload, with no separate "create the +session" step that could run out of order against it. + +```mermaid +sequenceDiagram + autonumber + participant FE + participant API + Note over FE: session_id created when the pane opens (front end owns it) + FE->>API: upload (multipart file) to the session attachments mount + API->>API: get_or_create attachments mount (idempotent on the session id), then write + API-->>FE: attachment resource returned + Note over FE,API: no separate create-session step that could run out of order. The mount is created on the first upload for this session_id +``` + +A file uploaded but never sent leaves an unused stored object. That is a cleanup concern, not an +ordering one, and Stage 1 handles it with two periodic passes over the session's attachments. The +sweep deletes the originals that no claim operation ever marked, once they are older than their +time-to-live; the reaper clears rows that never reached the ready state, along with any object their +interrupted upload had already written. Reference counting against the conversation records is the +later refinement, and it waits on the records carrying references reliably (see +[decisions.md](decisions.md), open questions). + +--- + +## Decision D9: can the inline-only version grow into the full version + +**The question.** If we shipped the inline-only version first, could we grow it into this full +design later, or would we have to rebuild? + +**The inline-only version.** Deliver the file inline to the model only, with no durable storage: the +runner reads the bytes off the wire and builds the content block, exactly at the same seam. This +fixes model perception and nothing else. It does not fix findability, the working copy, the storage +bloat, or the loss of the attachment on a cold start. + +**Does it grow.** Partly, and it is worth being honest about which part. The one seam that both +versions share is the model-facing prompt-builder: both replace the single hard-coded text block +with a resolved list of content blocks at the same call. Shipping the inline-only version first does not +force a rewrite of that seam later. That is the real benefit, and it is a narrow one. + +What the inline-only version does **not** save is the rest of the work. Adopting durable references later +is still a system-wide change that touches seven places: front-end persistence (saving a reference +instead of bytes), API storage ownership (the attachment resource, the upload route, the download +route), SDK wire types (the reference form of the content block), runner resolution (turning an id +into bytes), the record schema (a field to hold the reference), authorization (the session-binding +check), and cleanup (removing never-referenced uploads). None of that is avoided by shipping thin +first. So the accurate claim is that the inline-only version avoids rework at the prompt-builder seam, not +that it is a smaller version of a migration that the full version merely extends. + +**The one thing not to defer.** The reference-on-the-wire change should not be skipped for long. +The resend of the whole history, whose cost could grow quadratically in the worst case, is already +gone, independently of this design: since the session-storage rework (PR #5560) the front end sends +only the trailing message and the runner rebuilds prior turns from durable records. But the records store only text today, so an inline-only attachment +cannot survive a cold start: either the record schema stores the bytes verbatim, and every cold +start replays them and the log keeps them forever, or the rebuilt conversation loses the file. A +durable reference resolves that, and it also removes the base64 payload from the browser's saved +messages and from the traces. That is why the plan puts the storage and reference work in the +first release. + +--- + +## Decision D10: what the reference contains + +**The question.** What actually travels on the wire to name a shared file: its storage location, or +an opaque id? + +**Options.** + +- **A. Raw storage coordinates.** The reference carries the mount id, the object path, and the + client's own media type, filename, and size. The runner reads the object at those coordinates. +- **B. An opaque, server-issued id.** The upload returns an attachment resource, + `{attachment_id, filename, media_type, size}`, where the API owns the storage location and the + authoritative metadata. The wire and the records carry the `attachment_id` plus display metadata. + The runner resolves the id through the API, which enforces that the attachment belongs to the + session being run. + +**What breaks under A.** Two things. First, forged references: a client that knows or guesses +another session's storage coordinates can name them and read files that are not its own, because the +coordinates are the authority. Second, lying metadata: the client-supplied media type becomes +authoritative even though a client can send any media type it likes, so the server would trust a +value it never verified. + +**Decision.** Option B, the opaque server-issued id. The storage location stays private to the API, +so there is nothing for a client to forge. The server verifies the media type on upload, so the +authoritative metadata is the server's, not the client's. And the id makes the key authorization +check natural to express: does this attachment belong to this session. The runner never sees storage +coordinates at all; it hands the id to the API and gets back bytes only when the binding checks out. + +**Reason.** An opaque id is the difference between "trust what the client says about where the file +is and what it is" and "let the server own both." The second is the only one that is safe when the +wire is something a client can write. + +--- + +## Decision D12: what a cold replay delivers + +**The question.** On a cold start the runner rebuilds the conversation from the durable records. +How does a historical attachment appear in that rebuilt conversation? + +**Options.** + +- **A. Re-deliver historical attachments inline, within a size budget.** The rebuilt conversation + carries a native content block for each past attachment, capped by a count and byte budget so a + long conversation stays under the provider's per-request limits. Past the budget, a placeholder + represents the file. +- **B. Mention with the real path, and reserve inline delivery for the current turn.** The rebuilt + conversation represents each past attachment as a textual mention carrying the real filename and + the working-copy path (`cwd/attachments//`). At cold start the runner + re-materializes the working copy of every attachment referenced in the session's records, so + every mentioned path exists. Inline native delivery happens only for attachments referenced by + the turn actually being run, the same rule a warm turn follows. + +**What breaks under each.** Option A pays the token cost of re-delivering every historical image on +every cold start, a cost that grows with the length of the conversation, and it still needs a +budget against the provider's per-request size and block limits (see [research.md](research.md), +section 3). Past that budget the guarantee disappears anyway, so A guarantees perception only for +the attachments that fit, and the budget numbers become one more thing to pick, measure, and +defend. Option B gives up guaranteed perception of historical attachments: whether the model sees a +past image again depends on the agent choosing to read the mentioned file. + +**Decision.** Option B, mention-first. The mention carries the real filename and the real +working-copy path, the cold start restores every referenced working copy so the mentioned paths +exist, and inline native delivery is reserved for the attachments referenced by the turn being run. +The file the person is actively working with stays guaranteed-perceived; the history becomes files +the agent can reach. + +**Why.** The harnesses' read tools verifiably deliver an image from disk on all three harnesses, +and a PDF on Claude (see [research.md](research.md), section 4), so a mentioned file is one tool +call away whenever the conversation needs it again. Mention-first removes the recurring token cost +of re-delivering every historical image on every cold start, and it dissolves the budget problem +rather than solving it: nothing inline grows with the length of the history, so there is no budget +to pick. The accepted cost is that historical-image perception becomes agent-discretionary instead +of guaranteed. On Pi, documents lose nothing, because inline PDFs never worked there anyway; the +adapter renders a blob as a byte count (section 4). + +**A residual corner, accepted.** Only the cold start sweeps the records and restores working +copies. On a warm turn, a prose-only question that arrives after the agent deleted a working copy +still finds the file missing, because a prose-only turn carries no reference that would trigger the +restore-when-missing rule. This is accepted for now; revisit with evidence if it bites. + +--- + +## Decision log and open questions + +The compact decision log (D1 through D14) and the open questions are in +[decisions.md](decisions.md), so they can be updated without editing this design narrative. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/plan.md b/docs/design/agent-workflows/projects/agent-multi-modality/plan.md new file mode 100644 index 0000000000..37ee826af9 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/plan.md @@ -0,0 +1,457 @@ +# Plan + +This file turns the design into staged work. Each stage says what changes in each layer, who owns +each part, and how it is tested. Stage 1 deliberately packages storage and model perception into one +user-visible release, because shipping storage without perception would recreate the original bug in +a new form; the reason is spelled out in that stage. + +The layers, and who owns each: + +- **Front end** (`web/oss/src/components/AgentChatSlice/`, `web/packages/agenta-playground/`, + `web/oss/src/components/Drives/`): collects the file, gates by capability, uploads it, sends the + reference, and renders it. The collection and rendering surface is already merged dark (see "What + is already built" below); the remaining front-end work is transport and reference wiring. +- **API** (`api/oss/src/core/mounts/`, `api/oss/src/apis/fastapi/mounts/` and `.../sessions/`): + stores the bytes, serves them, and enforces permissions. +- **Runner** (`services/runner/src/`): reads the original, writes the working copy, builds the content + blocks, and gates on capability. +- **SDK** (`sdks/python/agenta/sdk/agents/`): carries the reference and the new `audio` block, and + maps capabilities. + +Stage 1 implements decision D11: durable agent input from day one (see +[decisions.md](decisions.md)). No inline-only version will be built, so the storage, reference, +record-schema, and findability work below is first-release work, not a later addition. + +## What is already built (merged dark) + +The front-end surface of this design shipped ahead of the backend, in PRs +[#5458](https://github.com/Agenta-AI/agenta/pull/5458) (voice input) and +[#5459](https://github.com/Agenta-AI/agenta/pull/5459) (attachments and drive uploads), both merged +2026-07-29. It is dark behind two flags, `NEXT_PUBLIC_AGENT_FILE_UPLOADS` (the composer attach +button, the attachment preview, and every drive-upload entry point) and +`NEXT_PUBLIC_AGENT_VOICE_INPUT` (dictation and voice messages), both off by default. All paths +verified against the code on 2026-07-31: + +- The composer attach UI and attachment chips (`AgentConversation.tsx`, `ComposerAttachments.tsx`). +- The limits object: `DEFAULT_ATTACHMENT_LIMITS` in + `web/oss/src/components/AgentChatSlice/assets/attachments.ts` allows 100 files per turn, 10 MB + per image, 15 MB per audio clip, and 10 MB per document, with per-kind media-type validation. +- The upload lifecycle hook `useAttachmentUploads`, with a deliberate transport seam: no uploader + is passed today, so files stage as ready-to-send; providing an uploader runs the whole + progress, failure, and retry flow with no other change. +- Attachment preview (the `AttachmentViewerDrawer`) and the Files drawer upload surfaces (upload + button, drop-to-upload, drop-to-stage on a recents peek) in `Drives/DriveExplorer.tsx`. +- The voice capture UI: dictation (`useVoiceInput`, the Web Speech API turning live speech into + composer text) and voice messages (`useAudioRecorder`, `VoiceInputButton`, `RecordingBar`), which + land a recording in the attachment tray like any file. + +Two consequences shape the stages below. First, Stage 1 front-end work shrinks to wiring: the +collection UI exists and only its transport and rendering change. Second, paste and drag-to-attach +on the composer predated the flags and ran ungated, which was the Stage 0 gap. Stage 0 has since +closed it, so every composer path now respects `NEXT_PUBLIC_AGENT_FILE_UPLOADS`. + +Drive uploads are deliberately not part of the attachment pipeline: a file dropped into the Files +drawer writes directly to that mount at that path, and only files shared through the composer +become attachments (decision D13 in [design.md](design.md)). + +## Stage 0: gate the ungated paste and drag path (complete) + +**Goal.** Stop the front end from accepting files until the runner can deliver them, by refusing an +attachment that would reach the dead end instead of accepting it. + +**Complete, 2026-07-31, in commit `f2aa193cb2` on +[PR #5604](https://github.com/Agenta-AI/agenta/pull/5604), and verified live in both flag states.** +What was done, the implicit decisions it forced, and the QA record are in +[protocols/stage-0.md](protocols/stage-0.md). + +- **Front end.** The attach button, preview, and drive uploads were already behind + `NEXT_PUBLIC_AGENT_FILE_UPLOADS`, but paste and drag-to-attach on the composer predated the flag + and ran ungated (`AgentConversation.tsx`, the paste and drop handlers). The shared + `attachmentsBlocked` guard now includes `!uploadsEnabled`, which gates the drag, drop, and paste + handlers through the mechanism the composer already used for its other blocked states. This + removes the trap where a pasted screenshot looks accepted and is then ignored. +- **Tests.** Covered by the live check in both flag states recorded in the protocol rather than by + a new front-end unit test. + +## Stage 1: the first user-visible release (images perceived, files travel as references) + +This stage packages two bodies of work (the attachment storage and reference handling, and the +runner's model-facing change) into one release, on purpose. + +**Goal.** The model perceives images, and files travel as references rather than bytes. A person +attaches an image, the model sees it, the image is also on the agent's disk, the original stays +findable, and the wire no longer carries base64. + +**Why these ship together and not separately.** Shipping storage alone, visibly, would let the UI +accept files that the model still ignores. That is the original bug in a new form: the person sees an +attachment go through and the agent acts as though it were not there. So the release is packaged so +that model perception and the attachment storage and reference handling land together, and the person +never sees an attachment UI that quietly does nothing. + +**The minimum security and limits work belongs in this release, not later.** Session-binding checks, +forged-reference rejection, server-side media-type verification, per-file and per-turn size and count +limits, and time-to-live cleanup of never-referenced uploads are all part of Stage 1. They are not +polish for a later stage, because the first release already accepts files from a browser and stores +them, and an unauthenticated or unbounded version of that is not shippable. On limits specifically: +enforcement today is client-side only (the merged `DEFAULT_ATTACHMENT_LIMITS`: 100 files per turn, +10 MB images, 15 MB audio, 10 MB documents), which a client can bypass, so the server-side check +belongs to the new upload route. The exact rules are settled: the matrix in +[design.md](design.md), "The media-type, validation, and limits matrix", defines the per-kind +formats, caps, and count, the inspected-type-wins mismatch rule, and the workspace-only delivery +rule for an original that exceeds a provider's inline cap. The gateway decision is part of it: the +compose gateway request-body cap rises from 10 MB to 32 MB, and the route's per-kind caps stay the +enforced truth (the config task is listed under (a) below). + +**Deployment order inside the release.** The components deploy independently, so the order matters, +and it is reader-first: every component that has to understand a new form ships before the component +that emits it. Four packages merge and deploy bottom to top, the API, the runner, the SDK with the +agent service, and the front end, and turning the flag on is a fifth act taken separately once the +first three are confirmed deployed. Reader-first is what keeps every intermediate state safe. A +content block the runner does not understand is structurally accepted and semantically ignored, so a +producer that shipped early would lose files silently; a consumer that ships early is dead code +nobody reaches yet. + +**(a) API first.** Build the attachment resource before anything depends on it. + +- The create-only upload route: get-or-create the attachments mount, validate the file against the + settled matrix ([design.md](design.md), "The media-type, validation, and limits matrix": media + type verified server-side by inspecting the file bytes, never by trusting the client; per-kind + size caps; unrecognizable bytes rejected with a structured error), write the bytes, and + return `{attachment_id, filename, media_type, size}`. Refuse an overwrite or a delete of an + existing attachment original, because `write_file` overwrites silently today + ([design.md](design.md), decision D7). +- Raise the compose gateway request-body cap to match the matrix: `client_max_body_size` from `10M` + to `32m` in `hosting/docker-compose/oss/nginx/nginx.conf` (the only nginx config in the compose + stack, mounted by the OSS gh compose files `docker-compose.gh.yml` and + `docker-compose.gh.local.yml` behind the `with-nginx` profile; the dev stacks and the EE compose + files run no nginx). The railway gateway already allows 32 MB + (`hosting/railway/oss/gateway/nginx.conf`, verified 2026-07-31). The route-level per-kind caps + are the enforced truth; the gateway is a ceiling. +- Upload idempotency and atomicity: a retried upload must not create a duplicate original and must + not leave a partial one. The client sends an idempotency key alongside the file, unique per project + and session. The route reads the body in bounded chunks, validates it, inserts a `pending` row + carrying that key, writes the object, and only then marks the row `ready`; downloads and claims + resolve `ready` rows and nothing else. So a retry with the same key after an ambiguous failure + returns the same attachment instead of a second original, an upload still in flight is refused + rather than raced, and a create that dies between the object write and the `ready` transition + leaves nothing anyone can reach, with the reaper removing the row and its object afterwards. A + resource that is never referenced by a sent turn is swept by the time-to-live cleanup below. +- The download route the runner uses to read an original, with the session-binding check: the + attachment must belong to the session being run, and a reference to another session's attachment is + rejected ([design.md](design.md), decision D10). +- Per-file size limits and the per-session quotas, enforced server-side at the create route, with + the numbers from the settled matrix: 10 MB per image, 15 MB per audio clip, 10 MB per document + and per other kind. The per-turn count of 100 is the runner's to enforce over the current user + turn, since each upload is its own request (the matrix in [design.md](design.md)). +- Time-to-live cleanup of uploads that are never referenced by a sent turn. +- The record-schema extension so a record can carry an attachment reference. Records are text-only + today (see [research.md](research.md), section 7), so without this a reference cannot survive in the + durable log. This is why the schema change is in the first release rather than deferred. + +**(b) Runner second.** Once the API can store and serve, teach the runner to resolve and deliver. + +- Give the runner a first-class notion of the current user turn, before anything else in this step. + It has none today: `resolvePromptText` scans backward for the most recent non-empty user text + (`protocol.ts`), so an attachment-only turn that follows a text turn resends the earlier text. The + same assumption sits in the tail-freshness check, in the history fingerprint (which hashes user + text and no attachment ids, so two turns differing only in their files collide), and in the + inbound-record persist guard. Every item below reads the current turn, so this abstraction lands + first; `resolvePromptText` stays afterwards as what it really is, the approval-resume fallback. +- Dual-read during rollout. Dual-read means the runner accepts both the old inline-byte form and the + new attachment-reference form during the rollout, so a runner deploy does not have to be + simultaneous with the front-end switch. +- Resolution through the API: hand the `attachment_id` to the API download route and receive the + bytes only when the binding checks out. The runner never sees storage coordinates. +- Materialize the working copy to the id-namespaced path `cwd/attachments//`, + restoring it only when missing and never overwriting an edited copy ([design.md](design.md), The + working-copy path and edited copies). +- Build ACP image blocks: replace the single text block at the `env.session.prompt(...)` call + (`run-turn.ts`, currently line 803, the one real call site) with the resolved list of content + blocks. The rejection of a text-free turn lives in `run-plan.ts`, and it reads the current turn's + attachments too, so an image-with-no-text turn becomes valid instead of rejected. +- Structured capability errors: gate native delivery on the full D5 three-layer intersection, not + on a single flag. The runner must check all three layers before building a native block: the ACP + transport flag the adapter advertises (already probed and mapped in `capabilities.ts`), the + adapter fidelity for that kind (a static fact per pinned adapter version, from + [research.md](research.md), section 4), and the selected model's modalities. The first two the + runner already holds or can hard-code per pin; the third is a new implementation input: the run + request already carries the selected model (`protocol.ts`, the `model` field), and the SDK's + model catalog records each model's modalities (`sdks/python/agenta/sdk/agents/model_catalog.py`, + the `modalities` field, backed by the data files in `sdks/python/agenta/sdk/agents/data/`), so + the wire needs to carry the selected model's modalities to the runner, or the runner needs an + equivalent lookup. On a contract violation fail the turn with a structured error code the front + end can render, reusing the `assertRequiredCapabilities` and `*_UNSUPPORTED_MESSAGE` pattern in + `capabilities.ts`. Never drop silently. +- The mention-first cold-replay policy (decision D12 in [design.md](design.md)). Cold replay + (rebuilding the conversation from the durable records on a cold start) represents a historical + attachment as a textual mention carrying the real filename and the working-copy path + (`cwd/attachments//`), never as inline re-delivery, so nothing inline + grows with the history and no size budget is needed. At cold start the runner re-materializes the + working copy of every attachment referenced in the session's records, restoring only what is + missing and never overwriting an edited copy, so every mentioned path exists. Inline native + delivery is reserved for the attachments referenced by the turn actually being run. Update the + cold-replay path (the record fold in `services/runner/src/sessions/reconstruct.ts`, which rebuilds + a past user turn as text only today, and the transcript flattening in `transcript.ts`) so a past + image becomes that mention instead of being dropped from the rebuilt turn (see the cold-replay + policy below). + +**(c) The SDK and the agent service third.** The SDK is what produces the new content block, so it +ships once the runner already understands it. A browser file part carrying an attachment id becomes +an attachment block on the way in; the resolved connection puts the selected model's input modalities +on the wire, which is the fact the runner's third capability layer reads; and the delivery outcome +travels back through `wire.py` and the Vercel stream to the browser. The wire shape is under "SDK / +wire" below. This is its own package because shipping it is two rollout events, a package publish and +an agent-service deploy, rather than one. + +**(d) Front end fourth.** Once the API stores, the runner resolves, and the SDK produces, switch the +browser over. The collection UI is already merged dark (see "What is already built"), so this is +wiring work, not UI work: + +- Wire the transport seam to the new upload route: pass a real uploader to `useAttachmentUploads` + (today none is passed, so the progress, failure, and retry flow is inert by design). +- Send references instead of base64 `data:` URLs: replace the inline flow (`files.ts` + `fileToPart`) with the uploaded attachment's reference in the message parts (`agentRequest.ts`). +- Render the person's own attachment by resolving the reference through the download route, not + from the base64 `data:` URL (`sessions.ts`), which is what removes the browser-storage pressure. +- Stop rejecting unknown kinds in the composer: today `validateIncoming` refuses a file whose type + maps to no kind ("isn't a supported file type", `attachments.ts`, line 131), which contradicts + decision D6. Accept every kind, and show the workspace-only notice instead, per the matrix in + [design.md](design.md). + +**(e) Turning the flag on, last and on its own.** `NEXT_PUBLIC_AGENT_FILE_UPLOADS` is configuration, +not code, so the front end merges with it still off in production and the flip is a separate act +taken once (a), (b), and (c) are confirmed deployed. Then the composer accepts a file, attaches it +(workspace-only with a visible notice when the capability intersection does not allow native +perception), and shows it. Stage 0's gating of the paste and drag path keys off the same flag, so +enabling it opens every entry point together. + +**Tracing.** Because the history now carries a reference, the Python span no longer holds base64. +Confirm the trace shows the reference, not the bytes (the Python agent span keeps `messages` on +purpose, and a reference there is small). + +**SDK / wire.** Add the attachment-reference form to the content block: it carries `attachment_id`, +`filename`, `media_type`, and `size`, and no bytes. Update `protocol.ts`, `wire.py`, both contract +tests, and the shared golden fixtures together, as the runner's wire contract requires (see +`services/runner/CLAUDE.md`). + +**Tests.** These are listed in one place at the end of this file, under "Tests across the release," +because they span the API, the runner, and the front end. + +## Stage 2: the audio release, documents when adapters allow + +**Goal.** The voice UI turns on with browser dictation as the only audio-to-text, a recorded clip +follows the D6 workspace-only path (decision D14 in [design.md](design.md)), and documents are +planned for when the adapters change. + +**Audio: turn on what is built. Not blocked.** Dictation needs no new work: the composer's +dictation mode (`useVoiceInput`, the Web Speech API) already turns live speech into composer text +and is merged dark (PR #5458). The Stage 2 audio work is: + +- Ensure a recorded voice message and an uploaded audio file follow the D6 workspace-only path: the + recording travels through the Stage 1 attachment pipeline unchanged (uploaded once, stored as an + immutable original, materialized as a working copy, referenced in the records), and the composer + shows the visible notice that the model will not hear it. No transcript is produced or sent. +- Turn on `NEXT_PUBLIC_AGENT_VOICE_INPUT`: the voice capture UI (dictation and voice messages) is + already merged dark (PR #5458) and lands its recording in the attachment tray like any file. + +**Server-side transcription is deferred, not Stage 2 work.** The first release adds no +transcription endpoint, no key resolver, no platform key, and no metering (decision D14). The +survey a follow-up starts from (the litellm call path, prices, who has a usable key) is in +[research.md](research.md), section 4 ("Server-side transcription, surveyed and deferred"). + +**Native audio is the last upgrade, not Stage 2 work.** The `audio` type on the Agenta content +block (mapped in the Vercel adapter `messages.py` and mirrored in `protocol.ts`, `wire.py`, the +contract tests, and the golden fixtures) and the runner's mapping to the ACP audio block wait until +an adapter advertises the `audio` capability. Neither pinned adapter does, and the Anthropic +Messages API has no audio input at all ([research.md](research.md), section 4), so that upgrade has +no date and nothing in this stage depends on it. + +**Documents: blocked on adapter work.** From [research.md](research.md), section 4: the Claude +adapter drops a document blob entirely and the Pi adapter renders it as a byte count, so native +document delivery cannot ship on the adapters we pin today. `claude-agent-acp` is maintained by Zed +and `pi-acp` by the Pi project, so unblocking documents means an upstream release, a fork we +maintain, or a different harness. What would unblock it: either adapter-native document handling +(an adapter that turns an embedded resource into a real document input), or a deliberate decision +to deliver documents as extracted text (the agent, or a pre-step, extracts the text and inlines +it), which sidesteps native document delivery entirely. Once unblocked, the runner maps a document +to whichever path that decision chooses, gated on the `documents` capability. + +**Shared Stage 2 work.** + +- **Capability names (alias rollout, not a simultaneous rename).** Retire the old `fileAttachments` and + `file_attachments` names in favor of `images`, `audio`, and `documents`, and map ACP + `embeddedContext` to `documents`. Do this as an alias rollout: introduce the new names alongside the + old, keep the old names accepted as aliases through the rollout, and remove them later once every + independently deployed component speaks the new names. A single rename landing in every component at + once would break the versions in between (see [design.md](design.md), decision D5). The removal + timing needs no separate decision: remove the aliases once every component's deployed version + speaks the new names, confirmed per component. The pip-installed SDK is the gating one, since + user-held versions upgrade on their own schedule, so the removal follows a standard deprecation + window rather than a fleet deploy. +- **Front-end limits.** Replace the static defaults (`attachments.ts` `DEFAULT_ATTACHMENT_LIMITS`) + with limits derived from the selected model's real limits (see [research.md](research.md), section + 3), passed down in place of the default, which the file was already written to allow. +- **Tests.** A recorded clip uploads as a normal attachment, the turn proceeds workspace-only with + the visible notice, and nothing audio-derived reaches the model inline. A capability-contract + test that the mapped set is consistent across the layers. Runner tests + for the document path and its gate, plus the SDK contract test for the `audio` block, once their + respective adapter work unblocks. + +## Stage 3: the Files-drawer listing and cleanup refinements + +**Goal.** The "Shared by you" origin in the Files drawer, the reference-counting refinement of cleanup, the read-only +credential scope, and the edit-then-find flow verified end to end. The basic time-to-live cleanup +already ships in Stage 1; this stage refines it. + +- **Front end (drawer).** Add "Shared by you" as a third origin over the attachments mount, reusing the + existing provenance tagging in `DriveExplorer` (see [research.md](research.md), section 2). It is a + new origin label, not a new panel. +- **API (cleanup refinement).** Stage 1 ships a time-to-live sweep of never-referenced uploads. Once + records reliably carry references, add reference counting against the conversation records so a + still-referenced upload is never swept and a truly orphaned one is removed promptly (the open + question in [decisions.md](decisions.md) tracks this). +- **Retention.** Covered by the session lifecycle, settled in [decisions.md](decisions.md), "Settled + by evidence": originals are deleted when the session is deleted and kept as long as it exists, + including while archived. The existing session-mount teardown already implements this + (`delete_session_mounts` and `archive_session_mounts` in `api/oss/src/core/mounts/service.py`), + so the Stage 3 work is one test asserting the attachments mount is included in that teardown. +- **Hardening.** Add a read-only credential scope for the attachments mount, the strongest + immutability guarantee and the path that would let the runner read the object store directly instead + of through the API download route ([design.md](design.md), decision D7). +- **Tests.** An end-to-end check that after the agent edits its working copy, the original still opens + unchanged under "Shared by you" and the agent's new file shows under the agent origin. + +## Why the plan is shaped this way + +The split of responsibilities, the reuse choices, and the sizing all follow from a few judgments, +stated here so a reviewer can challenge them. + +**Responsibilities are split cleanly.** The front end collects, gates, uploads, and renders. The API +stores, serves, and enforces permissions. The runner reads the original, materializes the working +copy, and turns references into model content. The SDK carries the reference and the block types. No +layer reaches across into another's job: the runner never talks to the browser's storage, the front +end never builds ACP blocks, the API never knows about ACP. The one place to watch is the capability +mapping, which touches four layers, so the layered contract in [design.md](design.md) is the single +source and the old names retire through an alias rollout, keeping mixed component versions working +during the change. + +**Engineering and architecture practice.** The design reuses the existing mounts substrate rather +than inventing storage, follows the established pattern of a separate mount for a separate lifecycle +(the agent-files precedent), and reuses the existing capability-gate pattern for the failure case. +Nothing here is a new architectural concept; it is a new use of concepts already in the code. + +**Tradeoffs are stated, not hidden.** Every major choice in [design.md](design.md) lists its options +and what breaks under each. The lifecycle tradeoff is the clearest example: an attachment could have +lived in a `cwd` subfolder on lifecycle grounds, and only the findability requirement forces its own +mount. + +**Scale and extensibility, sized for a first version with room to grow.** The resend cost, which +could grow quadratically in the worst case, is already gone, independently of this design: since the +session-storage rework (PR #5560) the front +end sends only the trailing message and the runner rebuilds prior turns from durable records. The +reference-on-the-wire change removes what remains: the browser storage pressure, the trace bloat, +and the attaching turn's own base64 weight, so the first version scales better than today. The materialize step runs once per file per session. +The design grows into the harder modalities by adding block types, not by reworking the model-facing +call. Shipping the inline-only version first would avoid rework at that one call, though not the rest +of the system-wide change (decision D9). What is deliberately left for later is stated in +[scope.md](scope.md): video, assistant-produced files, cross-session reuse, and storage optimizations +like deduplication and thumbnails. + +**Fit with the current architecture.** The change lives at the boundaries that already exist: the one +`session.prompt` call in the runner, the existing mount routes, the existing content-block adapter, +the existing capability probe, and the existing provenance tagging in the drawer. It does not +introduce a parallel system beside any of these. + +## Four guarantees the stages must not lose + +These four points are each covered inside a stage above, but they are the kind of detail that gets +dropped during implementation, so they are restated here as explicit checks. + +- **Immutability is enforced server-side, not only by hiding the mount.** Originals enter only + through the create-only upload route, the API refuses overwrite and delete of an original (because + `write_file` overwrites silently today), and no signed credentials are ever issued for the + attachments mount (because any signed credential is read-write today) ([design.md](design.md), + decision D7). A read-only credential scope is the Stage 3 hardening follow-up. +- **Re-materialization is defined behavior.** If the agent deleted its working copy and a later turn + references the file, the runner re-reads the original and writes the copy again; it never overwrites + an edited working copy (Stage 1). Model delivery never depends on the working copy, and always reads + the original. +- **The capability contract is one layered model across every layer.** The layered contract in + [design.md](design.md) is the single source, and the old capability names retire through an alias + rollout (Stage 2), never a rename that must land in every component at the same time, so no + independently deployed component is ever left + speaking a name the others do not. +- **An attach never silently drops, and a turn fails only on a contract violation.** An unsupported + kind becomes a workspace-only attachment with a visible notice, and the runner fails the turn only + when asked to deliver a native block the harness cannot accept ([design.md](design.md), decision D6; + Stage 1). + +## The tracing change + +The tracing improvement in Stage 1 is not a separate task; it follows directly from the +reference-based message format. Once the saved history carries a reference instead of bytes, the +Python agent span (which keeps `messages` on purpose and has no length cap) holds a small reference +rather than a base64 blob. Confirm this in Stage 1 rather than adding a truncation cap, since the +cause is removed at the source. + +## The cold-replay policy + +On a cold start the runner rebuilds the conversation from the durable records +(`services/runner/src/engines/sandbox_agent/reconstruct-history.ts`). The adopted policy is +mention-first (decision D12 in [design.md](design.md)): a historical attachment is represented by a +textual mention carrying its real filename and working-copy path, and the runner restores the +working copy of every attachment referenced in the session's records, so each mentioned path +exists. Inline native delivery happens only for attachments referenced by the turn being run. No +count or byte budget is needed, because nothing inline grows with the length of the history; a +mentioned file is one tool call away, since the harnesses' read tools deliver an image from disk on +all three harnesses, and a PDF on Claude (see [research.md](research.md), section 4). + +## Tests across the release + +These tests span the API, the runner, and the front end, so they are listed together rather than +under a single stage. + +**Model perception and the seam.** + +- An image-and-text turn produces an image block and a text block. +- An image-only turn is valid instead of rejected. +- A cold replay of a past image no longer emits the string "[image]"; it emits the mention with the + real filename and working-copy path, and that path exists after the cold-start sweep. +- Run integration tests against the two pinned adapters (`@agentclientprotocol/claude-agent-acp` and + `pi-acp`) confirming that an inline image reaches the model as a real image. + +**Storage, references, and the wire.** + +- Add an SDK contract test for the attachment-reference form against the golden fixture. +- Test that a picked file uploads once and the message carries a reference with no `data:` URL. +- Verify that a saved-and-reloaded message still renders the file from the reference. + +**Security and authorization.** + +- A reference to another session's attachment is rejected (foreign-session rejection). +- A forged reference (an id that does not resolve to the caller's session) is rejected. +- An overwrite attempt against an existing original is refused, and a delete attempt against an + original is refused. +- A browser-supplied media type that disagrees with the type the server inspected from the bytes does + not win; the server's verified type is authoritative. + +**Limits and cleanup.** + +- Per-file and per-turn size and count limits are enforced server-side, with the numbers and the + boundary behavior from the matrix in [design.md](design.md): a file over its kind's cap is + rejected at upload with a structured error, and unrecognizable bytes are rejected as an invalid + file. +- Deleting a session tears down its attachments mount with the other session mounts, and archiving + keeps the originals (the retention rule in [decisions.md](decisions.md), "Settled by evidence"). +- An upload retry after a transient failure does not create a duplicate or a partial original. +- An abandoned upload (never referenced by a sent turn) is swept by the time-to-live cleanup. + +**Resolution and materialization.** + +- Resolver failure paths: a missing attachment, expired authorization, and a timeout each surface a + structured error, not a silent drop. +- Materialization is atomic and never overwrites an edited working copy. +- Two attachments with the same filename in one session do not collide (the id-namespaced path). +- Warm resume and cold replay both work with references, under the mention-first policy above: a + rebuilt turn carries the mention with the real path, and every mentioned working copy exists after + the cold-start sweep. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/research.md b/docs/design/agent-workflows/projects/agent-multi-modality/research.md new file mode 100644 index 0000000000..5ef1b21d43 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/research.md @@ -0,0 +1,857 @@ +# Research + +This file collects the findings the design rests on. It is meant to be read top to bottom. Each +section answers one question. Every claim links to a file and line in our code, or to an outside +source. Line numbers drift as files change, so treat them as a starting point and confirm the +surrounding function name. + +Sections: + +1. Section 1 explains where the file is lost today and the side effects it causes. +2. Section 2 describes the mounts system: what it is, every endpoint, who may call each one, and how + the file panel uses them today. +3. Section 3 covers the kinds of files and how models handle them: which kinds a model perceives + natively, which are just files an agent reads with tools, and the size limits per provider. +4. Section 4 explains how ACP carries content and what each harness does with each content type. +5. Section 5 compares two ways to deliver a file to the model. +6. Section 6 surveys how other tools handle attachments: Zed, opencode, and others. +7. Section 7 covers records and replay: what exists and the direction for the runner to rebuild + context. +8. Section 8 covers assistant-produced files: what is declared and what is missing. +9. Section 9 explains the capability flags: where they came from and why they were never used. +10. Section 10 covers the front-end adapter and its current limits. + +--- + +## 1. Current state of the code + +The attachment path is wired from the chat box all the way to the runner. It breaks at the last +step before the harness. + +| Step | Where | Carries the file? | +| --- | --- | --- | +| Chat box picks, pastes, drops, previews the file | `web/oss/src/components/AgentChatSlice/assets/files.ts:20` (`fileToPart` reads the file into a `data:` URL) | Yes, as a base64 `data:` URL in an AI SDK `file` part | +| Request body carries the message parts | `web/packages/agenta-playground/src/state/execution/agentRequest.ts` (history assembled from message parts) | Yes | +| Our SDK parses the part into a content block | `sdks/python/agenta/sdk/agents/adapters/vercel/messages.py:97` (`_part_to_blocks`, the `file` case) | Yes, into a `ContentBlock` of type `image` or `resource` | +| Our SDK writes the wire payload | `sdks/python/agenta/sdk/agents/utils/wire.py` (`ContentBlock.to_wire`) | Yes, base64 intact | +| Runner receives the request | `services/runner/src/server.ts` | Yes, intact | +| **Runner hands the turn to the harness** | `services/runner/src/engines/sandbox_agent/run-turn.ts:742` | **No. It sends `[{ type: "text", text: turnText }]` only.** | + +The single real call that sends a turn to the harness is `env.session.prompt(...)` at +`run-turn.ts:742`. It is hard-coded to one text block. The text it sends comes from +`messageText()` at `protocol.ts:573`, which keeps only blocks whose `type` is `"text"` and joins +them. Everything that is not text is discarded there. + +Three concrete failures follow from this: + +1. **Image and text together.** The model receives only the text. When the conversation is later + rebuilt from the durable records on a cold start, the image is absent entirely, because the + records store only the text (see section 7). A history that still carries an image block (for + example a resume, where the front end sends the full history) is flattened to the literal + string `"[image]"` (`transcript.ts`). +2. **Image with no text.** The turn is rejected. `resolvePromptText()` (`protocol.ts:585`) returns + an empty string, and the run fails with "No user message to send" from `run-plan.ts`. +3. **No warning to the person.** The attach button, the attachment preview, and the drive-upload + surfaces are behind the `NEXT_PUBLIC_AGENT_FILE_UPLOADS` flag, off by default + (`web/oss/src/components/AgentChatSlice/assets/constants.ts`, verified 2026-07-31), but pasting + or dragging a file into the composer predates the flag, is not gated, and still reaches the dead + end. The person is never told the file will be ignored. + +### Side effects that a reference-based design removes + +- **Resending the whole conversation every turn (fixed since the session-storage rework).** The + front end used to resend the full message history on each turn, so a five megabyte image became + about seven megabytes of base64 re-uploaded on every later turn, a cost that could grow with the + square of the conversation length in the worst case. The session-storage rework (PR #5560, merged 2026-07-30, + on by default) removed this: the front end sends only the trailing user message + (`NEXT_PUBLIC_SESSIONS_LAST_MESSAGE_ONLY`, read in `agentRequest.ts`), the runner persists every + turn as durable records (`AGENTA_RECORDS_DURABLE`, `services/runner/src/sessions/persist.ts`), + and on a cold start it rebuilds the conversation from those records + (`AGENTA_SESSIONS_RECONSTRUCT`, + `services/runner/src/engines/sandbox_agent/reconstruct-history.ts`). What remains is the + attaching turn itself: its bytes still ride that one message as base64. +- **Filling the browser's local storage.** Saved messages hold the base64 inline + (`web/oss/src/components/AgentChatSlice/state/sessions.ts:31`, which notes "a conversation with + large files can approach the localStorage quota"). When the roughly five megabyte browser quota + is exceeded, `writeMessagesWithQuotaGuard` (`sessions.ts:409`) drops other sessions' saved + transcripts to make room. +- **Bloating the traces.** The Python agent span keeps the messages on purpose and there is no + length cap on exported span attributes, so a base64 image lands verbatim in a trace. + +Once the saved messages and the traces carry a small reference instead of the bytes, the remaining +costs go away too, because the bytes are stored once in the object store and never travel with the +messages again. + +--- + +## 2. The mounts system + +### What a mount is + +A mount is one named storage area in an S3 object store. Each mount has its own set of object +keys under a prefix and can hand out short-lived, prefix-scoped credentials so a client can read +and write its files directly. The code lives in `api/oss/src/core/mounts/service.py` (the service), +`api/oss/src/apis/fastapi/mounts/router.py` (the general routes), and +`api/oss/src/apis/fastapi/sessions/router.py` (the session-scoped routes, class +`SessionMountsRouter` near line 863). + +The object key layout is +`[/]mounts///` (`service.py:358`, `_storage_key`). The +`` segment is the tenant partition of the key. **It is not a "project mount."** There +is no project-scoped mount kind in the system today. Every mount in practice carries a session +scope or an agent scope. In `MountQuery` +(`api/oss/src/core/mounts/dtos.py:25`) both `session_id` and `agent_id` are optional, so the type +itself does not forbid a mount with neither scope. The current code never creates one, so the +accurate claim is that no project-scoped mount kind exists in practice, not that the type makes one +impossible. So in a +storage key like `mounts///photo.png`, the first segment only +says which tenant owns the bytes; the mount itself is still session-scoped. + +There are three kinds of mount today: + +- **A session `cwd` mount.** Backs the agent's working directory for one session. Created on first + use, keyed on the session id (`get_or_create_session_cwd`, `service.py:489`). Mounted into the + sandbox so the agent sees it as its working folder. +- **Extra named session mounts.** A session can have more than one mount, each with its own name + and its own storage prefix (`get_or_create_session_mount(session_id, name=...)`, `service.py:405`). + The comment there says any name other than `cwd` is "an additional session-scoped mount sharing + the same shape with its own prefix." This existing function is what the design uses to create a + separate attachments mount. +- **An agent mount.** Durable across all sessions of one agent. This is the `agent-files` folder + the agent sees. It exists because its lifecycle is different (it outlives the session). It is + mounted into the sandbox beside `cwd` and linked in as `agent-files/` + (`services/runner/src/engines/sandbox_agent/agent-mount.ts:16`, `AGENT_FILES_LINK_NAME`). + +The `cwd` mount is the agent's writable working folder. Its own README, written for the agent, +says: "Concurrent runs share this folder, so the last writer wins for each file" +(`agent-mount.ts:23`). In other words, the agent can overwrite or delete anything in `cwd`. That +fact drives a key design choice (see [design.md](design.md), the decision on where originals live). + +### Every mounts endpoint, who may call it, and what it does + +All of these sit behind the API's normal authentication. A caller is a person (through the web +app, carrying a JWT) or a program (carrying an API key), and either resolves to a user id and a +project id in `request.state`. On top of that, each route checks a role-based permission with +`check_action_access`. The runner is not a special caller: when it needs storage credentials, it +calls the sign endpoint with the same authorization the invocation carried. + +General mounts routes, registered in `MountsRouter.__init__` (`router.py:120`): + +| Route | Method | Permission | Arguments | What it does | +| --- | --- | --- | --- | --- | +| `/mounts/` | POST | `EDIT_MOUNTS` | mount body | Create a mount. | +| `/mounts/query` | POST | `VIEW_MOUNTS` | filters (`session_id`, `agent_id`) | List mounts. | +| `/mounts/{id}` | GET | `VIEW_MOUNTS` | mount id | Fetch one mount. | +| `/mounts/{id}` | PUT | `EDIT_MOUNTS` | mount body | Edit a mount. | +| `/mounts/{id}/sign` | POST | `USE_MOUNTS` | mount id | Mint short-lived S3 credentials scoped to this mount's prefix. | +| `/mounts/agents/sign` | POST | `USE_MOUNTS` | `artifact_id`, `name` | Get-or-create the agent mount and sign it. | +| `/mounts/agents/query` | POST | `VIEW_MOUNTS` | JSON body with `artifact_id`, `name` | Fetch the agent mount without creating it. The arguments arrive in a JSON body, not as query parameters. | +| `/mounts/{id}/archive` / `/unarchive` | POST | `EDIT_MOUNTS` | mount id | Soft archive or restore. | +| `/mounts/{id}/files` | GET | `VIEW_MOUNTS` | `path`, `read`, `order`, `limit`, `depth`, `git_aware`, `include_gitignored`, `with_counts` | List files or read one file's text. `with_counts` adds child counts to the listing. | +| `/mounts/{id}/files` | PUT | `EDIT_MOUNTS` | `path`, raw body | Write one file from a raw body. | +| `/mounts/{id}/files` | DELETE | `EDIT_MOUNTS` | `path` | Delete a file or a folder. | +| `/mounts/{id}/files/folder` | POST | `EDIT_MOUNTS` | `path` | Create an empty folder marker. | +| `/mounts/{id}/files/upload` | POST | `EDIT_MOUNTS` | multipart `file`, `path` | Upload a file through the API. | +| `/mounts/{id}/files/download` | GET | `VIEW_MOUNTS` | `path` | Download exact bytes as a binary response. | +| `/mounts/files/export` | POST | `VIEW_MOUNTS` | list of mounts | Stream a zip of many files ("download all"). | + +Session-scoped routes, registered in `SessionMountsRouter.__init__` (`sessions/router.py:863`). +These require the session permission **and** the mounts permission together (`_check` loops over +both, `sessions/router.py:925`): + +| Route | Method | Permissions | Arguments | What it does | +| --- | --- | --- | --- | --- | +| `/sessions/mounts/` | GET | `VIEW_SESSIONS` + `VIEW_MOUNTS` | `session_id` | List this session's mounts. | +| `/sessions/mounts/query` | POST | `VIEW_SESSIONS` + `VIEW_MOUNTS` | `session_id` | List with a body query. | +| `/sessions/mounts/sign` | POST | `RUN_SESSIONS` + `USE_MOUNTS` | `session_id`, `name` (default `cwd`) | Get-or-create the named session mount and sign it. | +| `/sessions/mounts/{id}/files/upload` | POST | `EDIT_SESSIONS` + `EDIT_MOUNTS` | multipart `file`, `path` | Upload a file into a session mount through the API. | +| `/sessions/mounts/{id}/files/download` | GET | `VIEW_SESSIONS` + `VIEW_MOUNTS` | `path` | Download exact bytes. | + +There is **no move or rename endpoint.** Moving a file is done by writing to the new path and +deleting the old one. Renaming works the same way. + +### Two ways bytes get in and out of a mount + +There are two separate mechanisms, and the difference matters for enforcing that a file cannot be +changed. + +- **Through the API.** The upload, write, download, and delete routes read or write bytes + server-side. Each one enforces its role permission. A person who has `VIEW` but not `EDIT` + cannot write. +- **Directly against S3 with signed credentials.** The sign endpoints hand out short-lived S3 + credentials scoped to the mount's prefix (`sign_mount_credentials`, `service.py:688`). The holder + can then read and write objects under that prefix directly, without going back through the API. + This is how the runner mounts a folder into the sandbox: it signs the `cwd` mount, and geesefs + uses those credentials to present the mount as a writable directory. + +Two consequences for this project: + +1. **The runner can read a mount's bytes without mounting it into the sandbox.** The signed + credentials are ordinary S3 credentials. The runner can do a plain object GET with them. It does + not have to make the mount appear as a folder for the agent. Note that for the attachments mount + the design does not use this path at first: because any signed credential is read-write (below), + decision D7 in [design.md](design.md) issues no signed credentials for that mount at all, and the + runner reads originals through the API download route; the direct read waits for the read-only + credential scope in Stage 3. +2. **Server-side "cannot be changed" is achievable.** If a mount's bytes only ever enter through + the upload API route, and the system never hands out write-capable signed credentials for that + mount, then nothing but a deliberate API write can change it. That is what makes an attachment + original immutable in fact, not just by convention. Today `sign_mount_credentials` + does not offer a read-only variant, so a read-only scope would be a small addition to the + short-lived storage credentials (STS) signing call. See the hardening step in [plan.md](plan.md), + Stage 3. + +### The write path, precisely + +Three facts about how a write actually happens today shape the design, so they are stated exactly. + +- **A signed credential is read-write.** `sign_mount_credentials` (`service.py:688`) has no + read-only variant, so any credential it issues grants write access to the whole mount prefix. + This matters because it means immutability cannot rest on signed credentials. As long as the + system signs a mount at all, the holder of that signature can write it. Immutability has to come + from not signing the mount and only writing it through the permission-checked upload route. +- **The upload route buffers the whole file in memory.** The API upload helper reads the entire + file into memory before writing it (`api/oss/src/apis/fastapi/mounts/utils.py:45`). This matters + because a large attachment would hold its full size in the API process. A streaming write path is + a later addition once large files are in play. +- **`write_file` overwrites silently.** `write_file` (`service.py:1297`) replaces any object + already at the same path without a check or an error. This matters because "the original is + create-only" cannot be assumed from the storage layer. The API has to enforce create-only + semantics itself, by refusing a write to a path that already holds an attachment original. + +### How the Files drawer uses these today + +The Files drawer is the `DriveExplorer` component (`web/oss/src/components/Drives/DriveExplorer.tsx`). +It shows a session's files as one tree, folding the `cwd` mount and the `agent-files` mount +together. It tags each top-level node by where it came from and shows the origin label only when +the tree mixes origins (`driveHasMixedOrigins` in `ContextRail.tsx`). For file management it calls +the same mount routes above: list to show the tree, upload to add a file, delete to remove one, +and write to save an edit. A "Shared by you" origin for attachments is a new third origin over +this existing tagging, not a new kind of panel. + +The service already hides runner-internal paths and dotfiles from the listings +(`_is_internal_mount_path` at `service.py:154`, `_is_hidden_path` at `service.py:190`), so an +internal attachments area would not clutter the normal view. + +### Responsibilities across a message-attachment flow + +Reading the endpoints above against the three flows that would use them: + +- **Managing files in the drawer** (upload, edit, delete, move). The front end owns this. It calls + the upload, write, delete, and list routes directly. The API enforces permissions and writes the + bytes. This flow exists today for the `cwd` and agent mounts. +- **Attaching a file to a message.** The front end uploads the file once (to an attachments mount) + and then sends only a reference on the wire. The API stores the bytes and later serves them to + the runner. The front end owns the upload and the reference; the API owns storage; the runner + owns turning the reference into model content. +- **Materializing a working copy for the agent.** The runner owns this. It reads the original from + the mount and writes a copy into the agent's `cwd` so the agent's tools can open it. + +--- + +## 3. Kinds of files and how models handle them + +This is the section that answers "which files are special and which are just files." The key idea: +a model has a small set of input types it perceives directly through its own encoder, and +everything else is just bytes that the agent must open with a tool. + +### Native modalities: the model perceives them directly + +For these, the bytes must be delivered to the model in a specific content slot. The model has a +built-in path that turns them into something it understands (an image encoder, an audio encoder, a +document parser). That slot is reached through message content, which includes a tool result: a +harness's own read tool can fill it for some kinds when the agent calls it, with coverage that +differs per harness (section 4). A tool that returns only text never reaches this path. + +- **Images** (PNG, JPEG, GIF, WebP). Perceived by Claude, the GPT-4o and GPT-5 family, and Gemini. +- **Audio** (for example WAV, MP3). Perceived natively by some models (Gemini, and the GPT audio + models). There is no reliable way to make a model "hear" audio by having the agent read the file, + so audio must be delivered as a native audio input. +- **PDF and other documents as a document input.** Claude accepts a PDF as a document content block + and reads both its text and its page images. This is a native path distinct from plain text. +- **Video.** Gemini perceives video natively. Claude and the current GPT chat models do not. Video + is out of scope for this project (see [scope.md](scope.md)). + +"Document" is not a modality of its own. Whether a PDF, a CSV, or a spreadsheet is perceived +natively depends on three things at once: the specific model, the harness adapter that hands the +bytes to that model, and the exact format. So document support is a per-format, per-adapter, +per-model matrix, not a single flag you can turn on. Section 4 shows how far apart the two adapters +we run actually are on this point. + +### Everything else: files an agent reads with tools + +For these, there is no special model path. The right handling is to place the file on the agent's +working directory and let the agent's tools open it, possibly after converting it. + +- **Plain text, source code, CSV, JSON, Markdown.** The agent reads them with its file tools. Small + text can also be inlined directly into the prompt as text, which every harness supports. +- **Office documents** (Excel, Word). The agent converts them first (for example with a library + like openpyxl for a spreadsheet) and then reads the result. The raw binary is not a native model + input for Claude or GPT. +- **Archives** (zip). The agent extracts them and works on the contents. +- **CAD and other binary formats, and folders.** The agent runs whatever tool understands them. + These are never native model inputs. +- **Very large files of any kind.** Even a native modality has size limits (below). Past those, a + file is handled as bytes on disk, not as a native input. + +### An important subtlety: a non-vision model can still use an image + +An agent whose model cannot see images is not useless with an image. Two separate things are +going on: + +1. **Model perception.** Whether the model itself sees the picture. This needs the native image + path and a vision-capable model. +2. **Tool use over the file.** Whether the agent can run a program on the image bytes, for example + to resize it, read its metadata, or feed it to an image library. This needs only the file on + disk. It works even when the model cannot see the image. + +So the file should be placed on disk for the agent regardless of whether the model can perceive it, +and the native delivery to the model should happen only when the model supports that modality. This +is exactly the "two outcomes at once" goal from [context.md](context.md). + +### Provider limits (verified 2026-07-31; re-verify before relying on them) + +From Anthropic's vision and PDF documentation: + +- **Whole request.** The Messages API caps a request at 32 megabytes on standard endpoints. + [Anthropic vision docs](https://platform.claude.com/docs/en/build-with-claude/vision). +- **Images.** 10 megabytes per image, measured on the base64-encoded data, on the Claude API + directly (5 megabytes base64 on Amazon Bedrock and Google Cloud). Maximum dimensions 8000x8000 + pixels, with a stricter limit near 2000 pixels per side when a request carries more than 20 + images. 100 images per request on 200k-context models, 600 otherwise. Formats: JPEG, PNG, GIF, + WebP. [Anthropic vision docs](https://platform.claude.com/docs/en/build-with-claude/vision). +- **PDF documents.** 32 megabytes maximum request size; 600 pages per request, dropping to 100 + pages when the request's context window is under 1M tokens; standard unencrypted PDF. + [Anthropic PDF support docs](https://platform.claude.com/docs/en/build-with-claude/pdf-support). + +OpenAI's vision input allows 512 megabytes of total payload and 1500 image inputs per request, +with no documented per-image cap (formats: PNG, JPEG, WebP, non-animated GIF). +[OpenAI images and vision guide](https://developers.openai.com/api/docs/guides/images-vision). + +Gemini perceives images, audio, and video natively. Inline data shares a 20 megabyte total request +budget, larger files go through its Files API, and a request can carry up to 3,600 images +(formats: PNG, JPEG, WebP, HEIC, HEIF). +[Gemini image understanding](https://ai.google.dev/gemini-api/docs/image-understanding), +[audio understanding](https://ai.google.dev/gemini-api/docs/audio), +[video understanding](https://ai.google.dev/gemini-api/docs/video-understanding). + +The important design takeaway is not the exact number, which changes, but the shape: there are hard +per-request and per-file limits, they differ by provider, and our front-end limits should be +derived from the selected model's real limits rather than from the transport-sized static defaults +we have today (see section 10). The settled server-side matrix these numbers feed is in +[design.md](design.md), "The media-type, validation, and limits matrix". + +--- + +## 4. How ACP carries content, and what each harness does with it + +### The ACP content model + +ACP is the Agent Client Protocol, an external standard published by Zed Industries. The runner +speaks it to the harness through two adapter packages it pins in `services/runner/package.json`: +`@agentclientprotocol/claude-agent-acp` at version `0.58.1` for the Claude harness +(`services/runner/package.json:23`), and `pi-acp` at version `0.0.29` for the Pi harness +(`services/runner/package.json:35`). Both pins verified against the file on 2026-07-31. A third +adapter, `@agentclientprotocol/codex-acp` at version `1.1.7`, arrives with the Codex harness in +PR #5509 (branch `feat/codex-harness`) and is covered below alongside the two pinned ones. The +specification is at `agentclientprotocol.com`. + +ACP defines these content block types for a prompt turn +([ACP content spec](https://agentclientprotocol.com/protocol/v1/content)): + +- **Text.** Field `text` (required). Every agent must support text. +- **Image.** Fields `data` (base64, required), `mimeType` (required), `uri` (optional). Requires the + `image` prompt capability. +- **Audio.** Fields `data` (base64, required), `mimeType` (required). Requires the `audio` prompt + capability. +- **Embedded resource.** Field `resource` (required), which is either a text variant (`uri` and + `text` required) or a blob variant (`uri` and `blob` base64 required). The spec calls this "the + preferred way to include context in prompts." Requires the `embeddedContext` prompt capability. +- **Resource link.** Fields `uri` (required) and `name` (required), plus optional `mimeType`, + `title`, `description`, `size`. This is a pointer to a resource the agent may fetch, not content + the model is guaranteed to perceive. + +The decisive fact for this project: for an image, an audio clip, or an embedded document, the bytes +are **required** and travel inline as base64 in the turn. There is no content type that hands the +model a bare URL and guarantees the model reads it. A resource link is only a pointer; whether the +model ever sees the file depends on the agent choosing to fetch it with a tool. + +Prompt capabilities are three flags the agent advertises at start-up: `image`, `audio`, and +`embeddedContext`. Each gates the matching content type +([ACP content spec](https://agentclientprotocol.com/protocol/v1/content)). + +### What our runner does with capabilities today + +The runner probes the harness's capabilities through the sandbox-agent library, which surfaces the +ACP capabilities as an `AgentInfo.capabilities` object. The runner maps that into its own +`HarnessCapabilities` shape (`capabilities.ts:68`, `mapCapabilities`), which includes `images` and +`fileAttachments` flags (`protocol.ts:281`). But it only ever branches on the tool-related flags +(`mcpTools`, `toolCalls`); the `images` and `fileAttachments` flags are assigned and then read +nowhere (a grep for `.images` in `services/runner/src` finds only the assignment at +`capabilities.ts:86`). So the runner receives the capability values but never uses them. This +project makes the runner enforce them. More on the history in section 9. + +### What the pinned adapters actually do + +This is where "the model perceives it" meets reality, because each adapter maps ACP content to its +own model API in its own way. Rather than infer this from documentation, the following comes from +reading the published packages themselves: the two the runner pins today, and the Codex adapter +from PR #5509. + +**The Claude adapter (`@agentclientprotocol/claude-agent-acp` 0.58.1).** The adapter file is +`node_modules/@agentclientprotocol/claude-agent-acp/dist/acp-agent.js`. + +- It advertises `promptCapabilities: { image: true, embeddedContext: true }` (near line 418). It + does **not** advertise audio. +- In its prompt conversion (near lines 3900 to 3925): an ACP image becomes a native Claude image + block; a `resource_link` becomes a text link; a text resource becomes a URI link plus a + `` text block; and a blob resource is **explicitly ignored**, with the code + comment "Ignore blob resources (unsupported)." +- The consequence is blunt. A PDF sent as an ACP blob resource never reaches Claude at all through + this adapter. It is dropped in the conversion. + +Why the blob drop exists: it is the adapter's choice, not an SDK constraint. The Claude Agent SDK +types a user message as an Anthropic `MessageParam`, and the Anthropic SDK's `ContentBlockParam` +includes a native `document` block that accepts a base64 PDF source +(`@anthropic-ai/sdk@0.115.0`, `resources/messages/messages.d.ts:555` and `:579`), so the adapter +could map a PDF blob the same way it already maps images. Nobody has asked upstream: searches of +`agentclientprotocol/claude-agent-acp` issues and PRs (open and closed) for "pdf", "blob", +"document", and "attachment" on 2026-07-31 found nothing requesting inbound blob or PDF prompt +support, and the ignore path is unchanged on the repo's main branch (`src/acp-agent.ts:6895`). + +Why audio is missing: the Anthropic Messages API has no audio input block at all. The string +"audio" appears nowhere in `ContentBlockParam` (`@anthropic-ai/sdk@0.115.0`, +`resources/messages/messages.d.ts`), so the adapter has nothing to map ACP audio onto; the +limitation is inherited from the API, not chosen by the adapter. An issue search for "audio" on the +adapter repo returned zero results on 2026-07-31. + +Newer versions: the latest published version is `0.64.0` as of 2026-07-31 +(`npm view @agentclientprotocol/claude-agent-acp version`). Modality behavior is unchanged between +`0.58.1` and `0.64.0`. The advertised capabilities are still `{ image: true, embeddedContext: true }` +with no audio (`0.64.0` tarball, `dist/acp-agent.js:622`), the blob branch still carries the comment +"Ignore blob resources (unsupported)" (`dist/acp-agent.js:5363`), and audio blocks are still skipped +under "Ignore audio and other unsupported types" (`dist/acp-agent.js:5387`). Upgrading the pin would +change none of the modality behavior described above. + +**The Pi adapter (`pi-acp` 0.0.29).** The adapter file is `node_modules/pi-acp/dist/index.js`. + +- It advertises `promptCapabilities: { image: true, audio: false, embeddedContext: + process.env.PI_ACP_ENABLE_EMBEDDED_CONTEXT === "true" }` (near line 1696). Audio is off, and + embedded context is off unless an environment variable turns it on. +- In its prompt conversion (near lines 1470 to 1495): images pass through as image inputs; a text + resource is inlined into the message text as `[Embedded Context] uri (mime)\n`; a blob + resource becomes only the line `[Embedded Context] uri (mime, N bytes)` with no content at all; + and an audio block, if one arrived despite the capability being off, becomes the text line + `[Audio] (mime, N bytes) not supported by pi-acp` (line 1493). +- The consequence is that a document sent as a blob to the Pi adapter reaches the model as a byte + count, not as the document. + +The adapter is community-maintained at [svkozak/pi-acp](https://github.com/svkozak/pi-acp), not by +earendil-works. Why the blob placeholder and the env gate exist: Pi's RPC prompt interface accepts +only a message string plus an images array, so the adapter has no first-class way to forward +arbitrary embedded resources. The maintainer gated `embeddedContext` behind +`PI_ACP_ENABLE_EMBEDDED_CONTEXT` in [pi-acp PR #12](https://github.com/svkozak/pi-acp/pull/12) +(merged) precisely to avoid "advertising broader support than we actually have" while blob +resources degrade to a placeholder. No issue tracks un-gating it or adding blob translation; a +sweep of all 92 issues and PRs in the repo on 2026-07-31 found nothing beyond PR #12 touching +`embeddedContext`. The `audio: false` flag follows from the same message-plus-images RPC shape; the +only issue matching "audio" (#58) is about session listing, not audio. + +The runner also applies a local patch to this pin (`services/runner/patches/pi-acp@0.0.29.patch`). +It only adds two environment variables for the skill and session directories; it does not touch +content conversion or capabilities. + +Newer versions: the latest published version is `0.0.33` as of 2026-07-31 +(`npm view pi-acp version`). Modality behavior is unchanged between `0.0.29` and `0.0.33`. The +advertised capabilities are byte-for-byte the same, including `audio: false` and the env-gated +`embeddedContext` (`0.0.33` tarball, `dist/index.js:1937`), and the conversion is the same code at +new offsets: the blob byte-count line at `dist/index.js:1671` and the audio not-supported line at +`dist/index.js:1681`. The only capability delta is a new session `delete` capability, which is not +modality related. + +A Pi extension can close one of these gaps. Pi loads extensions from its agent directory, and our +runner already installs one there for tool relay and tracing +(`services/runner/src/extensions/agenta.ts`, installed by +`services/runner/src/engines/sandbox_agent/pi-assets.ts`). The extension API lets an extension +register a custom tool (`registerTool`, `@earendil-works/pi-coding-agent` 0.80.6, +`dist/core/extensions/types.d.ts:874`) whose result carries real image content blocks back to the +model: `AgentToolResult.content` is typed as text or image content (pi-agent-core 0.80.6, +`dist/types.d.ts:306`), the agent loop copies that content verbatim into the tool-result message +(pi-agent-core, `dist/agent-loop.js:528`), and the Anthropic serializer emits real base64 image +blocks from it (pi-ai 0.80.6, `dist/api/anthropic-messages.js:875`). Pi's own read tool returns +exactly this shape for images (`dist/core/tools/read.js:183`). So a custom tool that renders PDF +pages to PNG images and returns them as image blocks works against the pinned version with no +change in Pi; the remaining work is ours (a PDF rasterizer available to the extension, and our +extension's relay currently returns text only, `services/runner/src/tools/dispatch.ts:71`). That +closes the tool-mediated document gap, at the agent's discretion like any tool call. The inline +gap is different. An extension can transform the incoming prompt and attach images to the very +user message being sent (the `input` event, `dist/core/extensions/types.d.ts:614` and `:628`, +delivered into the user message at `dist/core/agent-session.js:849`), but by the time it runs, a +blob's bytes are already gone: the pi-acp adapter has reduced the blob to the byte-count line +above, so the extension sees only the placeholder. Closing the inline document path therefore +needs changes in pi-acp, and a native document input would need changes in Pi itself, whose +message model carries only text, thinking, and image content (pi-ai 0.80.6, +`dist/types.d.ts:217`). + +**The Codex adapter (`@agentclientprotocol/codex-acp` 1.1.7).** This adapter arrives with the Codex +harness in PR #5509. That PR pins it at `1.1.7`, which bundles `@openai/codex` `0.145.0`, in the +runner image build (`services/runner/docker/Dockerfile.gh:100` on branch `feat/codex-harness`, via +`install-agent codex --agent-process-version 1.1.7`) and records the pin under `runtimeAgentPins` in +`services/runner/package.json:10` on that branch. The published package ships one bundle, +`dist/index.js`; line numbers below refer to the `1.1.7` tarball from npm. + +- It advertises `promptCapabilities: { embeddedContext: true, image: true }` (near line 28486). It + does **not** advertise audio. +- In its prompt conversion (`buildPromptItems`, near lines 26708 to 26744): an ACP image becomes a + native Codex image input (an `http`, `https`, or `data` URI passes through as a URL; inline base64 + becomes a data URL); a `resource_link` becomes a text link; a text resource becomes a URI link + plus a `` text block; a blob resource with an image MIME type becomes a native + image input; any **other** blob resource becomes a text block that contains the raw base64 inside + ``; and an audio block is dropped (the case + returns null). +- The consequence for documents differs from both other adapters. A PDF sent as a blob is neither + dropped nor reduced to a byte count; its base64 characters are pasted into the prompt as text. The + model receives characters, not a parsed document, so this is still not native document perception, + and it spends prompt tokens on base64. + +Newer versions: `1.1.7` is the latest published version as of 2026-07-31 +(`npm view @agentclientprotocol/codex-acp version`), so the pin and the latest are the same. + +### What the adapters support today + +Three consequences follow, stated plainly. + +- Image delivery works end to end on all three adapters. An inline ACP image reaches the model as a + real image it perceives. +- Native audio is unsupported by all three adapters today. None advertises the audio capability, so + there is nothing to deliver audio into. +- Native document delivery does not work today. The Claude adapter drops a blob resource entirely, + the Pi adapter renders it as a byte count, and the Codex adapter pastes its base64 into the prompt + as text. A PDF sent as a blob reaches none of the models as a document. + +Resource links are handled as pointers by all three. The agent may or may not fetch them. + +At the protocol level, ACP itself has no document content type; the `ContentBlock` union is exactly +text, image, audio, resource link, and embedded resource (`@agentclientprotocol/sdk@1.3.0`, +`dist/schema/types.gen.d.ts`), so a PDF can only ever travel as a blob resource. No spec issue or +discussion proposes a document type (searches of `agentclientprotocol/agent-client-protocol` for +"pdf", "blob resource", and "document" on 2026-07-31 found nothing on point). The one audio-related +thread, [ACP discussion #1591](https://github.com/orgs/agentclientprotocol/discussions/1591), asks +what channel clients should do with media agents cannot ingest and was open and unanswered as of +2026-07-31. So richer-media handling is an open question at the protocol level, and no adapter has +upstream work in flight to close these gaps. + +One note for [design.md](design.md): the D5 adapter-fidelity table describes two document behaviors +(Claude drops the blob, Pi renders a byte count). The Codex adapter adds a third behavior, base64 +pasted as text. None of the three is native document perception, so the D5 conclusion that document +delivery is blocked on adapter work stands unchanged; the table gains a column when PR #5509 lands. + +### What the harnesses' own read tools deliver from disk + +A file on disk is not invisible to the model. Each harness ships a tool that can turn a file the +agent reads into native model input, with different coverage per kind, verified against the pinned +versions on 2026-07-31, empirically on our stack for Claude and Pi (sessions `ebef4364` and +`54770cc0`) and from source for Codex. + +- **Claude Code reads images and PDFs natively.** Its Read tool returns an image file "as visual + content that Claude can see, not as raw bytes" + ([Claude Code tools reference](https://code.claude.com/docs/en/tools-reference#read-tool-behavior)) + and returns a PDF as a native document input (PDF support since v1.0.58; page ranges since + v2.1.30; limits 100 pages and 20 megabytes). This rests on an API fact: a `tool_result` may carry + `text`, `image`, and `document` blocks + ([handle tool calls](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls)). + On our stack (Claude Code 2.1.205 through the pinned adapter), an agent asked to read an uploaded + photo named its exact shapes and text, and read a compression-locked PDF verbatim, with no bytes + in the prompt. An earlier version of this document cited + [claude-code #35866](https://github.com/anthropics/claude-code/issues/35866) and + [#30925](https://github.com/anthropics/claude-code/issues/30925) as evidence that Read does not + deliver vision. Both are now closed, and neither supports that reading today: #30925 described + behavior that no longer exists, and #35866 concedes that Read sends the image and complains about + edge cases (delivery on Bedrock, media type detected from the file extension, corrupt files + breaking a session), several of which have since shipped fixes (format detection from bytes in + 2.0.65, a text fallback for mislabeled files in 2.1.144). +- **Pi reads images natively and nothing else.** Its read tool (pi-coding-agent 0.80.6, + `dist/core/tools/read.js`) detects jpg, png, gif, webp, and bmp by magic bytes and returns a real + image content block, omitted with a notice when the model has no vision. A PDF falls into the + text branch and reaches the model as a UTF-8 decode of raw bytes; on our stack the agent reported + the raw PDF source and could not recover the document's text. +- **Codex reads images natively through `view_image`.** The tool is enabled by default and returns + the file as an `input_image` content item (openai/codex, + `codex-rs/core/src/tools/handlers/view_image.rs`; shipped in rust-v0.26.0). No tool delivers a + local PDF natively; Codex's own skills shell out to converters. +- **No harness has any tool path for audio.** The Anthropic API accepts no audio input at all, Pi's + read tool has no audio branch, and every shipped Codex model declares text and image only. + +So the disk path does give perception for images on every harness, and for PDFs on Claude only, but +always at the agent's discretion: nothing forces the tool call, and a conversation rebuilt from +records does not re-read files. Section 5 weighs this against inline delivery. + +### Server-side transcription, surveyed and deferred + +Decision D14 ([design.md](design.md)) defers server-side transcription of recorded audio out of the +first release. This survey (2026-07-31) records what the deferred follow-up restarts from, so that +decision begins from facts rather than from scratch. + +- **The browser cannot fill the gap.** The Web Speech API transcribes only a live microphone + stream, which is why the composer's dictation mode works and a recorded blob cannot be + transcribed client-side. Chrome 139 can replay a blob through recognition, but that path is + Chrome-only, runs at real-time speed, and produces no server-side transcript. The limit is + structural, so recorded audio needs a server-side service or none at all. +- **The call path already exists in our dependencies.** The pinned litellm 1.92.0 (`api/uv.lock`, + `services/uv.lock`) ships `/audio/transcriptions` support for OpenAI, Mistral, and Deepgram, plus + Groq through its OpenAI-compatible route. So a person's stored OpenAI, Groq, or Mistral key could + transcribe through infrastructure Agenta already runs, and the same call path could carry an + Agenta platform key for people without keys. +- **Prices are negligible.** A 30-second note costs about $0.003 on OpenAI's transcription models, + about $0.0003 on Groq's Whisper large-v3-turbo ($0.04 per hour), and about $0.0015 on Mistral's + Voxtral Mini Transcribe. +- **Subscription-auth users have no bring-your-own path.** Anthropic has no transcription API, and + neither does the Codex OAuth surface, so a person authenticated through a Claude or ChatGPT + subscription holds no key that could transcribe. +- **Self-hosters could run it locally.** faster-whisper's small model in the services layer + transcribes a 30-second note in about 4 seconds of CPU. +- **Gaps at this pin.** AssemblyAI has no litellm support, and Gemini cannot route as transcription + at litellm 1.92.0. + +--- + +## 5. Two ways to deliver a file to the model, compared + +The design has to choose how the file reaches the model. There are two mechanisms, and they are not +interchangeable. This comparison answers the question of why the design delivers inline ACP content +rather than only writing the file into the sandbox and mentioning its path. + +**Mechanism A: inline ACP content blocks.** The runner reads the file and puts its bytes into an +ACP image, audio, or embedded-resource block in the turn. The harness maps that to the model's +native content path. + +- Gives the model real perception of the file. For images this is confirmed to work through both + pinned adapters (see section 4). +- Requires the bytes at prompt time, and requires the harness to advertise the matching capability. +- Is the only path that could ever work for audio, because there is no tool-read fallback for + audio at all. Note that neither pinned adapter supports native audio today (section 4), so audio + is blocked on adapter work even though inline delivery is its only possible route. + +**Mechanism B: write the file into the sandbox and mention its path in the prompt text.** The runner +writes the file into the agent's working directory and the prompt says something like "the file is +at attachments/photo.png." + +- Lets the agent's tools open, convert, or edit the file. This is real value for non-native + formats. +- Gives the model perception only when the agent chooses to call its read tool, and only for the + kinds that harness's tool handles: images on all three harnesses, PDFs on Claude Code only, audio + nowhere (section 4). The perception is real when it happens, but it is agent-mediated and uneven + across harnesses, and a turn replayed from records does not repeat it. +- Is the natural path for formats that are not native model inputs anyway (a zip, a CAD file), where + perception is not even the goal. + +The conclusion the design draws: use both, for different purposes. Inline delivery (Mechanism A) is +the only path that guarantees perception: it happens deterministically at the prompt-build seam the +runner owns, works the same on every harness that advertises the capability, and survives a replay. +The disk copy (Mechanism B) serves tool use, and on some harnesses it adds a second, agent-driven +route to perception, which is a useful fallback rather than a substitute. This is not redundant. The +two mechanisms serve the two separate outcomes from [context.md](context.md): perception and tool +use. + +--- + +## 6. How other tools handle attachments + +Looking at how mature tools handle this gives us patterns for the flow and for who owns each step. + +### Zed + +Zed's editor is the client in ACP. When a person attaches or pastes an image in Zed's agent panel, +Zed sends it as an ACP image content block, which the Claude Code adapter turns into a native Claude +image block. File mentions become resource references. So Zed's own behavior confirms the inline +content path for images. +[Zed on Claude Code via ACP](https://zed.dev/blog/claude-code-via-acp). + +### opencode + +opencode is a terminal-first coding agent. Its attachment behavior +([opencode attachments docs](https://v2.opencode.ai/attachments)): + +- **How files are attached.** By an attach action, by paste, or by drag and drop. In the terminal a + person references a file with `@filename`, which does a fuzzy search in the working directory and + inserts the file's content into the conversation. +- **How they reach the model.** Inline in the prompt, not as a path reference. Text files are decoded + and inserted as text. Images (PNG, JPEG, GIF, WebP) are passed as the provider's native image + input. SVG is treated as text. PDF, audio, and video are not sent to the model at all. +- **Where the bytes live.** opencode does not describe durable storage. Files are materialized at + prompt time from the path or the inline data URL. +- **Who owns what.** The client collects the file and applies a client-side size limit (up to 20 + mebibytes). The server validates the path and enforces a decoded-size limit per attachment. The + model provider applies its own format and size limits. + +The patterns worth taking from opencode: attachments are delivered inline to the model; only a small +set of image formats get native handling; everything else is text or is refused; and there is a clear +split where the client collects and pre-limits, the server validates, and the provider has the final +say on limits. + +### Others + +Claude Code supports `@`-file mentions that pull a file's content into the context, similar to +opencode. The general industry pattern across these tools is: images go as native image content; +plain text and code are inlined as text; unusual binaries are either refused or handled as files an +agent tool must open. None of them rely on handing the model a bare link and trusting it to fetch. + +--- + +## 7. Records and replay + +### What exists + +Our system already keeps a durable, per-session log called **records**. The runner posts every +agent event to the API's record-ingest endpoint, independent of whether any browser is listening +(`services/runner/src/sessions/persist.ts`, whose header comment describes the "producer-driven" +model and the `POST /sessions/records/ingest` endpoint). Each record is tagged with a +`record_source` of `"agent"` for engine events or `"user"` for the inbound user turn. The API has a +records sub-router with an ingest route and a query route (`RecordsRouter` at +`api/oss/src/apis/fastapi/sessions/router.py:435`; the file header lists +`POST /sessions/records/query`). + +Separately, when the runner starts a session that is not already warm, it rebuilds the conversation +from the durable records (`services/runner/src/engines/sandbox_agent/reconstruct-history.ts`, which +calls the fold in `services/runner/src/sessions/reconstruct.ts`) and then flattens it to text for +the harness (`transcript.ts`). This "cold replay" renders an image block as the literal +`"[image]"`, and a past user attachment does not even get that far: the records store only the +text, so the rebuilt user turn has no image block at all. + +### Records are text-only today + +The record path carries text and nothing else right now. The runner persists the inbound user +record as text (`services/runner/src/server.ts`, near line 1084), and the record event schema +allows only the shape `{ type: "message", text }` (`services/runner/src/protocol.ts`, near line +325). There is no field in a record for an attachment. On a warm turn the runner sends only the +latest user text; on a cold start it rebuilds prior turns from the records +(`reconstruct-history.ts`) and flattens the whole transcript to text (`run-turn.ts`, near line +196), where the rebuilt user turn carries no trace of the attachment at all. + +Two things follow for this project. First, carrying an attachment reference in a record requires +extending the record schema, because the schema has no place to put one today. Second, cold replay +loses past attachments entirely: a user turn rebuilt from records is text only, so the attachment +does not even appear as a placeholder. This is why the plan makes the record-schema change part of the first +release rather than a later addition: without it, a reference cannot survive in the durable log at +all. + +### The direction, now shipped + +The session-storage rework shipped and is on by default (PR #5560, merged 2026-07-30). The front +end sends only the last message on each turn instead of resending the whole history +(`NEXT_PUBLIC_SESSIONS_LAST_MESSAGE_ONLY`, read in `agentRequest.ts`). The runner owns rebuilding +the conversation from records when a session cannot be resumed warm, for example after a crash +(`AGENTA_SESSIONS_RECONSTRUCT`, +`services/runner/src/engines/sandbox_agent/reconstruct-history.ts`). Responsibility for history +moved from the front end to the runner. The reference-based attachment design fits this cleanly, +because a reference is exactly what a record should store for a shared file: the record points at +the durable original rather than carrying its bytes. + +### The tracking issue + +This direction was tracked in +[#5443, "(feat) Runner should rebuild session context from records when a session cannot be resumed warm"](https://github.com/Agenta-AI/agenta/issues/5443) +and shipped with PR #5560. +The related warm case is +[#5384, "(feat) Warm sessions should hold client tool calls open instead of replaying the turn"](https://github.com/Agenta-AI/agenta/issues/5384): +issue #5384 avoids a replay when the session is still alive, while the rebuild from records covers +the case when it is not. The rebuild itself is outside this project's scope; what this project +contributes is that records will carry small references to durable originals, which is what a +faithful rebuild needs. + +--- + +## 8. Assistant-produced files + +The runner protocol declares an event type for a file the agent produces: +`{ type: "file"; url: string; mediaType: string }` (`protocol.ts:379`), which the comment says maps +to a Vercel `file` part on the way to the front end. But **nothing emits it.** A search across the +runner source for that event type finds only its declaration, no producer. The protocol defines the +event, but no runner code emits it. + +How this normally works elsewhere: an agent writes the file into its working directory and +mentions it in its answer, but neither Zed nor opencode recognizes file references by scanning the +prose or the working directory. The affordance always rides structured data on the wire: ACP +tool-call `locations` entries carrying absolute paths, `resource_link` blocks the agent emits, or +(in Zed only) a strict existence check on backticked code spans against the worktree, with no +fuzzy matching. opencode renders no file links from agent text at all; every file affordance there +comes from tool-call inputs. For our product this means assistant-produced files are a +separate piece of work: making the agent write to `cwd` and surfacing the new file in the drawer is +already close to working (the drawer lists `cwd`), while a first-class "assistant returned this file" +inline chip is the not-yet-built part. This project treats assistant-produced files as out of scope +(see [scope.md](scope.md)). + +--- + +## 9. The capability flags + +The neutral capability set, including `images` and `file_attachments`, was introduced with the +original agent-runtime protocol work (`sdks/python/agenta/sdk/agents/dtos.py`, class +`HarnessCapabilities`; introduced in commit `b9e62f99aa`, "feat(sdk): agent runtime ports, adapters, +tool resolution, and messages protocol"). The matching TypeScript shape is `HarnessCapabilities` in +`protocol.ts:281`. + +The intent is documented right in the code. The docstring on the Python `HarnessCapabilities` says +adapters should "deliver tools over MCP only when `mcp_tools` is set, skip image blocks without +`images`" (`dtos.py`). The tool half of that intent was built: `assertRequiredCapabilities` +(`capabilities.ts:169`) refuses a run that carries tools when the harness lacks `mcpTools` and +`toolCalls`. The image half was not built: there is no code that skips or gates image blocks on the +`images` flag, because no code sends image blocks at all yet (section 1). So the flags are not so +much abandoned as built ahead of the feature that would use them. This project is that feature. It +should reuse the same "fail loud when a required capability is missing" pattern the tool gate already +uses, rather than invent a new one. + +Note the difference between two capability vocabularies that are easy to confuse: + +- **ACP prompt capabilities:** `image`, `audio`, `embeddedContext`. External. Advertised by the + harness. These gate ACP content types. +- **Our harness capability flags:** `images`, `fileAttachments`, and the rest in + `HarnessCapabilities`. Ours. Derived from the probe. The design maps the ACP prompt capabilities + into a clear internal set of `images`, `audio`, and `documents` and gates on that. The mapping and + the exact contract across layers are in [design.md](design.md) and [plan.md](plan.md). + +--- + +## 10. The front-end adapter and its current limits + +### What the adapter parses + +Our SDK's Vercel adapter turns an inbound `file` part into a content block +(`sdks/python/agenta/sdk/agents/adapters/vercel/messages.py:97`). The rule is: if the media type +starts with `image/`, the block type is `image`; otherwise it is `resource`. The block carries the +`uri` (from the part's `url`), the `data` (base64, if present), and the `mime_type`. On the way back +out, both `image` and `resource` blocks render as a Vercel `file` part (`_block_to_parts`, +`messages.py:310`). + +So the content-block vocabulary our system produces from the front end is: `text`, `image`, +`resource`, `tool_call`, `tool_result`. There is **no audio block.** Audio, video, and every +document all collapse to `resource` today. Adding a first-class `audio` block is part of this work. + +The front-end helper `fileKind` (`web/oss/src/components/AgentChatSlice/assets/files.ts:11`) already +recognizes image, audio, video, and file, so the front end can tell audio apart even though the wire +cannot carry it as audio yet. + +### The current limits and where they came from + +The front-end attachment limits live in +`web/oss/src/components/AgentChatSlice/assets/attachments.ts` (`DEFAULT_ATTACHMENT_LIMITS`, +verified 2026-07-31, merged in PR #5459): + +- At most 5 files per message, across all kinds. +- Per-kind size caps: 10 megabytes per image, 15 megabytes per audio clip, 10 megabytes per + document. +- Accepted kinds and types: image (`image/`), audio (`audio/`), document (`application/pdf`, + `text/`, `application/json`). + +The limits are one value object so they can later be replaced with limits derived from the selected +model and harness capabilities; the file's own comment names narrowing `kinds` as the seam that +capability gating plugs into. The numbers are sized against the transport, not against any model +limit: the file's header notes that while attachments ride the request body, the real ceiling is the +gateway's `client_max_body_size` (10 megabytes on the compose stack) plus base64 inflation of about +a third, which also means the 15 megabyte audio cap exceeds what the compose gateway accepts today. +Section 3 has the real provider limits the caps should eventually be derived from. These limits are +enforced client-side only; nothing checks them server-side today. The front end sends files inline +as base64 `data:` URLs (`files.ts`, `fileToPart`), which is exactly the mechanism the +reference-based design replaces. The upload lifecycle hook (`useAttachmentUploads`) ships with a +deliberate transport seam: no uploader is wired, so staged files sit ready-to-send and the +progress, failure, and retry flow activates when Stage 1 provides an uploader. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/scope.md b/docs/design/agent-workflows/projects/agent-multi-modality/scope.md new file mode 100644 index 0000000000..eaf275ea2c --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/scope.md @@ -0,0 +1,101 @@ +# Scope + +## In scope + +This scope implements decision D11 (see [decisions.md](decisions.md)): durable agent input from the +first release. No inline-only version will be built. + +- A person can attach an image, an audio clip, or a document to an agent message. This covers files + shared through the chat composer only: a file dragged into the Files drawer uploads directly to + that mount through the existing drive flow and is not an attachment + ([decisions.md](decisions.md), D13). +- The model perceives images natively (this works end to end on both pinned adapters). Native audio + and native document delivery do not work on the adapters we run today and are blocked on adapter + work (see [research.md](research.md), section 4, and [decisions.md](decisions.md), D8 and the + "Settled by evidence" section). For audio, the first release ships the composer's browser + dictation only (live speech becomes composer text, merged dark in PR #5458); a recorded voice + message or uploaded audio file is a workspace-only attachment with the D6 visible notice, and no + transcript reaches the model ([decisions.md](decisions.md), D14). Server-side transcription is a + deferred follow-up, and native ACP audio delivery is the last upgrade. +- The agent can always work on the file, because a working copy is written into its working directory + regardless of whether the model can perceive the file. +- Every shared file stays findable: the original never changes, renders inline in the conversation, + and can be downloaded as exact bytes. Listing it under a "Shared by you" origin in the Files drawer + is Stage 3 work. +- Files travel as an opaque, server-issued `attachment_id` on the wire, in the saved history, and in + the traces, not as bytes and not as raw storage coordinates ([decisions.md](decisions.md), D10). +- Attaching a file never silently does nothing. An unsupported kind is attached as a workspace-only + file with a visible notice, and the runner fails a turn only on a contract violation, such as a + stale front end asking for a native block the harness cannot accept + ([decisions.md](decisions.md), D6). + +## Out of scope (for now) + +- **Video.** No current model we target except Gemini perceives video, and ACP's content model has no + video type. Video needs new protocol and adapter support and waits until there is a clear need and a + supported harness. +- **Assistant-produced files as a first-class chip.** This project will not add an inline element for + files the agent produces, a first-class "here is a file" chip in its answer. The runner protocol + declares a `file` event type but nothing emits it (see [research.md](research.md), section 8). The + agent can already write files into its working directory and the drawer already lists them; the + missing piece is the inline chip, which is separate work. +- **Cross-session attachment reuse.** This project will not let a person pick a file shared in an + earlier session and reattach it. Attachments are session-scoped, so an "attach from a past session" + picker is a later feature. +- **Deduplication by content.** This project will not store one copy for identical uploads when the + same file is shared twice. This is a storage optimization, not a correctness need, so it waits. +- **Thumbnails and previews generated server-side.** The server will not generate preview images for + large files. This is a polish item that does not block the core flow. +- **Transcoding.** This project will not convert media files between formats before delivery (for + example turning a video into frames); that work belongs with future video support. Server-side + audio transcription is deferred separately ([decisions.md](decisions.md), D14), and when it ships + it will not be transcoding either: the recording stays stored and delivered unchanged, and the + transcript is an addition, not a replacement. +- **Server-side transcription of recorded audio.** Deferred out of the first release by D14: no + transcription endpoint, no key resolver, no platform key, no metering. Listed again under + follow-ups below with the pointer to the survey. + +## Follow-ups + +- **Large pasted text becomes a file.** When a person pastes a very large block of text into the chat + box, it should not be sent through the prompt as text. It should be saved as a file and referenced in + the prompt, the same way an attachment is. Where exactly it is saved is likely answered by the + attachments mount this project builds. This is a natural extension of the reference model and is + listed as a follow-up rather than in the core scope. +- **A clean reference chip in the composer.** When a file is attached, the composer should show the + reference in a clear, readable way (a chip with the filename and kind) rather than a raw preview. + This is a presentation improvement on top of the core flow. +- **A read-only credential scope for the attachments mount.** The strongest form of the immutability + guarantee, noted in [design.md](design.md), decision D7, and [plan.md](plan.md), Stage 3. +- **Front-end limits derived from real model limits.** Replace the static default limits with values + computed from the selected model (see [research.md](research.md), section 3). Started in Stage 2 and + can be refined further as provider limits change. +- **Server-side transcription of recorded audio.** The deferred middle step between browser + dictation and native ACP audio ([decisions.md](decisions.md), D14). The survey it restarts from + (the litellm call path Agenta already runs, prices, who has a usable key) is in + [research.md](research.md), section 4, "Server-side transcription, surveyed and deferred". +- **Per-workspace retention policies.** The settled retention rule ties originals to the session + lifecycle ([decisions.md](decisions.md), "Settled by evidence"). A policy such as "purge + attachments after N days" for compliance customers is a separate feature: it needs a policy + setting, a sweep job, and a story for the holes it leaves in old conversations whose references + no longer resolve. +- **User-initiated purge of a single original.** Removing one wrongly shared file before its + session ends needs a product surface plus a revoked-reference marker in the records, so a replay + cannot re-materialize the purged bytes. +- **Runner-side downscaling of oversized images.** Today an accepted image that exceeds a + provider's inline cap is delivered workspace-only with the D6 notice ([design.md](design.md), + "The media-type, validation, and limits matrix"). Downscaling it to fit the cap would restore + guaranteed perception for those files, at the cost of new image-processing machinery in the + runner. + +## Next steps + +- Implementation starts with Stage 0 or Stage 1 of [plan.md](plan.md); every product decision (D1 + through D14) is taken. +- The document-delivery question is answered: documents do not arrive natively today (the Claude + adapter drops blobs and the Pi adapter renders a byte count), so documents are a Stage 2 blocker on + adapter work, not an open harness check (see [research.md](research.md), section 4). +- The runner-rebuilds-context-from-records direction shipped with the session-storage rework + (PR #5560, tracked as [issue #5443](https://github.com/Agenta-AI/agenta/issues/5443)); the + rebuild itself stays outside this project's scope (see [research.md](research.md), section 7). +- Decide whether Stage 0 ships on its own or folds into Stage 1. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/status.md b/docs/design/agent-workflows/projects/agent-multi-modality/status.md new file mode 100644 index 0000000000..bdada05db7 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/status.md @@ -0,0 +1,87 @@ +# Status + +This file records the project's current state. + +## State + +The design is complete. Every decision, D1 through D14, is taken; the last four (D11 durable agent +input from day one, D12 mention-first cold replay, D13 two upload surfaces, D14 first-release audio +is browser dictation only) were decided on 2026-07-31. What remains open are implementation-time +questions, not product questions; see [decisions.md](decisions.md) for the list. The next step is +implementation, starting with Stage 0 or Stage 1 of [plan.md](plan.md). + +The one-line problem: agent workflows are text-only at the model. A file travels intact from the +chat box to the runner and is dropped at the one call that hands a turn to the harness +(`services/runner/src/engines/sandbox_agent/run-turn.ts`, the `session.prompt` call, currently near +line 742). The image fix is entirely on our side of that call, because both pinned adapters deliver +an image natively. Native audio and native documents are different: neither pinned adapter supports +them today, so those native paths are blocked on adapter work. For audio the first release ships +the composer's browser dictation only; a recorded clip is a workspace-only attachment with a +visible notice, and server-side transcription is deferred (decision D14). + +The front-end surface is already merged dark (PRs #5458 and #5459, merged 2026-07-29): the composer +attach UI, the attachment chips and preview, the per-kind limits object, the upload lifecycle hook +with its unwired transport seam, the voice capture UI, and the drive-upload surfaces, behind +`NEXT_PUBLIC_AGENT_FILE_UPLOADS` and `NEXT_PUBLIC_AGENT_VOICE_INPUT` (both off by default). Paste +and drag-to-attach predate the flags and are live ungated, which is the Stage 0 gap. The backend +(upload route, attachment resource, record schema, runner delivery) has not started; see "What is +already built (merged dark)" in [plan.md](plan.md). + +## Reading order + +See [README.md](README.md). In short: [context.md](context.md) for the plain story, +[research.md](research.md) for the findings, [design.md](design.md) for the design and its options, +[plan.md](plan.md) for the staged work, [scope.md](scope.md) for in and out, and +[decisions.md](decisions.md) for the log and open questions. + +## Stage tracker + +| Stage | Scope | State | +| --- | --- | --- | +| 0 | Close the silent-failure gap: gate the ungated paste and drag path on `NEXT_PUBLIC_AGENT_FILE_UPLOADS` | not started (optional) | +| 1 | First user-visible release: the attachment resource and storage, the record-schema extension, the runner's resolve-materialize-and-deliver seam for images, structured capability errors, the minimum security and limits work (per the settled matrix in [design.md](design.md), including the gateway raise to 32 MB), and the front-end transport and reference wiring | not started | +| 2 | The audio release: turn on the voice UI, dictation as the only audio-to-text, recordings on the D6 workspace-only path (D14); the document plan (blocked on adapter work), the capability-alias rollout, derived front-end limits | not started (documents blocked on adapter work) | +| 3 | Findability polish and cleanup: "Shared by you" origin, reference-counting cleanup refinement, read-only credential scope, verify the edit-then-find flow | not started | + +## Decisions taken + +See the decision log in [decisions.md](decisions.md) (D1 through D14, all decided). In brief: +deliver inline for perception and on disk for tool use; keep the original in a dedicated session +mount out of the sandbox; two copies, an unchanging original and a disposable working copy; compute +the capability as the intersection of transport, adapter fidelity, and model, and gate in the +composer and the runner; never silently drop, attach an unsupported kind as workspace-only, and fail +the turn only on a contract violation; enforce immutability through a create-only upload route with +no signed credentials for the mount; carry an opaque server-issued `attachment_id` on the wire +(D10); the first release promises durable agent input, not image perception alone (D11); cold replay +is mention-first (D12): a historical attachment is a textual mention with its real working-copy +path, every referenced working copy is restored at cold start, and inline delivery is reserved for +the running turn; drawer uploads and composer attachments are deliberately different pipelines +(D13); and first-release audio is browser dictation only, with recordings workspace-only under the +D6 notice, server-side transcription deferred, and native ACP audio as the last upgrade (D14). + +## Open questions + +See [decisions.md](decisions.md). All product decisions are taken. One implementation question +remains open: when cleanup moves from a time-to-live sweep to reference counting. Two former open +questions were settled by evidence on 2026-07-31 and moved to "Settled by evidence" in +[decisions.md](decisions.md): the media-type, validation, and limits matrix (the full matrix is in +[design.md](design.md), "The media-type, validation, and limits matrix", including the compose +gateway raise from 10 MB to 32 MB) and the retention rules (originals share the session lifecycle: +deleted with the session, kept while it exists, including archived). The capability-alias removal +is deliberately not on this list: it is rollout mechanics with a stated settle condition, recorded +in [plan.md](plan.md) Stage 2. + +## Next actions + +- Start implementation: Stage 1 of [plan.md](plan.md), optionally preceded by Stage 0. +- Decide whether Stage 0 ships on its own or folds into Stage 1. +- Build the Stage 1 upload route against the settled matrix ([design.md](design.md), "The + media-type, validation, and limits matrix"), including the compose gateway raise to 32 MB. + +## Artifacts + +- [README.md](README.md) · [context.md](context.md) · [research.md](research.md) · + [design.md](design.md) · [plan.md](plan.md) · [scope.md](scope.md) · [decisions.md](decisions.md) +- Branch: `docs/agent-multi-modality` (docs only). +- Merged front-end groundwork: PR #5458 (voice input), PR #5459 (attachments and drive uploads), + both dark behind flags.