[docs] Plan agent multi-modality - #5439
Conversation
dosubot
Bot
added
the
size:XL
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds documentation for agent multi-modality attachments. It covers current limitations, target storage and delivery architecture, staged implementation, capability constraints, decisions, status, scope, and validation requirements. ChangesAgent multi-modality design
Estimated code review effort: 2 (Simple) | ~15 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6e6cffaa-1cf6-4746-a5d0-51e72e498e22
📒 Files selected for processing (4)
docs/design/agent-workflows/projects/agent-multi-modality/README.mddocs/design/agent-workflows/projects/agent-multi-modality/context.mddocs/design/agent-workflows/projects/agent-multi-modality/proposal.mddocs/design/agent-workflows/projects/agent-multi-modality/status.md
| Key change is step 12–13: the runner resolves the reference and emits **real ACP blocks** | ||
| instead of `[{type:"text", text: turnText}]`. Steps 8–10 (materialize working copy) run once per | ||
| session per file — `cwd` is durable across turns, so later turns skip it. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Define re-materialization after working-copy deletion.
The design allows the agent to edit or delete the working copy, but later turns “skip” materialization unconditionally. Reusing the reference after deletion can therefore leave the agent without the file. Specify an existence check and rehydration policy, while ensuring existing edits are not overwritten unexpectedly.
| subgraph run["runner enforcement (defense in depth)"] | ||
| F --> J{capability present at prompt build?} | ||
| G --> J | ||
| H --> J | ||
| J -->|yes| K["emit ACP block"] | ||
| J -->|no| L["drop with explicit warning event<br/>(never silent)"] | ||
| end |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Reject unsupported modalities instead of dropping them.
“Drop with an explicit warning” still lets the run continue without content the user attached, violating the stated “never silently dropped” and honest-capability goals. Fail the turn with a user-visible structured capability error, rather than invoking the model with a partial prompt.
|
|
||
| Three concrete failure modes: | ||
|
|
||
| 1. **Image + text** → the model gets only the text. On cold replay it gets the literal string |
There was a problem hiding this comment.
pleaes let us not limit ourselves to image in this research. let us be as general as we can. images, zips, cad files, excel files, folders, videos.. different sizes etc..
There was a problem hiding this comment.
🤖 The AI agent says:
Done. research.md section 3 now covers images, audio, video, PDFs, office documents (Excel, Word), plain text and code, CSV and JSON, zip archives, CAD and other binary formats, folders, and very large files.
The organizing idea that came out of it: a model natively perceives only a small set of kinds (images, audio for some models, documents for some, video only for Gemini). Everything else is a file the agent opens with tools, sometimes after converting it (a spreadsheet through a library, a zip by extracting). The design therefore puts every attached file on the agent's disk for tool use, and additionally delivers the native kinds to the model inline when the model supports them.
| Capability flags were designed and abandoned mid-flight: | ||
|
|
||
| - Agenta flags exist in TS ([`protocol.ts:283`](../../../../../services/runner/src/protocol.ts) — | ||
| `images?`, `fileAttachments?`) and Python ([`dtos.py:187`](../../../../../sdks/python/agenta/sdk/agents/dtos.py)). |
There was a problem hiding this comment.
why do we have two types only / imges and fileattachements
There was a problem hiding this comment.
🤖 The AI agent says:
Because the flags were built ahead of the feature that would use them. The whole capability set, including images and file_attachments, landed with the original agent-runtime protocol commit (b9e62f99aa). The docstring even says adapters should "skip image blocks without images", but that skip was never built, because nothing sends image blocks yet. Only the tool flags (mcp_tools, tool_calls) ever got a real gate.
Changed in the PR: research.md section 9 has this history, and the design replaces the two-flag set with a layered capability model (protocol, adapter, model) described in design.md.
| - Agenta flags exist in TS ([`protocol.ts:283`](../../../../../services/runner/src/protocol.ts) — | ||
| `images?`, `fileAttachments?`) and Python ([`dtos.py:187`](../../../../../sdks/python/agenta/sdk/agents/dtos.py)). | ||
| - `dtos.py:187` documents the intent — adapters *"skip image blocks without `images`."* | ||
| **No such skip exists.** `capabilities.images` is assigned once at |
There was a problem hiding this comment.
again why capabilities are abut images only
There was a problem hiding this comment.
🤖 The AI agent says:
Same root cause as the previous comment: the runner probes the harness, maps only images and fileAttachments out of the ACP capability set (image, audio, embeddedContext), and then never reads them anywhere.
Changed in the PR: the design now requires the runner to map all three ACP capabilities, and treats the effective capability for a modality as the intersection of three layers: what the protocol can carry, what the harness adapter actually delivers natively, and what the selected model perceives. The layered contract table is in design.md.
| | Type | Owner | Modalities | Where | | ||
| | --- | --- | --- | --- | | ||
| | ACP `ContentBlock` | Zed (external) | text, image, **audio**, resource_link, resource | runner → harness ([`run-turn.ts:478`](../../../../../services/runner/src/engines/sandbox_agent/run-turn.ts)) | | ||
| | Agenta `ContentBlock` | ours | text, image, resource — **no audio** | FE → runner ([`protocol.ts:12`](../../../../../services/runner/src/protocol.ts), [`dtos.py:229`](../../../../../sdks/python/agenta/sdk/agents/dtos.py)) | |
There was a problem hiding this comment.
what is a ressource / ressource_link
why ours have no audio.
please update this whole thing and explain the different modalities and how they are usually handled. which ones are special which one are just files
There was a problem hiding this comment.
special as in native and require special path
There was a problem hiding this comment.
🤖 The AI agent says:
Definitions first. An embedded resource carries the file's bytes inline in the message (as text or base64), so the harness can hand the content to the model. A resource_link is only a pointer (a URI and a name); the agent may fetch it with a tool, but the model is never guaranteed to see it. Our content block simply has no audio variant today (audio collapses into resource); the plan adds one, because ACP represents audio as its own content type and there is no tool-read fallback for it.
Changed in the PR: research.md section 3 is now a full taxonomy of modalities: the native kinds, where the model's own encoder must receive the bytes through message content (images, audio, documents for some models, video for Gemini), versus everything else, which is just a file the agent opens with tools. Section 4 explains each ACP content type and what our two pinned harness adapters actually do with each one.
There was a problem hiding this comment.
🤖 The AI agent says:
Exactly. "Special" means native: the model's own encoder must receive the bytes through the message content, and no tool that reads a file from disk can substitute for that path. research.md section 3 uses this as the dividing line: native kinds need the dedicated delivery path, everything else is a normal file for the agent's tools.
| 2. **Agent operates on it.** The file is on the agent's filesystem so its tools can read, | ||
| transform, or edit it — whatever the conversation calls for. | ||
| 3. **Always findable.** Every file a user shares is a durable, immutable record surfaced inline | ||
| in the conversation and under a "Shared by you" origin in the Files drawer — never lost, even |
There was a problem hiding this comment.
not sure about never lost. if the file is uploaded this means it can be deleted
There was a problem hiding this comment.
🤖 The AI agent says:
You are right; the wording overclaimed. The guarantee is now stated precisely: the agent cannot change or delete the original, because the attachments mount is never exposed in the sandbox and originals only enter through a create-only upload route (the API refuses overwrite and delete for originals, and no signed storage credentials are issued for that mount). It is not immortality: Stage 1 defines cleanup for uploads that were never sent, and what happens on session archive or delete is an open retention question in decisions.md. design.md decision D7.
| after the agent runs. | ||
| 4. **Honest capability.** Unsupported modalities are refused at the composer, driven by the ACP | ||
| `promptCapabilities` the runner receives — never silently dropped. | ||
| 5. **No base64 on the hot path.** The wire, the persisted history, and traces carry a small |
There was a problem hiding this comment.
please be very precise here. what do you mean by hot path, what do you mean by history. to my knowledge we dont have an entity called history. and why do we care about the wire between the service and the runner not having base64, what is the value there?
There was a problem hiding this comment.
🤖 The AI agent says:
Rewritten without those words. Precisely: the wire is the request from the front end through the API to the runner. There is no entity called "history"; the two real things are the front end's saved message store (browser local storage) and the durable per-session records kept by the API. The docs now name those.
Why carrying base64 there hurts: the front end currently resends the whole conversation on every turn, so the same image bytes travel again on every later turn (the cost grows with the square of the conversation length); the saved bytes overflow browser storage and evict other sessions' chats; and the bytes land verbatim in traces. A small reference removes all three. That is the value, and it is separate from the model boundary, which still needs inline bytes at prompt time. research.md section 1.
| ## Non-goals (for now) | ||
|
|
||
| - **Video.** Absent from ACP's content model and from every model's metadata; net-new, deferred. | ||
| - **Assistant-emitted files.** A runner `file` run-event type is declared but has no emitter |
There was a problem hiding this comment.
can you explain in the research what are these assistant emitted files. how the flow look like there. where is this even used. i would expect an agent to just write the files to the mount and reference them in its answer (by the way how does the frontend know when to show a link to a file [in agenta but also in zed])
There was a problem hiding this comment.
🤖 The AI agent says:
research.md section 8 covers this. The runner protocol declares a file event ({type: "file", url, mediaType}), but no runner code emits it. Your expectation matches how Zed and opencode actually behave: the agent writes files into its working directory and mentions them in its answer, and the client shows the file because it can see the working directory, not because of a protocol event. In Agenta the drawer already lists cwd, so an agent-produced file is already findable there; the missing piece is an inline "here is the file" chip in the answer, which is separate work and listed in scope.md as out of scope here.
There was a problem hiding this comment.
can you please research how does zed and opencode do the mentions. I have noticed that sometimes the agent just sayys the name of the file but not the full path, yet zed seems to find that (at least I think so). How does that work.?
There was a problem hiding this comment.
🤖 The AI agent says:
Short answer: Zed does not find the file from a bare name. Nothing in Zed scans the assistant's prose for filenames. What you saw clickable was carrying a full absolute path underneath, you just could not see it. I read both codebases (Zed @ 5e1fd392f67e, opencode @ 32f278b48f1a) and the Claude Code ACP adapter (@agentclientprotocol/claude-agent-acp@0.58.1).
Zed. Three things make a file reference clickable, and only one of them is client-side guessing:
- Tool-call locations. Every ACP tool call can carry
locations: [{path, line}]. The Claude Code adapter fills that straight from the tool's own argument (dist/tools.js:65,102,123,139), and Claude'sfile_pathis always absolute. Zed refuses to resolve a non-absolute one (acp_thread.rs:1128-1159) and, when a call has exactly one location, makes the whole row a "go to file" target (thread_view.rs:10046-10081). The row shows a short name because Zed shortens it for display; the JSON says/abs/path/src/main.rs. This is almost certainly what you noticed. - Links the agent wrote. A
resource_linkblock becomes[@name](file:///abs/path)markdown (acp_thread.rs:1699-1704), and clicks run throughopen_link, which acceptsfile://URIs, bare absolute paths, andpath:12:5(thread_view.rs:12471-12613). - Backticked code spans, resolved strictly. If the agent writes
`src/main.rs:42`, Zed tries to resolve it (conversation_view.rs:3475-3640). This is a strict existence check, not a search: it tries the literal relative path and the path with the worktree root name stripped, and the entry must exist. There is no fuzzy or basename matching; Zed's fuzzy matcher is used only in the@-mention picker. So a backtickedmain.rsresolves only whenmain.rssits at a worktree root; if the file lives atsrc/main.rs, the bare name does not resolve. Text outside backticks never resolves at all.
opencode does less. It has no path detection anywhere. A backticked path-looking span gets a color and nothing more (markdown-inline-code-kind.ts, consumed only by CSS). Every file affordance comes from tool inputs (message-part.tsx:468-545). Its system prompt even asks the model to write file_path:line_number (prompt/default.txt:88-95) and nothing in the UI reads it.
User-side mentions, since it is the mirror image. Zed sends @file as an absolute file:///abs/path: a resource_link when the agent does not advertise embeddedContext, an embedded resource with the contents inlined when it does (message_editor.rs:2138). Directories go as a link only, so the agent reads them itself. opencode sends a file part with an absolute file:// URL plus a relative display name, leaves the literal @src/main.ts in the text, and then runs the read tool server-side and injects the contents as text (session/prompt.ts:808-903). Its @ picker is fuzzy, but the fuzziness happens at pick time, never at render time.
What this means for us. Neither product guesses at filenames, and we should not either. The pattern both converge on is to carry the path as structured data alongside the text, in both directions. Three concrete consequences. First, for user mentions, keep the resolved path and the character span on the part, the way opencode does, so rendering and replay never re-parse prose; that fits the mention-first replay decision in D12. Second, on the agent side, the affordance we want is already on the wire and we discard it: ACP tool calls carry locations with absolute paths, and grepping locations in services/runner/src returns nothing. Forwarding that field into our event stream is a small change that buys exactly what Zed has. Third, if we ever want more than that, copy Zed's code-span resolver rather than a fuzzy matcher: backticked spans only, strict existence check against the working directory, plain text when it does not resolve. Guessing produces dead links, which are worse than no link.
One correction that came out of this: research.md §8 claimed Zed and opencode recognize agent-written files "by reading the working directory". That was wrong and is now fixed in the doc.
|
|
||
| The wire stops carrying base64. A shared file becomes a small **attachment reference** | ||
| `{ mount_id, path, mediaType, filename, size }`. Behind that reference live **two objects with | ||
| different rights**, which is what lets "the user always finds it" and "the agent can do anything |
There was a problem hiding this comment.
what does the different right even mean
There was a problem hiding this comment.
🤖 The AI agent says:
Removed the phrase. The concrete difference: the original cannot be changed by the agent (it is outside the sandbox and only enters through a create-only API route), and the working copy can be changed freely (it sits in the working directory, which the agent fully controls). design.md decision D4 states it this way.
|
|
||
| ref -.addresses.-> orig | ||
| orig ==>|runner reads → base64| acp["ACP image / audio / document block<br/>→ model perceives"] | ||
| orig ==>|runner materializes copy| cwd |
There was a problem hiding this comment.
what does materializes copy here mean
There was a problem hiding this comment.
🤖 The AI agent says:
Defined in the README glossary and used consistently: to materialize means the runner writes a copy of the stored original into the agent's working directory (at cwd/attachments/<attachment_id>/<filename>) so the agent's tools can open it. Re-materialization only restores a missing copy; it never overwrites one the agent has edited.
| orig ==>|runner reads → base64| acp["ACP image / audio / document block<br/>→ model perceives"] | ||
| orig ==>|runner materializes copy| cwd | ||
| cwd ==>|Read / Bash / edit| tools["agent tools operate"] | ||
| orig ==>|render + download| ui["conversation + Shared-files drawer"] |
There was a problem hiding this comment.
what does render + download means
There was a problem hiding this comment.
🤖 The AI agent says:
Plain version: render means the conversation shows the file inline where it was sent (the front end resolves the reference to a download URL and displays it); download means the person gets the exact original bytes back from the download endpoint. Both are backed by the unchanging original. design.md, the findability section.
Design workspace for multi-modal agent input (images, audio, documents): the model perceives the content and the agent can operate on it, with every shared file a durable, findable record. Docs only. README, context (verified current-state audit), proposal (design + 8 interaction diagrams + decision log), status.
…ter review Restructure into the plan-feature layout (context, research, design, plan, scope, decisions, status; proposal.md absorbed), answer every review comment, verify the pinned adapter behavior (images native on both; audio and blob documents undelivered today), switch the wire reference to a server-issued attachment id, fix the upload flow to create-only API routes, make capability a three-layer intersection, and repackage the plan into one user-visible release with security and limits included. D11 (what the first release promises) is left open for the product owner. Claude-Session: https://claude.ai/code/session_01Jg92VZvmkMnm5VQXprTNEH
…e multi-modality design The docs argued from full-history resending, which PR #5560 removed (last-message wire, runner-owned records, cold-start reconstruction). Restate what the attachment reference still buys: survival of the file in the durable records, browser storage, and traces. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…s, upstream findings, and the Codex adapter Verify the pins, diff modality behavior against the latest published versions (unchanged for both), and add the codex-acp 1.1.7 adapter from PR #5509 with its third document behavior: base64 pasted as text. Record why blobs and audio are unsupported upstream: the Claude adapter chooses to drop blobs (the Anthropic SDK has a document block), Claude audio is a Messages API limit, Pi's RPC takes only message plus images, and ACP itself has no document type. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…D12 mention-first cold replay Live tests on our stack disproved 'Read does not reliably deliver vision': all three harnesses read images natively from disk, Claude reads PDFs too, audio has no path anywhere. D1's rationale becomes guarantee/uniformity/replay instead of blindness. New D12: cold replay mentions historical attachments by filename and working-copy path and restores the copies, reserving inline delivery for the running turn; this replaces the redelivery-budget open question. Also: a Pi extension can add tool-mediated PDF perception (registerTool accepts image content), but inline blobs stay a pi-acp limitation. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…pped frontend, and consolidate D11 decided: durable agent input from day one, no inline-only version. D13: drawer uploads write straight to the mount; the attachment pipeline is composer-only. D14: interim audio is transcription we own, since native audio is infeasible at both the adapter and the Anthropic API layer. Plan re-based on Arda's merged-dark frontend (#5458/#5459) so Stage 1 FE work is wiring, not building. CodeRabbit fixes: D3 now matches D7 (API download route, no signed reads), materialization is restore-when-missing everywhere, upload idempotency and the full three-layer runner gate added, lint and cost-claim fixes. Status and open questions rebuilt: all product decisions taken, five implementation-time questions remain. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…out mechanics It has a stated settle condition (every component speaks the new names, the pip-installed SDK gated by a deprecation window), so it is not an open question needing a decision. Four implementation-time questions remain. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…ver-side transcription Mahmoud's call: v1 audio is the shipped Web Speech dictation and nothing else. Recorded and uploaded audio follow the D6 workspace-only path with the visible notice; the agent's tools deal with the file. No transcription endpoint, key resolver, platform key, or metering in the first release. The full service survey (litellm 1.92.0 call path, prices, the no-key-user constraint, self-hosted whisper) is preserved in research.md so the deferred follow-up restarts from facts. Open questions drop to three. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…n question remains Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
… 1 plan The claim operation replaces download-side referencing, D7 widens to the protected-mount policy, the matrix gains the session quota and the bounded-read rule, D5's unknown modality defaults to workspace-only, and the deployment order is restated reader-first. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…he matrix The product owner raised the per-turn count from the dark-shipped default of 5 to 100, and the per-session count quota from 100 to 1,000 so one full turn cannot consume it; the byte quota stays the real bound. Claude-Session: https://claude.ai/code/session_01A1XQVjHPYJgVBHWSNUphtx
…esolution research
5cd13b4 to
9528c48
Compare
Railway Preview Environment
Updated at 2026-08-01T18:07:51.074Z |
2 participants
Context
Agent workflows are text-only at the model. A person can attach an image or a document in the agent chat, the bytes travel intact from the browser through the API and SDK to the runner, and the runner then drops everything except text at the single call that hands the turn to the harness (
run-turn.ts:742, hard-coded to one text block). The model never sees the file, an image-only message is rejected outright, and nothing warns the person. The prompt (non-agent) playground handles files correctly, so agents are behind the rest of the product and the gap is invisible until the agent ignores your file.This PR contains the design workspace for closing that gap, under
docs/design/agent-workflows/projects/agent-multi-modality/. It is documentation only: no production code changes.The design in one paragraph
The wire stops carrying file bytes. The front end uploads a file once through a create-only API route; the API stores it in a session-scoped attachments mount (our existing S3-backed mounts system) that is never exposed inside the sandbox, and returns an opaque, server-issued attachment id. Messages, records, and traces carry that id instead of base64. Behind the id sit two objects: an immutable original (what the model reads, what renders inline, what download returns) and a working copy the runner writes into the agent's working directory so tools can open or edit it. At prompt time the runner resolves the id, builds native ACP content blocks (the protocol requires inline bytes for the model to perceive a file), and replaces the hard-coded text block. Whether a modality reaches the model is the intersection of three layers (protocol transport, adapter fidelity, model modalities); a file the model cannot perceive is still attached as a workspace file with a visible notice, never silently dropped.
How to review
Read in this order:
claude-agent-acp0.58.1,pi-acp0.0.29) actually do with each block type, how Zed and opencode handle attachments, the records system, and the capability-flag history.The one decision this PR needs
D11 (open): what does attaching a file promise in the first release? Image perception only (smallest fix, storage migration later), or durable agent input (immutable original, working copy, references in records; recommended). The options, tradeoffs, and recommendation are in decisions.md and in the summary comment below. Everything else in the design is decided, with the reasoning recorded.
Verified facts worth knowing before reviewing
https://claude.ai/code/session_01Jg92VZvmkMnm5VQXprTNEH