docs(design): plan in-sandbox platform-tool MCP (Claude/Codex tools on Daytona, unified with Pi)

docs/design/agent-workflows/projects/in-sandbox-tool-mcp/README.md

@@ -0,0 +1,70 @@
+# In-sandbox platform-tool MCP
+
+Deliver Agenta gateway and platform tools to MCP-client harnesses (Claude Code today, Codex
+next) running in a Daytona sandbox, by running our own small MCP server inside the sandbox.
+Its `tools/call` handler writes to the file relay the runner already polls and executes
+server-side. Today that combination gets zero tools and the run is refused up front
+(`services/runner/src/engines/sandbox_agent/run-plan.ts:355`,
+`REMOTE_TOOLS_UNSUPPORTED_MESSAGE`).
+
+The design goal the owner cares about most is unification with Pi: one gateway-tool code
+path serving both the Pi extension and the new MCP server. That shared path is the relay
+client and file protocol that PR #5232 (event-driven-tool-relay) extracts as its slice 0;
+this project consumes those modules, so the relay protocol cannot fork per harness.
+
+## Glossary
+
+- **Runner**: the Node sidecar (`services/runner/`) that executes agent runs. It holds the
+  run's credentials and executes gateway and platform tools server-side.
+- **Harness**: the coding agent that runs inside the sandbox (Pi, Claude Code, later Codex).
+- **Sandbox**: the isolated environment the harness runs in. Local (same machine as the
+  runner) or Daytona (a remote cloud VM the runner reaches through a daemon API).
+- **Daemon**: the sandbox-agent process inside the sandbox that the runner drives over a
+  signed URL. It creates harness sessions and reads/writes sandbox files for the runner.
+- **File relay**: the file-based tool-call channel. An in-sandbox writer creates
+  `<id>.req.json` in a relay directory; the runner polls the directory, executes the call
+  with runner-held credentials, and writes `<id>.res.json` back.
+- **Gateway tools**: backend-resolved tools (Composio actions, workflow-as-tool, platform
+  operations) executed through Agenta's `/tools/call` with server-side credentials.
+- **Client tools**: browser-fulfilled tools (for example `request_connection`). A call
+  pauses the turn and a human answers it in the playground.
+- **Public spec**: the credential-free advertisement shape of a resolved tool (name,
+  description, input schema, kind). Private fields (`callRef`, code, scoped env, callback
+  auth) never leave runner memory.
+- **The shim**: this project's deliverable. A small, dependency-free MCP server process
+  inside the sandbox that advertises the public specs and forwards each `tools/call` to
+  the file relay.
+- **Warm reuse / park**: since PR #5225 a Daytona sandbox survives across turns. It stays
+  running for an idle window (park-to-running), then stops without deletion
+  (park-to-stopped) and restarts on the next turn.
+
+## Files and reading order
+
+1. [context.md](context.md): why the work exists, the settled owner decisions (including
+   the user-MCP HTTP-only policy), goals, non-goals, and neighboring projects.
+2. [research.md](research.md): the verified current state with file and line anchors, what
+   PR #4873 built and why it went stale, and the warm-reuse lifecycle facts the design
+   must survive.
+3. [plan.md](plan.md): the recommended design (transport choice, unification path,
+   lifecycle, security), the implementation slices, and the test plan.
+4. [open-questions.md](open-questions.md): the decisions the owner still needs to make.
+5. [status.md](status.md): progress and provenance. Source of truth for state.
+
+## Prior art this builds on (not duplicated here)
+
+- [../claude-daytona-tools/](../claude-daytona-tools/README.md): the full option analysis.
+  Option A (in-sandbox MCP front-end over the relay) recommended; Option B (tunnel the
+  runner's MCP) rejected.
+- [../remote-tools-delivery/specs.md](../remote-tools-delivery/specs.md): independent
+  reconfirmation; recommends the in-sandbox relay client.
+- [../mcp-delivery-architecture/](../mcp-delivery-architecture/README.md): the umbrella.
+  `gateway-mcp-location.md` records the 2026-07-11 owner decision that rejects the
+  API-hosted gateway and commits to the in-sandbox front-end.
+- [../gateway-tool-mcp/](../gateway-tool-mcp/README.md): the three-MCP-layer distinction
+  (user stdio / internal gateway channel / user HTTP) that this plan keeps separable.
+- PR #4873 (closed, unmerged): a working implementation of the stdio variant, pre-rename.
+  Mined in [research.md](research.md).
+
+## Status
+
+DESIGN ONLY. No runtime code changes in this PR. See [status.md](status.md).

docs/design/agent-workflows/projects/in-sandbox-tool-mcp/context.md

@@ -0,0 +1,105 @@
+# Context
+
+## What the user sees today
+
+A Claude Code run on Daytona that carries any custom tool fails immediately with
+`REMOTE_TOOLS_UNSUPPORTED_MESSAGE` (`services/runner/src/engines/sandbox_agent/run-plan.ts:355`).
+The same run works on the local sandbox, and the same tools work on Daytona with the Pi
+harness. The refusal is deliberate (PR #5047 replaced an earlier silent zero-tools success),
+but it leaves a hole in the product: the harness we want customers to use for serious agents
+(Claude, soon Codex) cannot use platform tools in the sandbox environment we want to be the
+default (Daytona, now fast thanks to warm reuse).
+
+The cause is an advertisement gap, not an execution gap. Tool execution on Daytona is solved
+and harness-agnostic: the runner polls a relay directory in the sandbox filesystem, executes
+each request with runner-held credentials, and writes the response back. Pi works because its
+bundled extension runs inside the sandbox and writes those request files. Claude takes tools
+only over MCP, and the only MCP server we run today binds to the runner's own loopback, which
+is unreachable from inside a remote sandbox. Nothing inside the sandbox speaks MCP for us.
+
+## Settled decisions (owner, 2026-07-11)
+
+These are decided. This workspace encodes them; it does not reopen them.
+
+1. **The sandbox talks only to the runner.** An API-hosted tool gateway (a platform MCP
+   endpoint the sandbox dials directly) was analyzed and rejected in
+   [../mcp-delivery-architecture/gateway-mcp-location.md](../mcp-delivery-architecture/gateway-mcp-location.md).
+   Warm sandboxes are the priority, and the committed model is: gateway-tool logic stays in
+   the runner, credentials never enter the sandbox, and delivery to MCP-client harnesses goes
+   through an in-sandbox front-end feeding the file relay.
+2. **User-declared MCP servers: HTTP transport only, permanently.** We will never host or
+   run arbitrary user stdio/npx MCP servers, on the runner host or in the sandbox. A user who
+   wants a local MCP server runs it themselves and gives us the URL and host. This supersedes
+   the L1 direction ("run user stdio MCP in the sandbox") in
+   [../mcp-delivery-architecture/directions.md](../mcp-delivery-architecture/directions.md).
+   Authentication for user HTTP MCP servers: an API key in a request header, which the
+   existing mechanism already supports (named secrets become headers,
+   `services/runner/src/engines/sandbox_agent/mcp.ts:119`). OAuth for user MCP servers is a
+   later feature, named as future work and out of scope here.
+3. **Platform tools (our own) are delivered by our own MCP server running inside the
+   sandbox**, feeding the existing file relay. This project designs that server.
+4. **The design goal that matters most is unification with Pi.** Today Pi gets tools through
+   its bundled in-sandbox extension (`registerTool`, writing relay request files) and local
+   Claude gets them through a runner-loopback HTTP MCP server. The owner wants one
+   gateway-tool logic serving both, in closely shared code, so the relay protocol and the
+   execution semantics cannot drift per harness.
+
+## The three MCP layers, kept separable
+
+The word "MCP" names three different things in this codebase
+(established in [../gateway-tool-mcp/README.md](../gateway-tool-mcp/README.md); conflating
+them caused the #4831 regression). This plan touches only the second.
+
+| Layer | Declared by | Status | This project |
+| --- | --- | --- | --- |
+| User stdio/npx MCP servers | The user (`transport: "stdio"`) | Disabled, now permanently (decision 2) | Untouched. The mechanism that spawns our shim must not relax this gate. |
+| Internal gateway-tool channel | Nobody; synthesized by the runner from the run's resolved tools | Local only (runner-loopback HTTP) | Extended into the sandbox. This is the whole project. |
+| User HTTP MCP servers | The user (`transport: "http"` + URL) | Built, SSRF-guarded, behind `AGENTA_AGENT_MCPS_ENABLED` (default off) | Untouched. API-key-in-header auth is the current answer; OAuth is future work. |
+
+## Goals
+
+1. A Claude run on Daytona with gateway/callback tools succeeds, and the tools actually
+   execute (the current refusal stops firing for that combination).
+2. The delivery is harness-agnostic on the sandbox side: any MCP-client harness (Codex next)
+   gets the same tools with no per-harness work beyond its ACP adapter.
+3. One shared implementation of "turn a tool call into a relay request" serves the Pi
+   extension, the local Claude channel, and the new in-sandbox server: the relay client
+   that PR #5232 extracts and pins with contract tests, consumed here.
+4. The lifecycle survives warm sandbox reuse (PR #5225): a parked-and-resumed sandbox, a
+   stopped-and-restarted sandbox, and a tool-set change between turns must all behave
+   correctly.
+5. No security invariant weakens: public specs and the relay directory are the only things
+   that enter the sandbox; no credential ever does; no new network surface opens.
+
+## Non-goals
+
+- Client tools (browser round-trip, `request_connection`) on Claude+Daytona in the first
+  slice. Their pause semantics depend on the client-tool continuation work
+  ([../agent-client-tool-cleanup/](../agent-client-tool-cleanup/),
+  [../mcp-client-tool-continuation/](../mcp-client-tool-continuation/)). Gateway/callback
+  tools land first; the plan states the sequencing.
+- Changing the relay file protocol or its polling mechanics. The sibling project
+  [../event-driven-tool-relay/](../event-driven-tool-relay/README.md) owns relay latency;
+  this plan only keeps the shim compatible with it.
+- Re-enabling user stdio MCP in any form (decision 2).
+- OAuth for user HTTP MCP servers (future work).
+- Any new remote sandbox provider. The fail-closed gate for non-Daytona remote providers
+  stays until delivery is proven per provider.
+- Changing how local Claude gets tools. The runner-loopback HTTP channel works and keeps its
+  client-tool pause behavior; it stays.
+
+## Neighboring projects
+
+- [../claude-daytona-tools/](../claude-daytona-tools/README.md) and
+  [../remote-tools-delivery/specs.md](../remote-tools-delivery/specs.md): the two prior
+  designs this plan reconciles into one implementation path.
+- [../mcp-delivery-architecture/](../mcp-delivery-architecture/README.md): the umbrella and
+  the decision record.
+- [../event-driven-tool-relay/](../event-driven-tool-relay/README.md): concurrent sibling.
+  It replaces relay polling with filesystem-event wakeups. The shim is a second writer of
+  the same relay files, so the two projects share the file contract; neither implements the
+  other's scope.
+- [../session-keepalive/](../session-keepalive/) and the warm-Daytona work (PR #5225): the
+  reuse lifecycle this design must survive.
+- [../agent-client-tool-cleanup/](../agent-client-tool-cleanup/): the park-and-resume
+  redesign client tools depend on.

docs/design/agent-workflows/projects/in-sandbox-tool-mcp/open-questions.md

@@ -0,0 +1,48 @@
+# Open questions
+
+Decisions the owner needs to make or confirm. Each states the recommendation so a yes is
+enough. Items 1 and 5 changed on 2026-07-11 when the Codex review was folded in; see
+[status.md](status.md) for that provenance.
+
+1. **Transport: approve A2 (harness-spawned stdio) conditionally, gated on the slice 0
+   restart spike?** The Codex review rejected the earlier "correct by construction" claim:
+   whether the pinned Claude ACP adapter respawns the shim on the `session/load` path after
+   a VM stop is external adapter behavior that must be proven. The plan now makes that
+   proof slice 0 and locks A2 only after it passes. Documented fallbacks: force cold
+   `createSession` for sessions containing the shim if restoration fails; A1 (HTTP
+   loopback) as last resort. Recommendation: approve the conditional flip.
+
+2. **Client tools on Claude+Daytona: keep failing loud until the bridge work lands?**
+   This project delivers executable tools only; a run carrying a client tool
+   (`request_connection`) on that path still refuses with a narrowed message. The
+   alternatives (drop the spec silently, or advertise it and return a synthetic error)
+   both mislead. Recommendation: fail loud, leave client tools to the future
+   Daytona client-tool bridge workspace.
+
+3. **Specs delivery: confirm the file, not the env variable.** Decided in the plan per the
+   Codex review: `AGENTA_AGENT_TOOLS_PUBLIC_SPECS` is unbounded and would be copied through
+   four exec-environment layers under A2, so the shim reads the specs from a file uploaded
+   next to the bundle, with the path in its per-server env. Pi keeps its env variable; the
+   spec content shape stays shared. The earlier "note a fallback, build it later" position
+   is out. Recommendation: confirm the file.
+
+4. **Gate posture for future remote providers: confirm fail-closed stays.** After slice 1
+   the refusal still fires for any non-Daytona remote provider (the in-flight E2B work
+   would need its own proven delivery before the gate opens for it). Recommendation:
+   confirm.
+
+5. **User HTTP MCP, API-key-now: is the existing mechanism the answer?** Named secrets
+   already become request headers on the user's HTTP entry
+   (`services/runner/src/engines/sandbox_agent/mcp.ts:119`), SSRF-guarded, behind
+   `AGENTA_AGENT_MCPS_ENABLED` (default off). The decision "API key in a header for now"
+   appears to be already built; the only open item is when to flip the flag default, which
+   is the separate S2 work (#4912), not this project. OAuth for user MCP stays named future
+   work. Recommendation: confirm this reading so the policy in context.md is complete.
+
+## Settled by the 2026-07-11 review fold (no longer questions)
+
+- **Codex-on-Daytona** is a follow-up verification task, cut from v1.
+- **Pi consuming the shim directly** (the old U2) is a follow-up decision, cut from v1.
+- **Snapshot bake** is a follow-up with its own decision; per-run upload ships first.
+- **Relay module ownership**: PR #5232 owns `relay-client.ts` / `relay-protocol.ts`
+  extraction as its slice 0; this project consumes them.