Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5da3d20
test(runner): add history-driven client-tool relay integration tests
ardaerzin Jul 8, 2026
66cc1a2
Merge remote-tracking branch 'origin/big-agents' into test/runner-cli…
ardaerzin Jul 8, 2026
2a0a6a1
feat(frontend): add "Other…" custom value to elicitation enum fields
ardaerzin Jul 9, 2026
06a4a0c
test(frontend): add elicitation E2E scaffold with SSE transport mock
ardaerzin Jul 9, 2026
dfa8cb0
Merge remote-tracking branch 'origin/big-agents' into test/runner-cli…
ardaerzin Jul 9, 2026
6bc3f51
test(frontend): address review — validate seed ids, consistent scenar…
ardaerzin Jul 9, 2026
963cd80
Merge branch 'big-agents' into fe-feat/elicitation-m1-followups
ardaerzin Jul 9, 2026
c2e189c
feat(frontend): support default values on elicitation form fields
ardaerzin Jul 10, 2026
0196a6d
feat(frontend): multi-select fields in elicitation forms
ardaerzin Jul 10, 2026
0affc05
feat(frontend): choice cards for context-ful elicitation options
ardaerzin Jul 10, 2026
d665f12
test(frontend): extract and pin the enum controls' decision logic
ardaerzin Jul 10, 2026
b8e01fb
test(frontend): one-click-accept E2E spec + dialect decision record
ardaerzin Jul 10, 2026
81f6da0
Merge remote-tracking branch 'origin/big-agents' into fe-feat/elicita…
ardaerzin Jul 10, 2026
e56d887
Merge remote-tracking branch 'origin/big-agents' into fe-feat/elicita…
ardaerzin Jul 10, 2026
f2ff411
fix(frontend): address review — fold misplaced array options, type-ma…
ardaerzin Jul 10, 2026
e37a178
Merge branch 'big-agents' into fe-feat/elicitation-m1-followups
ardaerzin Jul 10, 2026
8df62f5
Merge branch 'big-agents' into fe-feat/elicitation-m1-followups
ardaerzin Jul 10, 2026
a640dcf
fix(frontend): dogfooding round — collapsed defaults registered, empt…
ardaerzin Jul 10, 2026
c73c41d
fix(frontend): dogfooding polish — subtle selected card, no premature…
ardaerzin Jul 10, 2026
3f6bbf8
feat(frontend): elicitation form drafts survive a reload
ardaerzin Jul 10, 2026
45ed92c
Merge remote-tracking branch 'origin/big-agents' into fe-feat/elicita…
ardaerzin Jul 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions api/oss/src/core/workflows/static_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,18 @@ def _request_input_revision() -> WorkflowRevision:
"(subdomain, workspace) before request_connection; or collect schedule "
"details (frequency, time of day, timezone) before create_schedule. "
"`requestedSchema` must be a FLAT JSON object schema: top-level "
"string/number/integer/boolean properties only (enum, format and title "
"allowed) — no nested objects or arrays. Supported `format` values: "
"string/number/integer/boolean properties (enum, format, title and "
"default allowed). For a multi-pick question use {type: 'array', "
"items: {type: 'string', enum: [...]}} — the ONLY array shape "
"allowed; no nested objects or deeper arrays. When options need "
"explaining, replace `enum` with oneOf: [{const, title, description}] "
"(works inside `items` too) — such options render as selectable cards "
"with the description under each title. When you can "
"propose a sensible value, set it as the field's `default` (a "
"primitive): it prefills the form so the user can accept everything "
"in one click. Enum options are SUGGESTIONS, not a hard constraint — "
"the form lets the user type their own value, so keep enums short and "
"likely rather than exhaustive. Supported `format` values: "
"'date', 'date-time', 'email', 'uri', and 'multiline' — use 'multiline' "
"for any long or free-form text field (notes, a description, a message "
"body). NEVER request secrets "
Expand Down
26 changes: 26 additions & 0 deletions api/oss/tests/pytest/unit/workflows/test_static_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -953,12 +953,38 @@ def test_request_input_matches_golden_request_fixture():
assert payload_keys == set(tool["input_schema"]["properties"])
assert isinstance(golden["message"], str) and golden["message"]

def _assert_one_of(one_of, name):
# Context-ful options: [{const, title?, description?}] — consts must be strings.
assert isinstance(one_of, list) and one_of, name
for option in one_of:
assert isinstance(option["const"], str), name

requested = golden["requestedSchema"]
assert requested["type"] == "object"
for name, prop in requested["properties"].items():
if prop["type"] == "array":
# Multi-select: the ONE admitted array shape — string items, optional enum/oneOf.
items = prop["items"]
assert items["type"] == "string", name
assert "properties" not in items and "items" not in items, name
if "oneOf" in items:
_assert_one_of(items["oneOf"], name)
if "default" in prop:
assert isinstance(prop["default"], list), name
assert all(isinstance(v, str) for v in prop["default"]), name
continue
assert prop["type"] in _ELICITATION_PRIMITIVES, name
assert "properties" not in prop and "items" not in prop, name
if "oneOf" in prop:
_assert_one_of(prop["oneOf"], name)
if "default" in prop:
assert isinstance(prop["default"], (str, int, float, bool)), name
assert set(requested.get("required", [])) <= set(requested["properties"])
# The golden must exercise the dialect's optional shapes: a prefilled field (#5190),
# a multi-select array, and context-ful oneOf options (choice cards).
assert any("default" in prop for prop in requested["properties"].values())
assert any(prop["type"] == "array" for prop in requested["properties"].values())
assert any("oneOf" in prop for prop in requested["properties"].values())


def test_request_input_matches_golden_response_fixture():
Expand Down
27 changes: 27 additions & 0 deletions docs/design/agent-chat-interaction-kinds/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,33 @@ so they travel with the code.
surface with a retry cap. One card chrome; state carried by status line (`{state} · {next}`
pill vocabulary, always neutral).

## Elicitation dialect extensions (2026-07-10, PR #5177)

The M1 dialect ("top-level primitives/enums only") gained three extensions once template
questionnaires (Mahmoud's agent-templates work) hit its limits. The governing principle for
all three: **adopt the standard JSON Schema idiom over bespoke hints** — the payload author
is an LLM, and it already knows these shapes from training data.

- **`default`** (issue #5190): a proposed value prefilling the field, so the user can accept
the whole form in one click. Primitives on scalar fields; array-of-strings on multi-select.
Date/date-time fields ignore it (a wire default is an ISO string; antd DatePicker requires
dayjs — render-side guard, not a contract rule).
- **Multi-select**: exactly one array shape is admitted — `{type: "array", items: {type:
"string", enum?/oneOf?}}`. String leaves only, nothing deeper; the answer is an array of
strings. Renders as a chip picker (checkbox choice cards when options carry descriptions).
- **Context-ful options**: `oneOf: [{const, title, description}]` on single fields and array
items. Any option description upgrades the control from a Select to selectable choice
cards; titles alone upgrade Select labels. Parse canonicalizes consts into `enum`, so
downstream consumers never branch on the authored shape.

Rendering ruling that shipped with these: **enum options are SUGGESTIONS, not a hard
constraint** — the consumer of the answer is the agent, which handles off-menu values fine.
Every enum control therefore carries an explicit "Other…" escape hatch (free-text). This is
elicitation-only (`openEnums`/`formats` opt-in flags on the shared form builder); gateway
tool execution forms keep strict enums and `Form.List` arrays, because their schemas are
real API parameters. The `__ag_enum_other__` UI sentinel must never reach the settled
`content` — pinned by unit tests on both the control logic and the E2E resume body.

## Model-composed UI: deliberately OUT OF SCOPE, with a re-evaluation trigger

Evaluated and rejected for now (2026-07-04): letting the model freely compose dashboards /
Expand Down
6 changes: 5 additions & 1 deletion services/runner/src/protocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,11 @@ export type RenderHint =
// stamps it so the frontend renders the OAuth/API-key connect dialog when the tool pauses. No
// payload — the widget is fully described by the paused call's tool name + input. `wire.py` does
// not pin RenderHint (render rides as an opaque dict), so this member is TS-only.
| { kind: "connect" };
| { kind: "connect" }
// `elicitation` requests the built-in schema-driven form (interaction kinds M1): the `request_input`
// client tool stamps it so the frontend renders a form from the paused call's `requestedSchema`. Like
// `connect`, it carries no payload here and is TS-only (the render rides through as an opaque dict).
| { kind: "elicitation" };

export type AgentEvent =
| { type: "message"; text: string }
Expand Down
97 changes: 96 additions & 1 deletion services/runner/tests/unit/client-tools.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,14 @@
import { describe, it } from "vitest";
import assert from "node:assert/strict";

import type { AgentEvent } from "../../src/protocol.ts";
import type { AgentEvent, AgentRunRequest } from "../../src/protocol.ts";
import type { ClientToolVerdict, Responder } from "../../src/responder.ts";
import {
ApprovalResponder,
ConversationDecisions,
extractApprovalDecisions,
extractClientToolOutputs,
} from "../../src/responder.ts";
import type { ClientToolRelayRequest } from "../../src/tools/client-tool-relay.ts";
import { PendingApprovalLatch } from "../../src/permission-plan.ts";
import {
Expand Down Expand Up @@ -312,3 +318,92 @@ describe("buildClientToolRelay", () => {
]);
});
});

/**
* Layer-B integration: the relay driven by the REAL responder built from run history, exactly
* as a live /run wires it (extract{Approval,ClientTool} → ConversationDecisions → ApprovalResponder
* → buildClientToolRelay). The mock-responder tests above pin the relay's mechanics; this pins the
* whole emit/pause/resume decision end to end — the seam where the coalescing bug lived (a FE-only
* mock could not have caught it). See docs/design/agent-chat-interaction-kinds/decisions.md.
*/
describe("buildClientToolRelay with the real responder (history-driven)", () => {
const input = {
message: "What is your city?",
requestedSchema: { type: "object", properties: { city: { type: "string" } } },
};
const liveReq: ClientToolRelayRequest = {
id: "i-live",
toolCallId: "tc-live",
toolName: "request_input",
input,
spec: { name: "request_input", kind: "client", render: { kind: "elicitation" } },
};

// A prior turn's request_input call and its answer, correlated by id. The tool_result carries
// no args (the real wire shape) — the store recovers them from the tool_call via the shape index.
const priorTurn = (output: unknown) => [
{
role: "assistant" as const,
content: [{ type: "tool_call", toolCallId: "c-1", toolName: "request_input", input }],
},
{
role: "tool" as const,
content: [{ type: "tool_result", toolCallId: "c-1", toolName: "request_input", output }],
},
];

function realSeam(request: AgentRunRequest) {
const events: AgentEvent[] = [];
const paused: string[] = [];
const responder = new ApprovalResponder(
{ default: "allow", rules: [] },
new ConversationDecisions(
extractApprovalDecisions(request),
extractClientToolOutputs(request),
),
);
const relay = buildClientToolRelay({
responder,
run: { emitEvent: (e) => events.push(e) },
latch: new PendingApprovalLatch(),
pause: { markPausedToolCall: (id) => paused.push(id), pause: () => {} },
recordPendingInteraction: () => {},
});
return { relay, events, paused };
}

it("pauses (emits a fresh form) for a new identical call in a later turn, not reusing the prior answer", async () => {
const request: AgentRunRequest = {
sessionId: "s",
messages: [
{ role: "user", content: "ask" },
...priorTurn({ action: "accept", content: { city: "Berlin" } }),
{ role: "user", content: "ask again" }, // latest user msg: the prior answer is a past turn
],
};
const s = realSeam(request);
const outcome = await s.relay.onClientTool(liveReq);
assert.equal(outcome, "pendingApproval", "a fresh identical call must pause for a new answer");
assert.equal(s.events.length, 1, "the client_tool interaction is emitted for the new form");
assert.equal((s.events[0] as { kind: string }).kind, "client_tool");
assert.deepEqual((s.events[0] as { payload: { render: unknown } }).payload.render, {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Remove duplicate assertion line.

Line 389 is duplicated — the same assert.deepEqual call appears twice consecutively. This is a copy-paste artifact; the test still passes but the redundant line should be removed.

🧹 Proposed fix
     assert.equal((s.events[0] as { kind: string }).kind, "client_tool");
     assert.deepEqual((s.events[0] as { payload: { render: unknown } }).payload.render, {
-    assert.deepEqual((s.events[0] as { payload: { render: unknown } }).payload.render, {
       kind: "elicitation",
     });
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
assert.deepEqual((s.events[0] as { payload: { render: unknown } }).payload.render, {
assert.equal((s.events[0] as { kind: string }).kind, "client_tool");
assert.deepEqual((s.events[0] as { payload: { render: unknown } }).payload.render, {
kind: "elicitation",
});

kind: "elicitation",
});
assert.deepEqual(s.paused, ["tc-live"]);
});

it("fulfills an in-turn resume from its own output without emitting a new form", async () => {
const request: AgentRunRequest = {
sessionId: "s",
messages: [
{ role: "user", content: "ask" },
...priorTurn({ action: "accept", content: { city: "Berlin" } }),
],
};
const s = realSeam(request);
const outcome = await s.relay.onClientTool(liveReq);
assert.deepEqual(outcome, { output: { action: "accept", content: { city: "Berlin" } } });
assert.equal(s.events.length, 0, "a genuine resume must not emit a new form");
assert.deepEqual(s.paused, []);
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,12 @@ import {
buildDegradationErrorText,
deriveElicitationPartState,
parseElicitationPayload,
partitionElicitationDraft,
serializeElicitationContent,
} from "@agenta/shared/utils"
import {CheckCircle, Prohibit, Question, Warning, XCircle} from "@phosphor-icons/react"
import {Button, Form, Typography} from "antd"
import dayjs from "dayjs"

import type {ClientToolHandlerProps} from "./types"

Expand All @@ -29,6 +31,9 @@ const {Text} = Typography
/** ElicitationResult → the settle channel's Record shape (interfaces carry no index signature). */
const toOutput = (result: ElicitationResult) => ({...result}) as Record<string, unknown>

/** In-progress field values survive a reload (localStorage draft keyed by the toolCallId). */
const draftKeyFor = (toolCallId: string) => `agenta:elicitation-draft:${toolCallId}`

/** Settled/parked single-line chip — one chrome for every terminal state (design: settled chip). */
const Chip = ({
icon,
Expand Down Expand Up @@ -64,6 +69,47 @@ const ElicitationWidget = ({meta, settle, degradedEarlierInTurn}: ClientToolHand
settle({errorText: buildDegradationErrorText(parsed.reason)})
}, [parsed, parked, meta.settled, settle])

// Draft persistence: typed values live only in antd Form state, so a reload would lose them.
const draftKey = draftKeyFor(meta.toolCallId)
const clearDraft = () => {
try {
localStorage.removeItem(draftKey)
} catch {
// storage unavailable — drafts are best-effort
}
}
const persistDraft = (values: Record<string, unknown>) => {
try {
localStorage.setItem(draftKey, JSON.stringify(values))
} catch {
// storage unavailable — drafts are best-effort
}
}
const settleAndClear: typeof settle = (args: Parameters<typeof settle>[0]) => {
clearDraft()
settle(args as {output: Record<string, unknown>})
}
const restoredRef = useRef(false)
useEffect(() => {
if (restoredRef.current || !parsed.ok || parked || meta.settled) return
restoredRef.current = true
try {
const raw = localStorage.getItem(draftKey)
if (!raw) return
const {plain, dates} = partitionElicitationDraft(
parsed.payload,
JSON.parse(raw) as Record<string, unknown>,
)
// DatePicker rejects strings — revive persisted ISO strings to dayjs.
form.setFieldsValue({
...plain,
...Object.fromEntries(Object.entries(dates).map(([k, v]) => [k, dayjs(v)])),
})
} catch {
// unreadable draft — fall back to schema defaults
}
}, [parsed, parked, meta.settled, draftKey, form])

const partState = deriveElicitationPartState({
state: meta.state,
output: meta.output,
Expand Down Expand Up @@ -142,7 +188,9 @@ const ElicitationWidget = ({meta, settle, degradedEarlierInTurn}: ClientToolHand
try {
const values = await form.validateFields()
const content = serializeElicitationContent(parsed.payload, values)
settle({output: toOutput(buildAcceptResult(content, "Provided the requested input."))})
settleAndClear({
output: toOutput(buildAcceptResult(content, "Provided the requested input.")),
})
} catch {
// antd surfaces inline field errors; Accept stays enabled for retry.
} finally {
Expand Down Expand Up @@ -171,6 +219,8 @@ const ElicitationWidget = ({meta, settle, degradedEarlierInTurn}: ClientToolHand
schema={parsed.payload.requestedSchema as unknown as Record<string, unknown>}
form={form}
formats
openEnums
onValuesChange={persistDraft}
/>

<div className="flex items-center gap-2">
Expand All @@ -180,7 +230,9 @@ const ElicitationWidget = ({meta, settle, degradedEarlierInTurn}: ClientToolHand
<Button
type="text"
onClick={() =>
settle({output: toOutput(buildDeclineResult("Declined the request."))})
settleAndClear({
output: toOutput(buildDeclineResult("Declined the request.")),
})
}
>
Decline
Expand All @@ -189,7 +241,9 @@ const ElicitationWidget = ({meta, settle, degradedEarlierInTurn}: ClientToolHand
type="text"
className="ml-auto opacity-60"
onClick={() =>
settle({output: toOutput(buildCancelResult("Dismissed the request."))})
settleAndClear({
output: toOutput(buildCancelResult("Dismissed the request.")),
})
}
>
Dismiss
Expand Down
55 changes: 55 additions & 0 deletions web/oss/tests/playwright/acceptance/agent-chat/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Agent chat — Elicitation E2E (interaction kinds M1, layer A)

Deterministic Playwright coverage for the `elicitation` interaction kind: emit → render → settle →
resume → replay, driven by a **transport mock** rather than a live LLM.

## How it works

Auth, the ephemeral project, the seeded agent revision, and the playground shell stay **real**. Only
the agent run (`**/invoke*`) is intercepted (`mockElicitationInvoke` in `tests.ts`) and fulfilled with
byte-accurate AI SDK v6 SSE (`assets/elicitationStream.ts`):

- **1st run** → a paused turn: `request_input` left `input-available` (no output) + the sibling
`data-render` part (`{kind: "elicitation"}`) → the FE renders the form.
- **2nd run** (the auto-resume after the form settles) → a normal text turn echoing the values.

The SSE shapes are pinned against the real producer,
`sdks/python/agenta/sdk/agents/adapters/vercel/{stream,sse}.py`. The mocked response **must** set
`Content-Type: text/event-stream` or the FE's negotiating fetch parses it as batch JSON.

## Specs (`index.ts`)

1. **Round-trip** — form renders, accept resumes with the submitted values (asserts the resume POST
carried the settled output). Mock-only, no reload.
2. **Required-field gate** — empty Accept shows an inline error and does not resume. Mock-only.
3. **Settled replay** — after accept, a reload shows the read-only chip.
4. **Reload-while-pending** — reload with the form pending, then accept.

## First-run seams (resolve against the live stack — do NOT assume)

These are isolated on purpose; the SSE/transport core above is solid, these need one live pass:

- **`seedAgentChatApp` (`tests.ts`) — the one hard blocker.** The playground mounts `AgentChatPanel`
only for an `is_agent` workflow (`Playground.tsx:106`); the base fixture only seeds
completion/chat. Since `/invoke` is mocked, the agent's config is irrelevant — a **minimal rendered
`is_agent` revision** is all that's needed. Seed it by replaying the product's create-agent API
calls (mirror `apiHelpers.createApp`) or by driving the create flow once and capturing the appId +
latest revision id. The fixture currently throws until this is done.
- **Composer + field selectors** — `sendChatMessage` and `getByLabel("First Name")` are best-effort
against `RichChatInput` and the `SchemaForm` DOM; confirm and adjust on first run.
- **Reload rehydration source (specs 3 & 4)** — a mocked run records no server-side session
transcript. If reload rehydrates the chat from client persistence these pass as-is; if it loads from
the server transcript, the mock must also serve the session-history endpoint. Confirm first.

## Run one spec (from `web/tests/`, with the stack up)

```bash
AGENTA_LICENSE=oss \
AGENTA_WEB_URL="http://localhost:3000" \
AGENTA_API_URL="http://localhost:3000/api" \
AGENTA_TEST_LLM_PROVIDER=mock \
npx playwright test ../oss/tests/playwright/acceptance/agent-chat/elicitation.spec.ts \
--workers=1 --retries=0 --headed
```

Start with spec 1 (round-trip) — it needs no reload and proves the transport-mock pattern end to end.
Loading
Loading