Skip to content

Commit 933a86f

Browse files
cemililikclaude
andcommitted
test(core): add the 1.U end-to-end harness suite — the M2 proof
1.U is the proof the engine works end-to-end before any surface exists. It composes the already-landed pieces behind the @relavium/llm seam (1.K FallbackChain, 1.N run loop + RunEventBus, 1.O AgentRunner, 1.P node handlers, 1.Q human gate, 1.R checkpoint/resume, 1.S node retry, 1.T ToolRegistry) on the in-memory ExecutionHost reference — zero platform imports, no live network/keys, deterministic. No engine code and no new ADR: the harness uses only already-exported @relavium/core symbols. A scenario suite (the seed the Phase-2 CLI regression harness 2.K grows from): - happy-path — the literal 3-node input -> agent(+tool call) -> output: live token streaming, per-attempt cost (attributed to the model), a gap-free sequenceNumber stream that validates against the canonical RunEventSchema. - flagship — input -> agent -> human_gate -> output: in ONE run across a process boundary, the agent's forced provider error -> node retry (ADR-0040) -> failover to the second chain entry (1.K), per-attempt cost attributed to the fallback model; then a pause at the gate (the durable mid-run checkpoint persisted to the SQLite-shaped store); then a FRESH engine resumes via resumeFromCheckpoint, runs output to run:completed, and reproduces the final output with sequenceNumber continuing gap-free. The agent is NOT re-run on resume (its output is restored). - determinism — a re-run yields an identical event signature + final output (the no-wall-clock/no-RNG ban the risk table binds to the harness). Design decision (maintainer-approved): the resume demonstration uses a human gate as the durable suspend point, because the Phase-1 engine resumes ONLY from a gate/budget pause — a gate-less interrupted run is reconciled to run:failed (ADR-0036; verified by the resume-model review + an adversarial refutation). The literal 3-node topology is preserved by the happy-path member; the flagship adds the gate. Recorded as the §1.U "Harness shape" clarification in the phase plan (no new ADR — composes decided mechanisms). Exit criterion #1 annotated to point resume at the gated flagship. Per-attempt cost is asserted as distinct cost:updated events (not only the terminal cumulative); all LLM cost is incurred pre-gate, so the plain-human-gate cost-restore deferral (deferred-tasks.md) is off this path. 1.U / M2 will be marked Done only after this PR merges (roadmap-done-after-merge). Full gate (format/lint/typecheck/test/build) green — core 683 tests; Leakwatch clean. Refs: §1.U, M2, ADR-0036, ADR-0040, ADR-0003 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 8087d85 commit 933a86f

2 files changed

Lines changed: 425 additions & 1 deletion

File tree

docs/roadmap/phases/phase-1-engine-and-llm.md

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -842,6 +842,26 @@ The proof that the engine works before any surface exists.
842842
then fallback with the run still completing; resume from a mid-run checkpoint
843843
reproduces the same final output — **M2 achieved**.
844844

845+
> **Harness shape (decided 2026-06-16, implementing 1.U).** The harness is a **scenario suite** with a
846+
> reusable driver (not a single test) — the seed the Phase-2 CLI regression harness (2.K) grows from, and
847+
> the suite 1.AB's `condition`/`transform` scenario and the determinism ban plug into. Its **happy-path**
848+
> member is the literal 3-node `input → agent → output` (a clean run: live token streaming + a tool call +
849+
> per-attempt cost + a gap-free `sequenceNumber` stream that validates against the canonical
850+
> [`RunEventSchema`](../../reference/contracts/sse-event-schema.md)). Its **flagship** member inserts a
851+
> `human_in_the_loop` gate — `input → agent → human_gate → output` — as the **durable mid-run checkpoint**,
852+
> because the Phase-1 engine resumes **only** from a durable suspend point (a human/budget gate): a gate-less
853+
> interrupted run is reconciled to `run:failed`, never resumed
854+
> ([ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); the derived
855+
> `Checkpointer`, ADR-0003). The flagship drives, **in one run** across a process boundary: the agent's
856+
> forced provider error → **node retry** ([ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)
857+
> Part A) → **failover** to the second chain entry (1.K), streaming + a tool call + per-attempt cost; then the
858+
> gate pause (the persisted checkpoint); then a **fresh engine** resumes via `resumeFromCheckpoint`, runs
859+
> `output` to `run:completed`, and reproduces the same final output with `sequenceNumber` continuing gap-free.
860+
> All LLM cost is incurred **pre-gate**, so the plain-human-gate cost-restore deferral
861+
> ([deferred-tasks.md](../deferred-tasks.md)) is off this path. **No new engine code or ADR** — the harness
862+
> composes 1.K/1.N/1.O/1.P/1.Q/1.R/1.S/1.T behind the `@relavium/llm` seam and uses only already-exported
863+
> `@relavium/core` symbols.
864+
845865
### Agent-first sub-spine (1.V–1.AA) — additive, parallel to the M2 critical path
846866

847867
These build the `AgentSession` entry point ([ADR-0024](../../decisions/0024-agent-first-entry-point-agentsession.md)). They run **parallel** to 1.L–1.U and do **not** feed the 1.U workflow harness — each is proven by its own harness (1.AA). The `WorkflowEngine` is unchanged; `AgentSession` is an additional entry point on the same substrate.
@@ -1212,7 +1232,8 @@ All must be true to start Phase 2 (CLI):
12121232
1. A `relavium`-equivalent invocation from the Node harness (1.U) runs a 3-node
12131233
sequential workflow end-to-end: live token streaming, in-process emission of the
12141234
canonical run events with monotonic `sequenceNumber`, SQLite-shaped checkpointing,
1215-
and resume from a checkpoint.
1235+
and resume from a checkpoint (the resume rides the suite's gated **flagship** scenario —
1236+
the engine resumes only from a durable gate checkpoint; see §1.U *Harness shape*).
12161237
2. Forcing a provider error triggers node retry and then a fallback to the next
12171238
provider in the chain, with the run completing and **per-attempt** cost recorded
12181239
correctly.

0 commit comments

Comments
 (0)