[AGE-4063] test(qa): make continuity a real dimension of the agent release gate - #5697
Merged
mmabrouk merged 2 commits intoAug 3, 2026
Merged
Conversation
…lease gate The Codex subscription bug in #5692 shipped through a gate that could not see it: the only multi-turn journey stayed on the warm daemon, and the cell labelled "Codex subscription" actually ran the Pi harness with an OpenAI-compatible subscription provider, so the code that broke had no coverage at all. Adds warm / cold1 / cold2 journeys (the tier vocabulary from reports/warm-approvals-qa.md), running in every cell, over a store-backed durable cwd: - cold1 forces an eviction from the client by changing the agent's instructions (a configFingerprint input), so the runner unmounts and remounts the cwd. - cold2 replaces the runner replica through an operator hook (SIGKILL + owner TTL wait); local sandboxes must refuse, remote ones resume cold. - The store is a precondition: the journeys resolve the session's durable mount over the API and SKIP (FAIL with --require-store) when there is none, so a green result can never mean "the store was not in play". - Evidence is store-side, not prose: the client reads the agent's token straight out of the object store and writes a store-only file the agent must read back, plus the turn ledger's agent_session_id/sandbox_id and any 0-byte objects in the cwd. Cells: S2 (codex + runtime_provided — the genuine subscription cell), X2 (codex on daytona, so a completed cold 2 is observable), S1 relabelled as what it is. Also folds in the QA-infrastructure fix left over from the v0.108.0 release run: P2 needs connection mode `agenta` plus the full `<slug>/custom/<model>` key under v0.107.x resolver semantics. Refs #5692
dosubot
Bot
added
the
size:XL
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe release gate adds store-backed warm, cold1, and cold2 continuity journeys. It expands Codex coverage, validates durable working-directory state through store-side checks, adds runtime controls, and documents the related failure lesson and operator requirements. ChangesDurable continuity release gate
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ReleaseGate
participant DurableMount
participant ObjectStore
participant SessionRunner
ReleaseGate->>DurableMount: Start continuity journey
DurableMount->>ObjectStore: Persist cwd files and session state
SessionRunner->>DurableMount: Resume after warm, eviction, or replacement
ReleaseGate->>ObjectStore: Verify files, tokens, ledgers, and zero-byte objects
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f64b12dc-9601-43cc-8f22-0a00cd80e5a3
📒 Files selected for processing (8)
.agents/skills/agent-release-gate/SKILL.md.agents/skills/agent-release-gate/resources/LESSONS.md.agents/skills/agent-release-gate/resources/coverage.md.agents/skills/agent-release-gate/resources/qa_product.py.agents/skills/agent-release-gate/resources/seeds/README.mddocs/design/codex-harness/reports/durable-cwd-entries-lesson.mddocs/design/codex-harness/status.mddocs/designs/testing/README.md
- Bound the cold2 operator hook (3 min) and wait the owner TTL plus a 20s margin. The key lapses at the boundary and the replacement replica may still be starting, so a resume timed exactly on the TTL races both and fails for the misleading reason this journey exists to avoid. Matches what the approval-matrix driver already does. - An empty turn ledger no longer reads as a stable warm run. `_turn_ledger` returns [] on any non-200, and "at most one id" let missing evidence pass as evidence. Require exactly one, record `ledger_available`, and say so in the failure reason. - Single-source the local cold-2 refusal text as LOCAL_NOT_OWNER_MARKER, quoted in coverage.md and in the journey's own `why`, so the doc and the assertion cannot drift. Coerce error entries to str before the substring test. Refs #5692
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The Codex subscription bug in #5692 reached a release through a gate that could not have caught it. Three things lined up.
The gate cell labelled "the Codex subscription path" (S1) runs the Pi harness with the
openai-codexprovider. Pi never loads codex-acp and never assembles a.codexhome, so the code that broke had no coverage. The cell that does run the Codex harness (X1) uses a managed vault key, and managed mode is deliberately file-free with noauth.jsonat all. "Codex harness plus your own login" was a supported, documented configuration with zero gate coverage, in a matrix that read as though it had some.The pre-merge QA that did cover that configuration was four single-turn checks (
docs/design/codex-harness/reports/m4-implementation-notes.md). The failure only appears on turn two.And the gate's one multi-turn journey (
warm) fired three quick turns against a live daemon on a live mount. The working directory is a geesefs mount over S3, and it only makes the round trip through the object store when something unmounts and remounts it. geesefs serves a symlink correctly while the mount is up, so a warm-only journey never takes the trip where the symlink turns into a 0-byte object.The bug class is bigger than one symlink: an entry inside the durable working directory that the object store cannot represent. It has now bitten symlinks, and it was pre-empted twice before (SQLite write-ahead logging, which is why
CODEX_SQLITE_HOMEis split onto container-local disk, and hard links).Changes
Continuity is now a dimension of the gate rather than a single journey, using the tier names the approvals QA already established in
reports/warm-approvals-qa.md: warm (same daemon, same live mount), cold 1 (session evicted, runner alive), cold 2 (runner replica replaced). All three run in every cell, so Codex, Claude and Pi are covered across the credential modes that matter.Each journey is multi-turn over a store-backed working directory, and the transition in the middle is the only difference.
cold1forces the eviction from the client. Changing the agent's instructions changes the config fingerprint, the runner evicts the pooled session (mismatch (config) ...; evict + cold), and the rebuild unmounts and remounts the durable cwd. That is a real store round trip driven by a real product action (editing an agent mid-session), and it leaves the transcript untouched, so the history guard stays out of it.cold2needs the runner replica replaced, which no HTTP client can do, so it takes an operator hook.--cold2-replace-cmdmust SIGKILL the replica, neverdocker stop(on SIGTERM the runner destroys every sandbox it owns, including the session under test), and the driver then waits out the session-owner TTL. The expected result differs per sandbox, exactly aswarm-approvals-qa.mdworked out: a local sandbox must REFUSE (local sandbox requires a single runner), a remote one resumes cold. It SKIPs loudly without the hook.The store is a precondition, not a detail.
mount.tsdegrades silently to an ephemeral directory when the sign call 503s, and every turn still looks fine, so a continuity pass on a storeless deployment would be green and meaningless. The journeys resolve the session's durable mount throughGET /api/sessions/mounts/?session_id=...before asserting anything, and SKIP without one, or FAIL with--require-store.The assertions are store-side, not "turn two replied":
GET /api/mounts/{id}/files?read=..., which goes to S3, not to the mount). That proves the cwd is durable and teaches the client the true token.catit. Content that only ever existed as an object cannot reach the agent unless that turn's cwd really resolves to that store prefix.agent_session_idandsandbox_id(POST /api/sessions/turns/query, the only continuity signal a client can see).warmfails if either changed, since a rebuilt session means the turn was not warm.Every result prints the runner-log grep that settles the tier definitively, because nothing about warm versus cold reaches the SSE stream.
Cells changed:
runtime_providedcredentials. The genuine Codex subscription cell, and the only one that exercisesCODEX_HOMEpointed at<cwd>/.codexwithauth.jsonsymlinked into the durable cwd.Docs:
coverage.mdgains the continuity section, the stated cold-2 method, and a table of what each cell needs beyond the three env vars (which cells need a store-backed deployment, which need the subscription sidecar).SKILL.mdtells a release conductor about--require-storeand the cold-2 hook. The lesson is recorded as #16 inLESSONS.mdand as a one-page retrospective indocs/design/codex-harness/reports/durable-cwd-entries-lesson.md.Folded in from the release run
Cell P2 was corrected during the v0.108.0 release and the fix was never committed. It touches the same file, so splitting it would only create a conflict. Under v0.107.x resolver semantics a named custom connection needs connection mode
agentaand a provider-less config, and--custom-slugmust send the full<slug>/custom/<model>key (a bare id that also exists in the shared catalog gets its provider inferred first, and the pair check then rejects the run).Two other corrections described as pending from that release,
CLAUDE_CONFIG_DIRwith a writable login copy for C1 and the shared runner token for the sidecar, are not present as uncommitted changes in the release worktree. Only the P2 fix and two one-off probe scripts are there, so only P2 is included. The probe scripts are release-night one-offs with a hardcoded vault slug. The X2 cell above is the durable version of one of them.Tests / notes
ruff formatandruff checkare clean, anduv run qa_product.py --helpworks with the cell and journey lists updated.invokeandapi_callstubbed: warm passes on a store-backed deployment and fails when the ledger shows a rebuilt session, both cold tiers pass on a healthy store round trip and fail when the durable file never reaches the store, the storeless case SKIPs and becomes a FAIL under--require-store, and cold 2 SKIPs without the hook, passes on a local refusal, and fails if a local sandbox resumes instead. That stub harness is scratch and is not committed.--store-settle, default 45s, since geesefs uploads on close), and whether the deployment pinsAGENTA_RUNNER_REPLICA_ID, which would make a local cold 2 resume instead of refusing.Refs #5692