fix(systems): resolve templates in the deploy dry run (#1841) - #1845
Merged
Conversation
`dry_run: true` returned `status: "valid"` with zero warnings for a manifest whose agent referenced a template that cannot resolve — the same manifest then 404s that agent on the real deploy. The preview validated manifest SHAPE only, so the cheapest mistake to make (a typo'd or renamed template id) was exactly the one it could not catch. That gap is worth closing rather than documenting because recovery from a partial deploy is manual: the deploy path's own warning says re-running the manifest creates suffixed duplicates of the agents that already succeeded, so the operator fixes it agent by agent. A preview is the control that avoids paying that, and #1793/#1759 had just made an unresolvable `local:` template a hard 404 at create time — validation-time resolution is the matching half. The preflight reuses the CREATE path's `_resolve_local_template` on a throwaway config rather than re-deriving "does this template exist", so the preview cannot drift from the deploy: the reason string and status code are produced by the same code that will produce them for real. A test asserts that equality directly. Scope, deliberately: `local:` is resolved (a filesystem read); `github:` is NOT probed, since validating it means a network call to GitHub with the platform PAT on a preview endpoint — slow, rate-limited, and a new outbound call on a path that had none. A dry run therefore still cannot promise a github-template manifest deploys, and the tests pin that the resolver is never reached for one. `status` gains `invalid` for a preview with blockers, matching the deploy path's existing `partial` / `failed` vocabulary rather than reporting success next to a populated `failed[]`. `agents_to_create` still lists the full plan. Response model comment and the MCP `deploy_system` tool description updated (Invariant #13). Tests: 4 new cases (flagged failure with the real reason + status code, clean manifest stays `valid`, github not probed, preview/deploy reason equality), verified failing 2/2 against the pre-fix path. The ent125 fixture now stubs the resolver seam, since the preflight reads the real catalog root and pytest has none. Verified live against a running instance: the manifest from the issue now returns `status: "invalid"` with the verbatim 404 reason, where it returned `"valid"` with no warnings before. Related to #1841
Contributor
Author
|
Full unit suite on this branch: Same 4 environmental failures this repo shows on an untouched checkout: the first is a missing |
This was referenced Jul 28, 2026
vybe
approved these changes
Jul 28, 2026
vybe
left a comment
Contributor
There was a problem hiding this comment.
Validated via /validate-pr: diff reviewed, security greps clean, touched unit tests pass locally on a combined worktree with all four post-release fix PRs (167 passed). Merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The deploy dry run now resolves each
local:template, so a manifest that cannot deploy stops reportingvalid.Related to #1841
Before / after, same manifest
Before —
status: "valid",warnings: [],failed: []. After, on a live instance:{ "status": "invalid", "failed": [{ "name": "sysprobe2-bad", "short_name": "bad", "template": "local:this-template-does-not-exist", "reason": "Local template 'this-template-does-not-exist' was not found. Check the id against GET /api/templates — ...", "status_code": 404 }], "agents_to_create": ["sysprobe2-good", "sysprobe2-bad"] }Why this and not a doc note
Recovery from a partial deploy is manual. The deploy path's own warning spells it out — re-running the manifest creates suffixed duplicates of the agents that already succeeded — so the fix is per-agent by hand. The preview is the control that avoids paying that, and the failure it was blind to (typo'd or renamed template id) is the cheapest mistake available. #1793/#1759 had just made an unresolvable
local:template a hard 404 at create time; validation-time resolution is the matching half of that change.Approach
The preflight calls the create path's own
_resolve_local_templateon a throwawayAgentConfig, instead of re-deriving "does this template exist". Two consequences worth reviewing:test_preflight_reason_matches_what_the_real_deploy_reportsasserts that equality directly rather than hard-coding an expected message.template.yaml), so the manifest's own config is never passed to it.The import is lazy, matching the existing
_default_create_agent_fnseam — a module-level import would close a cycle.github:is deliberately not probed. Validating it means a network call to GitHub with the platform PAT on a preview endpoint: slow, rate-limited, and a new outbound call on a path that had none. So a dry run still cannot promise a github-template manifest deploys, and a test pins that the local resolver is never reached for one. Worth a second opinion during review — the alternative is an opt-in?probe_remote=true.statusgainsinvalidfor a preview with blockers, matching the deploy path's ownpartial/failedvocabulary rather than claiming success beside a populatedfailed[].agents_to_createstill lists the full plan, so the preview shows what was asked for and what blocks it. The response-model comment and the MCPdeploy_systemtool description are updated (Invariant #13).Verification
4 new cases; reverted the preflight with the tests in place → 2 failed (the flagged-failure case and the preview/deploy equality case).
One fixture change to flag: the ent125
envfixture now stubs the resolver seam. The preflight reads the real catalog root (/agent-configs/templates), which does not exist under pytest, so without the stub every pre-existing dry-run test would have started failing — the same "new contract, old test" trap that blocked #1822.Full unit suite posted below.
🤖 Generated with Claude Code
Fixes #1841