Skip to content

fix(systems): resolve templates in the deploy dry run (#1841) - #1845

Merged
vybe merged 1 commit into
devfrom
fix/1841-dryrun-template-resolution
Jul 28, 2026
Merged

fix(systems): resolve templates in the deploy dry run (#1841)#1845
vybe merged 1 commit into
devfrom
fix/1841-dryrun-template-resolution

Conversation

@dolho

@dolho dolho commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

What

The deploy dry run now resolves each local: template, so a manifest that cannot deploy stops reporting valid.

Related to #1841

Before / after, same manifest

name: sysprobe
agents:
  good: {template: local:dd-intake}
  bad:  {template: local:this-template-does-not-exist}

Beforestatus: "valid", warnings: [], failed: []. After, on a live instance:

{
  "status": "invalid",
  "failed": [{
    "name": "sysprobe2-bad",
    "short_name": "bad",
    "template": "local:this-template-does-not-exist",
    "reason": "Local template 'this-template-does-not-exist' was not found. Check the id against GET /api/templates — ...",
    "status_code": 404
  }],
  "agents_to_create": ["sysprobe2-good", "sysprobe2-bad"]
}

Why this and not a doc note

Recovery from a partial deploy is manual. The deploy path's own warning spells it out — re-running the manifest creates suffixed duplicates of the agents that already succeeded — so the fix is per-agent by hand. The preview is the control that avoids paying that, and the failure it was blind to (typo'd or renamed template id) is the cheapest mistake available. #1793/#1759 had just made an unresolvable local: template a hard 404 at create time; validation-time resolution is the matching half of that change.

Approach

The preflight calls the create path's own _resolve_local_template on a throwaway AgentConfig, instead of re-deriving "does this template exist". Two consequences worth reviewing:

  • The preview cannot drift from the deploy. The reason string and status code an operator sees in a dry run are produced by the same code that will produce them for real — test_preflight_reason_matches_what_the_real_deploy_reports asserts that equality directly rather than hard-coding an expected message.
  • The throwaway config matters: that resolver mutates the object it is handed (type/resources/tools/runtime from template.yaml), so the manifest's own config is never passed to it.

The import is lazy, matching the existing _default_create_agent_fn seam — a module-level import would close a cycle.

github: is deliberately not probed. Validating it means a network call to GitHub with the platform PAT on a preview endpoint: slow, rate-limited, and a new outbound call on a path that had none. So a dry run still cannot promise a github-template manifest deploys, and a test pins that the local resolver is never reached for one. Worth a second opinion during review — the alternative is an opt-in ?probe_remote=true.

status gains invalid for a preview with blockers, matching the deploy path's own partial / failed vocabulary rather than claiming success beside a populated failed[]. agents_to_create still lists the full plan, so the preview shows what was asked for and what blocks it. The response-model comment and the MCP deploy_system tool description are updated (Invariant #13).

Verification

pytest tests/unit/test_ent125_resilient_system_deploy.py
19 passed

4 new cases; reverted the preflight with the tests in place → 2 failed (the flagged-failure case and the preview/deploy equality case).

One fixture change to flag: the ent125 env fixture now stubs the resolver seam. The preflight reads the real catalog root (/agent-configs/templates), which does not exist under pytest, so without the stub every pre-existing dry-run test would have started failing — the same "new contract, old test" trap that blocked #1822.

Full unit suite posted below.

🤖 Generated with Claude Code

Fixes #1841

`dry_run: true` returned `status: "valid"` with zero warnings for a manifest
whose agent referenced a template that cannot resolve — the same manifest then
404s that agent on the real deploy. The preview validated manifest SHAPE only,
so the cheapest mistake to make (a typo'd or renamed template id) was exactly
the one it could not catch.

That gap is worth closing rather than documenting because recovery from a
partial deploy is manual: the deploy path's own warning says re-running the
manifest creates suffixed duplicates of the agents that already succeeded, so
the operator fixes it agent by agent. A preview is the control that avoids
paying that, and #1793/#1759 had just made an unresolvable `local:` template a
hard 404 at create time — validation-time resolution is the matching half.

The preflight reuses the CREATE path's `_resolve_local_template` on a throwaway
config rather than re-deriving "does this template exist", so the preview cannot
drift from the deploy: the reason string and status code are produced by the
same code that will produce them for real. A test asserts that equality
directly.

Scope, deliberately: `local:` is resolved (a filesystem read); `github:` is NOT
probed, since validating it means a network call to GitHub with the platform PAT
on a preview endpoint — slow, rate-limited, and a new outbound call on a path
that had none. A dry run therefore still cannot promise a github-template
manifest deploys, and the tests pin that the resolver is never reached for one.

`status` gains `invalid` for a preview with blockers, matching the deploy path's
existing `partial` / `failed` vocabulary rather than reporting success next to a
populated `failed[]`. `agents_to_create` still lists the full plan. Response
model comment and the MCP `deploy_system` tool description updated (Invariant
#13).

Tests: 4 new cases (flagged failure with the real reason + status code, clean
manifest stays `valid`, github not probed, preview/deploy reason equality),
verified failing 2/2 against the pre-fix path. The ent125 fixture now stubs the
resolver seam, since the preflight reads the real catalog root and pytest has
none.

Verified live against a running instance: the manifest from the issue now
returns `status: "invalid"` with the verbatim 404 reason, where it returned
`"valid"` with no warnings before.

Related to #1841
@dolho

dolho commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Full unit suite on this branch:

5042 passed, 16 skipped, 4 failed  (27:26)

FAILED tests/unit/test_1472_schedule_validation.py::test_accepts_valid[0 4 * * *-Europe/Kiev]
FAILED tests/unit/test_admin_email_login.py::test_admin_logs_in_with_username
FAILED tests/unit/test_admin_email_login.py::test_admin_logs_in_with_registered_email
FAILED tests/unit/test_admin_email_login.py::test_email_identifier_is_normalized

Same 4 environmental failures this repo shows on an untouched checkout: the first is a missing tzdata module, the other three trace to a bcrypt has no attribute '__about__' version mismatch that also appears in the backend boot log. Neither is touched by this PR — identical set on #1843 and #1847.

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated via /validate-pr: diff reviewed, security greps clean, touched unit tests pass locally on a combined worktree with all four post-release fix PRs (167 passed). Merging.

@vybe
vybe merged commit c04fb8c into dev Jul 28, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants