fix(tern): transient storage errors no longer terminalize in-flight applies

[aparajon]

Why this matters

SchemaBot's own storage database can be briefly unreachable — a credential rotation, a failover, a network blip. Two recovery paths treated that brief outage as if the schema change itself had failed, and permanently marked the apply failed even though the real work (a checkpointed copy on the target database, or a live PlanetScale deploy request) was still running fine. The user sees "failed" on their PR while the change actually completes — or keeps running with nothing tracking it.

What it does

1. Crash recovery no longer fails an apply just because it couldn't read the plan.
When recovery picks up an apply, it loads the plan to rebuild the reviewed DDL. Previously "the read errored" and "the plan row doesn't exist" were handled identically: mark the apply failed. Now they're distinguished:

• Read error (storage blip): this recovery attempt gives up and releases its claim. Nothing is marked failed; the next poll retries once storage is healthy.
• Plan row confirmed missing: the apply still fails — the DDL genuinely can't be rebuilt — but now through the shared failure helper, so its tasks fail with it, a completion time is recorded, and the waiting PR check is notified.

2. A running Vitess apply is no longer abandoned over a bookkeeping write.
Once PlanetScale accepts the apply, the deploy request is live on the provider. If the follow-up write that saves the resume state (the record that lets a restart reattach) fails, the apply used to be marked failed — orphaning the live deploy. Now it pauses as failed_retryable, and the retry reattaches to the existing deploy request instead of abandoning it. This matches how the same save failure was already handled during progress polling.

3. If a Stop() lands at the same moment as a failure, the reported outcome is the Stop.
The failure helpers already refused to overwrite a concurrently-settled terminal state in storage, but they still notified the PR check with the stale in-flight state they were holding. Now they adopt the settled state first, so the check reports what actually happened (e.g. stopped, not a leftover running).

Before / after

storage blip during recovery          storage blip after PlanetScale
(plan read fails)                     accepts (resume-state save fails)

before: apply → failed ❌             before: apply → failed ❌
        (work still running)                  (live deploy orphaned)

after:  claim released,               after:  paused failed_retryable,
        retried next poll ✅                  retry reattaches to the
                                              live deploy request ✅

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens Tern apply recovery by ensuring transient storage failures don’t incorrectly terminalize in-flight applies (which can orphan engine-side work like checkpointed copies or live Vitess deploy requests), aligning storage-error handling with the intended “fail closed but retryable” operational model.

Changes:

• Split recovery plan-load handling so storage read errors return an error (release claim / retry later) while only a confirmed-missing plan row terminalizes the apply.
• After a grouped Vitess apply is accepted, treat failures to persist engine resume state as retryable (failed_retryable) instead of terminal failure, preventing orphaned deploy requests.
• Add targeted tests covering both the transient plan-load error path and the post-accept resume-state save-failure path.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
pkg/tern/local_control_resume.go	Separates plan-load storage errors from confirmed-missing plan rows during recovery, returning errors for transient failures.
pkg/tern/local_control_resume_test.go	Adds recovery tests verifying storage errors remain recoverable and missing plans terminalize with observer notification.
pkg/tern/local_client_test.go	Extends test storage with Plans() and adds a grouped-apply test asserting post-accept save failures pause as failed_retryable.
pkg/tern/local_apply_grouped.go	Switches post-accept resume-state save failures to failed_retryable for Vitess grouped applies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.