fix(server): make remote updates rollback-safe

[t3dotgg]

Remote server updates had accumulated multiple version owners and handoff paths, so a failed candidate could leave the service pinned to a broken version.

This replaces that implementation with one stable systemd launcher that owns version selection and durable update state. Candidates are staged and checked against the exact migration manifest, complete all fallible startup before reporting prepared, then activate only after the launcher commits. Clients correlate reconnects with the launcher-generated update ID, while foreground servers keep the explicit manual-update path.

Verification

• 281 focused tests across 16 files
• Server, client-runtime, and contracts typechecks
• Targeted lint and formatting
• Production server bundle
• Standalone launcher dependency check
• Missing-database preflight check

Built with GPT-5.6-sol in the Codex harness via T3 Code.

---

Note

High Risk
Changes core boot/update infrastructure: systemd units, version pinning, DB migration gating, and process handoff—failures can brick startup or leave the service on a bad runtime until rollback.

Overview
Replaces remote self-update paths that pointed systemd at a versioned bin.mjs (and foreground respawn) with a stable service-launcher under ~/.t3/runtime, durable service-state.json, and IPC between the running server and the launcher.

Boot service now installs the launcher artifact, writes initial state, and sets ExecStart to node …/service-launcher.mjs (with KillMode=control-group). Pinned runtimes are npm-installed into a staging tree, validated (including __service-preflight against the live DB migration manifest), then atomically published; blocked preflights surface the local npx t3@… service update path instead of mutating the DB remotely.

Remote update RPC stages the target version, runs preflight on the staged entry, then requestUpdate on the launcher and returns a launcher-generated updateId for client correlation—no direct unit rewrite or detached respawn from the server process.

Startup ordering adds ServerActivation / forkParked so background reactors and side effects (runtime state, Tailscale, cloud link reconcile) wait until activation; HTTP ingress is parked behind awaitCommandReady. Launcher trial children skip WAL and migrations so preflight stays read-only.

Servers without an installed launcher-managed service can no longer self-update remotely; the error directs users to t3 service install.

^{Reviewed by Cursor Bugbot for commit ff26b5a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Make remote server updates rollback-safe via a stable systemd launcher and staged installs

• Introduces a new serviceLauncher.ts process that owns update state and child process lifecycle. The systemd unit now points to a stable launcher file instead of a versioned CLI entry, so rollbacks don't require rewriting the unit.
• pinnedRuntime.ts now stages npm installs into a temp directory, validates via a preflight check, writes a sentinel, then atomically publishes via rename. Staging is cleaned up on failure or interruption.
• A new __service-preflight CLI command checks that DB migrations and launcher protocol match before an update is committed; if migrations are required, the update is blocked with a message to run npx t3@<version> service update locally.
• The client now correlates the post-restart ready lifecycle event using a launcher-provided updateId and immediately surfaces a ServerUpdateTerminalError on rollback/failure instead of waiting for a generic reconnect timeout (extended to 4 minutes).
• Introduces a forkParked/ServerActivation pattern so background fibers (tailscale, cloud link reconcile, runtime state persistence) are held until the activation gate is signaled, preventing races during startup.
• Risk: removePinnedRuntimeInstallation is removed as a public API; callers that relied on it will break at compile time.

^{Macroscope summarized ff26b5a.}