Use UFFD only for initial snapshot fork restores

[sjmiller609]

Summary

• add a one-shot Firecracker UFFD restore metadata flag
• arm UFFD only for the first restore of a new fork from standby snapshot state
• clear the flag after restore/standby/stop so later direct resumes use file-backed restore

Validation

• go test ./lib/instances ./lib/hypervisor/firecracker ./lib/uffdpager
• validated on deft with local UFFD + Firecracker concurrency merge:
  • single browser fork -> CDP, standby, file-backed restore -> CDP succeeds
  • 25-way Running fork burst: 25/25 succeeded; all requests started together; no source-state/vsock copy races

Note: the runtime validation used the Firecracker concurrency branch merged locally, since that branch removes the source alias/fork race this behavior relies on for burst fanout.

---

Note

High Risk
Changes Firecracker snapshot memory, fork fanout, and UFFD/systemd integration on a critical VM lifecycle path; incorrect backing or locking could affect restore correctness or concurrent forks.

Overview
This PR limits Firecracker UFFD to a one-shot path on standby snapshot fanout, instead of using UFFD on every restore when the backend is configured for UFFD.

New forks from standby (instance or snapshot) can skip copying snapshot-latest/memory, record deferred backing and FirecrackerUseUFFDOnNextRestore, and restore once via the pager against shared source memory. Standby then materializes local memory via SnapshotOptions.DeferredMemoryBackingPath before diff snapshot; later restores use file-backed memory. Metadata and lifecycle paths clear the one-shot flag and UFFD session state after restore, standby, and stop.

Fork/copy gains optional path skips, CopyRegularFile, and stricter skipping of stale *.sock artifacts. Restore accepts SnapshotMemoryBackingPath; the UFFD supervisor moves env to /run/hypeman/uffd with CI-scoped systemd instance names. CI/Makefile install the pager unit template, pass pager binary env vars, and add an integration test for the full UFFD lifecycle.

^{Reviewed by Cursor Bugbot for commit c74c539. Bugbot is set up for automated code reviews on this repo. Configure here.}

[firetiger-agent]

Created a monitoring plan for this PR.

What this PR does: Speeds up VM fork fanout in the hot pool by deferring the large snapshot memory file copy when creating Firecracker forks with UFFD enabled — forked VMs start immediately using the source's memory as backing, and materialize their own copy only when entering standby.

Intended effect:

• Hot pool warm failures (kernel_hotpool_warm_failure_total): baseline 40K–76K/hr (active hours); confirmed if no sustained increase post-deploy
• Hot pool hit rate (kernel_hotpool_hit_total): baseline 2.3M–3.7M/hr (active hours); confirmed if rate holds steady or improves within a same-day-of-week window
• API error rate: baseline ~0.13–0.17% during active hours; confirmed if it stays below 0.5%

Note: hypeman has no OTel telemetry — logs route to Railway stdout only. Hot pool metrics proxied through the kernel API are the observable signal. CI pass/fail is the primary authority for correctness.

Risks:

• Deferred path pointing to deleted source — if a source snapshot is deleted while a fork still holds FirecrackerDeferredSnapshotMemoryPath, standby fails with "materialize deferred snapshot memory" ERROR in Railway stdout; alert if hot pool warm failures exceed 100K/hr for 2+ hours
• Standby memory materialize failure — large memory file copy at standby time could fail on I/O error; manifests as "snapshot failed, attempting to resume VM" in Railway stdout and elevated hot pool warm failures
• Hot pool hit rate regression — if the one-shot optimization path has a bug causing forks to fail more often; alert if hit rate drops >50% vs same-day-of-week baseline for 2+ consecutive active hours

Status updates will be posted automatically on this PR as monitoring progresses.

View monitor

[hiroTamada]

nice work — the one-shot design cleanly confines the fragile pager/backing dependency to a single post-fork restore, and the lifecycle clearing + the TestFCUFFDOneShotLifecycle integration test (disk + /dev/shm files across the full fork→standby→restore→chained-fork chain) give good confidence on the snapshot-isolation risk.

both bugbot findings look already resolved on this head: the chained-fork backing path via the firecrackerDeferredSnapshotMemoryPath helper (regression-tested at firecracker_test.go:817), and the session close on the network-reconfigure failure path (restore.go:337).

a few non-blocking nits below, mostly around maintainability. approving.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c74c539. Configure here.}