feat(qdrant-backup-restore): per-shard backups with verified restores

[ddakv]

What

The legacy backup snapshots every collection on every peer — with 3 nodes and RF 3 that's ~3× the logical data per run, with no retention: the bucket fills, backups fail, and no verified-restorable set exists.

This adds a per-shard mode to qdrant_backup_recovery.sh (create_snap_shards, recover_snap_shards, prune_snap [--legacy]) plus a DR RUNBOOK and staged k8s cutover manifests. Legacy paths are frozen — deployments behave identically until the operator cuts over. Result per run: one snapshot per shard (≈1× logical data), retention-bounded usage, restores that end in hard verification.

How it works

• Backup: preconditions (green cluster, Active replica per shard) → one shard-snapshot per shard, written by Qdrant natively to S3 → manifest uploaded last (config, aliases, per-shard keys with ETag+size) — a set exists completely or not at all. Job fails loudly on any collection failure.
• Retention: keep newest N sets, delete manifest-first; orphan sweep behind four independent guards (listing-error abort, zero-manifest skip, 48 h grace, 50 % delete cap). prune_snap --legacy = day-0 bucket unblock.
• Restore: pick newest complete set (or set-id filter) → integrity pre-flight (ETag+size vs manifest) before any destructive step → recreate collection from manifest → per-shard presigned recover with durable cross-pod resume → verification: green + counts + replica sets + payload sampling + search + alias/index recreation. Any failure ⇒ FAILED, nonzero exit.
• Cutover is staged: manifests carry the new env (optional: true secrets) while args still run legacy; the RUNBOOK gates each step — day-0 prune → shadow run → restore canary → cutover. CI stamps commit provenance into the ConfigMap (byte-identity to the script is test-enforced).

Review guide

Read top-down from the three *_task entrypoints; every decision routes through a small pure function (stdin→stdout, contract in its header). The frozen legacy zone (≤ line ~946) is diff-verifiably unchanged. Spend your attention on five spots: (1) restore pre-flight ordering (before state gate/FORCE), (2) the FORCE path + resume precedence, (3) the sweep's four guards, (4) manifest-last ordering, (5) the version gate. Every protective layer exists because we reproduced its failure live (e.g. Qdrant accepts corrupted snapshot archives — qdrant#3372 — so the tool verifies integrity itself).

Guarantees & caveats

• Fails loud, never lies: every silent-failure path found by review or red-teaming ends fail-closed. Trust the tool's verdict line, not cluster color (Qdrant reports green in broken states — reproduced).
• Pinned to Qdrant v1.15.1 empirics; the drill suite re-runs against the customer's exact version before rollout (gate).
• Merge = prod-affecting (ConfigMap injector ships main): merge only after BA IDST go. Opening/reviewing this PR ships nothing.
• Per-shard restore needs a cluster-mode target with working cluster-info discovery (no static-hosts mode — documented, follow-up ticket for a fallback). Five further legacy-parity deltas are documented in the RUNBOOK, with loud warnings where old env would silently no-op.
• Restore stages the shard archive on the target node: plan free disk ≥ largest shard. restore-job.yaml as shipped intentionally fails fast until the RUNBOOK's flip; cron 0 0 */7 * 0 actually fires ~9-10×/month (pre-existing, comment now truthful — cadence decision open).
• Follow-ups (non-gating) are listed in the RUNBOOK/stacked PR: discovery-fallback spike, extra committed tests, legacy-hygiene ticket, doc dedup.

Testing

Evidence (test code in the stacked PR):

• ~350 unit assertions, green on every commit; mutation-tested where load-bearing.
• 17 live failure drills on a real 3-node v1.15.1 cluster + S3 — golden paths, kill-mid-restore resume, retention/sweep, legacy coexistence, and two corruption regressions (pre-flight aborts before FORCE deletion; blinded served corruption caught by the sampling layer by name). 17/17, repeatedly, two independent runners.
• Red-teamed against 31 community-reported failure modes (qdrant#3372, #7851, #2620, #10135 …) — one real gap found, fixed, pinned as permanent drills.
• Kubernetes e2e (kind): the shipped ConfigMap + Job manifests as real pods — backup Job → exact S3 census → delete collection → restore Job → VERIFIED (points=3000/3000), plus a corruption negative leg failing the Job with the target untouched.
• Reviews passed: whole-branch principal-engineer (approve, zero must-fix), code-quality (sign-off grade; shellcheck: zero findings in the script), legacy-parity audit (complete replacement, exceptions documented).
• Still pending, after this PR: re-run the drill suite pinned to the customer's exact Qdrant version; validation on the BA IDST test system; then the staged production rollout from the RUNBOOK (shadow run → restore canary → cutover).