Rework save system with ZarrV3 two-tier storage by JoeyBF · Pull Request #6 · JoeyBF/sseq

JoeyBF · 2026-04-07T00:26:42Z

Summary

Replaces the ad-hoc save format with a ZarrV3-backed layout via the zarrs crate. Clean break from the old format — no migration script.

The full design doc is in the commit message; this PR description is just the headline points.

Why

Problem	Master	This PR
File count	720K files at stem 200; OOM on 40 TB at stem 300	Sharding consolidates the long tail; well below master
Compression	None at write time, external zstd	zstd in-format; CRC32C on every chunk
Integrity	Adler32 tail	CRC32C per chunk
Format	Hand-rolled `to_bytes`/`from_bytes` plumbing	Structured zarr arrays matching the actual data shape

Architecture

Two tiers within one zarr store:

Shard tier (small kinds)

kernel, differential, augmentation_qi, nassau_differential, secondary_*, chain_map, chain_homotopy. One sharded vlen-bytes array per kind, shape (N_SPAN, S_SPAN[, IDX_SPAN]) = (4096, 1024[, 256]), shard (8, 8[, 8]), inner chunk [1, 1[, 1]]. CRC32C per shard, no zstd. Tens of thousands of zarr elements collapse into hundreds of shard files per kind.

Stream tier (large kinds, structured)

res_qi and nassau_qi get groups, not single arrays:

res_qi/: pivots/ (1D i64) + rows/ (2D u8 [image_dim, num_limbs * 8], chunked over rows). ResQiReader walks pivots and fetches matrix chunks on demand.
nassau_qi/: commands/ (1D vlen-bytes, one element per state-machine command — signature change, fix, or pivot operation with embedded lift+image). Header (target_dim, zero_mask_dim, subalgebra_profile, num_commands, finished) lives in group attributes. NassauQiReader yields a typed NassauCommand enum one at a time.

The finished group attribute is the source of truth for atomicity: writers dropped before finish() leave it false and the matching reader treats the QI as missing so callers recompute. Memory during read or write is bounded by chunk shape, regardless of total payload size — the multi-GB nassau_qi case never materialises.

Coordinates

Shard arrays are indexed by (n, s) (= MultiDegree<2>::coords()), not (s, t). Tighter square bound, generalises to higher-N gradings. Negative n is supported via a hidden N_MIN = -1024 offset (zarr v3 has no native negative indices); the bound is generous enough that no caller needs to know about it.

Public coords API is the const-generic SaveCoords<const N: usize> trait, with Bidegree: SaveCoords<2> and BidegreeGenerator: SaveCoords<3> — methods that only make sense in one dimension reject the other type at compile time.

Subgroups for named homomorphisms

Named ResolutionHomomorphism saves into products/{name}/; named ChainHomotopy saves into homotopies/{left}__{right}/. Anonymous endpoints disable saving entirely (matching master and fixing a latent collision bug from examples/massey.rs). The four secondary kinds therefore appear in three places: at the root for the main resolution, under products/{name}/ for a named hom's secondary lift, and under homotopies/{l}__{r}/ for a chain homotopy's secondary lift. Subgroups share the underlying FilesystemStore via Arc clone — they're just a path prefix, not a separate store.

Concurrency

zarrs::Array::store_array_subset documents (since 0.14) that callers must serialise concurrent invocations on regions sharing chunks — the per-chunk locks were removed because the old default deadlocked. We honour the contract with a per-SaveKind Arc<Mutex<()>>. We also pass CodecOptions::with_concurrent_target(1) on sharded writes, because the sharding codec uses into_par_iter internally and a rayon worker holding our std::sync::Mutex could otherwise deadlock via work-stealing.

Verified by reproduction: without these two fixes, secondary_massey --features nassau,concurrent panics on resume from a persistent save dir within a few iterations with Expected header with 1 elements, got 0 from the vlen-bytes codec. With them, 10 consecutive runs produce identical md5s.

Other notes

bitcode (with the serde feature) replaces hand-rolled to_bytes/from_bytes for everything in the shard tier; the relevant fp types get #[derive(Serialize, Deserialize)]. aligned-vec gains its serde feature.
FpVector::num_limbs becomes pub so structured row buffers can be sized without poking at private internals.
QuasiInverse::stream_quasi_inverse, MilnorSubalgebra::{to,from}_bytes, the Magic enum, the SaveDirectory::Split HPC workaround, and ZarrWriter/ZarrReader (the io::Write/io::Read blob shims) are all gone.

Test plan

All 6 tests/save_load_resolution.rs tests pass (debug + release)
Full cargo test is green across the workspace
Manual reproduction at S_2 stem 30/60 shows file count well below master
secondary_massey --features nassau,concurrent against a persistent save dir, 10 consecutive runs → identical md5s
Production run at stem 200+ to confirm the 40 TB scenario is fixed
Production-scale nassau_qi (multi-GB) read/write end-to-end

🤖 Generated with Claude Code

Rebases the zarrs-save-rework branch (6 save-system commits) onto master. Master had since split ext::secondary into per-primary submodules (SpectralSequences#254) and made the secondary lift entry points return Result (SpectralSequences#241, SpectralSequences#242), so the conflicts were confined to ext/src/secondary.rs. Resolution: - Adopt the store-based save API (store.read/write/exists/delete) in the shared SecondaryHomotopy / SecondaryLift code while preserving master's Result-returning try_compute_homotopy_step. - Drop the stale monolithic duplicates of SecondaryResolution, SecondaryResolutionHomomorphism, and SecondaryChainHomotopy; those types now live in submodules under resolution.rs / chain_homotopy.rs / resolution_homomorphism.rs. - Remove the per-new() SaveKind::secondary_data() create_dir loops from the moved constructors: the reworked store creates its shard arrays lazily on first write, and create_dir no longer exists on SaveKind. Dropped from the original branch: the throwaway "experimental rayon build" commit (patched rayon to a personal fork) and the earlier "tracing to parallel guard" commit (master ships a superior version). All 7 save_load_resolution tests and the ext lib unit tests pass.

Two hot-path improvements to ZarrSaveStore that remove overhead the rebase inherited, with no change to the on-disk format or public API. 1. Cache opened Array handles (`arrays: DashMap<SaveKind, Arc<ShardArray>>`). Previously every read/write/delete called `zarrs::array::Array::open`, which re-reads and re-parses the array's `zarr.json` through the storage backend on each call. The handle holds no chunk data and its methods take `&self`, so a cached `Arc<Array>` is safe to share for concurrent reads and (shard-serialized) writes. Now the metadata is parsed at most once per kind per store. Read/delete of a not-yet-created kind still returns None/Ok as before. 2. Key the write lock by `(kind, shard coords)` instead of by kind alone. The zarrs read-modify-write contract only requires serializing writes that share a shard; the old per-kind lock needlessly serialized every write of a kind across the whole store. Writes to different shards touch disjoint chunk files, so per-shard locking restores the cross-shard parallelism a parallel resolution depends on while still honouring the contract. Kept `with_concurrent_target(1)` so the sharding codec stays sequential and the rayon-join deadlock the original guarded against cannot occur. Verified on S_2 through-stem (n=70, s=40), release + concurrent, 4 cores: best-of-3 wall time 15.1s -> 13.0s (~14% faster) with identical Ext output. Save+resume roundtrip re-checked for both the standard and nassau backends; all lib and save_load_resolution tests pass.

…_LEVEL The rebased branch hardcoded zstd level 19 for the stream tier (res_qi, nassau_qi), which is on the hot write path — every differential quasi-inverse is compressed there. Benchmarked against master's uncompressed old format (S_2 through-stem n=70 s=40, release + concurrent, 4 cores, best of 3): master (old, uncompressed): 2.55s write, 51M on disk (real bytes) zarr, zstd level 19: 13.0s write, 23M zarr, zstd level 3: 3.65s write, 24M Level 19 was ~5x slower than master for a 4% smaller store than level 3 on this dense, high-entropy data. Level 3 (zstd's own default) keeps nearly all the compression — still less than half of master's on-disk size — at a fraction of the write cost, cutting the regression from ~5x to ~1.4x. The read/resume path is unchanged and already at parity with master (~0.10s). Compression still matters for very large runs, so the level is read once from the EXT_SAVE_ZSTD_LEVEL environment variable (clamped to zstd's [1, 22]; unparseable values warn and fall back to the default). Set e.g. EXT_SAVE_ZSTD_LEVEL=19 when the on-disk footprint, not save time, is the binding constraint.

A line beginning with "+ serde_json parse" was parsed as a markdown "+" list marker, so clippy (-D warnings in CI's lint job) flagged the following lines as list items without indentation. Reword to "and a serde_json parse". No code change.

Address the low-risk, verified subset of CodeRabbit's review on SpectralSequences#260: - ResQiReader::apply / into_quasi_inverse: replace assert!(got, ...) with anyhow::ensure!(...). Both are already anyhow::Result-returning, so a truncated stream now propagates an error instead of aborting the caller. - NassauQiReader::parse: reject odd-length Signature payloads before chunks_exact(2) silently drops a trailing byte. - Stream-kind guards in write/read/exists/delete: promote debug_assert! to assert! so misusing a stream kind (ResQi/NassauQi) on the shard-tier API is caught in release too, not just debug. Deliberately not applied from the same review: - "Also assert target_res_dimension in the differential load path" — this is a false positive: test_load_smaller legitimately resumes into a smaller target range, so the read-time resolution dimension differs from the saved one (the rows carry their own widths). The original code asserts only target_cc_dimension for exactly this reason. - Prime check in bind_to_algebra — redundant: magic() already encodes the prime (p << 16), so a prime mismatch already fails the existing magic check.

…From) Addresses the remaining actionable review finding on SpectralSequences#260. Store creation can fail (bad path, permissions, unreadable existing metadata), but the old From<Option<PathBuf>> swallowed that with .expect(), so new_with_save's anyhow::Result could never surface it. Replace the impl with TryFrom<Option<PathBuf>> (Error = anyhow::Error) and thread the fallible conversion through the entry points that take a save dir: Resolution / nassau::Resolution::new_with_save and utils::construct{,_nassau, _standard} now take `impl TryInto<SaveDirectory, Error = anyhow::Error>` and `?`-propagate. Every existing call site passes Option<PathBuf>, so this is source-compatible; a bad save path now returns an Err instead of panicking. Not applied from the same review (left for the author's call): name sanitization for subgroup paths, dimension-framing headers for chain-map/homotopy payloads, and per-complex store fingerprinting.

CI's lint job runs clippy on nightly, whose chunks_exact_to_as_chunks lint (not yet in the stable clippy I checked locally) rejects chunks_exact(2) with a constant size. Switch the NassauQi signature decode to as_chunks::<2>(); the even-length check just above guarantees an empty remainder, so this is equivalent. as_chunks is stable since 1.88, well under the toolchains CI tests.

Follow-up to the TryFrom change: the bound `TryInto<SaveDirectory, Error = anyhow::Error>` was stricter than the original `Into<SaveDirectory>` because the reflexive `TryInto<SaveDirectory>` for `SaveDirectory` has `Error = Infallible`, so a caller passing an already-built `SaveDirectory` was rejected. Widen the bound to `Error: Into<anyhow::Error>` (accepts both `Option<PathBuf>` with `anyhow::Error` and `SaveDirectory` with `Infallible`) and convert via `.map_err(Into::into)`, restoring the original API surface while keeping the fallible path for store creation.

- chain_homotopy.rs: restore the original ChainHomotopy field order (homotopies before save_dir). The rework had swapped them for no functional reason (field order here doesn't affect drop semantics or the constructor); revert to minimize churn. - save.rs: the wasm rationale said the filesystem store "doesn't compile for WASM", but that's specific to wasm32-unknown-unknown — wasm32-unknown-emscripten is `unix` and unaffected. Reword the module doc and the parallel inline comment to name the target precisely.

@hoodmane

Addresses @hoodmane's review: gate on what the dependencies actually require rather than target_arch = "wasm32", and stop repeating the cfg everywhere. The zarr `filesystem` feature pulls in `positioned-io::RandomAccessFile`, which is itself `cfg(any(windows, unix))`; the `zstd` feature's `zstd-sys` C build needs the same POSIX platform. So `any(windows, unix)` — not `target_arch = "wasm32"` — is the authoritative condition: `wasm32-unknown- emscripten` is `unix` and keeps the filesystem+zstd path, while `wasm32-unknown-unknown` (the web frontend) and `wasm32-wasi` fall back to the in-memory store with CRC32C-only codecs. - Cargo.toml: select the `zarrs` feature set on `cfg(any(windows, unix))` / `cfg(not(any(windows, unix)))`. - save.rs: collapse the ~14 scattered `#[cfg(target_arch = "wasm32")]` sites into two `#[cfg]`-gated `platform` submodules exposing `open_store`, `root_group_missing`, and `stream_tier_codecs` (plus the zstd-level machinery and the wasm `Send`/`Sync` impls). The rest of the file is now cfg-free. Behaviour on the two built targets (native, wasm32-unknown-unknown) is unchanged; only the previously-untargeted emscripten/wasi cases move. Verified: native check + nightly clippy/fmt, wasm32-unknown-unknown build of sseq_gui, and the save_load + lib test suites.

Addresses CodeRabbit's "bind to the complex" finding. Previously a save directory only recorded the algebra (magic + prime), so reusing one directory for a *different* module over the same algebra — e.g. resolve S_2, then point the same dir at C2 — would silently load the first module's cached differentials/kernels/quasi-inverses for the second: structurally valid, semantically wrong, and CRC-clean. - Add `ChainComplex::fingerprint()`: a stable content hash of the complex's structure (each bounded module's per-degree dimensions and the algebra action on every basis element, plus each differential's action — basis-sensitive, exactly what the resolution algorithm consumes). It walks `s` until the cached zero module (since `next_homological_degree` is `i32::MAX` for a `FiniteChainComplex`) under a safety cap, and skips unbounded modules. Hashing uses a fixed FNV-1a so the value is stable across runs and toolchains, unlike `std::hash::DefaultHasher`. - Rename `ZarrSaveStore::bind_to_algebra` -> `bind_to_complex`, taking the fingerprint and storing it (as a hex string, to survive the JSON round-trip exactly). On resume the stored fingerprint must match; a mismatch — including a store predating this attribute — fails loudly instead of mixing data. - New `test_wrong_complex`: S_2 then C2 in the same dir errors with "different complex". Existing save/load + resume tests still pass (same complex hashes equal, so resume is unaffected).

Supersedes the previous complex-fingerprint approach (commit f436040), which bound a save directory to its complex by storing an opaque 64-bit content hash. Storing the module's JSON spec instead does the same identity job while being readable, inspectable, and reusable for reconstruction — one artifact serving three purposes. - Revert `ChainComplex::fingerprint()`/`Fnv` and restore `bind_to_algebra` (algebra magic/prime/prefix). The complex-identity gate now lives in the new `ZarrSaveStore::bind_module_spec`, called from `construct`/`construct_nassau` where the spec is actually known: on a fresh store it records `module_spec`, on an existing one it must match exactly, else it bails. `test_wrong_complex` (S_2 then C2 in one dir) still fails loudly, now via the spec comparison. - `construct_from_save(dir)`: rebuild and resume a resolution from a save directory alone — reads the recorded `module_spec` + `algebra_prefix`, rebuilds the `Config`, and resolves back into the same dir. No need to re-supply what the directory already knows it holds. - `set_complex_name`: record the short human-readable label (the module spec the user typed, e.g. "S_2") as a `complex_name` attribute, a concise companion to the full `module_spec`, so `zarr.json` says what it resolves. - Fix a broken rustdoc intra-doc link (`FilesystemStore`) that failed the docs build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJt3nb5Fp6okurqLzcQbCa

The `set_name` doc comment calls the `set_complex_name` store write "best-effort and purely informational", but it used `.expect()`, so a transient store failure would abort the whole resolution over a label that doesn't guard anything (the real load guard is `bind_module_spec`, which still propagates its errors). Log a warning and continue instead. Applies to both the standard and Nassau `set_name`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJt3nb5Fp6okurqLzcQbCa

JoeyBF force-pushed the zarrs-save-rework branch 5 times, most recently from 6b16e00 to ca421eb Compare April 12, 2026 02:06

JoeyBF force-pushed the zarrs-save-rework branch from ca421eb to 28ff3ff Compare April 17, 2026 10:56

JoeyBF force-pushed the master branch 2 times, most recently from 133b60c to ef263a9 Compare June 22, 2026 01:53

claude added 3 commits July 8, 2026 01:31

JoeyBF force-pushed the zarrs-save-rework branch from e0657b7 to 4979714 Compare July 8, 2026 02:28

claude added 10 commits July 8, 2026 02:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework save system with ZarrV3 two-tier storage#6

Rework save system with ZarrV3 two-tier storage#6
JoeyBF wants to merge 13 commits into
masterfrom
zarrs-save-rework

JoeyBF commented Apr 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

JoeyBF commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Architecture

Shard tier (small kinds)

Stream tier (large kinds, structured)

Coordinates

Subgroups for named homomorphisms

Concurrency

Other notes

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JoeyBF commented Apr 7, 2026 •

edited

Loading