Summary
The filesystem store is constructed bare, so writes are not atomic:
// src/store/filesystem.rs
impl TryInto<ReadableWritableListableStorage> for &FilesystemStoreConfig {
fn try_into(self) -> Result<ReadableWritableListableStorage, Self::Error> {
let store = Arc::new(
FilesystemStore::new_with_options(self.root.clone(), self.opts.clone())
.map_py_err::<PyRuntimeError>()?,
);
Ok(store)
}
}zarrs/zarrs#421 added AtomicWriteStorageAdapter, which writes through <key>.tmp and renames. Wrapping the store in it would mean a process killed mid-write leaves the previous complete value rather than a truncated one.
Why this matters to us
We run per-position reconstruction jobs on a preemptible Slurm partition, writing sharded OME-Zarr to Lustre through zarr-python with this codec pipeline. A job killed mid-write left a truncated shard, and since our shard grid rounds up past the array bound, every later write to that shard had to read it back first and died on its checksum:
RuntimeError: the checksum is invalid
RuntimeError: The encoded shard is smaller than the expected size of its index.
Deterministic, so retries and resumes failed identically — one preemption could take out a whole run. Details in czbiohub-sf/iohub#415.
We have worked around it downstream (deleting the shards a write owns before writing, czbiohub-sf/iohub#455), but atomic writes fix it at the source: the truncated file would never exist.
Suggested change
Wrap the store, probably behind an option rather than unconditionally:
use zarrs::storage::storage_adapter::atomic_write::AtomicWriteStorageAdapter;
let store = Arc::new(FilesystemStore::new_with_options(...)?);
Ok(if self.atomic_writes { Arc::new(AtomicWriteStorageAdapter::new(store)) } else { store })surfaced the same way as the existing knobs (validate_checksums, direct_io, chunk_concurrent_*) via CodecPipelineImpl and zarr.config.
No dependency bump appears necessary: zarrs = "0.23.6" requires zarrs_storage = "^0.4.3", and the adapter shipped in zarrs_storage 0.4.5, so it already resolves.
Things worth weighing
- Cost. Every write becomes a temp write plus a rename, and #421 notes an atomic full-value fallback for partial writes. For whole-shard writes like ours that is close to free, but it is not free in general — which argues for opt-in, at least initially.
direct_io. Worth checking the interaction, since that path is O_DIRECT with page-size alignment and the temp-then-rename indirection is new to it.- Other backends.
obstore.rs and http.rs are out of scope here; AtomicRenameStorageTraits is implemented for FilesystemStore, which is the backend where a torn file is observable this way.
Happy to open a PR if the shape above looks right — mainly wanted to check whether opt-in or default-on is preferred before writing it.
Summary
The filesystem store is constructed bare, so writes are not atomic:
zarrs/zarrs#421 added
AtomicWriteStorageAdapter, which writes through<key>.tmpand renames. Wrapping the store in it would mean a process killed mid-write leaves the previous complete value rather than a truncated one.Why this matters to us
We run per-position reconstruction jobs on a preemptible Slurm partition, writing sharded OME-Zarr to Lustre through zarr-python with this codec pipeline. A job killed mid-write left a truncated shard, and since our shard grid rounds up past the array bound, every later write to that shard had to read it back first and died on its checksum:
Deterministic, so retries and resumes failed identically — one preemption could take out a whole run. Details in czbiohub-sf/iohub#415.
We have worked around it downstream (deleting the shards a write owns before writing, czbiohub-sf/iohub#455), but atomic writes fix it at the source: the truncated file would never exist.
Suggested change
Wrap the store, probably behind an option rather than unconditionally:
surfaced the same way as the existing knobs (
validate_checksums,direct_io,chunk_concurrent_*) viaCodecPipelineImplandzarr.config.No dependency bump appears necessary:
zarrs = "0.23.6"requireszarrs_storage = "^0.4.3", and the adapter shipped inzarrs_storage0.4.5, so it already resolves.Things worth weighing
direct_io. Worth checking the interaction, since that path isO_DIRECTwith page-size alignment and the temp-then-rename indirection is new to it.obstore.rsandhttp.rsare out of scope here;AtomicRenameStorageTraitsis implemented forFilesystemStore, which is the backend where a torn file is observable this way.Happy to open a PR if the shape above looks right — mainly wanted to check whether opt-in or default-on is preferred before writing it.