@matear & @e-peacey
RE: the conversation about how to best use zarr on Gadi I have my own issue here that I've not yet had the chance to make progress on. write code to convert zarr collections to zarr-zipstore
While I don't yet have any direct experience with using zarr-ZipStore my understanding is it doesn't appear to effect performance much ( this needs to be tested ) and it solves the inode problem that cloud optimised formats like zarr have on linux filesystems. Normally zarr looks like "one file per chunk" to a linux FS which is typically a much larger inode footprint than netcdf.
See: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore
@matear & @e-peacey
RE: the conversation about how to best use
zarronGadiI have my own issue here that I've not yet had the chance to make progress on. write code to convert zarr collections to zarr-zipstoreWhile I don't yet have any direct experience with using
zarr-ZipStoremy understanding is it doesn't appear to effect performance much ( this needs to be tested ) and it solves theinodeproblem that cloud optimised formats likezarrhave on linux filesystems. Normally zarr looks like "one file per chunk" to a linux FS which is typically a much largerinodefootprint thannetcdf.See: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore