Build Out Store Offerings

[ilan-gold]

We currently only do file system store: https://github.com/ilan-gold/zarrs-python/blob/main/src/codec_pipeline_store_filesystem.rs but should expand to, at the minimum, HTTP.

• Integrate HTTP Store feat: add HTTP store support #61
• Flag for using Opendal (better performance? default for cloud urls like s3:// and gcs:// ?)
• Cache store? Somehow enabling https://docs.rs/zarrs/latest/zarrs/array/trait.ChunkCache.html if we detect an LRUCacheStore on the python side?
• Consolidated metadata: Add partial Zarr V3 consolidated metadata support zarrs#55
• Investigate exposing the zarr-python ByteGetter ByteSetter to Rust to potentially support any store (fallback only?)

[kylebarron]

👋 Long time no see!

I've been working on a library that may be helpful here: pyo3-object_store. It's designed to define builders once, so that many Python-Rust bindings can reuse the same ObjectStore integration.

It enables your functions that use object store to be as simple as:

#[pyfunction]
pub fn use_object_store(store: PyObjectStore) {
    let store: Arc<dyn ObjectStore> = store.into_inner();
}

I've been meaning to update and publish a new release of pyo3-object_store. It would be cool to prototype an integration here.

[ilan-gold]

Hi @kylebarron funny how that happens.

I think (@LDeakin can correct me) that one would need to implement the necessary traits on the object_store object from https://docs.rs/zarrs_storage/0.3.0/zarrs_storage/index.html to get a compatible store for zarrs (to work with https://github.com/ilan-gold/zarrs-python/blob/main/src/store/manager.rs#L17).

But where that happens and how is probably more a matter of discussion. That would be very cool though!

[kylebarron]

That's this, right? https://docs.rs/zarrs_object_store/latest/zarrs_object_store/

[ilan-gold]

Ah ok, @LDeakin is very comprehensive!

[LDeakin]

Cool stuff @kylebarron! Yeah, I'd be interested to see an integration

[ljstrnadiii]

What is the priority to support S3? Is there any interest? Any rough outline of what would need to be done to enable?

[LDeakin]

Is there any interest?

Yea! At the Zarr summit, there was a lot of interest to support more stores in zarrs-python.

Any rough outline of what would need to be done to enable

Two options:

1. It could be wrapped similarly to the HTTPStore
   • S3 has quite a few options compared to a HTTP store
   • This is very tedious to do for each store, and is brittle
2. We integrate with pyo3-object_store in some way so we can just use any obstore store

The latter is preferable, and perhaps @kylebarron could give some advice if you (or anyone else) would be willing to attempt it.

[kylebarron]

pyo3-object_store provides helpers to give you an Arc<dyn ObjectStore> in your Python API.

So you can create a Python function that creates a Zarr store backend with

use zarrs_object_store::AsyncObjectStore;

#[pyfunction]
pub fn use_object_store(store: AnyObjectStore) {
    let store: Arc<dyn ObjectStore> = store.into_dyn();
    let zarr_backend = AsyncObjectStore::new(store);
}

I don't know the layout of zarrs-python well enough to know how to integrate it there

[ljstrnadiii]

Sweet--thanks for the guidance! I am a total Rust newb, but looking for a project to help me learn. I might take a stab at this over the next few weeks, but do not let me hold anyone back from implementing.

[ilan-gold]

@ljstrnadiii Something to note here - we do not have an async API at the moment. All io (i.e., from s3) will be parallel but not concurrent. So you may not see any grand performance improvement. The route to an async API is a little hairy but I've poked around at it a bit - I don't see any fundamental limitations here, though, but a first-time rust PR, it might not be.

[kylebarron]

What's the blocker for async? I don't understand zarr-python well enough to know how this integration works.

It looks like here:

zarrs-python/src/store.rs

Lines 22 to 26 in fbe8827

	pub enum StoreConfig {
	Filesystem(FilesystemStoreConfig),
	Http(HttpStoreConfig),
	// TODO: Add support for more stores
	}

Zarrs-python is storing configs rather than instances.

I think we may have discussed this before; but is there a central requirement to have the configs as you have them? Also does it change anything now that there's an obstore integration in zarr-python?

[ilan-gold]

What's the blocker for async? I don't understand zarr-python well enough to know how this integration works.

More than anything, no one has had the time to do it. Lachlan, Phil, and I all work with local data.

In terms of code, https://github.com/zarrs/zarrs-python/blob/main/src/lib.rs#L298 was giving me problems because of the lifetime + tokio. Maybe someone who has done more async rust could find a way around it, maybe just Arcing it works, not sure. If there's no way around it, we might have to create our own "lazy" NDBuffer class, but I assume this is a me-limitation and not a rust-limitation.