Currently the zarr storage API comprises the MutableMapping interface from the Python standard library, plus some optional methods listdir, rmdir and getsize. This issue is intended for discussion of possible further optional extensions to the storage API to support two different but related functionalities. I personally don't have immediate concrete use cases for this, but I expect these will arise at some point, so this issue is intended just to park some initial thoughts.
Batch operations
The first functionality is supporting batch operations. For example, when a user is writing data to a region of a zarr array that spans more than one chunk, e.g.:
store = # any zarr-compatible store
z = zarr.array(shape=100, chunks=10, store=store)
z[:] = 42
...currently the zarr core module will communicate the modification for each chunk via a separate API call to the storage layer. In concrete terms, this means multiple calls to the __setitem__() method on the store, one for each key/value pair comprising the new encoded data for each modified chunk.
There are a number scenarios where a store implementation might be able to improve performance or provide some other useful functionality if it were made aware that multiple keys were being updated as part of the same high-level operation. For example, a cloud store or any store where network communication is involved might be able to reduce latency overheads by batching multiple key/value updates into a single request.
Similarly, when a user is reading data from a region of a zarr array that spans more than one chunk, currently the data for each chunk is retrieved from the storage layer via a separate API call to the __getitem__() method on the store. A store might be able to provide some optimisation if it were passed all keys to retrieve in a single API call.
In order to support batch writes, the MutableMapping interface already provides the update() method, and this would provide a possible implementation path. To make use of this, stores that could perform batch writes would provide an appropriate implementation of update(). We could then modify the zarr core module to make use of update() rather than making multiple __setitem__() calls.
In order to support batch reads, there is no appropriate method on the MutableMapping interface, so we'd have to define an optional method. Again, some modification to the zarr core module would be needed to take advantage of it.
Transactions
The second functionality is supporting transactions. In general, a user might want to use a store that supports transactions, and have full control over the granularity (i.e., beginning and end) of transactions, to fit whatever is the logic of the application. Where a store supports transactions, it might be useful to have a standard API, so a user could do something like:
store = # some store that supports transactions
root = zarr.group(store=store)
with store.transaction():
a = root.create('foo', shape=100, chunks=10)
a[:] = 42
b = root.create('spam/eggs', shape=1000000, chunks=100000)
b[:500] = 1
# and anything else that should be committed together
Discussion
This is just an initial sketch of some thoughts for discussion, please feel free to comment and share ideas. I'm aware the language above is also a bit vague, so please feel free to suggest clarifications or cleaner ways of thinking about these related issues.
Currently the zarr storage API comprises the MutableMapping interface from the Python standard library, plus some optional methods
listdir,rmdirandgetsize. This issue is intended for discussion of possible further optional extensions to the storage API to support two different but related functionalities. I personally don't have immediate concrete use cases for this, but I expect these will arise at some point, so this issue is intended just to park some initial thoughts.Batch operations
The first functionality is supporting batch operations. For example, when a user is writing data to a region of a zarr array that spans more than one chunk, e.g.:
...currently the zarr core module will communicate the modification for each chunk via a separate API call to the storage layer. In concrete terms, this means multiple calls to the
__setitem__()method on the store, one for each key/value pair comprising the new encoded data for each modified chunk.There are a number scenarios where a store implementation might be able to improve performance or provide some other useful functionality if it were made aware that multiple keys were being updated as part of the same high-level operation. For example, a cloud store or any store where network communication is involved might be able to reduce latency overheads by batching multiple key/value updates into a single request.
Similarly, when a user is reading data from a region of a zarr array that spans more than one chunk, currently the data for each chunk is retrieved from the storage layer via a separate API call to the
__getitem__()method on the store. A store might be able to provide some optimisation if it were passed all keys to retrieve in a single API call.In order to support batch writes, the
MutableMappinginterface already provides theupdate()method, and this would provide a possible implementation path. To make use of this, stores that could perform batch writes would provide an appropriate implementation ofupdate(). We could then modify the zarr core module to make use ofupdate()rather than making multiple__setitem__()calls.In order to support batch reads, there is no appropriate method on the
MutableMappinginterface, so we'd have to define an optional method. Again, some modification to the zarr core module would be needed to take advantage of it.Transactions
The second functionality is supporting transactions. In general, a user might want to use a store that supports transactions, and have full control over the granularity (i.e., beginning and end) of transactions, to fit whatever is the logic of the application. Where a store supports transactions, it might be useful to have a standard API, so a user could do something like:
Discussion
This is just an initial sketch of some thoughts for discussion, please feel free to comment and share ideas. I'm aware the language above is also a bit vague, so please feel free to suggest clarifications or cleaner ways of thinking about these related issues.