Manage dependencies with pixi

[maxrjones]

Moves from using conda environment files to using pixi to store all dependency information in pyproject.toml

Check out the updated contributing guide to get started.

TODO:

• Update readthedocs build setup
• Add quickstart to PR description
• Set up environments for minimum and upstream tests
• Update contributor guide
• Check that release workflow doesn't need updating
• Add release notes

Closes #310

[TomNicholas]

At the risk of being really annoying, should we be switching to pixi or to uv here?

[maxrjones]

At the risk of being really annoying, should we be switching to pixi or to uv here?

It doesn't matter to me as long as we find a way to install all dependencies without using conda environment files. Pip installing h5py doesn't install the netcdf4/hdf5 C libraries which will give the user a module not found error when trying to use those readers. So, in this PR I use the conda distributions of those libraries in

VirtualiZarr/pyproject.toml

Lines 121 to 124 in a2b5461

	[tool.pixi.feature.hdf5-lib.dependencies]
	netcdf4 = ">=1.7.2,<2"
	hdf5 = ">=1.14.4,<2"
	h5netcdf = ">=1.5.0,<2"

Do you know if the wheels for the underlying C libraries are available via PyPI such that we could meet this goal using uv without pixi?

[TomNicholas]

Thanks for the clarification.

There seem to be some wheels listed to download for all of these projects:

https://pypi.org/project/netCDF4/#files
https://pypi.org/project/h5py/#files
https://pypi.org/project/h5netcdf/#files

[maxrjones]

There seem to be some wheels listed to download for all of these projects:

good to know, thanks. since this PR will really change the developer experience I'll just open RFC PRs for both pixi and uv and see which people prefer. Probably won't get to it until over the weekend though

[TomNicholas]

Thank you @maxrjones ! Really appreciate your efforts here.

[abarciauskas-bgse]

LGTM! I'll let @TomNicholas provide official approval.

[abarciauskas-bgse]

Should zarr-python from github be an upstream dependency? I know it's also in default dependencies but just curious because zarr-python also a dependency of icechunk.

[maxrjones]

Since we're using Zarr's internal metadata classes in the develop branch, I think it's important to explicitly test for compatibility with the latest, unreleased changes in Zarr-Python.

[TomNicholas]

This looks great, I just have a couple of questions so I can understand the choices better.

[TomNicholas]

What defines the min-deps dependencies now? The dependencies in the pyproject.toml?

[maxrjones]

yeah it's in pyproject.toml and defined is the combination of these dependency sets:

VirtualiZarr/pyproject.toml

Line 171 in efcb153

min-deps = ["dev", "hdf", "hdf5", "hdf5-lib"] # VirtualiZarr/conftest.py using h5py, so the minimum set of dependencies for testing still includes hdf libs

It's unfortunate the s3 (included as part of dev and the hdf libraries are required right now for running any tests (this was already the case) and would be good to refactor the conftest.py modules so this isn't the case.

I also added a line to the contributing guide showing how to list the installed environments.

[TomNicholas]

Why is it necessary to make running the tests a pixi thing, rather than just installing dependencies using pixi and then running tests using pytest directly?

[maxrjones]

This is a shorthand for running a command in a specified shell. I added a note on how to explicitly start a shell to run pytest in 86ff959 (#407)

[TomNicholas]

Oh that's cool. Will that work with #486?

[maxrjones]

yeah, unfortunately they'll have some merge conflicts to resolve but it uses the same sphinx-autobuild extension they reference.

[TomNicholas]

Amazing @maxrjones thank you so much!