Build requirements are too strict

[mtelka]

I'd like to package pytest-mypy for OpenIndiana, but it is very hard to do so because build requirements for pytest-mypy are these (from pyproject.toml):

[build-system]
requires = ['setuptools ~= 50.3.0', 'setuptools-scm[toml] ~= 5.0.0', 'wheel ~= 0.36.0']
build-backend = 'setuptools.build_meta'

The latest acceptable versions of required tools are about two years old now.

Could you please update pytest-mypy to relax its build requirements?

Thank you.

[dmtucker]

We can certainly update it (and possibly relax the pins).

What versions do you have/need?

[mtelka]

I prefer requirements without upper bounds, or something obvious, for example setuptools-scm<8 (or python<4). I do not care about lower bounds, because I can always upgrade. The aim of OpenIndiana is to provide latest versions of all projects so any lower bounds is okay. BTW, our currently packaged versions are these: setuptools-65.5.1, setuptools-scm-7.0.5, wheel-0.38.4.

Thank you.

[dmtucker]

Thanks!
Just for clarity/posterity, are using the PEP 517 build process (i.e. building in an isolated virtualenv)?
Also, are you not able to access PyPI during the build?

[mtelka]

PEP 517 (AFAIK) does not require strictly isolated build environment. We build in real (current) environment because built packages will be run in real environment too[1]. This is the reason why we cannot easily provide multiple versions of single package (e.g. setuptools) to fulfill strict requirements of packages like pytest-mypy. That would be possible with virtualenv-like environments only and we want to avoid that if possible.

Since we build in real (current) environment we are able to access anything we want (including PyPI). We also test in real (current) environment too to make sure the package integration is correct on works as expected. For tox based testing (this will include pytest-mypy once packaged) we use tox-current-env plugin to force (or fool) tox to not create isolated test environments, but use the real (current) one.

Thank you.

[1] I believe other distros like Fedora or Ubuntu does it similarly.

[dmtucker]

We build in real (current) environment because built packages will be run in real environment too

FWIW, this project does not depend on any of its build dependencies at runtime.

we cannot easily provide multiple versions of single package (e.g. setuptools) to fulfill strict requirements of packages like pytest-mypy.

Avoiding this is an expressly stated goal of PEP 517 isolation:

It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment

Anyways... Give 0.10.2 a go when you get a chance 🙂

[mtelka]

We build in real (current) environment because built packages will be run in real environment too

FWIW, this project does not depend on any of its build dependencies at runtime.

Yes, sure, but for building the build dependencies needs to be installed on a build machine so build dependencies are de facto runtime dependencies needed during the build.

we cannot easily provide multiple versions of single package (e.g. setuptools) to fulfill strict requirements of packages like pytest-mypy.

Avoiding this is an expressly stated goal of PEP 517 isolation:

It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment

Yes, exactly, the "global environment" is the key here. To solve that we would need to build in isolated/virtualized environment and we want to try to avoid that because it would make the build environment too complex. And so far it looks viable because vast majority of Python projects seems to be already flexible enough with their deps. Where it is not the case (like it was for pytest-mypy before 0.10.2) it is usually straightforward to relax the deps. BTW, the number of projects with too strict build deps is about 1-2% only so it looks like it is better to ask such projects to relax their build deps (for their own benefit too: they could build with newer build tools after the change to possibly get some new features and bugs fixed) than to implement virtual build env at our side.

Anyways... Give 0.10.2 a go when you get a chance slightly_smiling_face

It is in queue waiting for its dependencies to get packaged, namely mypy. But at least we can now satisfy build requirements. So big thank you for relaxing the build deps and releasing new version! I really appreciate.

[dmtucker]

NP, thanks for reaching out! I've not had much experience with OS packaging before.

we want to try to avoid that because it would make the build environment too complex.

What exactly is the build process? Do you use build? Since this project does not have any platform-specific dependenices I would think you could use the pre-built wheel instead of building at all 🤔

BTW, the number of projects with too strict build deps is about 1-2%

It's definitely a controversial practice. I'll give you that 😋

for their own benefit too: they could build with newer build tools after the change to possibly get some new features and bugs fixed

This is a trade-off, though, because it means builds that use the latest available versions will start to fail once the build tools make a backwards-incompatible release.

[mtelka]

NP, thanks for reaching out! I've not had much experience with OS packaging before.

we want to try to avoid that because it would make the build environment too complex.

What exactly is the build process? Do you use build? Since this project does not have any platform-specific dependenices I would think you could use the pre-built wheel instead of building at all thinking

The build process exactly is this: https://github.com/OpenIndiana/oi-userland/blob/f3877e672d82783b9d41a0c81f55d3dd01a85067/make-rules/pyproject.mk#L72.
Yes, we use build (then install using installer). We prefer sdists for many reasons. One of them is that they often contains tests and we like to test our packages to make sure everything is okay. OpenIndiana is not mainstream OS that everybody test their projects on so we need to do that.

BTW, the number of projects with too strict build deps is about 1-2%

It's definitely a controversial practice. I'll give you that yum

Maybe, but that's on discretion of every particular project. And, as I said, majority of them are not very strict.

for their own benefit too: they could build with newer build tools after the change to possibly get some new features and bugs fixed

This is a trade-off, though, because it means builds that use the latest available versions will start to fail once the build tools make a backwards-incompatible release.

Sure. To avoid that a project could set some reasonable upper boundary and once it is crossed (by setuptools, for example) the project could (should) adapt and set new upper boundary. I think the best way how to handle that is to use a build backend with strict semver (I'm not sure if setuptools uses semver or not) to avoid surprises.

Thank you.