Cannot use ninja from a venv

[stephanlachnit]

I'm currently having an issue using meson-python in an environment where I cannot install ninja system-wide.

In particular, I'm in a venv

python3 -m venv venv
source venv/bin/activate
pip install -e . # has `requires = ["meson-python"]`

Which results in the following ninja:

Found ninja-1.11.1.git.kitware.jobserver-1 at /tmp/pip-build-env-3_c2bhz3/normal/bin/ninja

However, this doesn't work in the venv:

>>> import constellation
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1078, in _find_spec
  File "/home/stephan/Projects/constellation/venv/lib/python3.11/site-packages/_constellation_editable_loader.py", line 271, in find_spec
    tree = self.rebuild()
           ^^^^^^^^^^^^^^
  File "/home/stephan/Projects/constellation/venv/lib/python3.11/site-packages/_constellation_editable_loader.py", line 312, in rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
  File "/usr/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-env-3_c2bhz3/normal/bin/ninja'

I also tried installing ninja in the venv and setting the NINJA env var, but this also doesn't work:

meson-python: error: Could not find ninja version 1.8.2 or newer.

which I think is because this is called outside of the venv - ninja --version in the venv is fine.

[eli-schwartz]

pip install creates a new venv to use instead, due to "build isolation".

meson-python will anyways install ninja as a pip-based build dependency if you don't have one on PATH, so no need to do so manually.

That's actually the problem. Since you're using an editable install, the ninja it installs inside the build isolation venv no longer exists when you try to rebuild the package.

You CANNOT use build isolation in combination with editable installs. Sadly, pip defaults to doing this so you have to magically know it doesn't work and disable it.

Editable installs are... advanced use cases, basically.

[rgommers]

You have to use:

pip install -e . --no-build-isolation

See the docs in https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html for the background on why --no-build-isolation is necessary.

[stephanlachnit]

Ah ok - so even when I want to use ninja from a venv, I need to pass --no-build-isolation?

[stephanlachnit]

Ah right, and with --no-build-isolation I need to install meson-python beforehand - quite annoying, thanks for your help!

[eli-schwartz]

Ah right, and with --no-build-isolation I need to install meson-python beforehand - quite annoying, thanks for your help!

Yeah but if you don't pass no-build-isolation then pip installs meson-python plus meson, compiles the project once, and uninstalls meson-python plus meson.

You might have meson already installed. I guarantee many other users of editable installs don't. :D

[stephanlachnit]

Yeah but if you don't pass no-build-isolation then pip installs meson-python plus meson, compiles the project once, and uninstalls meson-python plus meson.

I guess this is the annoying part - it would be nice if one could tell pip to leave those package persistently

[rgommers]

it would be nice if one could tell pip to leave those package persistently

pip devs don't seem to have much interest in this, but someone should just write a small wrapper to install build deps listed in pyproject.toml into the development environment and then invoke pip install -e . --no-build-isolation. This would be the most sensible behavior for any project with compiled code, independent of which build backend is used.

[peppedilillo]

Dears, I try to comment here, will open a separate issue if that's better. I'm looking for help with editable modes. Using pip install -e . --no-build-isolation I can correctly install the package and compile my C code, which is called through ctypes.
However once I try to use the package it fails to find the shared libraries:

  File "/home/deppep/Dropbox/Progetti/hbstools/hbstools/hbstools/triggers/bft_cwrap.py", line 16, in <module>
    clib_bft = ctypes.CDLL(_LIBCFOCUS)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/deppep/Dropbox/Progetti/hbstools/hbstools/hbstools/triggers/.sharedlibs/lib-pfocus.so: cannot open shared object file: No such file or directory

The _LIBCFOCUS above is implemented here:

def _library_path(libname: str) -> str:
    """Gets the path to the C shared library, with appropriate extension.
    Assumes the library and this file to be installed in the same directory"""
    from pathlib import Path
    import sys

    if sys.platform.startswith("win32"):
        suffix = ".dll"
    elif sys.platform.startswith("linux"):
        suffix = ".so"
    elif sys.platform.startswith("darwin"):
        suffix = ".dylib"
    else:
        raise OSError("System not supported")

    return str(Path(__file__).parent / f"{libname}{suffix}")


_LIBCFOCUS = _library_path(".sharedlibs/lib-pfocus")

The problem is that, once in editable mode, the code looks for the shared libraries in the wrong place. While not in editable mode, the libraries are installed in my local environment directory (site-packages/hbstools/triggers/.sharedlibs/lib-pfocus.so). However, once in editable-mode, I do not understand where they are actually installed, but they are expected to be in my source directory.

Is it possible to install my project in editable mode? Thank you.

[eli-schwartz]

ctypes can be tricky to get correct.

For actual (C-API) bindings, on Linux, meson will set up RPATH entries for editable install usage so that loading a C-API extension works. And IIRC meson-python uses patchelf to ensure those entries are relativized in the wheel as well. macOS needs a different approach, and Windows simply does not support RPATH at all so you need custom, per-library code.

Use of ctypes cannot handle this automatically the way RPATH can, because it's all manually coded paths (basically the same as Windows). Your best bet is to have bft_cwrap.py in the build directory (e.g. with meson fs_mod.copyfile()) and try both the .sharedlibs location (for standard wheel installs) and look for the library next to __file__ (for editable installs).

[rgommers]

@peppedilillo if @eli-schwartz's reply is not enough yet to help you solve the problem, please copy your original question, the reply and what you want to ask next to a new discussion.

(try to not comment on closed issues unless you're very sure it's the same thing, which this is not)