BUG: Hang on Windows in scikit-learn with 1.13rc1 and 1.14.dev (maybe due to OpenBLAS 0.3.26 update?)

[lesteve]

Describe your issue.

This was originally seen in MNE-Python CI roughly a week ago (March 13) using the scipy developement wheel (scientific-python-nightly-wheel) and reported in scikit-learn see scikit-learn/scikit-learn#28625 for more details.

This can also be reproduced with scipy 1.13rc1. Could it be due to updating OpenBLAS to 0.3.26?

The scikit-learn code does nested parallelism, OpenBLAS within OpenMP. Setting OMP_NUM_THREADS=1 or OPENBLAS_NUM_THREADS=1 avoids the hang. A similar work-around can be used with threadpoolctl.

Insights or suggestions more than welcome!

Reproducing Code Example

from sklearn.metrics._pairwise_distances_reduction import ArgKmin
import numpy as np
import threadpoolctl

# Commenting any of these two lines avoids the hang
# threadpoolctl.threadpool_limits(limits=1, user_api='blas')
# threadpoolctl.threadpool_limits(limits=1, user_api='openmp')
X = np.zeros((20, 14000))
ArgKmin.compute(X=X, Y=X, k=10, metric='euclidean')

Error message

Hang, i.e. the script never completes. With one of the work-around it completes in less than one second.

SciPy/NumPy/Python version and system information

1.13.0rc1 1.26.4 sys.version_info(major=3, minor=12, micro=2, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= ZEN MAX_THREADS=24
    pc file directory: c:/opt/64/lib/pkgconfig
    version: 0.3.26
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= ZEN MAX_THREADS=24
    pc file directory: c:/opt/64/lib/pkgconfig
    version: 0.3.26
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.9
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  pythran:
    include directory: C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-t0bobn5y\overlay\Lib\site-packages/pythr
an
    version: 0.15.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-bxqwrdgx\cp312-win_amd64\build\venv\Scripts\python.exe
  version: '3.12'

[lucascolley]

cc @mattip

[lesteve]

Hmmm actually debugging further this does seem like an issue in OpenBLAS 0.3.26. I can reproduce the hang with conda-forge packages and scipy 1.12. With OpenBLAS 0.3.25 the snippet runs fine.

Conda env created with:

mamba create -n lof-issue-conda-2 scikit-learn ipython openblas=0.3.26 blas=*=*openblas* -y

Problematic environment info with OpenBLAS 0.3.26 (it does not seem like OpenBLAS 0.3.26 is mentioned somehow in the output below so I added sklearn.show_versions() below ...)

1.12.0 1.26.4 sys.version_info(major=3, minor=12, micro=2, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/include
    lib directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/lib
    name: blas
    openblas configuration: unknown
    pc file directory: C:\bld\scipy-split_1706041678996\_h_env\Library\lib\pkgconfig
    version: 3.9.0
  lapack:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/include
    lib directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/lib
    name: lapack
    openblas configuration: unknown
    pc file directory: C:\bld\scipy-split_1706041678996\_h_env\Library\lib\pkgconfig
    version: 3.9.0
  pybind11:
    detection method: pkgconfig
    include directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/include
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    commands: clang-cl
    linker: lld-link
    linker args: --target=x86_64-pc-windows-msvc, -nostdlib, -Xclang, --dependent-lib=msvcrt,
      -fuse-ld=lld, -Wl,-defaultlib:C:\bld\scipy-split_1706041678996\_build_env/Library/lib/clang/17/lib/windows/clan
g_rt.builtins-x86_64.lib
    name: clang-cl
    version: 17.0.6
  c++:
    commands: clang-cl
    linker: lld-link
    linker args: --target=x86_64-pc-windows-msvc, -nostdlib, -Xclang, --dependent-lib=msvcrt,
      -fuse-ld=lld, -Wl,-defaultlib:C:\bld\scipy-split_1706041678996\_build_env/Library/lib/clang/17/lib/windows/clan
g_rt.builtins-x86_64.lib
    name: clang-cl
    version: 17.0.6
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.7
  fortran:
    args: -D_CRT_SECURE_NO_WARNINGS, -D_MT, -D_DLL, --target=x86_64-pc-windows-msvc,
      -nostdlib
    commands: flang-new
    linker: lld-link
    linker args: --target=x86_64-pc-windows-msvc, -nostdlib, -Xclang, --dependent-lib=msvcrt,
      -fuse-ld=lld, -Wl,-defaultlib:C:\bld\scipy-split_1706041678996\_build_env/Library/lib/clang/17/lib/windows/clan
g_rt.builtins-x86_64.lib,
      -D_CRT_SECURE_NO_WARNINGS, -D_MT, -D_DLL, --target=x86_64-pc-windows-msvc, -nostdlib
    name: flang
    version: 17.0.6
  pythran:
    include directory: ..\..\_h_env\Lib\site-packages\pythran
    version: 0.15.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\bld\scipy-split_1706041678996\_h_env\python.exe
  version: '3.12'

sklearn.show_versions()

System:
    python: 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:42:31) [MSC v.1937 64 bit (AMD64)]
executable: C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.4.1.post1
          pip: 24.0
   setuptools: 69.2.0
        numpy: 1.26.4
        scipy: 1.12.0
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.3.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 4
         prefix: libomp
       filepath: C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Library\bin\libomp.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 4
         prefix: libblas
       filepath: C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Library\bin\libblas.dll
        version: 0.3.26
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 4
         prefix: vcomp
       filepath: C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\vcomp140.dll
        version: None

[mattip]

@martin-frbg does this look familiar?

The scikit-learn code does nested parallelism

@lesteve do you know how many threads/processes are opened?

[martin-frbg]

Not familiar, also not aware of any change in 0.3.26 that would be likely to cause it if the library was built with OpenMP support. (If not, there is a small chance that it might be related to OpenMathLib/OpenBLAS#4359 - but again nothing of the sort known to date)

[lesteve]

Not familiar, also not aware of any change in 0.3.26 that would be likely to cause it if the library was built with OpenMP support.

I believe that both for wheels and conda-forge OpenBLAS is built with pthreads as the threading layer and not OpenMP. At least the scikit-learn output shows threading_layer: pthreads for both the wheel and conda-forge package.

I checked the threadpoolctl code and the threading_layer info is from openblas_get_parallel (following https://github.com/OpenMathLib/OpenBLAS/wiki/Faq#how-can-i-find-out-at-runtime-what-options-the-library-was-built-with-)

[martin-frbg]

would need to revert 4359 then to test - or simply copy the 0.3.25 driver/others/blas_server_win32.c into 0.3.26

[mseminatore]

@lesteve can you please clarify 'The scikit-learn code does nested parallelism' for me to help narrow this down? Am I correct in interpreting that to mean that the code invokes exec_blas_async(), rather than exec_blas(), multiple times in a row to queue up several batches of work asynchronously?

I will take a look when I can free up from other tasks. It will probably be about a week before I can dig into this so having a clear repro scenario will help speed things up.

[lesteve]

@lesteve can you please clarify 'The scikit-learn code does nested parallelism' for me to help narrow this down?

There is an outer OpenMP parallel loop and inside the OpenMP parallel loop there are some OpenBLAS calls. This is coming from Python so I am not sure about your question about exec_blas_async vs exec_blas. I may try to have a simpler reproducer in Cython at one point, but this may take some time ...

Another angle of attack I have tried and got stuck is to make sure OpenMathLib/OpenBLAS#4359 is the issue. I am compiling OpenBLAS on Windows inside a MSYS2 terminal with make. At the end of the compilation I get a cygopenblas.dll. The command I use is:

make shared NOFORTRAN=1 NO_LAPACKE=1

I was naively hoping to copy this dll to my Python site-packages to replace libblas.dll (somewhere like C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-3\Library\bin\libblas.dll). if I do this numpy is failing to import so I must be doing something wrong. Any insights about this, let me know!

The ImportError looks like this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\__init__.py", line 149, in <module>
    from . import lib
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\lib\__init__.py", line 23, in <module>
    from . import index_tricks
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\lib\index_tricks.py", line 12, in <module>
    import numpy.matrixlib as matrixlib
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\matrixlib\__init__.py", line 4, in <module>
    from . import defmatrix
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\matrixlib\defmatrix.py", line 12, in <module>
    from numpy.linalg import matrix_power
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\linalg\__init__.py", line 73, in <module>
    from . import linalg
  File "C:\Users\rjxQE\AppData\Local\miniforge3\envs\lof-issue-conda-2\Lib\site-packages\numpy\linalg\linalg.py", line 35, in <module>
    from numpy.linalg import _umath_linalg
ImportError: DLL load failed while importing _umath_linalg: The specified module could not be found.

[martin-frbg]

Problematic environment info with OpenBLAS 0.3.26 (it does not seem like OpenBLAS 0.3.26 is mentioned somehow in >the output below so I added sklearn.show_versions() below ...)

1.12.0 1.26.4 sys.version_info(major=3, minor=12, micro=2, releaselevel='final', serial=0)
Build Dependencies:
blas:
detection method: pkgconfig
found: true
include directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/include
lib directory: C:/Users/rjxQE/AppData/Local/miniforge3/envs/lof-issue-conda-2/Library/lib
name: blas
openblas configuration: unknown
pc file directory: C:\bld\scipy-split_1706041678996_h_env\Library\lib\pkgconfig
version: 3.9.0

this looks strangely like it may have picked up a libblas from the "netlib" Reference-LAPACK project rather than OpenBLAS. Could it be that the various Python bits loaded multiple different BLAS implementations into the process ?

[ogrisel]

@lesteve can you try to mamba install "libopenblas=*=*openmp*" in your conda-forge env on windows to see if that fixes the problem on your reproducer? This command should switch the threading layer from pthreads to openmp and it the problem comes from nesting pthreads calls under OpenMP calls that might fix it.

And if it does, it might reveal a thread-safety problem of openblas' handling of its native threading layer under Windows.

The code in scikit-learn calls OpenBLAS under an OpenMP parallel loop. To avoid oversubscription, it does call openblas_set_num_threads(1) (via threadpoolctl.threadpool_limits) prior to entering the OpenMP parallel loop.

Maybe the recent changes in OpenMathLib/OpenBLAS#4425 introduced a thread-safety problem in this threading layer when globally changing the number of threads at runtime?

[lesteve]

@ogrisel it looks like libopenblas with openmp is not available on conda-forge for Windows, only openmp with pthreads is, for example:

$ mamba search 'libopenblas>=0.3.25'

Loading channels: done
# Name                       Version           Build  Channel
libopenblas                   0.3.25 pthreads_hc140b1d_0  conda-forge
libopenblas                   0.3.26 pthreads_hc140b1d_0  conda-forge