Missing tbbpool

Dave2112 · June 6, 2023, 10:09pm

Hi. I am wanting to set up a threading layer, and so I’m using Numba 0.57.0 along with tbb-2021.9.0. When I run numba -s, and I look at the status of tbb, it reports:

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : False
+--> Disabled due to Unknown import problem.
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

So, in digging in around with what numba -s is doing, I set breakpoints and I see: ("cannot import name 'tbbpool' from 'numba.np.ufunc' (<root>/projects/customers/drc/tss-azure/venv/lib/python3.9/site-packages/numba/np/ufunc/__init__.py)",). When I look at what ended up in my venv’s site_packages, I don’t see tbbpool.cpp nor omppool.cpp in what came down in the pip install I was doing for Numba. Any help would be greatly appreciated.

Hannes · June 7, 2023, 6:20am

Hi,

tbb is an optional dependency, you need to install it manually with either pip or conda. The Threading Layers — Numba 0.57.0+0.g4fd4e39c6.dirty documentation

Dave2112 · June 7, 2023, 8:38am

Hi. Thanks for getting back to me. I guess I should have made my post a little clearer, but tbb (and omp for that matter) are both installed (via pip). But the error it’s giving me, is for missing files from Numba, specifically, tbbpool.cpp and omppool.cpp that you can see here. For me, I just see this in this in my site_packages for ./numba/np/ufunc:
Screenshot 2023-06-07 at 4.31.42 AM
Somehow, both those cpp files aren’t there. So it appears to me that pip install is not installing everything it should? I must be missing something… (well, other than the actual files, I should say )

Hannes · June 7, 2023, 9:51am

Hi,

and I suppose I should have read you initial post a bit more carefully

FWIW: I cannot reproduce your problem on a CentOS 7 machine. If I create a new virtual environment with Python 3.9 and pip install numba==0.57 and tbb it just works.

Did you install numba and tbb in one shot or one after another? (Technically this should work plugin-style, but who knows with these things) Could you try making a new clean venv and try to install just these two packages to ensure nothing else is getting in the way? Are you installing pre-compiled wheels or building numba from source?

Hannes · June 7, 2023, 10:01am

Addendum: The cpp files you are looking for do not exist in my venv either (neither would I expect them since I installed a pre-built wheel) but I can manually import tbbpool from numba.np.ufunc in a python session.

Maybe it would be helpful for the core maintainers, if you actually provide the full output of numba -s here, ot could contain a smoking gun.

Dave2112 · June 7, 2023, 1:23pm

Hannes, thanks so much for getting back to me! So, I’m confused by something you said: the cpp files don’t exist for you either in your venv. But, when you do numba -s it looks like that process is looking for the presence of the tbbpool and ombpool packages: (as here, throws an exception for me where it complains that: ("cannot import name 'tbbpool' from 'numba.np.ufunc' (/Users/chariot/projects/customers/rtm/tss-azure/venv/lib/python3.9/site-packages/numba/np/ufunc/__init__.py)",). When you do numba -s it doesn’t complain about an “Unknown import problem”?

So, given that, I still went ahead and created a play venv, and just installed numba. So I did:

➜  test_venv source ./venv/bin/activate
(venv) ➜  test_venv pip install numba
Collecting numba
  Using cached numba-0.57.0-cp311-cp311-macosx_10_9_x86_64.whl (2.5 MB)
Collecting llvmlite<0.41,>=0.40.0dev0
  Using cached llvmlite-0.40.0-cp311-cp311-macosx_10_9_x86_64.whl (30.4 MB)
Collecting numpy<1.25,>=1.21
  Using cached numpy-1.24.3-cp311-cp311-macosx_10_9_x86_64.whl (19.8 MB)
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.40.0 numba-0.57.0 numpy-1.24.3

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
(venv) ➜  test_venv ls -tal ./venv/lib/python3.11/site-packages/numba/np/ufunc  
total 584
drwxr-xr-x  12 drc  staff    384 Jun  7 09:07 __pycache__
drwxr-xr-x  17 drc  staff    544 Jun  7 09:07 .
drwxr-xr-x  18 drc  staff    576 Jun  7 09:07 ..
-rw-r--r--   1 drc  staff  27203 Jun  7 09:07 wrappers.py
-rw-r--r--   1 drc  staff  48792 Jun  7 09:07 workqueue.cpython-311-darwin.so
-rw-r--r--   1 drc  staff  14325 Jun  7 09:07 ufuncbuilder.py
-rw-r--r--   1 drc  staff   1846 Jun  7 09:07 sigparse.py
-rw-r--r--   1 drc  staff  26122 Jun  7 09:07 parallel.py
-rw-r--r--   1 drc  staff  48308 Jun  7 09:07 omppool.cpython-311-darwin.so
-rw-r--r--   1 drc  staff   6117 Jun  7 09:07 gufunc.py
-rw-r--r--   1 drc  staff  12268 Jun  7 09:07 dufunc.py
-rw-r--r--   1 drc  staff  30651 Jun  7 09:07 deviceufunc.py
-rw-r--r--   1 drc  staff   6107 Jun  7 09:07 decorators.py
-rw-r--r--   1 drc  staff  16873 Jun  7 09:07 array_exprs.py
-rw-r--r--   1 drc  staff   8864 Jun  7 09:07 _num_threads.cpython-311-darwin.so
-rw-r--r--   1 drc  staff  21124 Jun  7 09:07 _internal.cpython-311-darwin.so
-rw-r--r--   1 drc  staff   1070 Jun  7 09:07 __init__.py

So, you can see those resources aren’t there. Here’s the output from numba -s

System info:

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2023-06-07 09:21:48.588785
UTC start time                                : 2023-06-07 13:21:48.588790
Running time (s)                              : 0.5661

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : skylake
CPU Count                                     : 12
Number of accessible CPUs                     : ?
List of accessible CPUs cores                 : ?
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit adx aes avx avx2 bmi bmi2
                                                clflushopt cmov crc32 cx16 cx8
                                                f16c fma fsgsbase fxsr invpcid
                                                lzcnt mmx movbe pclmul popcnt
                                                prfchw rdrnd rdseed rtm sahf sgx
                                                sse sse2 sse3 sse4.1 sse4.2 ssse3
                                                xsave xsavec xsaveopt xsaves

Memory Total (MB)                             : 32768
Free Memory (MB)                              : 2096

__OS Information__
Platform Name                                 : macOS-13.3.1-x86_64-i386-64bit
Platform Release                              : 22.4.0
OS Name                                       : Darwin
OS Version                                    : Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64
OS Specific Version                           : 13.3.1   x86_64
Libc Version                                  : ?

__Python Information__
Python Compiler                               : Clang 14.0.0 (clang-1400.0.29.202)
Python Implementation                         : CPython
Python Version                                : 3.11.3
Python Locale                                 : en_US.UTF-8

__Numba Toolchain Versions__
Numba Version                                 : 0.57.0
llvmlite Version                              : 0.40.0

__LLVM Information__
LLVM Version                                  : 14.0.6

__CUDA Information__
CUDA Device Initialized                       : False
CUDA Driver Version                           : ?
CUDA Runtime Version                          : ?
CUDA NVIDIA Bindings Available                : ?
CUDA NVIDIA Bindings In Use                   : ?
CUDA Minor Version Compatibility Available    : ?
CUDA Minor Version Compatibility Needed       : ?
CUDA Minor Version Compatibility In Use       : ?
CUDA Detect Output:
None
CUDA Libraries Test Output:
None

__NumPy Information__
NumPy Version                                 : 1.24.3
NumPy Supported SIMD features                 : ('MMX', 'SSE', 'SSE2', 'SSE3', 'SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2')
NumPy Supported SIMD dispatch                 : ('SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2', 'AVX512F', 'AVX512CD', 'AVX512_KNL', 'AVX512_SKX', 'AVX512_CLX', 'AVX512_CNL', 'AVX512_ICL')
NumPy Supported SIMD baseline                 : ('SSE', 'SSE2', 'SSE3')
NumPy AVX512_SKX support detected             : False

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : False
+--> Disabled due to Unknown import problem.
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
NUMBA_THREADING_LAYER                         : TBB

__Conda Information__
Conda not available.

__Installed Packages__
Package    Version
---------- -------
llvmlite   0.40.0
numba      0.57.0
numpy      1.24.3
pip        23.0.1
setuptools 67.6.1

No errors reported.


__Warning log__
Warning (cuda): CUDA driver library cannot be found or no CUDA enabled devices are present.
Exception class: <class 'numba.cuda.cudadrv.error.CudaSupportError'>
Warning: Conda not available.
 Error was [Errno 2] No such file or directory: 'conda'

Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.
--------------------------------------------------------------------------------
If requested, please copy and paste the information between
the dashed (----) lines, or from a given specific section as
appropriate.

=============================================================
IMPORTANT: Please ensure that you are happy with sharing the
contents of the information present, any information that you
wish to keep private you should remove before sharing.
=============================================================

THANKS AGAIN!

Dave2112 · June 7, 2023, 2:32pm

Hi Hannes. So when you do: numba -s in that clean venv you created, do you not also see:

Disabled due to Unknown import problem

I ask because numba -s appears to look for both the tbbpool and omppool resources (so putting a breakpoint, here results in

("cannot import name 'tbbpool' from 'numba.np.ufunc' (/Users/drc/projects/customers/rtm/tss-azure/venv/lib/python3.9/site-packages/numba/np/ufunc/__init__.py)",)

for me. So how are you not seeing this too? What tbbpool (or omppool) is it resolving for you? And I am bringing down a wheel as well, but why wouldn’t those resources be there if they are needed by Numba? I mean, they are here (well, I guess at least cpp files are there…)… you must be picking up some other artifact if this is all working for you? (and btw, not sure what you meant by " just works": do you mean that ‘numba -s’ shows that can be found, or that some application you have doesn’t crash like mine does because of thread safety related issues – because it can’t install a thread layer?)

As requested, this is what I see in a quick venv I made:

➜  test_venv source ./venv/bin/activate
(venv) ➜  test_venv pip install numba
Collecting numba
  Using cached numba-0.57.0-cp311-cp311-macosx_10_9_x86_64.whl (2.5 MB)
Collecting llvmlite<0.41,>=0.40.0dev0
  Using cached llvmlite-0.40.0-cp311-cp311-macosx_10_9_x86_64.whl (30.4 MB)
Collecting numpy<1.25,>=1.21
  Using cached numpy-1.24.3-cp311-cp311-macosx_10_9_x86_64.whl (19.8 MB)
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.40.0 numba-0.57.0 numpy-1.24.3

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
(venv) ➜  test_venv ls -tal ./venv/lib/python3.11/site-packages/numba/np/ufunc  
total 584
drwxr-xr-x  12 drc  staff    384 Jun  7 09:07 __pycache__
drwxr-xr-x  17 drc  staff    544 Jun  7 09:07 .
drwxr-xr-x  18 drc  staff    576 Jun  7 09:07 ..
-rw-r--r--   1 drc  staff  27203 Jun  7 09:07 wrappers.py
-rw-r--r--   1 drc  staff  48792 Jun  7 09:07 workqueue.cpython-311-darwin.so
-rw-r--r--   1 drc  staff  14325 Jun  7 09:07 ufuncbuilder.py
-rw-r--r--   1 drc  staff   1846 Jun  7 09:07 sigparse.py
-rw-r--r--   1 drc  staff  26122 Jun  7 09:07 parallel.py
-rw-r--r--   1 drc  staff  48308 Jun  7 09:07 omppool.cpython-311-darwin.so
-rw-r--r--   1 drc  staff   6117 Jun  7 09:07 gufunc.py
-rw-r--r--   1 drc  staff  12268 Jun  7 09:07 dufunc.py
-rw-r--r--   1 drc  staff  30651 Jun  7 09:07 deviceufunc.py
-rw-r--r--   1 drc  staff   6107 Jun  7 09:07 decorators.py
-rw-r--r--   1 drc  staff  16873 Jun  7 09:07 array_exprs.py
-rw-r--r--   1 drc  staff   8864 Jun  7 09:07 _num_threads.cpython-311-darwin.so
-rw-r--r--   1 drc  staff  21124 Jun  7 09:07 _internal.cpython-311-darwin.so
-rw-r--r--   1 drc  staff   1070 Jun  7 09:07 __init__.py

…and when I do numba -s:

System info:

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2023-06-07 09:21:48.588785
UTC start time                                : 2023-06-07 13:21:48.588790
Running time (s)                              : 0.5661

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : skylake
CPU Count                                     : 12
Number of accessible CPUs                     : ?
List of accessible CPUs cores                 : ?
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit adx aes avx avx2 bmi bmi2
                                                clflushopt cmov crc32 cx16 cx8
                                                f16c fma fsgsbase fxsr invpcid
                                                lzcnt mmx movbe pclmul popcnt
                                                prfchw rdrnd rdseed rtm sahf sgx
                                                sse sse2 sse3 sse4.1 sse4.2 ssse3
                                                xsave xsavec xsaveopt xsaves

Memory Total (MB)                             : 32768
Free Memory (MB)                              : 2096

__OS Information__
Platform Name                                 : macOS-13.3.1-x86_64-i386-64bit
Platform Release                              : 22.4.0
OS Name                                       : Darwin
OS Version                                    : Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64
OS Specific Version                           : 13.3.1   x86_64
Libc Version                                  : ?

__Python Information__
Python Compiler                               : Clang 14.0.0 (clang-1400.0.29.202)
Python Implementation                         : CPython
Python Version                                : 3.11.3
Python Locale                                 : en_US.UTF-8

__Numba Toolchain Versions__
Numba Version                                 : 0.57.0
llvmlite Version                              : 0.40.0

__LLVM Information__
LLVM Version                                  : 14.0.6

__CUDA Information__
CUDA Device Initialized                       : False
CUDA Driver Version                           : ?
CUDA Runtime Version                          : ?
CUDA NVIDIA Bindings Available                : ?
CUDA NVIDIA Bindings In Use                   : ?
CUDA Minor Version Compatibility Available    : ?
CUDA Minor Version Compatibility Needed       : ?
CUDA Minor Version Compatibility In Use       : ?
CUDA Detect Output:
None
CUDA Libraries Test Output:
None

__NumPy Information__
NumPy Version                                 : 1.24.3
NumPy Supported SIMD features                 : ('MMX', 'SSE', 'SSE2', 'SSE3', 'SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2')
NumPy Supported SIMD dispatch                 : ('SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2', 'AVX512F', 'AVX512CD', 'AVX512_KNL', 'AVX512_SKX', 'AVX512_CLX', 'AVX512_CNL', 'AVX512_ICL')
NumPy Supported SIMD baseline                 : ('SSE', 'SSE2', 'SSE3')
NumPy AVX512_SKX support detected             : False

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : False
+--> Disabled due to Unknown import problem.
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
NUMBA_THREADING_LAYER                         : TBB

__Conda Information__
Conda not available.

__Installed Packages__
Package    Version
---------- -------
llvmlite   0.40.0
numba      0.57.0
numpy      1.24.3
pip        23.0.1
setuptools 67.6.1

No errors reported.


__Warning log__
Warning (cuda): CUDA driver library cannot be found or no CUDA enabled devices are present.
Exception class: <class 'numba.cuda.cudadrv.error.CudaSupportError'>
Warning: Conda not available.
 Error was [Errno 2] No such file or directory: 'conda'

Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.

…and for good measure:

(venv) ➜  venv pip freeze     
llvmlite==0.40.0
numba==0.57.0
numpy==1.24.3
tbb==2021.9.0

SO: just to set the context straight. All I wish to do is in an existing Python project (that runs in an Azure Function), is to parallelize an expensive range loop, in a threadsafe way. When I see numba -s complain the way it does, I get a runtime error saying I need to install a thread layer. So I am assuming that if it can’t see the right things when I do numba -s, that this will always fail at runtime as well.

Hannes · June 8, 2023, 2:16pm

Hi,

Unfortunately I cannot dig deep right now, and I am not at the machine I used yesterday.

I think you should not be worried about those missing .cpp files. Python will never import a cpp file directly.
But when a python package uses extension modules (i.e. modules not written in Python) they are compiled and the Python import machinery “imports” symbols from pre-compiled binaries, usually some kind of .so or .dll .

By just works I mean that numba -s resolves TBB and OMP successfully and shows them as available- I have not run any multithreaded cocde for testing.

However: Your numba -s record shows that you set the ENV variable NUMBA_THREADING_LAYER = TBB although tbb is clearly not available. I am not surprised that this fails.

The fact that you run on Azure could be important, that will always make things harder to analyse I guess. But afaik the numba test suite actually runs on Azure (at least partially) so in prinicple that should not be a roadblock. Not sure what Azure functions are, some kind of “serverless” service?

I am not sure I will be able to fix this, so if a core-dev could chime in, that’d be great.

Dave2112 · June 8, 2023, 2:45pm

Hey Hannes. Ok, I absolutely love the quotes around “serverless”. lol. That’s precisely what Azure is, it’s Microsoft’s answer to “serverless” that as it turns out, isn’t really serverless. ;). LOL.

I did set that ENV in an attempt to get things moving, but of course that didn’t work. I don’t think my issue really hasn’t anything to do with Azure, as it appears that for whatever reason, my build isn’t able to resolve internal Numba things it needs: For example,when ‘numba s’ is invoked, it looks for a numba package for tbbpool (the forum won’t let me post a link here for some reason…) . But here it is, from numba_sysinfo.py:

    try:
        # check import is ok, this means the DSO linkage is working
        from numba.np.ufunc import tbbpool  # NOQA
        # check that the version is compatible, this is a check performed at
        # runtime (well, compile time), it will also ImportError if there's
        # a problem.
        from numba.np.ufunc.parallel import _check_tbb_version_compatible
        _check_tbb_version_compatible()
        sys_info[_tbb_thread] = True
    except ImportError as e:
        # might be a missing symbol due to e.g. tbb libraries missing
        sys_info[_tbb_thread] = False
        sys_info[_tbb_error] = parse_error(e, 'tbbpool')

    try:
        from numba.np.ufunc import omppool
        sys_info[_openmp_thread] = True
        sys_info[_openmp_vendor] = omppool.openmp_vendor

Perhaps there needs to be some step to compile tbbpool,cpp? Anywho, Is there some recommended way for me to get this escalated to other eyes? I really need to resolve this.

Topic		Replies	Views
Windows, Python 3.8 and TBB support Support: How do I do ...?	1	868	October 12, 2020
Suppress TBB Warning Support: What is this error message?	10	5686	January 12, 2022
[ANN] Numba 0.56.0 Announcements	2	866	July 30, 2022
Library not loaded: '@rpath/libiomp5.dylib' Support: How do I do ...?	1	729	July 9, 2023
How to install numba.openmp Numba	14	2352	August 1, 2023

Missing tbbpool

Related Topics