Omp use in manylinux wheels

Numba Community Support
1

In a project of mine I’ve been investigating numba to speedup some scientific calculations. I was using @jit with parallel=True in nopython mode. The test suite for my package was working fine on my local computer (macOS) and in CI on mac and win, but was hanging in CI on Linux.

When I tried running the test suite locally in a docker container I found that the hang was coming from a test that investigates the use of the jitted function in a multiprocessing.Pool context. In the main thread there is no problems with calculation, but in a Pool the jitted function hangs. I managed to fix the hang by changing the numba.config.THREADING_LAYER = 'forksafe', previously numba was trying to use omp.

The numba documentation states:

Due to compatibility issues with manylinux1 and other portability concerns, the OpenMP threading layer is disabled in the Numba binary wheels on PyPI.

I was installing numba via pip from PyPI, so why was the omp threading layer being used?

2

I note that installing Numba with pip and running numba -s also indicates that it is available:

OpenMP Threading Layer Available              : True
+-->Vendor: GNU

I will mention this in the triage meeting or public dev meeting (depending on time) today.

3

Can you link to this statement in the docs please?

4

Also - could you please open an issue with a set of steps to reproduce the issue please? We should be able to follow on discussion there - Sign in to GitHub · GitHub

5

The statement is at the bottom of “User Manual → The threading layers → Which threading layers are available?”

I would make a link, but this discourse doesn’t allow me to do so.

I’ll try and make a MWE, but time is needed to distil the issue without having a bunch of other packages.

1 Like
6

Ah, thanks - for reference, this is at: The Threading Layers — Numba 0.59.0dev0+179.ga4664180.dirty documentation