Numba cache and lowering error with multi-processing (v0.56.4)

Numba Support: What is this error message?
1

Hi,

I’m currently struggling with some sneaky error involving caching and parallel processing (Numba v0.56.4).

The original issue can be seen at convst issue #24, with a linked issue at convst issue #34. The problem seems to be that some function are not cached properly and need to be recompiled at each run of the algorithm.

The only error message I manage to get is (it happens multiple times for different functions):

NumbaDebugInfoWarning: Could not find source for function: <function __numba_parfor_gufunc_0x7f84a8f89760 at 0x7f84a91631f0>. Debug line information may be inaccurate

When executing the same method multiple time and looking at the timings, we see that some runs are doing more work than others. Additionally, when looking at the cache files, the same function is cached multiple times (e.g. method_name.py38.1.nbc, method_name.py38.2.nbc, etc.). Though, I don’t know if this is an expected behaviour.

The issue of “recompilation” seems to happen when new processes are created, each using numba with parallel=True options down the way (threads are limited to avoid going above number of cpu).

Is there some known issue I am unaware of, or I am doing something wrong with the multiprocessing ?

Thank you in advance.

2

To have a look at the code:

The warning happens in this function, and a lowering error (issue 24) was raised at line 303 of the same file.

The multiprocessing is launched by “RDST Ensemble”.

You may try to reproduce the issue by installing the module with pip install convst and use the code mentioned in issue #34.