Sys.path issues with objmode() and cProfile

Numba Support: What is this error message?
1

I have numba installed in a custom directory, which works fine when I use sys.path.insert to explicitly add it to the path (I do not want to use virtual environments here). However, it doesn’t seem to work with a combination of objmode() and cProfile. Here is a minimal example:

import sys
sys.path.insert(0, '/tmp/test/lib64/python3.8/site-packages')
import numba

#@numba.njit
def testpath():
    with numba.objmode():
        print(sys.path)
        #path = sys.path

testpath()

A) If I run this script directly (numba_profile_test.py), it works fine – it prints out the value of sys.path as expected. (I can also uncomment @numba.njit and it works fine).

B) If I run this script with cProfile (python -m cProfile -o cprofile.out numba_profile_test.py), it works fine.

C1) If I uncomment @numba.njit and run this script with cProfile, I get the following error:

Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.8/cProfile.py", line 196, in <module>
    main()
  File "/usr/lib64/python3.8/cProfile.py", line 189, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib64/python3.8/cProfile.py", line 19, in runctx
    return _pyprofile._Utils(Profile).runctx(statement, globals, locals,
  File "/usr/lib64/python3.8/profile.py", line 62, in runctx
    prof.runctx(statement, globals, locals)
  File "/usr/lib64/python3.8/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)
  File "numba_profile_test.py", line 13, in <module>
    testpath()
NameError: global name 'sys' is not defined

C2) If I also uncomment the line path = sys.path and run with cProfile, I get a different error:

Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.8/cProfile.py", line 196, in <module>
    main()
  File "/usr/lib64/python3.8/cProfile.py", line 189, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib64/python3.8/cProfile.py", line 19, in runctx
    return _pyprofile._Utils(Profile).runctx(statement, globals, locals,
  File "/usr/lib64/python3.8/profile.py", line 62, in runctx
    prof.runctx(statement, globals, locals)
  File "/usr/lib64/python3.8/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)
  File "numba_profile_test.py", line 13, in <module>
    testpath()
SystemError: CPUDispatcher(<function testpath at 0x7f65c1ff11f0>) returned NULL without setting an error

Why does objmode() seem to interfere with cProfile here? And what does the NULL error mean?

2

Hi @kartiksubbarao

Thanks for raising this. Which version of Numba are you using? I can’t reproduce this on the current development builds. To answer C2) specifically, SystemError means something has gone wrong in the CPython interpreter, in the case above, it’s probably because something has erroneously returned NULL to signal error but there’s no exception set. It’s probably a bug/bad interaction somewhere.

3

I’m using numba-0.51.2, the current version available from pip. Here’s how I populated /tmp/test:

pip install --prefix /tmp/test numba

If this issue has since gone away in the development builds, that’s good to hear. If there’s a quick way I can test out the dev version let me know (apart from having to build/install LLVM etc :slight_smile: ). Otherwise I guess I can just wait to see if this is fixed in the next release.