Contributing to Numba with no compiler or LLVM experience

Do you want to contribute to Numba, but haven’t worked on compilers and/or LLVM before?

This post provides a couple of pointers to resources that may be helpful - working through them should provide some background that will help to make sense of the way Numba generates code, particularly around the lowering of its IR to LLVM and code generation. Numba’s frontend is quite specific to Python and Numba, but some of the concepts you learn from the resources below might help to understand it in general.

  • My First Language Frontend with LLVM Tutorial - this tutorial walks step-by-step through implementing a compiler for a toy language that uses LLVM for code generation and JIT compilation. This tutorial uses C++.
  • Eli Bendersky’s Python / llvmlite version of the LLVM tutorial - because the above tutorial uses C++, Eli Bendersky implemented the tutorial code using llvmlite in Python - llvmlite is the library used by Numba for access to LLVM. Following the code of this tutorial in conjunction with the C++ LLVM tutorial may be helpful for understanding it better, and being able to understand Numba’s use of llvmlite. Note that this implementation is a few years old, so there might be some issues with using it with a current version of llvmlite, but it should still provide a good idea of how to use llvmlite.
  • Github repo for the Python / llvmlite version of the tutorial - Code for the Python version of the tutorial is here.
  • Mapping High Level Constructs to LLVM IR - this is a good “how-to” reference for when you know what you want to do in LLVM IR, but don’t know how to accomplish it. For example, if you’re wondering “How do I access a member of a struct?”, the section Accessing a Structure Member is very helpful and explains how to use the getelementptr instruction (or gep, as it’s written in llvmlite) to get a pointer to the member for loading / storing.

If you work through any of these resources, please do post back your experience working through it, and mention any issues working through them / translating the concepts to understanding Numba - I’d like to help out anyone who is working towards contributing to Numba - posting your experience will also help find ways to make it easier for other new contributors.

Other areas to look at to get started contributing:

  • Contributing to Numba in the manual explains how to get set up with a development environment, run tests, process / modes of communication (Gitter, Github, etc.), documentation, code reviews, etc.
  • Good First Issues - issues tagged “Good First Issue” are those that are considered good starting points for new contributors to work on. You can have a look at the list and see if you can find an issue that you think you can attempt.
  • Good Second Issues - those that are a little trickier than the “Good First Issues” - worth a look once you’ve tackled one or two issues already.

(Note: my intention is to update the list of resources as and when appropriate, to keep it relevant / improve it as time goes by)

8 Likes

A new addition to the list of useful resources:

This is the answer to my very first question for contributing to Numba. Thanks!

@gmarkall Can we have a similar contribution guide for numba-examples.

The numba-examples repo is a bit of a varied collection of items, so I’m not really sure I can write a concise and specific contribution guide for it. A couple of starting points might be to look at current and previous PRs (e.g. Add example calling cuRAND by gmarkall · Pull Request #40 · numba/numba-examples · GitHub) to see what a contribution looks like, and also to check out the Making a Benchmark section of the README.

I think the scope for numba-examples is fairly broad, so in my opinion any self-contained example of code that solves a problem in an interesting way or is a common requirement would make a good candidate for adding to the numba-examples repo.

1 Like

Thanks for the info @gmarkall

Sorry if this is not the right place to ask but how do I solve this issue? I am installing llvmlite and numba from source. Do I have to install the latest libstdc++ from source? I don’t know why it complains that my version is older?

hbina085@cm3588-nas ~/g/numba (main) [1]> python -m pip freeze | grep -E "llvmlite|numba"                                                                                           (base) 
DEPRECATION: Loading egg at /tank/miniconda3/lib/python3.12/site-packages/libcpuid-0.1.0-py3.12-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
DEPRECATION: Loading egg at /tank/miniconda3/lib/python3.12/site-packages/numba-0.62.0.dev0+15.g8348b5d84-py3.12-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
llvmlite==0.45.0.dev0+10.g69b8c1b
numba==0.62.0.dev0+15.g8348b5d84
hbina085@cm3588-nas ~/g/numba (main)> python hello.py                                                                                                                               (base) 
Traceback (most recent call last):
  File "/tank/miniconda3/lib/python3.12/site-packages/llvmlite/binding/ffi.py", line 141, in __getattr__
    return self._fntab[name]
           ~~~~~~~~~~~^^^^^^
KeyError: 'LLVMPY_AddSymbol'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tank/miniconda3/lib/python3.12/site-packages/llvmlite/binding/ffi.py", line 122, in _load_lib
    self._lib_handle = ctypes.CDLL(str(lib_path))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /tank/miniconda3/bin/../lib/libstdc++.so.6:

When my libstdc++ seems to be up to date

hbina085@cm3588-nas ~/g/numba (main)> strings /lib/aarch64-linux-gnu/libstdc++.so.6 | grep GLIBCXX                                                                                  (base) 
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBCXX_3.4.29
GLIBCXX_3.4.30
GLIBCXX_3.4.31
GLIBCXX_3.4.32
GLIBCXX_3.4.33
GLIBCXX_TUNABLES
GLIBCXX_DEBUG_MESSAGE_LENGTH

You shouldn’t need to install libstdc++ from source.

Can you share details of your setup / platform and the steps you’ve followed to build and install llvmlite please?

Sorry…I didn’t see that I am getting any replies so this is a very, very late response.

So my setup currently is my CM3588 board that I SSH in.

When I use numba and llmvlite installed via pip, it works fine

hbina085@cm3588-nas ~/g/numba (main)> conda activate py312                                                                                                                                                           (py312) 
hbina085@cm3588-nas ~/g/numba (main)> python hello.py                                                                                                                                                                (py312) 
hello
10-10-hello-True

Then I installed llvmlite and numba from the cloned repo using

LLVM_CONFIG=/usr/bin/llvm-config-15 python setup.py install

I am getting this error

hbina085@cm3588-nas ~/g/llvmlite (main)> python -m pip freeze | grep -E "llvmlite|numba"                                                                                                                          (numbaenv) 
DEPRECATION: Loading egg at /tank/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba-0.62.0.dev0+15.g8348b5d84-py3.12-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
llvmlite==0.43.0
numba==0.62.0.dev0+15.g8348b5d84
hbina085@cm3588-nas ~/g/llvmlite (main)> cd ../numba/                                                                                                                                                             (numbaenv) 
hbina085@cm3588-nas ~/g/numba (main)> python hello.py                                                                                                                                                             (numbaenv) 
Traceback (most recent call last):
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/llvmlite/binding/ffi.py", line 136, in __getattr__
    return self._fntab[name]
           ~~~~~~~~~~~^^^^^^
KeyError: 'LLVMPY_AddSymbol

Edit:

Okay…after revisiting this problem, I do notice that my llvmlite version is not a dev? So I uninstalled this and reinstalled from the repo.

Now I am getting a different error, but at least its not libc related?

hbina085@cm3588-nas ~/g/numba (main)> python hello.py                                                                                                                                                             (numbaenv) 
Traceback (most recent call last):
  File "/tank/git/numba/hello.py", line 1, in <module>
    import numba
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/__init__.py", line 73, in <module>
    from numba.core import config
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 601, in <module>
    _env_reloader = _EnvReloader()
                    ^^^^^^^^^^^^^^
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 145, in __init__
    self.reset()
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 149, in reset
    self.update(force=True)
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 178, in update
    self.process_environ(new_environ)
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 413, in process_environ
    ENABLE_AVX = _readenv("NUMBA_ENABLE_AVX", int, avx_default)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 209, in _readenv
    return default() if callable(default) else default
           ^^^^^^^^^
  File "/home/hbina085/miniconda3/envs/numbaenv/lib/python3.12/site-packages/numba/core/config.py", line 403, in avx_default
    cpu_name = CPU_NAME or ll.get_host_cpu_name()
                           ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'llvmlite.binding' has no attribute 'get_host_cpu_name

Oh dear is this related to how aarch64 CPUID works…

Edit2:

Honestly, I am so utterly confused. I tried to make sure my installation of llvmlite is fine first, but the tests are failing?

hbina085@cm3588-nas ~/g/llvmlite (main)> conda activate numbaenv                                                                                                                                                  (numbaenv) 
hbina085@cm3588-nas ~/g/llvmlite (main)> LLVM_CONFIG=/usr/bin/llvm-config-15 python setup.py install                                                                                                              (numbaenv) 
/tank/miniconda3/lib/python3.12/site-packages/setuptools/dist.py:294: InformationOnly: Normalizing '0.45.0dev0+10.g69b8c1b' to '0.45.0.dev0+10.g69b8c1b'
  self.metadata.version = self._normalize_version(self.metadata.version)
running install
/tank/miniconda3/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running build
got version from VCS {'version': '0.45.0dev0+10.g69b8c1b', 'full': '69b8c1b370185c96cdbbd9de2d30e54803c30d43'}
running build_ext
/tank/miniconda3/bin/python /tank/git/llvmlite/ffi/build.py
LLVM version... 15.0.7

SVML not detected
Makefile.linux:22: warning: ignoring prerequisites on suffix rule definition
make: Nothing to be done for 'all'.
running build_py
copying llvmlite/utils.py -> build/lib.linux-aarch64-cpython-312/llvmlite
copying llvmlite/__init__.py -> build/lib.linux-aarch64-cpython-312/llvmlite
copying llvmlite/binding/linker.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/value.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/newpassmanagers.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/common.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/analysis.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/module.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/context.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/executionengine.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/passmanagers.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/transforms.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/object_file.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/targets.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/__init__.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/ffi.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/orcjit.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/initfini.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/options.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/typeref.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/binding/dylib.py -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
copying llvmlite/ir/module.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/types.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/transforms.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/__init__.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/_utils.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/context.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/values.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/instructions.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/ir/builder.py -> build/lib.linux-aarch64-cpython-312/llvmlite/ir
copying llvmlite/tests/refprune_proto.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/customize.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/test_ir.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/__main__.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/test_binding.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/test_refprune.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/__init__.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/tests/test_valuerepr.py -> build/lib.linux-aarch64-cpython-312/llvmlite/tests
copying llvmlite/binding/libllvmlite.so -> build/lib.linux-aarch64-cpython-312/llvmlite/binding
UPDATING build/lib.linux-aarch64-cpython-312/llvmlite/_version.py
running install_lib
copying build/lib.linux-aarch64-cpython-312/llvmlite/_version.py -> /tank/miniconda3/lib/python3.12/site-packages/llvmlite
copying build/lib.linux-aarch64-cpython-312/llvmlite/binding/libllvmlite.so -> /tank/miniconda3/lib/python3.12/site-packages/llvmlite/binding
byte-compiling /tank/miniconda3/lib/python3.12/site-packages/llvmlite/_version.py to _version.cpython-312.pyc
running install_egg_info
running egg_info
writing llvmlite.egg-info/PKG-INFO
writing dependency_links to llvmlite.egg-info/dependency_links.txt
writing top-level names to llvmlite.egg-info/top_level.txt
reading manifest file 'llvmlite.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching 'CMakeCache.txt' found anywhere in distribution
adding license file 'LICENSE'
adding license file 'LICENSE.thirdparty'
writing manifest file 'llvmlite.egg-info/SOURCES.txt'
removing '/tank/miniconda3/lib/python3.12/site-packages/llvmlite-0.45.0.dev0+10.g69b8c1b-py3.12.egg-info' (and everything under it)
Copying llvmlite.egg-info to /tank/miniconda3/lib/python3.12/site-packages/llvmlite-0.45.0.dev0+10.g69b8c1b-py3.12.egg-info
running install_scripts
hbina085@cm3588-nas ~/g/llvmlite (main)> python runtests.py                                                                                                                                                       (numbaenv) 
EEE.....
======================================================================
ERROR: llvmlite.tests.test_binding (unittest.loader._FailedTest.llvmlite.tests.test_binding)
----------------------------------------------------------------------
ImportError: Failed to import test module: llvmlite.tests.test_binding
Traceback (most recent call last):
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 141, in __getattr__
    return self._fntab[name]
           ~~~~~~~~~~~^^^^^^
KeyError: 'LLVMPY_AddSymbol'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 122, in _load_lib
    self._lib_handle = ctypes.CDLL(str(lib_path))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /tank/miniconda3/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /tank/git/llvmlite/llvmlite/binding/libllvmlite.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tank/miniconda3/lib/python3.12/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "/tank/git/llvmlite/llvmlite/tests/test_binding.py", line 17, in <module>
    from llvmlite import binding as llvm
  File "/tank/git/llvmlite/llvmlite/binding/__init__.py", line 4, in <module>
    from .dylib import *
  File "/tank/git/llvmlite/llvmlite/binding/dylib.py", line 36, in <module>
    ffi.lib.LLVMPY_AddSymbol.argtypes = [
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 144, in __getattr__
    cfn = getattr(self._lib, name)
                  ^^^^^^^^^
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 136, in _lib
    self._load_lib()
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 130, in _load_lib
    raise OSError("Could not find/load shared object file") from e
OSError: Could not find/load shared object file


======================================================================
ERROR: llvmlite.tests.test_ir (unittest.loader._FailedTest.llvmlite.tests.test_ir)
----------------------------------------------------------------------
ImportError: Failed to import test module: llvmlite.tests.test_ir
Traceback (most recent call last):
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 141, in __getattr__
    return self._fntab[name]
           ~~~~~~~~~~~^^^^^^
KeyError: 'LLVMPY_AddSymbol'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 122, in _load_lib
    self._lib_handle = ctypes.CDLL(str(lib_path))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /tank/miniconda3/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /tank/git/llvmlite/llvmlite/binding/libllvmlite.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tank/miniconda3/lib/python3.12/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "/tank/git/llvmlite/llvmlite/tests/test_ir.py", line 14, in <module>
    from llvmlite import binding as llvm
  File "/tank/git/llvmlite/llvmlite/binding/__init__.py", line 4, in <module>
    from .dylib import *
  File "/tank/git/llvmlite/llvmlite/binding/dylib.py", line 36, in <module>
    ffi.lib.LLVMPY_AddSymbol.argtypes = [
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 144, in __getattr__
    cfn = getattr(self._lib, name)
                  ^^^^^^^^^
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 136, in _lib
    self._load_lib()
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 130, in _load_lib
    raise OSError("Could not find/load shared object file") from e
OSError: Could not find/load shared object file


======================================================================
ERROR: llvmlite.tests.test_refprune (unittest.loader._FailedTest.llvmlite.tests.test_refprune)
----------------------------------------------------------------------
ImportError: Failed to import test module: llvmlite.tests.test_refprune
Traceback (most recent call last):
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 141, in __getattr__
    return self._fntab[name]
           ~~~~~~~~~~~^^^^^^
KeyError: 'LLVMPY_AddSymbol'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 122, in _load_lib
    self._lib_handle = ctypes.CDLL(str(lib_path))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /tank/miniconda3/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /tank/git/llvmlite/llvmlite/binding/libllvmlite.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tank/miniconda3/lib/python3.12/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/miniconda3/lib/python3.12/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "/tank/git/llvmlite/llvmlite/tests/test_refprune.py", line 3, in <module>
    from llvmlite import binding as llvm
  File "/tank/git/llvmlite/llvmlite/binding/__init__.py", line 4, in <module>
    from .dylib import *
  File "/tank/git/llvmlite/llvmlite/binding/dylib.py", line 36, in <module>
    ffi.lib.LLVMPY_AddSymbol.argtypes = [
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 144, in __getattr__
    cfn = getattr(self._lib, name)
                  ^^^^^^^^^
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 136, in _lib
    self._load_lib()
  File "/tank/git/llvmlite/llvmlite/binding/ffi.py", line 130, in _load_lib
    raise OSError("Could not find/load shared object file") from e
OSError: Could not find/load shared object file


----------------------------------------------------------------------
Ran 8 tests in 0.001s

FAILED (errors=3)

I can’t tell what’s going on specifically, but I think your environment is messed up and it would be best to just start with a fresh one.

For what you’re doing (building llvmlite on AArch64), there is one issue to keep in mind: LLVM needs to be patched on AArch64 because of RuntimeDyldELF doesn't clear `GOTOffsetMap` in `finalizeLoad()`, leading to invalid GOT relocations on AArch64 · Issue #61402 · llvm/llvm-project · GitHub, and your system-installed LLVM will almost certainly not have the required patch applied ([RuntimeDyld] RuntimeDyldELF: Clear GOTOffsetMap when resetting GOT s… · llvm/llvm-project@2e1b838 · GitHub). Therefore, you have three options:

  • Use conda for environment management and install llvmdev from the numba channel (with conda install numba::llvmdev=15.0.7), then build llvmlite. I’d suggest this route as you seem to already have a conda setup.
  • Build LLVM yourself locally, having applied the above patch, then build llvmlite. This has the advantage that you’re also free to do a debug build, and be able to step into LLVM with a debugger if you wish, but is a lot more work to set up.
  • Don’t build llvmlite, but just conda install it (e.g. conda install numba/label/dev::llvmlite to get a nightly build of it), and only work on Numba. Whether this is appropriate depends on your interests, but is the simplest path if you’re aiming to make changes to Numba rather than llvmlite.

To create a new environment, I’d suggest something like

conda create -n numbadev python=3.12 numpy=2.0 scipy jinja2 cffi
conda activate numbadev

to get an environment with appropriate dependencies. You can use Python 3.13 and NumPy 2.1, but I tend to not use the latest versions of everything myself. SciPy is needed for full Numba functionality (some linear algebra operations) and cffi is supported by Numba but may not be required.

Then from there, build / install llvmlite and Numba and try to validate that everything works (and in future sessions, make sure to always run the conad activate script to ensure you’re working in that environment).

If it doesn’t work on aarch64 then I can just work on it on my VM instead. Thanks for the heads up. I will try to follow your guide.

Just to clarify - Numba is fully supported on AArch64 both on Linux and macOS - it’s just that there’s some nuance in building it because the required LLVM bug fix is not backported in most packages / distributions.