Setuptools can't find my GCC compiler

Numba Support: What is this error message?
1

I am attempting to include my AOT-compiled extensions in my setup.py.

from setuptools import setup

import sys
import os

script_path = os.path.dirname(os.path.realpath(__file__))

# where my modules are
sys.path.insert(0, os.path.join(script_path,'src'))

# cc_calculator = CC("cc_calculator"), ...
from onta.engine.static.calculator import cc_calculator
from onta.engine.static.formulae import cc_formulae
from onta.engine.static.paths import cc_paths

setup(
    ext_modules=[
        cc_calculator.distutils_extension(),
        cc_formulae.distutils_extension(),
        cc_paths.distutils_extension(),
    ]
)

This has all worked fine and dandy for the past week or so. I have built the wheel multiple times, no problems. All of the sudden, I started receiving this message when attempting python -m build,

File "/tmp/build-env-vh52ukyr/lib/python3.8/site-packages/numba/pycc/cc.py", line 65, in __init__
    self._toolchain = Toolchain()
  File "/tmp/build-env-vh52ukyr/lib/python3.8/site-packages/numba/pycc/platform.py", line 78, in __init__
    self._raise_external_compiler_error()
  File "/tmp/build-env-vh52ukyr/lib/python3.8/site-packages/numba/pycc/platform.py", line 121, in _raise_external_compiler_error
    raise RuntimeError(msg)
RuntimeError: Attempted to compile AOT function without the compiler used by `numpy.distutils` present. If using conda try:

#> conda install gcc_linux-64 gxx_linux-64

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

I have been banging my head against the wall trying to figure out what the problem is, since I definitely have a C/C++ compiler on my system,

> gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

> g++ --version
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I have tried to take the error message’s hint and install the compilers through conda, but I get the same message. I think there is something going on with venv when the package is building. What’s strange to me is I can compile the modules outside of the setup.py, e.g.,

cc_calculator.compile()
cc_formulae.compile()
cc_paths.compile()

Will give me the appropriate .so files for my system, e.g. cc_calculator.cpython-38-x86_64-linux-gnu.so.

I did notice there is a deprecation warning on the numpy site about the distutils compiler, and I also saw there was release to 1.23.3 within the last day, so I am wondering if there was an update that has affected numba in some way? The release notes didn’t seem to include anything about this though.

I have no idea what’s going on, or why it was working before and then suddenly stopped working. Any guidance would be much appreciated.

3

Numba tries to compile something using the compiler used by distutils (the error message claims numpy.distutils, but I don’t see any evidence of that in the code). The check is here: numba/platform.py at main · numba/numba · GitHub

Unfortunately whenever something goes wrong Numba swallows the exception, hiding the real cause. To get to the bottom of what might be going wrong, you could try running the following, extracted and modified from platform.py, to see if it produces errors:

from contextlib import contextmanager
from distutils.ccompiler import new_compiler
from distutils.sysconfig import customize_compiler
from pathlib import Path
from tempfile import mkdtemp
import os


@contextmanager
def _gentmpfile(suffix):
    # windows locks the tempfile so use a tempdir + file, see
    # https://github.com/numba/numba/issues/3304
    try:
        tmpdir = mkdtemp()
        ntf = open(os.path.join(tmpdir, "temp%s" % suffix), 'wt')
        yield ntf
    finally:
        try:
            ntf.close()
            os.remove(ntf)
        except:  # noqa: E722
            pass
        else:
            os.rmdir(tmpdir)


def check_external_compiler():
    # see if the external compiler bound in numpy.distutil is present
    # and working
    compiler = new_compiler()
    customize_compiler(compiler)
    for suffix in ['.c', '.cxx']:
        with _gentmpfile(suffix) as ntf:
            simple_c = "int main(void) { return 0; }"
            ntf.write(simple_c)
            ntf.flush()
            ntf.close()
            # *output_dir* is set to avoid the compiler putting temp files
            # in the current directory.
            compiler.compile([ntf.name], output_dir=Path(ntf.name).anchor)


check_external_compiler()
4

See also `numpy.distutils` is deprecated in NumPy 1.23 and issues with `setuptools>=65`. · Issue #8355 · numba/numba · GitHub

Are you using NumPy 1.23 now?

5

Howdy. Thanks for the reply.

I saw the GitHub issue about this and thought it might apply here. I ran the script you provided and it produced no errors. Which means it can my find my compiler, correct?

Also, in answer to your second question, I did not explicitly have a numpy dependency in my setup.cfg, so it was pulling in whatever numba depended on. I did attempt to explicilty lock numpy at 1.19 and at 1.22, but that resulted in the same error.

What is confusing to me is, I have tried building the wheel on three different computers besides the one I develop on. And the first time, it always compiles fine. No issues whatsoever. And then, for whatever reason, if I have to recompile, it throws the error. Every time, same behavior. I’ve tried it on two different Ubuntu distributions and MacOS. It’s like it’s starting from a blank state and then after it’s been initialized, it’s defaulting to a configuration with incorrect information.

6

What version of setuptools do you presently have? Can you try with a setuptools version that’s <60 if you presently have a later version?