TypingError in nopython with np.concatenate within QGIS

I initially posted this as a question in the Numba Gitter channel, during that chat I was asked to post details here on the issue.

The high level issue, I have a module in my codebase accelerated with numba’s jit_module that have worked without issue for months. I recently started writing a QGIS python plugin that uses this module, and when I try to run the code within QGIS I get the following exception/stacktrace that I truncated a bit below:

File “/home/andrew/anaconda3/envs/pangeo/lib/python3.8/site-packages/numba/core/typed_passes.py”, line 104, in run_pass
typemap, return_type, calltypes, errs = type_inference_stage(
File “/home/andrew/anaconda3/envs/pangeo/lib/python3.8/site-packages/numba/core/typed_passes.py”, line 82, in type_inference_stage
errs = infer.propagate(raise_errors=raise_errors)
File “/home/andrew/anaconda3/envs/pangeo/lib/python3.8/site-packages/numba/core/typeinfer.py”, line 1071, in propagate
raise errors[0]
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Use of unsupported NumPy function ‘numpy.concatenate’ or unsupported use of the function.

The curious bit, is that outside of QGIS/QGIS python console, I don’t have this issue and the jit_module compilation works as is.

I am unsure of what else to look at to diagnose the issue, but I can attach the pycharm debugger to QGIS so I am able to set breakpoints within the numba compiler code to try to diagnose things further, if I can get suggestions for what to look at.

A critical detail here is that I installed numba and QGIS within the same conda environment. Below I list the versions on some libraries I think would be relevant to numba developers.

QGIS conda depends

Name Version Build Channel
qgis 3.18.1 py38h34f39cb_0 installed
exiv2 0.27.1 had08079_0 installed
expat 2.3.0 h9c3ff4c_0 installed
future 0.18.2 py38h578d9bd_3 installed
gdal 3.2.1 py38hc0b2d6b_7 installed
geos 3.9.1 h9c3ff4c_2 installed
gsl 2.6 h294904e_0 installed
httplib2 0.19.1 pyhd8ed1ab_0 installed
icu 68.1 h58526e2_0 installed
jinja2 2.11.2 pyh9f0ad1d_0 installed
laz-perf 1.5.0 he1b5a44_0 installed
libgcc-ng 9.3.0 h24d8f2e_14 installed
libpq 13.2 hfd2b0eb_2 installed
libprotobuf 3.15.8 h780b84a_0 installed
libspatialindex 1.9.3 he1b5a44_3 installed
libspatialite 5.0.1 he52d314_3 installed
libstdcxx-ng 9.3.0 hdf63c60_14 installed
libzip 1.7.3 he9f05b3_0 installed
markupsafe 1.1.1 py38h1e0a361_1 installed
mock 4.0.3 py38h578d9bd_0 installed
nose2 0.9.2 py_0 installed
owslib 0.20.0 py_0 installed
pdal 2.2.0 h638e970_7 installed
plotly 4.14.3 pyh44b312d_0 installed
postgresql 13.2 h6303168_2 installed
proj 7.2.0 h277dcde_2 installed
psycopg2 2.8.6 py38h497a2fe_2 installed
pygments 2.6.1 py_0 installed
pyproj 3.0.1 py38h16ecdd7_0 installed
pyqt 5.12.3 py38ha8c2ead_3 installed
pyqtwebkit 5.212 py38hd669dca_1 installed
python 3.8.5 h7579374_1 installed
python-dateutil 2.8.0 py_0 installed
python_abi 3.8 1_cp38 installed
pytz 2020.1 pyh9f0ad1d_0 installed
pyyaml 5.3.1 py38h1e0a361_0 installed
qca 2.2.1 h73816c6_3 installed
qjson 0.9.0 h73816c6_1006 installed
qscintilla2 2.11.2 py38h63a9b5b_4 installed
qt 5.12.9 hda022c4_4 installed
qtkeychain 0.12.0 h2264404_0 installed
qtlocation 5.12.9 he1b5a44_0 installed
qtserialport 5.9.8 h73816c6_1 installed
qtwebkit 5.212 h8f65c2e_1 installed
qwt 6.1.6 h7ec6b3e_0 installed
qwtpolar 1.1.1 h73816c6_7 installed
requests 2.24.0 pyh9f0ad1d_0 installed
six 1.15.0 pyh9f0ad1d_0 installed
sqlite 3.35.5 h74cdb3f_0 installed
yaml 0.2.5 h516909a_0 installed

Numba conda depends

Name Version Build Channel
numba 0.53.1 py38h0e12cce_0 installed
libgcc-ng 9.3.0 h24d8f2e_14 installed
libstdcxx-ng 9.3.0 hdf63c60_14 installed
llvmlite 0.36.0 py38h4630a5e_0 installed
numpy 1.20.2 py38h9894fe3_0 installed
python 3.8.5 h7579374_1 installed
python_abi 3.8 1_cp38 installed
setuptools 49.2.1 py38h32f6830_0 installed

The numba accelerated functions are (with usual imports left out) below. The inspected input for interpolate_line_string is a (2,N) float array (can be the coordinates from a shapely linestring object).

from numba import jit_module
import numpy as np

def interpolate_line(x0, y0, x1, y1, last=False):
    diffx = np.fabs(x1 - x0)
    diffy = np.fabs(y1 - y0)
    num_samples = int(2 * np.sqrt(diffx ** 2 + diffy ** 2))+1
    x = np.linspace(x0, x1, num_samples)
    y = np.linspace(y0, y1, num_samples)
    if not last:
        # we are somewhere in the middle, we need to emulate endpoint=False behavior
        x = x[:-1]
        y = y[:-1]
    return np.vstack((x, y)).T

def interpolate_line_string(feature):
    lf = len(feature)
    res = np.zeros((2, 2), dtype=np.float64)
    ls = int(lf - 1)
    for i in range(ls):
        s = feature[i:i + 2]
        int_seg = interpolate_line(s[0][0], s[0][1], s[1][0], s[1][1], last=(i+1 == ls))
        res = np.concatenate((res, int_seg))
    # if c is the last, then set end_point to true
    return res[2:]

jit_module(nopython=True, fastmath=True, error_model="numpy")

thanks -Andrew

HI Andrew and thank you for posting this. I am not sure either what to recommend. It is very strange. It all appears as though QGIS may somehow have an older version of Numba or something along those lines. My recommendation would be to write a small script that imports numba and llvmlite and simply prints their versions. This will check which version QGIS thinks it has. Or, maybe use grep or find on your anaconda environment to look for a doubly installed Numba (not unheard of). Hope it helps and do report back if neither of those approaches yield anything insightful.

@AndrewAnnex there’s also a load of debug that can be dumped out when Numba runs, but as @esc suggests, it’s best first to double check that the environment is as expected. If nothing leaps out as the problem we’ll point you to some further debug options.

Sorry for the long delay on this, I was able to make enough progress without numba, so I needed to make some headway before looking at this issues again.
So far, I verified that inside and outside of QGIS the same versions of numba (0.53.1) and llvmlite (0.36.0) are being used, and inside the conda environment I only saw those versions. I did, however see numpy 1.19.4 and 1.20.2 in the pkgs folder, although 1.20.2 is being used inside qgis/python. I think numpy.concatenate has been in the api for a while so I don’t see how that would be the issue. Are there any cache directories that numba looks into that I could clear? I’ve definitely experimented with numba on this computer for a few years… Otherwise I may try to inspect the python path, maybe I have a system level package leaking into the conda environment???

Numpy may be an issue here, Numba is known to have some issues with the 1.20 series:

oh, wow I hadn’t considered that possibility/I hadn’t seen that issue before. I can try downgrading numpy to one of the 1.19 releases.

which did not work… although I did not attempt to delete other versions of numpy from the conda environment, but again I don’t see how that would be the issue as the older version was imported into qgis

OK, so probably not Numpy then.