I initially posted this as a question in the Numba Gitter channel, during that chat I was asked to post details here on the issue.
The high level issue, I have a module in my codebase accelerated with numba’s jit_module that have worked without issue for months. I recently started writing a QGIS python plugin that uses this module, and when I try to run the code within QGIS I get the following exception/stacktrace that I truncated a bit below:
File “/home/andrew/anaconda3/envs/pangeo/lib/python3.8/site-packages/numba/core/typed_passes.py”, line 104, in run_pass
typemap, return_type, calltypes, errs = type_inference_stage(
File “/home/andrew/anaconda3/envs/pangeo/lib/python3.8/site-packages/numba/core/typed_passes.py”, line 82, in type_inference_stage
errs = infer.propagate(raise_errors=raise_errors)
File “/home/andrew/anaconda3/envs/pangeo/lib/python3.8/site-packages/numba/core/typeinfer.py”, line 1071, in propagate
raise errors[0]
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Use of unsupported NumPy function ‘numpy.concatenate’ or unsupported use of the function.
The curious bit, is that outside of QGIS/QGIS python console, I don’t have this issue and the jit_module compilation works as is.
I am unsure of what else to look at to diagnose the issue, but I can attach the pycharm debugger to QGIS so I am able to set breakpoints within the numba compiler code to try to diagnose things further, if I can get suggestions for what to look at.
A critical detail here is that I installed numba and QGIS within the same conda environment. Below I list the versions on some libraries I think would be relevant to numba developers.
QGIS conda depends
Name Version Build Channel
──────────────────────────────────────────────────
qgis 3.18.1 py38h34f39cb_0 installed
exiv2 0.27.1 had08079_0 installed
expat 2.3.0 h9c3ff4c_0 installed
future 0.18.2 py38h578d9bd_3 installed
gdal 3.2.1 py38hc0b2d6b_7 installed
geos 3.9.1 h9c3ff4c_2 installed
gsl 2.6 h294904e_0 installed
httplib2 0.19.1 pyhd8ed1ab_0 installed
icu 68.1 h58526e2_0 installed
jinja2 2.11.2 pyh9f0ad1d_0 installed
laz-perf 1.5.0 he1b5a44_0 installed
libgcc-ng 9.3.0 h24d8f2e_14 installed
libpq 13.2 hfd2b0eb_2 installed
libprotobuf 3.15.8 h780b84a_0 installed
libspatialindex 1.9.3 he1b5a44_3 installed
libspatialite 5.0.1 he52d314_3 installed
libstdcxx-ng 9.3.0 hdf63c60_14 installed
libzip 1.7.3 he9f05b3_0 installed
markupsafe 1.1.1 py38h1e0a361_1 installed
mock 4.0.3 py38h578d9bd_0 installed
nose2 0.9.2 py_0 installed
owslib 0.20.0 py_0 installed
pdal 2.2.0 h638e970_7 installed
plotly 4.14.3 pyh44b312d_0 installed
postgresql 13.2 h6303168_2 installed
proj 7.2.0 h277dcde_2 installed
psycopg2 2.8.6 py38h497a2fe_2 installed
pygments 2.6.1 py_0 installed
pyproj 3.0.1 py38h16ecdd7_0 installed
pyqt 5.12.3 py38ha8c2ead_3 installed
pyqtwebkit 5.212 py38hd669dca_1 installed
python 3.8.5 h7579374_1 installed
python-dateutil 2.8.0 py_0 installed
python_abi 3.8 1_cp38 installed
pytz 2020.1 pyh9f0ad1d_0 installed
pyyaml 5.3.1 py38h1e0a361_0 installed
qca 2.2.1 h73816c6_3 installed
qjson 0.9.0 h73816c6_1006 installed
qscintilla2 2.11.2 py38h63a9b5b_4 installed
qt 5.12.9 hda022c4_4 installed
qtkeychain 0.12.0 h2264404_0 installed
qtlocation 5.12.9 he1b5a44_0 installed
qtserialport 5.9.8 h73816c6_1 installed
qtwebkit 5.212 h8f65c2e_1 installed
qwt 6.1.6 h7ec6b3e_0 installed
qwtpolar 1.1.1 h73816c6_7 installed
requests 2.24.0 pyh9f0ad1d_0 installed
six 1.15.0 pyh9f0ad1d_0 installed
sqlite 3.35.5 h74cdb3f_0 installed
yaml 0.2.5 h516909a_0 installed
Numba conda depends
Name Version Build Channel
───────────────────────────────────────────────
numba 0.53.1 py38h0e12cce_0 installed
libgcc-ng 9.3.0 h24d8f2e_14 installed
libstdcxx-ng 9.3.0 hdf63c60_14 installed
llvmlite 0.36.0 py38h4630a5e_0 installed
numpy 1.20.2 py38h9894fe3_0 installed
python 3.8.5 h7579374_1 installed
python_abi 3.8 1_cp38 installed
setuptools 49.2.1 py38h32f6830_0 installed
The numba accelerated functions are (with usual imports left out) below. The inspected input for interpolate_line_string is a (2,N) float array (can be the coordinates from a shapely linestring object).
from numba import jit_module
import numpy as np
def interpolate_line(x0, y0, x1, y1, last=False):
diffx = np.fabs(x1 - x0)
diffy = np.fabs(y1 - y0)
num_samples = int(2 * np.sqrt(diffx ** 2 + diffy ** 2))+1
x = np.linspace(x0, x1, num_samples)
y = np.linspace(y0, y1, num_samples)
if not last:
# we are somewhere in the middle, we need to emulate endpoint=False behavior
x = x[:-1]
y = y[:-1]
return np.vstack((x, y)).T
def interpolate_line_string(feature):
lf = len(feature)
res = np.zeros((2, 2), dtype=np.float64)
ls = int(lf - 1)
for i in range(ls):
s = feature[i:i + 2]
int_seg = interpolate_line(s[0][0], s[0][1], s[1][0], s[1][1], last=(i+1 == ls))
res = np.concatenate((res, int_seg))
# if c is the last, then set end_point to true
return res[2:]
jit_module(nopython=True, fastmath=True, error_model="numpy")
thanks -Andrew