My goal is to run the modified besselfunction of second kind, scipy.special.kv in no-pythonmode. Currently seeing ValueError ‘No function ’ __pyx_fuse_0pdtr’ found in pyx_capi of scipy.special.cython_special’ when running from numba_scipy import special as nsp, and I understand from here that the error possibly comes from version mismatch.
Best,
Okay! Interesting. How can I possibly write a compilable version of scipy.special.kv(v,z) without numba-scipy?
I didn’t have time to look into this, but @stuartarchibald gave me great advice here.
It may not be exactly the same as what you’re doing but at the very least it should help you understand which functions a given version of scipy.special is exporting.
Also, I don’t have an immediate handle on it but there was a project out there that reimplemented parts of scipy.special in numba’d python. I don’t know if it has the kv() function you’re interested in.
Interesting, thanks alot for pointing to the advice. I think I found a solution for now which I detail below.
The short version is:
from numba import njit
from numba.extending import get_cython_function_address
import ctypes
addr = get_cython_function_address('scipy.special.cython_special', '__pyx_fuse_1kv')
functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double, ctypes.c_double)
ckv = functype(addr)
@njit
def ckv_njit(v,z):
return ckv(v,z)
On this website I see the function I want to run in no-python-mode ( scipy.special.kv(v,z) ). Its available as scipy.special.cython_special.kv.
I inspect cython_special to find the functionname for kv:
In [37]: [(x, cython_special.__pyx_capi__[x]) for x in cython_special.__pyx_capi__ if 'kv' in x]
Out [37]: [('__pyx_fuse_0kv',
<capsule object "__pyx_t_double_complex (double, __pyx_t_double_complex, int __pyx_skip_dispatch)" at 0x7f16c2956730>),
('__pyx_fuse_1kv',
<capsule object "double (double, double, int __pyx_skip_dispatch)" at 0x7f16c29558f0>),
('__pyx_fuse_0kve',
<capsule object "__pyx_t_double_complex (double, __pyx_t_double_complex, int __pyx_skip_dispatch)" at 0x7f16c2954b40>),
('__pyx_fuse_1kve',
<capsule object "double (double, double, int __pyx_skip_dispatch)" at 0x7f16c29561c0>)]
According to its types it seems the function I want is connected to the functionname __pyx_fuse_1kv
In [46]: cython_special.__pyx_capi__["__pyx_fuse_1kv"]
Out [46]: <capsule object "double (double, double, int __pyx_skip_dispatch)" at 0x7f16c29558f0>
From reading the documentation the following seems to work for me:
from numba import njit
from numba.extending import get_cython_function_address
import ctypes
addr = get_cython_function_address('scipy.special.cython_special', '__pyx_fuse_1kv')
functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double, ctypes.c_double)
ckv = functype(addr)
@njit
def ckv_njit(v,z):
return ckv(v,z)
I only have one follow-up question:
/tmp/ipykernel_365/431863534.py:1: NumbaWarning: Cannot cache compiled function "ckv_njit" as it uses dynamic globals (such as ctypes pointers and large global arrays)
Hi @ofk123
Yes, there is a way around this, unfortunately not an easy one. If there is an easier way, I’d be very interested to know it too.
Anyway, you can look at this C extension file that exposes the function pointers in a way so that you can embed the call to the Cython-generated C function with llvmlite, as shown here.
This example is even a bit uglier than it will be for you because it calls Cython functions with complex arguments. This is quite nasty due to ABI incompatibilities. If anyone has a better solution for this, I’d be grateful for advice as well.
If you really want to go down this dirty path, I can guide you a bit more. Perhaps I should also revisit this problem to find a cleaner solution…
Hi @sschaer, thanks for your suggestions here. And for sharing that code 
Depending on how the code I am running performs without this function being cacheable, I might very well have to go down this path at some point.
Hi @ofk123
I’ve thought about it a bit, and depending on your needs, I may have a slightly better solution for you. If you have a C++ compiler available and e.g. portability is less important to you, you can use the C++ function cyl_bessel_k. Create a shared object file that exports the function and call it in an intrinsic. You may need to customize this a little. But you can test it as it is on Google Colab.
import numba as nb
from numba.extending import intrinsic
from numba.core.cgutils import get_or_insert_function
from llvmlite import ir
from llvmlite.binding import load_library_permanently
def gpp_make_shared(src, filename):
import subprocess
with open(filename, "w") as file:
file.write(src)
cmd = f"g++ -shared -fPIC -o {filename.replace('.cpp', '.so')} {filename} -lm"
subprocess.run(cmd, shell=True, check=True)
src = """
#include <math.h>
extern "C" double kv(double x, double nu) {
return std::cyl_bessel_k(x, nu);
}
"""
gpp_make_shared(src, "cyl_bessel_k.cpp")
load_library_permanently("./cyl_bessel_k.so")
@intrinsic
def ikv(typingctx, v, z):
if not all(isinstance(x, nb.types.Float) for x in (v, z)):
raise TypingError("arguments must be floats")
def codegen(context, builder, sig, args):
double = ir.DoubleType()
fnty = ir.FunctionType(double, [double, double])
fn = get_or_insert_function(builder.module, fnty, "kv")
v = builder.fpext(args[0], double)
z = builder.fpext(args[1], double)
return builder.call(fn, [v, z])
return nb.double(v, z), codegen
import numpy as np
import scipy.special
@nb.vectorize("f8(f8, f8)", cache=True)
def kv(v, z):
return ikv(v, z)
va = np.random.rand(1_000_000)
za = np.random.rand(1_000_000)
assert np.allclose(kv(va, za), scipy.special.kv(va, za))
%timeit scipy.special.kv(va, za)
%timeit kv(va, za)
@sschaer Thanks alot, nice to see there are other approaches, though C++ has been a bit outside of my knowledge. I could not run the code on my set-up (/bin/sh: 1: g++: not found, detailed below), but it runs fine on Google Colab.
On my setup, this line:
gpp_make_shared(src, "cyl_bessel_k.cpp")
Produces this message:
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
Cell In[2], line 1
----> 1 gpp_make_shared(src, "cyl_bessel_k.cpp")
Cell In[1], line 15, in gpp_make_shared(src, filename)
12 file.write(src)
14 cmd = f"g++ -shared -fPIC -o {filename.replace('.cpp', '.so')} {filename} -lm"
---> 15 subprocess.run(cmd, shell=True, check=True)
File /srv/conda/envs/notebook/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
524 retcode = process.poll()
525 if check and retcode:
--> 526 raise CalledProcessError(retcode, process.args,
527 output=stdout, stderr=stderr)
528 return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command 'g++ -shared -fPIC -o cyl_bessel_k.so cyl_bessel_k.cpp -lm' returned non-zero exit status 127.
kv on several workers in parallel. Regarding this, I have an obstacle detailed here:. If you have any experience with running ctypes.CFUNCTYPE.<locals>.CFunctionType then I would be very interested in your take on this.g++ is the c++ compiler I used which you don’t have or at least not on your path. In principle, you can use any compatible (e.g. clang does not support std::cyl_bessel_k) C++ compiler.
I am not an expert on that but I strongly suspect that the other problem you mentioned is a direct result of the caching issue. The function needs to be serialized in order to use it in other processes. However, since the function pointer may be invalid after deserialization, you get this error. Good thing this is detected so early, otherwise it would be hard to debug. 
For this to work, you need to make caching work.
Thanks @sschaer
Since I dont have direct installer-rights on the environment I use, I could for example go through my conda list to see if some C++compiler already exists.
Do you happen to know the name of some other compatible python - C++ compilers, that I can try searching for?
# packages in environment at /srv/conda/envs/notebook:
#
# Name Version Build Channel
_ipython_minor_entry_point 8.7.0 hb6b4a82_0 conda-forge
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
adal 1.2.7 pyhd8ed1ab_0 conda-forge
adlfs 2022.11.2 pyhd8ed1ab_0 conda-forge
affine 2.3.1 pyhd8ed1ab_0 conda-forge
aiobotocore 2.4.0 pyhd8ed1ab_0 conda-forge
aiofiles 22.1.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.3 py310h5764c6d_1 conda-forge
aioitertools 0.11.0 pyhd8ed1ab_0 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
aiosqlite 0.18.0 pyhd8ed1ab_0 conda-forge
alembic 1.9.1 pyhd8ed1ab_0 conda-forge
anyio 3.6.2 pyhd8ed1ab_0 conda-forge
aom 3.5.0 h27087fc_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
apprise 1.2.1 pyhd8ed1ab_0 conda-forge
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py310h5764c6d_3 conda-forge
argopy 0.1.12 pyhd8ed1ab_0 conda-forge
arrow-cpp 10.0.1 h27aab58_3_cpu conda-forge
asciitree 0.3.3 py_2 conda-forge
asgi-lifespan 1.0.1 pyhd8ed1ab_5 conda-forge
astropy 5.2 py310h0a54255_0 conda-forge
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
async-exit-stack 1.0.1 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
async_generator 1.10 py_0 conda-forge
asyncpg 0.27.0 py310h5764c6d_1 conda-forge
atk-1.0 2.38.0 hd4edc92_1 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
av 10.0.0 py310h2d7f6f1_2 conda-forge
aws-c-auth 0.6.21 h774e2f3_1 conda-forge
aws-c-cal 0.5.20 hff2c3d7_3 conda-forge
aws-c-common 0.8.5 h166bdaf_0 conda-forge
aws-c-compression 0.2.16 hf5f93bc_0 conda-forge
aws-c-event-stream 0.2.17 h52dae97_0 conda-forge
aws-c-http 0.6.29 hf21410f_0 conda-forge
aws-c-io 0.13.11 h4f448d1_2 conda-forge
aws-c-mqtt 0.7.13 hefb3e95_10 conda-forge
aws-c-s3 0.2.1 h927de71_2 conda-forge
aws-c-sdkutils 0.1.7 hf5f93bc_0 conda-forge
aws-checksums 0.1.14 h6027aba_0 conda-forge
aws-crt-cpp 0.18.16 ha7c8eef_6 conda-forge
aws-sdk-cpp 1.9.379 h32f6703_7 conda-forge
awscli 1.25.60 py310hff52083_0 conda-forge
azure-core 1.26.1 pyhd8ed1ab_0 conda-forge
azure-datalake-store 0.0.51 pyh9f0ad1d_0 conda-forge
azure-identity 1.12.0 pyhd8ed1ab_0 conda-forge
azure-storage-blob 12.14.1 pyhd8ed1ab_0 conda-forge
babel 2.11.0 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bcrypt 3.2.2 py310h5764c6d_1 conda-forge
beautifulsoup4 4.11.1 pyha770c72_0 conda-forge
binutils_impl_linux-64 2.39 he00db2b_1 conda-forge
binutils_linux-64 2.39 h5fc0e48_11 conda-forge
bleach 5.0.1 pyhd8ed1ab_0 conda-forge
blinker 1.5 pyhd8ed1ab_0 conda-forge
blosc 1.21.3 hafa529b_0 conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
boost-cpp 1.78.0 h75c5d50_1 conda-forge
boto3 1.24.59 pyhd8ed1ab_0 conda-forge
botocore 1.27.59 pyhd8ed1ab_0 conda-forge
bottleneck 1.3.5 py310hde88566_1 conda-forge
bounded-pool-executor 0.0.3 pyhd8ed1ab_0 conda-forge
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
brotlipy 0.7.0 py310h5764c6d_1005 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
c-blosc2 2.6.1 hf91038e_0 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.2.0 pyhd8ed1ab_0 conda-forge
cachey 0.2.1 pyh9f0ad1d_0 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
cartopy 0.21.1 py310hcb7e713_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
certipy 0.1.3 py_0 conda-forge
cf_xarray 0.7.6 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h255011f_3 conda-forge
cfgrib 0.9.10.3 pyhd8ed1ab_0 conda-forge
cfitsio 4.2.0 hd9d235c_0 conda-forge
cftime 1.6.2 py310hde88566_1 conda-forge
cgen 2020.1 py_0 conda-forge
charls 2.3.4 h9c3ff4c_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
ciso 0.2.0 py310hde88566_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
colorcet 3.0.1 pyhd8ed1ab_0 conda-forge
comm 0.1.2 pyhd8ed1ab_0 conda-forge
commonmark 0.9.1 py_0 conda-forge
configobj 5.0.6 py_0 conda-forge
contourpy 1.0.6 py310hbf28c38_0 conda-forge
coolname 2.0.0 pyhd8ed1ab_0 conda-forge
croniter 1.3.8 pyhd8ed1ab_0 conda-forge
cryptography 39.0.0 py310h34c0648_0 conda-forge
curl 7.87.0 hdc1c0ab_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cython 0.29.32 py310hd8f1fbe_1 conda-forge
cytoolz 0.12.0 py310h5764c6d_1 conda-forge
dask 2022.12.0 pyhd8ed1ab_0 conda-forge
dask-core 2022.12.0 pyhd8ed1ab_0 conda-forge
dask-gateway 2022.10.0 pyh8af1aa0_0 conda-forge
dask-glm 0.2.0 py_1 conda-forge
dask-labextension 6.0.0 pyhd8ed1ab_0 conda-forge
dask-ml 2022.5.27 pyhd8ed1ab_0 conda-forge
dataclasses 0.8 pyhc8e2a94_3 conda-forge
datashader 0.14.3 pyh1a96a4e_0 conda-forge
datashape 0.5.4 py_1 conda-forge
dav1d 1.0.0 h166bdaf_1 conda-forge
debugpy 1.6.4 py310hd8f1fbe_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
descartes 1.1.0 py_4 conda-forge
distributed 2022.12.0 pyhd8ed1ab_0 conda-forge
docker-py 6.0.0 pyhd8ed1ab_0 conda-forge
docopt 0.6.2 py_1 conda-forge
docrep 0.3.2 pyh44b312d_0 conda-forge
docutils 0.15.2 py310hff52083_6 conda-forge
donfig 0.7.0 pyhd8ed1ab_1 conda-forge
earthdata 0.2.2 pyhd8ed1ab_0 conda-forge
eccodes 2.27.1 h7f7619e_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
eofs 1.4.0 py_0 conda-forge
erddapy 1.2.1 pyhd8ed1ab_0 conda-forge
esmf 8.4.0 mpi_mpich_hc592774_102 conda-forge
esmpy 8.4.0 mpi_mpich_py310h515c5ea_101 conda-forge
exceptiongroup 1.1.0 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
expat 2.5.0 h27087fc_0 conda-forge
fastapi 0.88.0 pyhd8ed1ab_0 conda-forge
fasteners 0.17.3 pyhd8ed1ab_0 conda-forge
fastjmd95 0.2.1 pyh44b312d_0 conda-forge
fastprogress 1.0.3 pyhd8ed1ab_0 conda-forge
ffmpeg 5.1.2 gpl_h8dda1f0_105 conda-forge
findlibs 0.0.2 pyhd8ed1ab_0 conda-forge
fiona 1.8.22 py310ha325b7b_5 conda-forge
flit-core 3.8.0 pyhd8ed1ab_0 conda-forge
flox 0.6.5 pyhd8ed1ab_0 conda-forge
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.1 hc2a2eb6_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.38.0 py310h5764c6d_1 conda-forge
freeglut 3.2.2 h9c3ff4c_1 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
frozenlist 1.3.3 py310h5764c6d_0 conda-forge
fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge
ftfy 6.1.1 pyhd8ed1ab_0 conda-forge
future 0.18.2 pyhd8ed1ab_6 conda-forge
gcc_impl_linux-64 12.2.0 hcc96c02_19 conda-forge
gcc_linux-64 12.2.0 h4798a0e_11 conda-forge
gcm_filters 0.3.0 pyhd8ed1ab_0 conda-forge
gcsfs 2022.11.0 pyhd8ed1ab_0 conda-forge
gdal 3.6.1 py310hc1b7723_2 conda-forge
gdk-pixbuf 2.42.10 h05c8ddd_0 conda-forge
geocube 0.3.3 pyhd8ed1ab_0 conda-forge
geogif 0.1.3 pyhd8ed1ab_0 conda-forge
geographiclib 1.52 pyhd8ed1ab_0 conda-forge
geopandas 0.12.2 pyhd8ed1ab_0 conda-forge
geopandas-base 0.12.2 pyha770c72_0 conda-forge
geopy 2.3.0 pyhd8ed1ab_0 conda-forge
geos 3.11.1 h27087fc_0 conda-forge
geotiff 1.7.1 h7157cca_5 conda-forge
geoviews-core 1.9.5 pyha770c72_0 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
gh 2.21.1 ha8f183a_0 conda-forge
gh-scoped-creds 4.1 pyhd8ed1ab_0 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
git-lfs 3.3.0 ha770c72_0 conda-forge
gitdb 4.0.10 pyhd8ed1ab_0 conda-forge
gitpython 3.1.30 pyhd8ed1ab_0 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.7.8 hf3e180e_0 conda-forge
google-api-core 2.11.0 pyhd8ed1ab_0 conda-forge
google-auth 2.15.0 pyh1a96a4e_0 conda-forge
google-auth-oauthlib 0.8.0 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge
google-cloud-storage 2.7.0 pyh1a96a4e_0 conda-forge
google-crc32c 1.1.2 py310he8fe98e_4 conda-forge
google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge
googleapis-common-protos 1.57.0 pyhd8ed1ab_3 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
graphviz 7.0.5 h2e5815a_0 conda-forge
greenlet 2.0.1 py310hd8f1fbe_0 conda-forge
griffe 0.21.0 pyhd8ed1ab_1 conda-forge
grpcio 1.51.1 py310hc32fa93_0 conda-forge
gsw 3.4.0 py310hde88566_3 conda-forge
gtk2 2.24.33 h90689f9_2 conda-forge
gts 0.7.6 h64030ff_2 conda-forge
h11 0.14.0 pyhd8ed1ab_0 conda-forge
h2 4.1.0 pyhd8ed1ab_0 conda-forge
h3-py 3.7.4 py310hd8f1fbe_1 conda-forge
h5netcdf 1.1.0 pyhd8ed1ab_0 conda-forge
h5py 3.7.0 nompi_py310h416281c_102 conda-forge
harfbuzz 6.0.0 h8e241bc_0 conda-forge
hdf4 4.2.15 h9772cbc_5 conda-forge
hdf5 1.12.2 mpi_mpich_h5d83325_1 conda-forge
heapdict 1.0.1 py_0 conda-forge
holoviews 1.15.3 pyhd8ed1ab_0 conda-forge
hpack 4.0.0 pyh9f0ad1d_0 conda-forge
httpcore 0.16.3 pyhd8ed1ab_0 conda-forge
httpx 0.23.2 pyhd8ed1ab_0 conda-forge
hvplot 0.8.2 pyhd8ed1ab_0 conda-forge
hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagecodecs 2022.12.24 py310h17758e3_0 conda-forge
imageio 2.23.0 pyhfa7a67d_0 conda-forge
importlib-metadata 6.0.0 pyha770c72_0 conda-forge
importlib_metadata 6.0.0 hd8ed1ab_0 conda-forge
importlib_resources 5.10.2 pyhd8ed1ab_0 conda-forge
iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge
intake 0.6.6 pyhd8ed1ab_0 conda-forge
intake-esm 2022.9.18 pyhd8ed1ab_0 conda-forge
intake-geopandas 0.4.0 pyhd8ed1ab_0 conda-forge
intake-stac 0.4.0 pyhd8ed1ab_0 conda-forge
intake-xarray 0.6.1 pyhd8ed1ab_0 conda-forge
ipykernel 6.19.4 pyh210e3f2_0 conda-forge
ipyleaflet 0.17.2 pyhd8ed1ab_0 conda-forge
ipyspin 1.0.1 pyhd8ed1ab_0 conda-forge
ipython 8.8.0 pyh41d4057_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipytree 0.2.2 pyhd8ed1ab_0 conda-forge
ipyurl 0.1.2 pyh3684270_1 conda-forge
ipywidgets 8.0.3 pyhd8ed1ab_0 conda-forge
isodate 0.6.1 pyhd8ed1ab_0 conda-forge
jasper 2.0.33 ha77e612_0 conda-forge
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h166bdaf_2 conda-forge
json-c 0.16 hc379101_0 conda-forge
json5 0.9.5 pyh9f0ad1d_0 conda-forge
jsonpatch 1.32 pyhd8ed1ab_0 conda-forge
jsonpointer 2.0 py_0 conda-forge
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
jupyter-panel-proxy 0.1.0 py_0 conda-forge
jupyter-resource-usage 0.6.4 pyhd8ed1ab_0 conda-forge
jupyter-server-mathjax 0.2.6 pyh5bfe37b_1 conda-forge
jupyter-server-proxy 3.2.2 pyhd8ed1ab_0 conda-forge
jupyter_client 7.4.8 pyhd8ed1ab_0 conda-forge
jupyter_core 5.1.2 py310hff52083_0 conda-forge
jupyter_server 1.23.4 pyhd8ed1ab_0 conda-forge
jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge
jupyterhub-base 3.1.0 pyh2a2186d_0 conda-forge
jupyterhub-singleuser 3.1.0 pyh2a2186d_0 conda-forge
jupyterlab 3.5.1 pyhd8ed1ab_0 conda-forge
jupyterlab-git 0.41.0 pyhd8ed1ab_1 conda-forge
jupyterlab-s3-browser 0.12.0 pypi_0 pypi
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_server 2.18.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 3.0.5 pyhd8ed1ab_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
kealib 1.5.0 ha7026e8_0 conda-forge
kerchunk 0.0.9 pyhd8ed1ab_0 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py310hbf28c38_1 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
It’s not about compatibility with Python, it’s about having a compiler that provides the necessary math functions. The compiler is also not something you will find in your conda list.
If you don’t feel comfortable fiddling around with compilers and stuff, there yet one other solution. You can always pass the function pointer as an additional argument and then call the Cython function via this pointer:
from llvmlite import ir
from numba import extending
from numba import types
from numba import TypingError
from numba.extending import get_cython_function_address
double_t = ir.DoubleType()
kv_t = ir.FunctionType(double_t, [double_t, double_t])
kvp_t = kv_t.as_pointer()
@extending.intrinsic
def kv(typingctx, addr, v, z):
if not isinstance(addr, types.Integer):
raise TypingError("'addr' must be an integer")
if not all(isinstance(x, types.Float) for x in (v, z)):
raise TypingError("'v' and 'z' must be floats")
def codegen(context, builder, sig, args):
[addr, v, z] = args
kv = builder.inttoptr(addr, kvp_t)
v = builder.fpext(v, double_t)
z = builder.fpext(z, double_t)
return builder.call(kv, [v, z])
return types.double(addr, v, z), codegen
You can use it like that:
import numpy as np
import numba as nb
import scipy.special
@nb.njit(cache=True)
def test(addr, v, z):
out = np.empty(v.size)
for i in range(v.size):
out[i] = kv(addr, v[i], z[i])
return out
v = np.random.rand(100_000)
z = np.random.rand(100_000)
addr = get_cython_function_address("scipy.special.cython_special", "__pyx_fuse_1kv")
assert np.allclose(test(addr, v, z), scipy.special.kv(v, z))
Thanks alot for another suggestion @sschaer
@nb.njit(cache=True)
def test(addr, v, z):
return kv(addr, v, z)
v, z = 1.5, 10.5
test(addr, 1.5, 10.5) outputs 6.260706303843087. Your code clearly works.
I set up a local scheduler with distributed’s LocalCluster(). (This is still on the same machine, but gives the possibility to distribute tasks to local workers, (in this case, to 4 workers)):
client.run(test, addr, v, z) outputs CommClosedError.
from distributed import Client, LocalCluster
cluster=LocalCluster()
client=Client(cluster)
client.run(test, addr, v, z)
outputs:
2023-11-22 12:34:28,781 - distributed.scheduler - ERROR - broadcast to tcp://127.0.0.1:39139 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://127.0.0.1:47412 remote=tcp://127.0.0.1:39139>: Stream is closed
2023-11-22 12:34:28,788 - distributed.scheduler - ERROR - broadcast to tcp://127.0.0.1:34545 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://127.0.0.1:35376 remote=tcp://127.0.0.1:34545>: Stream is closed
2023-11-22 12:34:28,792 - distributed.nanny - WARNING - Restarting worker
2023-11-22 12:34:28,794 - distributed.nanny - WARNING - Restarting worker
2023-11-22 12:34:28,809 - distributed.scheduler - ERROR - broadcast to tcp://127.0.0.1:33269 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://127.0.0.1:52468 remote=tcp://127.0.0.1:33269>: Stream is closed
2023-11-22 12:34:28,814 - distributed.nanny - WARNING - Restarting worker
2023-11-22 12:34:28,853 - distributed.scheduler - ERROR - broadcast to tcp://127.0.0.1:33645 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://127.0.0.1:35752 remote=tcp://127.0.0.1:33645>: Stream is closed
2023-11-22 12:34:28,859 - distributed.nanny - WARNING - Restarting worker
---------------------------------------------------------------------------
CommClosedError Traceback (most recent call last)
Cell In[40], line 1
----> 1 client.run(test, addr, v, z)
File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/client.py:2901, in Client.run(self, function, workers, wait, nanny, on_error, *args, **kwargs)
2818 def run(
2819 self,
2820 function,
(...)
2826 **kwargs,
2827 ):
2828 """
2829 Run a function on all workers outside of task scheduling system
2830
(...)
2899 >>> c.run(print_state, wait=False) # doctest: +SKIP
2900 """
-> 2901 return self.sync(
2902 self._run,
2903 function,
2904 *args,
2905 workers=workers,
2906 wait=wait,
2907 nanny=nanny,
2908 on_error=on_error,
2909 **kwargs,
2910 )
File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/utils.py:339, in SyncMethodMixin.sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
337 return future
338 else:
--> 339 return sync(
340 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
341 )
File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/utils.py:406, in sync(loop, func, callback_timeout, *args, **kwargs)
404 if error:
405 typ, exc, tb = error
--> 406 raise exc.with_traceback(tb)
407 else:
408 return result
File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/utils.py:379, in sync.<locals>.f()
377 future = asyncio.wait_for(future, callback_timeout)
378 future = asyncio.ensure_future(future)
--> 379 result = yield future
380 except Exception:
381 error = sys.exc_info()
File /srv/conda/envs/notebook/lib/python3.10/site-packages/tornado/gen.py:769, in Runner.run(self)
766 exc_info = None
768 try:
--> 769 value = future.result()
770 except Exception:
771 exc_info = sys.exc_info()
File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/client.py:2806, in Client._run(self, function, nanny, workers, wait, on_error, *args, **kwargs)
2803 continue
2805 if on_error == "raise":
-> 2806 raise exc
2807 elif on_error == "return":
2808 results[key] = exc
CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://127.0.0.1:52468 remote=tcp://127.0.0.1:33269>: Stream is closed
So when running test(addr, v, z) in a distributed fashion, the scheduler logs that the workers-processes are shut down by some signal 11, resulting in a CommClosedError:
{'tcp://127.0.0.1:33565': (('INFO',
"2023-11-22 12:33:05,043 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:43347'"),
('INFO',
"2023-11-22 12:33:05,045 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:45013'"),
('INFO',
"2023-11-22 12:33:05,050 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:44561'"),
('INFO',
"2023-11-22 12:33:05,054 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:38947'"),
('INFO',
'2023-11-22 12:33:36,404 - distributed.nanny - INFO - Worker process 1353 was killed by signal 11'),
('INFO',
'2023-11-22 12:33:36,408 - distributed.nanny - INFO - Worker process 1360 was killed by signal 11'),
('INFO',
'2023-11-22 12:33:36,412 - distributed.nanny - INFO - Worker process 1357 was killed by signal 11'),
('WARNING',
'2023-11-22 12:33:36,419 - distributed.nanny - WARNING - Restarting worker'),
('WARNING',
'2023-11-22 12:33:36,423 - distributed.nanny - WARNING - Restarting worker'),
('WARNING',
'2023-11-22 12:33:36,428 - distributed.nanny - WARNING - Restarting worker'),
('INFO',
'2023-11-22 12:33:36,461 - distributed.nanny - INFO - Worker process 1362 was killed by signal 11'),
('WARNING',
'2023-11-22 12:33:36,465 - distributed.nanny - WARNING - Restarting worker'),
('INFO',}
I would be interested in your thoughts about why this could be happening? Since this is moving away from numba-topics I should consider moving this post to some other forum though.