Which version of scipy is numba-scipy currently most compatible with?

  1. Would scipy==1.7.3 & numba-scipy==0.3.1 & numba==0.58.1 be the versions to use?

My goal is to run the modified besselfunction of second kind, scipy.special.kv in no-pythonmode. Currently seeing ValueError ‘No function ’ __pyx_fuse_0pdtr’ found in pyx_capi of scipy.special.cython_special’ when running from numba_scipy import special as nsp, and I understand from here that the error possibly comes from version mismatch.


do you need numba-scipy? I used scipy directly with numba.

Okay! Interesting. How can I possibly write a compilable version of scipy.special.kv(v,z) without numba-scipy?

I didn’t have time to look into this, but @stuartarchibald gave me great advice here.
It may not be exactly the same as what you’re doing but at the very least it should help you understand which functions a given version of scipy.special is exporting.

Also, I don’t have an immediate handle on it but there was a project out there that reimplemented parts of scipy.special in numba’d python. I don’t know if it has the kv() function you’re interested in.

1 Like

Interesting, thanks alot for pointing to the advice. I think I found a solution for now which I detail below.

The short version is:

from numba import njit
from numba.extending import get_cython_function_address
import ctypes
addr = get_cython_function_address('scipy.special.cython_special', '__pyx_fuse_1kv')
functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double, ctypes.c_double)
ckv = functype(addr)

def ckv_njit(v,z):
    return ckv(v,z)
Longer version in case others end up here:

On this website I see the function I want to run in no-python-mode ( scipy.special.kv(v,z) ). Its available as scipy.special.cython_special.kv.

I inspect cython_special to find the functionname for kv:

In [37]: [(x, cython_special.__pyx_capi__[x]) for x in cython_special.__pyx_capi__ if 'kv' in x]

Out [37]: [('__pyx_fuse_0kv',
  <capsule object "__pyx_t_double_complex (double, __pyx_t_double_complex, int __pyx_skip_dispatch)" at 0x7f16c2956730>),
  <capsule object "double (double, double, int __pyx_skip_dispatch)" at 0x7f16c29558f0>),
  <capsule object "__pyx_t_double_complex (double, __pyx_t_double_complex, int __pyx_skip_dispatch)" at 0x7f16c2954b40>),
  <capsule object "double (double, double, int __pyx_skip_dispatch)" at 0x7f16c29561c0>)]

According to its types it seems the function I want is connected to the functionname __pyx_fuse_1kv

In [46]: cython_special.__pyx_capi__["__pyx_fuse_1kv"]
Out [46]: <capsule object "double (double, double, int __pyx_skip_dispatch)" at 0x7f16c29558f0>

From reading the documentation the following seems to work for me:

from numba import njit
from numba.extending import get_cython_function_address
import ctypes
addr = get_cython_function_address('scipy.special.cython_special', '__pyx_fuse_1kv')
functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double, ctypes.c_double)
ckv = functype(addr)

def ckv_njit(v,z):
    return ckv(v,z)

I only have one follow-up question:

  1. I would ideally like to be able to cache this function but I get the message below. Did you ever get around this somehow?

/tmp/ipykernel_365/431863534.py:1: NumbaWarning: Cannot cache compiled function "ckv_njit" as it uses dynamic globals (such as ctypes pointers and large global arrays)

Hi @ofk123

Yes, there is a way around this, unfortunately not an easy one. If there is an easier way, I’d be very interested to know it too.

Anyway, you can look at this C extension file that exposes the function pointers in a way so that you can embed the call to the Cython-generated C function with llvmlite, as shown here.

This example is even a bit uglier than it will be for you because it calls Cython functions with complex arguments. This is quite nasty due to ABI incompatibilities. If anyone has a better solution for this, I’d be grateful for advice as well.

If you really want to go down this dirty path, I can guide you a bit more. Perhaps I should also revisit this problem to find a cleaner solution…

1 Like

Hi @sschaer, thanks for your suggestions here. And for sharing that code :+1:

Depending on how the code I am running performs without this function being cacheable, I might very well have to go down this path at some point.

Hi @ofk123

I’ve thought about it a bit, and depending on your needs, I may have a slightly better solution for you. If you have a C++ compiler available and e.g. portability is less important to you, you can use the C++ function cyl_bessel_k. Create a shared object file that exports the function and call it in an intrinsic. You may need to customize this a little. But you can test it as it is on Google Colab.

import numba as nb 
from numba.extending import intrinsic
from numba.core.cgutils import get_or_insert_function
from llvmlite import ir
from llvmlite.binding import load_library_permanently

def gpp_make_shared(src, filename):
  import subprocess

  with open(filename, "w") as file:

  cmd = f"g++ -shared -fPIC -o {filename.replace('.cpp', '.so')} {filename} -lm"
  subprocess.run(cmd, shell=True, check=True)

src = """
#include <math.h>

extern "C" double kv(double x, double nu) {
    return std::cyl_bessel_k(x, nu);

gpp_make_shared(src, "cyl_bessel_k.cpp")


def ikv(typingctx, v, z):
  if not all(isinstance(x, nb.types.Float) for x in (v, z)):
    raise TypingError("arguments must be floats")

  def codegen(context, builder, sig, args):
    double = ir.DoubleType()
    fnty = ir.FunctionType(double, [double, double])
    fn = get_or_insert_function(builder.module, fnty, "kv")

    v = builder.fpext(args[0], double)
    z = builder.fpext(args[1], double)
    return builder.call(fn, [v, z])

  return nb.double(v, z), codegen

import numpy as np 
import scipy.special

@nb.vectorize("f8(f8, f8)", cache=True) 
def kv(v, z):
  return ikv(v, z)

va = np.random.rand(1_000_000)
za = np.random.rand(1_000_000)

assert np.allclose(kv(va, za), scipy.special.kv(va, za))

%timeit scipy.special.kv(va, za)
%timeit kv(va, za)

@sschaer Thanks alot, nice to see there are other approaches, though C++ has been a bit outside of my knowledge. I could not run the code on my set-up (/bin/sh: 1: g++: not found, detailed below), but it runs fine on Google Colab.

  1. What would I need in order to get around the error?
Details on (`/bin/sh: 1: g++: not found`)

On my setup, this line:

gpp_make_shared(src, "cyl_bessel_k.cpp")

Produces this message:

CalledProcessError                        Traceback (most recent call last)
Cell In[2], line 1
----> 1 gpp_make_shared(src, "cyl_bessel_k.cpp")

Cell In[1], line 15, in gpp_make_shared(src, filename)
     12     file.write(src)
     14 cmd = f"g++ -shared -fPIC -o {filename.replace('.cpp', '.so')} {filename} -lm"
---> 15 subprocess.run(cmd, shell=True, check=True)

File /srv/conda/envs/notebook/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    524     retcode = process.poll()
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command 'g++ -shared -fPIC -o cyl_bessel_k.so cyl_bessel_k.cpp -lm' returned non-zero exit status 127.
  1. Also: I plan on running the scalar-input version of kv on several workers in parallel. Regarding this, I have an obstacle detailed here:. If you have any experience with running ctypes.CFUNCTYPE.<locals>.CFunctionType then I would be very interested in your take on this.

g++ is the c++ compiler I used which you don’t have or at least not on your path. In principle, you can use any compatible (e.g. clang does not support std::cyl_bessel_k) C++ compiler.

I am not an expert on that but I strongly suspect that the other problem you mentioned is a direct result of the caching issue. The function needs to be serialized in order to use it in other processes. However, since the function pointer may be invalid after deserialization, you get this error. Good thing this is detected so early, otherwise it would be hard to debug. :slight_smile: For this to work, you need to make caching work.

1 Like

Thanks @sschaer

Since I dont have direct installer-rights on the environment I use, I could for example go through my conda list to see if some C++compiler already exists.

Do you happen to know the name of some other compatible python - C++ compilers, that I can try searching for?

conda list
# packages in environment at /srv/conda/envs/notebook:
# Name                    Version                   Build  Channel
_ipython_minor_entry_point 8.7.0                hb6b4a82_0    conda-forge
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
adal                      1.2.7              pyhd8ed1ab_0    conda-forge
adlfs                     2022.11.2          pyhd8ed1ab_0    conda-forge
affine                    2.3.1              pyhd8ed1ab_0    conda-forge
aiobotocore               2.4.0              pyhd8ed1ab_0    conda-forge
aiofiles                  22.1.0             pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.3           py310h5764c6d_1    conda-forge
aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
aiosqlite                 0.18.0             pyhd8ed1ab_0    conda-forge
alembic                   1.9.1              pyhd8ed1ab_0    conda-forge
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
aom                       3.5.0                h27087fc_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
apprise                   1.2.1              pyhd8ed1ab_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
argopy                    0.1.12             pyhd8ed1ab_0    conda-forge
arrow-cpp                 10.0.1           h27aab58_3_cpu    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
asgi-lifespan             1.0.1              pyhd8ed1ab_5    conda-forge
astropy                   5.2             py310h0a54255_0    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
async-exit-stack          1.0.1              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
async_generator           1.10                       py_0    conda-forge
asyncpg                   0.27.0          py310h5764c6d_1    conda-forge
atk-1.0                   2.38.0               hd4edc92_1    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
av                        10.0.0          py310h2d7f6f1_2    conda-forge
aws-c-auth                0.6.21               h774e2f3_1    conda-forge
aws-c-cal                 0.5.20               hff2c3d7_3    conda-forge
aws-c-common              0.8.5                h166bdaf_0    conda-forge
aws-c-compression         0.2.16               hf5f93bc_0    conda-forge
aws-c-event-stream        0.2.17               h52dae97_0    conda-forge
aws-c-http                0.6.29               hf21410f_0    conda-forge
aws-c-io                  0.13.11              h4f448d1_2    conda-forge
aws-c-mqtt                0.7.13              hefb3e95_10    conda-forge
aws-c-s3                  0.2.1                h927de71_2    conda-forge
aws-c-sdkutils            0.1.7                hf5f93bc_0    conda-forge
aws-checksums             0.1.14               h6027aba_0    conda-forge
aws-crt-cpp               0.18.16              ha7c8eef_6    conda-forge
aws-sdk-cpp               1.9.379              h32f6703_7    conda-forge
awscli                    1.25.60         py310hff52083_0    conda-forge
azure-core                1.26.1             pyhd8ed1ab_0    conda-forge
azure-datalake-store      0.0.51             pyh9f0ad1d_0    conda-forge
azure-identity            1.12.0             pyhd8ed1ab_0    conda-forge
azure-storage-blob        12.14.1            pyhd8ed1ab_0    conda-forge
babel                     2.11.0             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.2           py310h5764c6d_1    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
binutils_impl_linux-64    2.39                 he00db2b_1    conda-forge
binutils_linux-64         2.39                h5fc0e48_11    conda-forge
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
blinker                   1.5                pyhd8ed1ab_0    conda-forge
blosc                     1.21.3               hafa529b_0    conda-forge
bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
boost-cpp                 1.78.0               h75c5d50_1    conda-forge
boto3                     1.24.59            pyhd8ed1ab_0    conda-forge
botocore                  1.27.59            pyhd8ed1ab_0    conda-forge
bottleneck                1.3.5           py310hde88566_1    conda-forge
bounded-pool-executor     0.0.3              pyhd8ed1ab_0    conda-forge
branca                    0.6.0              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
c-blosc2                  2.6.1                hf91038e_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.2.0              pyhd8ed1ab_0    conda-forge
cachey                    0.2.1              pyh9f0ad1d_0    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
cartopy                   0.21.1          py310hcb7e713_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
certipy                   0.1.3                      py_0    conda-forge
cf_xarray                 0.7.6              pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
cfgrib                     pyhd8ed1ab_0    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
cftime                    1.6.2           py310hde88566_1    conda-forge
cgen                      2020.1                     py_0    conda-forge
charls                    2.3.4                h9c3ff4c_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
ciso                      0.2.0           py310hde88566_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               2.2.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
colorcet                  3.0.1              pyhd8ed1ab_0    conda-forge
comm                      0.1.2              pyhd8ed1ab_0    conda-forge
commonmark                0.9.1                      py_0    conda-forge
configobj                 5.0.6                      py_0    conda-forge
contourpy                 1.0.6           py310hbf28c38_0    conda-forge
coolname                  2.0.0              pyhd8ed1ab_0    conda-forge
croniter                  1.3.8              pyhd8ed1ab_0    conda-forge
cryptography              39.0.0          py310h34c0648_0    conda-forge
curl                      7.87.0               hdc1c0ab_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    0.29.32         py310hd8f1fbe_1    conda-forge
cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
dask                      2022.12.0          pyhd8ed1ab_0    conda-forge
dask-core                 2022.12.0          pyhd8ed1ab_0    conda-forge
dask-gateway              2022.10.0          pyh8af1aa0_0    conda-forge
dask-glm                  0.2.0                      py_1    conda-forge
dask-labextension         6.0.0              pyhd8ed1ab_0    conda-forge
dask-ml                   2022.5.27          pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
datashader                0.14.3             pyh1a96a4e_0    conda-forge
datashape                 0.5.4                      py_1    conda-forge
dav1d                     1.0.0                h166bdaf_1    conda-forge
debugpy                   1.6.4           py310hd8f1fbe_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
descartes                 1.1.0                      py_4    conda-forge
distributed               2022.12.0          pyhd8ed1ab_0    conda-forge
docker-py                 6.0.0              pyhd8ed1ab_0    conda-forge
docopt                    0.6.2                      py_1    conda-forge
docrep                    0.3.2              pyh44b312d_0    conda-forge
docutils                  0.15.2          py310hff52083_6    conda-forge
donfig                    0.7.0              pyhd8ed1ab_1    conda-forge
earthdata                 0.2.2              pyhd8ed1ab_0    conda-forge
eccodes                   2.27.1               h7f7619e_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
eofs                      1.4.0                      py_0    conda-forge
erddapy                   1.2.1              pyhd8ed1ab_0    conda-forge
esmf                      8.4.0           mpi_mpich_hc592774_102    conda-forge
esmpy                     8.4.0           mpi_mpich_py310h515c5ea_101    conda-forge
exceptiongroup            1.1.0              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                h27087fc_0    conda-forge
fastapi                   0.88.0             pyhd8ed1ab_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
fastjmd95                 0.2.1              pyh44b312d_0    conda-forge
fastprogress              1.0.3              pyhd8ed1ab_0    conda-forge
ffmpeg                    5.1.2           gpl_h8dda1f0_105    conda-forge
findlibs                  0.0.2              pyhd8ed1ab_0    conda-forge
fiona                     1.8.22          py310ha325b7b_5    conda-forge
flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
flox                      0.6.5              pyhd8ed1ab_0    conda-forge
folium                    0.14.0             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.1               hc2a2eb6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.38.0          py310h5764c6d_1    conda-forge
freeglut                  3.2.2                h9c3ff4c_1    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
freexl                    1.0.6                h166bdaf_1    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
frozenlist                1.3.3           py310h5764c6d_0    conda-forge
fsspec                    2022.11.0          pyhd8ed1ab_0    conda-forge
ftfy                      6.1.1              pyhd8ed1ab_0    conda-forge
future                    0.18.2             pyhd8ed1ab_6    conda-forge
gcc_impl_linux-64         12.2.0              hcc96c02_19    conda-forge
gcc_linux-64              12.2.0              h4798a0e_11    conda-forge
gcm_filters               0.3.0              pyhd8ed1ab_0    conda-forge
gcsfs                     2022.11.0          pyhd8ed1ab_0    conda-forge
gdal                      3.6.1           py310hc1b7723_2    conda-forge
gdk-pixbuf                2.42.10              h05c8ddd_0    conda-forge
geocube                   0.3.3              pyhd8ed1ab_0    conda-forge
geogif                    0.1.3              pyhd8ed1ab_0    conda-forge
geographiclib             1.52               pyhd8ed1ab_0    conda-forge
geopandas                 0.12.2             pyhd8ed1ab_0    conda-forge
geopandas-base            0.12.2             pyha770c72_0    conda-forge
geopy                     2.3.0              pyhd8ed1ab_0    conda-forge
geos                      3.11.1               h27087fc_0    conda-forge
geotiff                   1.7.1                h7157cca_5    conda-forge
geoviews-core             1.9.5              pyha770c72_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
gh                        2.21.1               ha8f183a_0    conda-forge
gh-scoped-creds           4.1                pyhd8ed1ab_0    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
git-lfs                   3.3.0                ha770c72_0    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.30             pyhd8ed1ab_0    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.7.8                hf3e180e_0    conda-forge
google-api-core           2.11.0             pyhd8ed1ab_0    conda-forge
google-auth               2.15.0             pyh1a96a4e_0    conda-forge
google-auth-oauthlib      0.8.0              pyhd8ed1ab_0    conda-forge
google-cloud-core         2.3.2              pyhd8ed1ab_0    conda-forge
google-cloud-storage      2.7.0              pyh1a96a4e_0    conda-forge
google-crc32c             1.1.2           py310he8fe98e_4    conda-forge
google-resumable-media    2.4.0              pyhd8ed1ab_0    conda-forge
googleapis-common-protos  1.57.0             pyhd8ed1ab_3    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
graphviz                  7.0.5                h2e5815a_0    conda-forge
greenlet                  2.0.1           py310hd8f1fbe_0    conda-forge
griffe                    0.21.0             pyhd8ed1ab_1    conda-forge
grpcio                    1.51.1          py310hc32fa93_0    conda-forge
gsw                       3.4.0           py310hde88566_3    conda-forge
gtk2                      2.24.33              h90689f9_2    conda-forge
gts                       0.7.6                h64030ff_2    conda-forge
h11                       0.14.0             pyhd8ed1ab_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
h3-py                     3.7.4           py310hd8f1fbe_1    conda-forge
h5netcdf                  1.1.0              pyhd8ed1ab_0    conda-forge
h5py                      3.7.0           nompi_py310h416281c_102    conda-forge
harfbuzz                  6.0.0                h8e241bc_0    conda-forge
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          mpi_mpich_h5d83325_1    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
holoviews                 1.15.3             pyhd8ed1ab_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
httpcore                  0.16.3             pyhd8ed1ab_0    conda-forge
httpx                     0.23.2             pyhd8ed1ab_0    conda-forge
hvplot                    0.8.2              pyhd8ed1ab_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
imagecodecs               2022.12.24      py310h17758e3_0    conda-forge
imageio                   2.23.0             pyhfa7a67d_0    conda-forge
importlib-metadata        6.0.0              pyha770c72_0    conda-forge
importlib_metadata        6.0.0                hd8ed1ab_0    conda-forge
importlib_resources       5.10.2             pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
intake                    0.6.6              pyhd8ed1ab_0    conda-forge
intake-esm                2022.9.18          pyhd8ed1ab_0    conda-forge
intake-geopandas          0.4.0              pyhd8ed1ab_0    conda-forge
intake-stac               0.4.0              pyhd8ed1ab_0    conda-forge
intake-xarray             0.6.1              pyhd8ed1ab_0    conda-forge
ipykernel                 6.19.4             pyh210e3f2_0    conda-forge
ipyleaflet                0.17.2             pyhd8ed1ab_0    conda-forge
ipyspin                   1.0.1              pyhd8ed1ab_0    conda-forge
ipython                   8.8.0              pyh41d4057_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipytree                   0.2.2              pyhd8ed1ab_0    conda-forge
ipyurl                    0.1.2              pyh3684270_1    conda-forge
ipywidgets                8.0.3              pyhd8ed1ab_0    conda-forge
isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
jasper                    2.0.33               ha77e612_0    conda-forge
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter-panel-proxy       0.1.0                      py_0    conda-forge
jupyter-resource-usage    0.6.4              pyhd8ed1ab_0    conda-forge
jupyter-server-mathjax    0.2.6              pyh5bfe37b_1    conda-forge
jupyter-server-proxy      3.2.2              pyhd8ed1ab_0    conda-forge
jupyter_client            7.4.8              pyhd8ed1ab_0    conda-forge
jupyter_core              5.1.2           py310hff52083_0    conda-forge
jupyter_server            1.23.4             pyhd8ed1ab_0    conda-forge
jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
jupyterhub-base           3.1.0              pyh2a2186d_0    conda-forge
jupyterhub-singleuser     3.1.0              pyh2a2186d_0    conda-forge
jupyterlab                3.5.1              pyhd8ed1ab_0    conda-forge
jupyterlab-git            0.41.0             pyhd8ed1ab_1    conda-forge
jupyterlab-s3-browser     0.12.0                   pypi_0    pypi
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.18.0             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.5              pyhd8ed1ab_0    conda-forge
jxrlib                    1.1                  h7f98852_2    conda-forge
kealib                    1.5.0                ha7026e8_0    conda-forge
kerchunk                  0.0.9              pyhd8ed1ab_0    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge

It’s not about compatibility with Python, it’s about having a compiler that provides the necessary math functions. The compiler is also not something you will find in your conda list.

If you don’t feel comfortable fiddling around with compilers and stuff, there yet one other solution. You can always pass the function pointer as an additional argument and then call the Cython function via this pointer:

from llvmlite import ir 
from numba import extending
from numba import types 
from numba import TypingError
from numba.extending import get_cython_function_address

double_t = ir.DoubleType()

kv_t = ir.FunctionType(double_t, [double_t, double_t])
kvp_t = kv_t.as_pointer()

def kv(typingctx, addr, v, z):
    if not isinstance(addr, types.Integer):
        raise TypingError("'addr' must be an integer")
    if not all(isinstance(x, types.Float) for x in (v, z)):
        raise TypingError("'v' and 'z' must be floats")

    def codegen(context, builder, sig, args):
        [addr, v, z] = args
        kv = builder.inttoptr(addr, kvp_t)
        v = builder.fpext(v, double_t)
        z = builder.fpext(z, double_t)
        return builder.call(kv, [v, z])
    return types.double(addr, v, z), codegen

You can use it like that:

import numpy as np 
import numba as nb 
import scipy.special

def test(addr, v, z):
    out = np.empty(v.size)
    for i in range(v.size):
        out[i] = kv(addr, v[i], z[i])
    return out 

v = np.random.rand(100_000)
z = np.random.rand(100_000)
addr = get_cython_function_address("scipy.special.cython_special", "__pyx_fuse_1kv")

assert np.allclose(test(addr, v, z), scipy.special.kv(v, z))
1 Like

Thanks alot for another suggestion @sschaer

def test(addr, v, z):
    return kv(addr, v, z)
v, z = 1.5, 10.5

test(addr, 1.5, 10.5) outputs 6.260706303843087. Your code clearly works.

I set up a local scheduler with distributed’s LocalCluster(). (This is still on the same machine, but gives the possibility to distribute tasks to local workers, (in this case, to 4 workers)):
client.run(test, addr, v, z) outputs CommClosedError.

from distributed import Client, LocalCluster
client.run(test, addr, v, z)


2023-11-22 12:34:28,781 - distributed.scheduler - ERROR - broadcast to tcp:// failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp:// remote=tcp://>: Stream is closed
2023-11-22 12:34:28,788 - distributed.scheduler - ERROR - broadcast to tcp:// failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp:// remote=tcp://>: Stream is closed
2023-11-22 12:34:28,792 - distributed.nanny - WARNING - Restarting worker
2023-11-22 12:34:28,794 - distributed.nanny - WARNING - Restarting worker
2023-11-22 12:34:28,809 - distributed.scheduler - ERROR - broadcast to tcp:// failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp:// remote=tcp://>: Stream is closed
2023-11-22 12:34:28,814 - distributed.nanny - WARNING - Restarting worker
2023-11-22 12:34:28,853 - distributed.scheduler - ERROR - broadcast to tcp:// failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp:// remote=tcp://>: Stream is closed
2023-11-22 12:34:28,859 - distributed.nanny - WARNING - Restarting worker
CommClosedError                           Traceback (most recent call last)
Cell In[40], line 1
----> 1 client.run(test, addr, v, z)

File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/client.py:2901, in Client.run(self, function, workers, wait, nanny, on_error, *args, **kwargs)
   2818 def run(
   2819     self,
   2820     function,
   2826     **kwargs,
   2827 ):
   2828     """
   2829     Run a function on all workers outside of task scheduling system
   2899     >>> c.run(print_state, wait=False)  # doctest: +SKIP
   2900     """
-> 2901     return self.sync(
   2902         self._run,
   2903         function,
   2904         *args,
   2905         workers=workers,
   2906         wait=wait,
   2907         nanny=nanny,
   2908         on_error=on_error,
   2909         **kwargs,
   2910     )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/utils.py:339, in SyncMethodMixin.sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    337     return future
    338 else:
--> 339     return sync(
    340         self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    341     )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/utils.py:406, in sync(loop, func, callback_timeout, *args, **kwargs)
    404 if error:
    405     typ, exc, tb = error
--> 406     raise exc.with_traceback(tb)
    407 else:
    408     return result

File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/utils.py:379, in sync.<locals>.f()
    377         future = asyncio.wait_for(future, callback_timeout)
    378     future = asyncio.ensure_future(future)
--> 379     result = yield future
    380 except Exception:
    381     error = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.10/site-packages/tornado/gen.py:769, in Runner.run(self)
    766 exc_info = None
    768 try:
--> 769     value = future.result()
    770 except Exception:
    771     exc_info = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.10/site-packages/distributed/client.py:2806, in Client._run(self, function, nanny, workers, wait, on_error, *args, **kwargs)
   2803     continue
   2805 if on_error == "raise":
-> 2806     raise exc
   2807 elif on_error == "return":
   2808     results[key] = exc

CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp:// remote=tcp://>: Stream is closed

So when running test(addr, v, z) in a distributed fashion, the scheduler logs that the workers-processes are shut down by some signal 11, resulting in a CommClosedError:

Fate of the 4 worker-processes
{'tcp://': (('INFO',
   "2023-11-22 12:33:05,043 - distributed.nanny - INFO -         Start Nanny at: 'tcp://'"),
   "2023-11-22 12:33:05,045 - distributed.nanny - INFO -         Start Nanny at: 'tcp://'"),
   "2023-11-22 12:33:05,050 - distributed.nanny - INFO -         Start Nanny at: 'tcp://'"),
   "2023-11-22 12:33:05,054 - distributed.nanny - INFO -         Start Nanny at: 'tcp://'"),
   '2023-11-22 12:33:36,404 - distributed.nanny - INFO - Worker process 1353 was killed by signal 11'),
   '2023-11-22 12:33:36,408 - distributed.nanny - INFO - Worker process 1360 was killed by signal 11'),
   '2023-11-22 12:33:36,412 - distributed.nanny - INFO - Worker process 1357 was killed by signal 11'),
   '2023-11-22 12:33:36,419 - distributed.nanny - WARNING - Restarting worker'),
   '2023-11-22 12:33:36,423 - distributed.nanny - WARNING - Restarting worker'),
   '2023-11-22 12:33:36,428 - distributed.nanny - WARNING - Restarting worker'),
   '2023-11-22 12:33:36,461 - distributed.nanny - INFO - Worker process 1362 was killed by signal 11'),
   '2023-11-22 12:33:36,465 - distributed.nanny - WARNING - Restarting worker'),

I would be interested in your thoughts about why this could be happening? Since this is moving away from numba-topics I should consider moving this post to some other forum though.

Moved the part about caching scipy.special.cython_special-functions, to a separate discussion.