I’m seeing this error on both windows and linux with python 3.9, 3.10 on numba==0.60.0 as well as earlier versions.
The jitted function calls work fine for a while, but eventually end up with this error:
Traceback (most recent call last):
File “D:\work\git1\ims_core\playground\numba_demos\signature_error.py”, line 58, in
raise e
File “D:\work\git1\ims_core\playground\numba_demos\signature_error.py”, line 54, in
tester(arg1, arg2)
File “D:\work\git1\mims\venv\lib\site-packages\numba\core\dispatcher.py”, line 658, in _explain_matching_error
raise TypeError(msg)
TypeError: No matching definition for argument type(s) RequiresType(int64 x 1), ProducesType(int64 x 1)
There are ways to work around the error… shorter field names, different field names for the namedtuple arguments. I haven’t found anything that feels like a reliable story of how to reason about this error. Minimal reproducer below.
import gc
import sys
from collections import namedtuple
from time import sleep
import numba
from numba.core import types
from numba.core.environment import Environment
from numba.core.target_extension import jit_registry
print(f"Python {sys.version.split(' ')[0]} is at -->{sys.executable}")
print(f'{numba.__version__=}')
def get_env():
env = Environment
# print(dir(env))
print('env._memo key count =', len([x for x in env._memo.keys()]), flush=True)
for key in sorted(env._memo.keys()):
print(key, flush=True)
print(f'{len(jit_registry)=}')
fieldname = 'this_is_a_very_long_field_name'
RequiresType = namedtuple(
'RequiresType',
fieldname,
module=__name__
)
ProducesType = namedtuple(
'ProducesType',
fieldname,
module=__name__
)
req_type = numba.typeof(RequiresType(1))
prod_type = numba.typeof(ProducesType(1))
func_type = types.void(req_type, prod_type)
@numba.njit(func_type)
def tester(arg1, arg2):
pass
arg1 = RequiresType(1)
arg2 = ProducesType(2)
for i in range(200):
print('loop', i, flush=True)
try:
get_env()
tester(arg1, arg2)
except TypeError as e:
print('got the error', flush=True)
get_env()
raise e
sleep(1)
gc.collect()
Some observations: looking at fingerprints of the arguments arg1 and arg2 (via numba._dispatcher.compute_fingerprint(arg1)), one can see that for sufficiently long field names the fingerprint bytes string becomes corrupted.
For short fields names fingerprints are something like RequiresType(field_name). For longer names, they look like garbage bytes.
Digging deeper into the source, I noticed that for long fingerprints (> 40 bytes) the buffer into which the fingerprint is written into gets expanded but the old stuff is lost.
Re-building numba with the buffer re-allocated and copied seems to fix the issue.
Thanks for finding this @nelson2005 and for debugging this @milton! I’ve added the PR #9119 (from above) to the 0.61 milestone, i.e. the next release. Based on the report in this thread, locally, I’ve got a reproducer that is consistent, this can be used as a test for the PR.
Thanks for planning the fix in the next numba release @stuartarchibald! However, if I try your updated test, it passes for me. On the other hand, I got hit every time when running the code from @nelson2005.
import sys
import numba
from collections import namedtuple
from numba import jit, types, typeof
def test_memcpy_typeof_buffer():
# https://github.com/numba/numba/issues/9097
# bug is fixed if the code below compiles
longname = 1024 * "a"
AType = namedtuple("AType", longname)
BType = namedtuple("BType", longname)
args = (AType(1), BType(1))
@jit(types.void(*(typeof(x) for x in args)))
def foo(arg1, arg2):
pass
foo(*args)
print("It ran ok!")
print(f"Python {sys.version.split(' ')[0]} is at -->{sys.executable}")
print(f'{numba.__version__=}')
print(f'{numba.__path__=}')
test_memcpy_typeof_buffer()
Here is the output:
Python 3.9.13 is at -->/ofs/envs/prod/py-20240816/bin/python
numba.__version__='0.56.3'
numba.__path__=['/ofs/envs/prod/py-20240816/lib/python3.9/site-packages/numba']
It ran ok!
It looks to me like writing the strings in one shot instead of char-by-char should be more efficient. Is there a particular reason it is the way it is now?
these long strings are added char-by-char