Namedtuple seems to cause a TypeError after a while

I’m seeing this error on both windows and linux with python 3.9, 3.10 on numba==0.60.0 as well as earlier versions.
The jitted function calls work fine for a while, but eventually end up with this error:

Traceback (most recent call last):
File “D:\work\git1\ims_core\playground\numba_demos\signature_error.py”, line 58, in
raise e
File “D:\work\git1\ims_core\playground\numba_demos\signature_error.py”, line 54, in
tester(arg1, arg2)
File “D:\work\git1\mims\venv\lib\site-packages\numba\core\dispatcher.py”, line 658, in _explain_matching_error
raise TypeError(msg)
TypeError: No matching definition for argument type(s) RequiresType(int64 x 1), ProducesType(int64 x 1)

There are ways to work around the error… shorter field names, different field names for the namedtuple arguments. I haven’t found anything that feels like a reliable story of how to reason about this error. Minimal reproducer below.

import gc
import sys
from collections import namedtuple
from time import sleep

import numba
from numba.core import types
from numba.core.environment import Environment
from numba.core.target_extension import jit_registry

print(f"Python {sys.version.split(' ')[0]} is at -->{sys.executable}")
print(f'{numba.__version__=}')

def get_env():
    env = Environment
    # print(dir(env))
    print('env._memo key count =', len([x for x in env._memo.keys()]), flush=True)
    for key in sorted(env._memo.keys()):
        print(key, flush=True)
    print(f'{len(jit_registry)=}')


fieldname = 'this_is_a_very_long_field_name'
RequiresType = namedtuple(
    'RequiresType',
    fieldname,
    module=__name__
)

ProducesType = namedtuple(
    'ProducesType',
    fieldname,
    module=__name__
)


req_type = numba.typeof(RequiresType(1))
prod_type = numba.typeof(ProducesType(1))
func_type = types.void(req_type, prod_type)


@numba.njit(func_type)
def tester(arg1, arg2):
    pass


arg1 = RequiresType(1)
arg2 = ProducesType(2)

for i in range(200):
    print('loop', i, flush=True)
    try:
        get_env()
        tester(arg1, arg2)
    except TypeError as e:
        print('got the error', flush=True)
        get_env()
        raise e
    sleep(1)
    gc.collect()

Some observations: looking at fingerprints of the arguments arg1 and arg2 (via numba._dispatcher.compute_fingerprint(arg1)), one can see that for sufficiently long field names the fingerprint bytes string becomes corrupted.

For short fields names fingerprints are something like RequiresType(field_name). For longer names, they look like garbage bytes.

Digging deeper into the source, I noticed that for long fingerprints (> 40 bytes) the buffer into which the fingerprint is written into gets expanded but the old stuff is lost.

Re-building numba with the buffer re-allocated and copied seems to fix the issue.

Thanks, that’s a great find! @esc or @gmarkall can you confirm?

 💣 zsh» python issue_disc_2744.py
Python 3.12.4 is at -->/Users/esc/miniconda3-arm64/envs/numba_3.12/bin/python
numba.__version__='0.61.0dev0+236.g301ba23116.dirty'
loop 0
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 1
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 2
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 3
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 4
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 5
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 6
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 7
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 8
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 9
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 10
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 11
env._memo key count = 1
_ZN08NumbaEnv8__main__6testerB2v1B38c8tJTIeFIjxB2IKSgI4CrvQClQZ6FczSBAA_3dE31RequiresType_28int64_20x_201_2931ProducesType_28int64_20x_201_29
len(jit_registry)=1
loop 12
env._memo key count = 1

and patch:

diff --git i/numba/_typeof.cpp w/numba/_typeof.cpp
index 48916d4913..871d296fbd 100644
--- i/numba/_typeof.cpp
+++ w/numba/_typeof.cpp
@@ -102,8 +102,10 @@ string_writer_ensure(string_writer_t *w, size_t bytes)
     newsize = (w->allocated << 2) + 1;
     if (newsize < bytes)
         newsize = bytes;
-    if (w->buf == w->static_buf)
+    if (w->buf == w->static_buf){
         w->buf = (char *) malloc(newsize);
+        strncpy(w->buf, w->static_buf, 40);
+    }
     else
         w->buf = (char *) realloc(w->buf, newsize);
     if (w->buf) {

Thanks for the quick work! Can it be back ported into existing versions? Or will only appear in a future version?

We don’t do backports, future version only.

Ref: `memcpy` static buffer content into newly allocated buffer by guilhermeleobas · Pull Request #9119 · numba/numba · GitHub

Thanks for the report and reproducer, @nelson2005. The other one that I had was a bit big and sort of non-deterministic.

This bug has been around for quite a bit in Numba. Hopefully the fix can land soon.

Thanks for finding this @nelson2005 and for debugging this @milton! I’ve added the PR #9119 (from above) to the 0.61 milestone, i.e. the next release. Based on the report in this thread, locally, I’ve got a reproducer that is consistent, this can be used as a test for the PR.

Thanks for planning the fix in the next numba release @stuartarchibald! However, if I try your updated test, it passes for me. On the other hand, I got hit every time when running the code from @nelson2005.

import sys
import numba
from collections import namedtuple
from numba import jit, types, typeof

def test_memcpy_typeof_buffer():
    # https://github.com/numba/numba/issues/9097
    # bug is fixed if the code below compiles

    longname = 1024 * "a"

    AType = namedtuple("AType", longname)
    BType = namedtuple("BType", longname)

    args = (AType(1), BType(1))

    @jit(types.void(*(typeof(x) for x in args)))
    def foo(arg1, arg2):
        pass

    foo(*args)

    print("It ran ok!")

print(f"Python {sys.version.split(' ')[0]} is at -->{sys.executable}")
print(f'{numba.__version__=}')
print(f'{numba.__path__=}')

test_memcpy_typeof_buffer()

Here is the output:

Python 3.9.13 is at -->/ofs/envs/prod/py-20240816/bin/python
numba.__version__='0.56.3'
numba.__path__=['/ofs/envs/prod/py-20240816/lib/python3.9/site-packages/numba']
It ran ok!

Can we add an explicit test that compute_fingerprint of a long namedtuple returns the bytes string we expect?

Also, while we’re on the topic :slight_smile: these long strings are added char-by-char here and here when string_writer_put_string is available and can directly consume the null-terminated char * produced by PyBytes_AsStringAndSize

It looks to me like writing the strings in one shot instead of char-by-char should be more efficient. Is there a particular reason it is the way it is now?