Incompatible functions with "nested arrays"

Numba Support: What is this error message?
1

I have encountered a regression when upgrading form version 0.53.1 to version 0.54.0
The following code is a reduction of the problem :

import numpy as np
import numba as nb

DTYPE = np.dtype([(‘X1’, np.float64, 4)])

@nb.njit
def fun():
structured_array = np.zeros(5, dtype=DTYPE)
X1 = np.copy(structured_array[0][‘X1’])
print(X1)
return

fun()

The problem here is related to the structured array in numpy which I use a lot (maybe not in the right way)
When executing this code with V0.54.0 I obtain the following error:

No implementation of function Function(<function copy at 0x7fb32f4873a0>) found for signature:
’ >>> copy(nestedarray(float64, (4,)))

I had no problem with this code in V0.53.1
I also noticed that other numpy functions seem to struggle when dealing with a “nestedarray” instead of a regular array

Is this a bug or an expected behavior, and if so, how can I solve it ?
Thank you

2

This may be this issue reported earlier: BUG cannot set items in structured array data types in 0.54.0 · Issue #7355 · numba/numba · GitHub

3

Thanks for your reply, yes this bug report seems related.

4

@Anthony-C31 thanks for reporting this, as @gmarkall noted it does look like a variation on the above mentioned issue.

You also note that there are other functions that now “seem to struggle” with nestedarrays, to ensure we get a good fix for this regression, is there any chance you could please post some examples of these functions that are having issues either here on on the github issue? Many thanks for your help.

5

Yes I have other examples of the regression:

The following code produces the same type of error:

@nb.njit
def fun():
structured_array = np.zeros(5, dtype=DTYPE)
X1 = structured_array[0][‘X1’].flatten() #doesn’t work in 0.54.0
X1 = np.ascontiguousarray(structured_array[0][‘X1’]) #doesn’t work in 0.54.0
print(X1)
return
fun()

This code compiles in 0.54.0 but the printed result is not ok for the last line… (ok in 0.53.1)

@nb.njit
def fun():
structured_array = np.zeros(5, dtype=DTYPE)
print(structured_array) #OK
print(structured_array[0][‘X1’]) #OK
print(np.min(structured_array[0][‘X1’])) #OK
print(np.max(structured_array[0][‘X1’])) #NOK: -9.868309920863368e+148
return

6

@Anthony-C31 thanks very much for those, much appreciated. I’ve transferred them to the issue BUG cannot set items in structured array data types in 0.54.0 · Issue #7355 · numba/numba · GitHub.

7

This seems to be the same kind of issue but it’s reading rather than writing structured array data types. Should I enter a new bug?

numbafail.py:

import sys
import numpy as np
import numba

@numba.jit(nopython=True)
def test1(result, content, ncols):
    int16LE = np.dtype('<h')
    ndata = 0
    for item in content:
        j = ndata
        k = j+ncols
        result[j:k] = item['data'][:2*ncols].view(int16LE)
        ndata += ncols

content = np.zeros(5,dtype=[('data', 'u1', (255,))])
notable_content_base = np.array([10,20,30,40,50,60,70,80])
ncols = len(notable_content_base)
nrows = content.shape[0]
for k in range(nrows):
    content[k]['data'][:ncols*2:2] = notable_content_base+k
    content[k]['data'][1:ncols*2:2] = 100
result = np.zeros(nrows*ncols, dtype=np.dtype('<h'))
print('python %s\nnumpy %s\nnumba %s' % (sys.version, np.__version__, numba.__version__))
test1(result, content, ncols)
print(result)

Numba 0.53.1 works:

>python numbafail.py
python 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
numpy 1.19.2
numba 0.53.1
[25610 25620 25630 25640 25650 25660 25670 25680 25611 25621 25631 25641
 25651 25661 25671 25681 25612 25622 25632 25642 25652 25662 25672 25682
 25613 25623 25633 25643 25653 25663 25673 25683 25614 25624 25634 25644
 25654 25664 25674 25684]

Numba 0.54 fails:

>python numbafail.py
python 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
numpy 1.19.2
numba 0.54.0
Traceback (most recent call last):
  File "numbafail.py", line 24, in <module>
    test1(result, content, ncols)
  File "c:\app\python\anaconda\3\envs\py3-2\lib\site-packages\numba\core\dispatcher.py", line 482, in _compile_for_args
    error_rewrite(e, 'typing')
  File "c:\app\python\anaconda\3\envs\py3-2\lib\site-packages\numba\core\dispatcher.py", line 423, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
←[1m←[1mNo implementation of function Function(<built-in function getitem>) found for signature:

 >>> getitem(nestedarray(uint8, (255,)), slice<a:b>)

There are 22 candidate implementations:
←[1m   - Of which 20 did not match due to:
   Overload of function 'getitem': File: <numerous>: Line N/A.
     With argument(s): '(nestedarray(uint8, (255,)), slice<a:b>)':←[0m
←[1m    No match.←[0m
←[1m   - Of which 2 did not match due to:
   Overload in function 'GetItemBuffer.generic': File: numba\core\typing\arraydecl.py: Line 162.
     With argument(s): '(nestedarray(uint8, (255,)), slice<a:b>)':←[0m
←[1m    Rejected as the implementation raised a specific error:
      TypeError: __init__() got an unexpected keyword argument 'ndim'←[0m
  raised from c:\app\python\anaconda\3\envs\py3-2\lib\site-packages\numba\core\types\abstract.py:66
←[0m
←[0m←[1mDuring: typing of intrinsic-call at numbafail.py (12)←[0m
←[1m
File "numbafail.py", line 12:←[0m
←[1mdef test1(result, content, ncols):
    <source elided>
        k = j+ncols
←[1m        result[j:k] = item['data'][:2*ncols].view(int16LE)
←[0m        ←[1m^←[0m←[0m