Iterate over array and copy

Numba Community Support
1

I again tried to get numba working on my slowest code.

I shorted the code to some code should represent my current code problems, also it does not make any sense at all.

The numba compiler seams to have problems with the with the range function over a shape of an array and I am not allowed to copy an array. I am using numba 0.52.0 from conda-forge.

It seam I have to use a signatur or the compiler would not be able to infer the Dictionary.

@jit([(u2[::], DictType(u2, DictType(u2,u2))),
      (u2[::], u2[::])], nopython=False)
def _func(io_array, in_percentage_dict):
    io_main_class_percentage = io_array.copy()
    io_percentage_data = io_array.copy()

    for x in range(io_array.shape[1]):
        for y in range(io_array.shape[2]):
            io_percentage_data[0, x, y] = in_percentage_dict[1][5]

    return io_percentage_data, io_main_class_percentage

The errors are:

:1: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function “_func” failed type inference due to: Internal error at <numba.core.typeinfer.StaticGetItemConstraint object at 0x000001886AAEE4C0>.
tuple index out of range
During: typing of static-get-item at (7)
Enable logging at debug level for details.

File “”, line 7:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit([(u2[::], DictType(u2, DictType(u2,u2))),
:1: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function “_func” failed type inference due to: Cannot determine Numba type of <class ‘numba.core.dispatcher.LiftedLoop’>

File “”, line 7:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit([(u2[::], DictType(u2, DictType(u2,u2))),
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:151: NumbaWarning: Function “_func” was compiled in object mode without forceobj=True, but has lifted loops.

File “”, line 4:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaWarning(warn_msg,
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:161: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit Deprecation Notices — Numba 0.50.1 documentation

File “”, line 4:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaDeprecationWarning(msg,
:1: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function “_func” failed type inference due to: Internal error at <numba.core.typeinfer.StaticGetItemConstraint object at 0x000001886B095D00>.
tuple index out of range
During: typing of static-get-item at (7)
Enable logging at debug level for details.

File “”, line 7:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit([(u2[::], DictType(u2, DictType(u2,u2))),
:1: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function “_func” failed type inference due to: Cannot determine Numba type of <class ‘numba.core.dispatcher.LiftedLoop’>

File “”, line 7:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit([(u2[::], DictType(u2, DictType(u2,u2))),
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:151: NumbaWarning: Function “_func” was compiled in object mode without forceobj=True, but has lifted loops.

File “”, line 4:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaWarning(warn_msg,
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:161: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit Deprecation Notices — Numba 0.50.1 documentation

File “”, line 4:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaDeprecationWarning(msg,

2

hi @Meresmata ,

about the problem with the array: you declared arr to be 1-dimensional u2[::] but then attempted to find the size of the second dimension .shape[1]. The compiler is telling you that shape does not have the element you requested tuple index out of range.

I recommend that you drop the signature and let numba infer the inputs. It always makes a better or equal job than a human.

Hope this helps,
Luk

3

thanks at @luk-f-a,

I have changed the annotation from u2[::] to u2[:,:,:] and tried again without adding a signature.

As a final attempt I also used the inspect_types() method, but the report is rather long and I can’t really trace the errors.

  1. Attempt: Correcting the annotation results into the error, that no canditate matches the given input types?

Code:
@jit([(u2[:,:,:], DictType(u2, DictType(u2,u2))),
(u2[:,:,:], u2[:,:,:])], nopython=False)
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
io_percentage_data = io_array.copy()

    for x in range(io_array.shape[1]):
        for y in range(io_array.shape[2]):
            io_percentage_data[0, x, y] = in_percentage_dict[1][5]

    return io_percentage_data, io_main_class_percentage

Warnings/Errors:

:1: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function “_func” failed type inference due to: No implementation of function Function() found for signature:

setitem(array(uint16, 3d, C), Tuple(Literalint, int64, int64), array(uint16, 1d, A))

There are 16 candidate implementations:
- Of which 16 did not match due to:
Overload of function ‘setitem’: File: : Line N/A.
With argument(s): ‘(array(uint16, 3d, C), UniTuple(int64 x 3), array(uint16, 1d, A))’:
No match.

During: typing of setitem at (9)

File “”, line 9:
def _func(io_array, in_percentage_dict):

for y in range(io_array.shape[2]):
io_percentage_data[0, x, y] = in_percentage_dict[0][5]
^

@jit([(u2[:,:,:], DictType(u2, DictType(u2,u2))),
:1: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function “_func” failed type inference due to: Cannot determine Numba type of <class ‘numba.core.dispatcher.LiftedLoop’>

File “”, line 7:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit([(u2[:,:,:], DictType(u2, DictType(u2,u2))),
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:151: NumbaWarning: Function “_func” was compiled in object mode without forceobj=True, but has lifted loops.

File “”, line 4:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaWarning(warn_msg,
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:161: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit …

File “”, line 4:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaDeprecationWarning(msg,

  1. When not using the singnature I use the test_input:
    in_arr = np.ones((1, 5, 5))
    in_dict = {0: {5: 1}}

    @jit()
    def _func(io_array, in_percentage_dict):
    io_main_class_percentage = io_array.copy()
    io_percentage_data = io_array.copy()

     for x in range(io_array.shape[1]):
     for y in range(io_array.shape[2]):
         io_percentage_data[0, x, y] = in_percentage_dict[0][5]

 return io_percentage_data, io_main_class_percentage

Errors:

:1: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function “_func” failed type inference due to: non-precise type pyobject
During: typing of argument at (3)

File “”, line 3:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

@jit()
:1: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function “_func” failed type inference due to: Cannot determine Numba type of <class ‘numba.core.dispatcher.LiftedLoop’>

File “”, line 6:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit()
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:151: NumbaWarning: Function “_func” was compiled in object mode without forceobj=True, but has lifted loops.

File “”, line 3:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaWarning(warn_msg,
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:161: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit …

File “”, line 3:
def _func(io_array, in_percentage_dict):
io_main_class_percentage = io_array.copy()
^

warnings.warn(errors.NumbaDeprecationWarning(msg,
:1: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function “_func” failed type inference due to: non-precise type pyobject
During: typing of argument at (6)

File “”, line 6:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

@jit()
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:151: NumbaWarning: Function “_func” was compiled in object mode without forceobj=True.

File “”, line 6:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

warnings.warn(errors.NumbaWarning(warn_msg,
C:\Users\remote\anaconda3\envs\COP4EE-Current_SeH\lib\site-packages\numba\core\object_mode_passes.py:161: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit …

File “”, line 6:
def _func(io_array, in_percentage_dict):

for x in range(io_array.shape[1]):
^

warnings.warn(errors.NumbaDeprecationWarning(msg,

The code than perfectly runs in object mode.

4

Thanks for the reproducible example, I could run it and see the problem for myself.
I would recommend that you drop the signature and use njit instead of jit. The latter is trying too hard to accept your input instead of telling you what is wrong.
Once I did that I got this error message

- argument 1: Cannot determine Numba type of <class 'dict'>

The problem is the nested dictionary. You can pass a python dictionary but not a python dictionary with a dictionary inside. To do that you have to use a Typed Dict, Supported Python features — Numba 0.52.0-py3.7-linux-x86_64.egg documentation

from numba.typed import Dict
d = Dict()
d1 = Dict()
d1[5]=1
d[0]=d1
_func(in_arr, d)

(array([[[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]]]), array([[[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]]]))
5

@luk-f-a Thanks. That looks awesome.

I didn’t know. That the dict could not be converted. :frowning:

Is there a a function to convert a dict with inner dict to a typedDict?

From the Manual I have seen, that typed Dicts can have Intitial values.

But the example ownly shows “flat” Dicts and only within a jitted function. But not something like:

from numba.typed import Dict as nbDict
d2 = nbDict({5:1})

Doesn’t work. Did I do a mistake, or is this behaviour intended?

When using Python dicts I always use the update Method, which numba does not support.
When using dict[key] = value, I often override, which doesn’t happen with the Method.

With kind regards!

6

I think the typed Dict does not yet support construction from a python dictionary. It’s a case of a missing feature, rather than a mistake or intended behaviour. It just needs someone to sit down and implement it.

It’s the same case of update, it has not been implemented yet.

cheers,
Luk