Issue with a where function 3D numpy array

Hi guys !
Thanks to the forum I managed to work on optimizing my python code which is now pretty fast in regards to what it was before.
My only concern is that now I’m trying to optimize another part of it but I’m being a bit lost.
I don’t understand if it is a problem due to numba not implementing a certain part of the where function or something that has to do with my code.

Thank you very much if you can help me sort out what’s the issue and how I can get over it !
Being still quite new to python/numba, I tried to look for infos on the internet but didn’t find anything that could be helpful…

Best regards,
Damien

What I am trying to do:

    a, b, c= where((voxelNormalized >= threshold) & (mask))

The parameters (theoretically):

voxelNormalized: NumpyArray[0:82][0:512][0:512] # Array that contains an array that contains an array itself (float values).
threshold: float # fixed value
mask: NumpyArray[0:82][0:512][0:512] # boolean values

The error:

numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<intrinsic length_of_iterator>) found for signature:
 
 >>> length_of_iterator(zip(iter(array(int64, 1d, C)), iter(array(int64, 1d, C)), iter(array(int64, 1d, C))))
 
There are 2 candidate implementations:
   - Of which 2 did not match due to:
   Intrinsic in function 'length_of_iterator': File: numba\cpython\rangeobj.py: Line 181.
     With argument(s): '(zip(iter(array(int64, 1d, C)), iter(array(int64, 1d, C)), iter(array(int64, 1d, C))))':
    Rejected as the implementation raised a specific error:
      TypingError: Unsupported iterator found in array comprehension, try preallocating the array and filling manually.
  raised from C:\Python310\lib\site-packages\numba\cpython\rangeobj.py:235

During: resolving callee type: Function(<intrinsic length_of_iterator>)
During: typing of call at my_numbaUtils.py (41)


File "services\numbaUtils.py", line 41:
def getSpotsNucleus(nucleusId, threshold, maskNucleus, voxelNormalized, frameId, penality):
    spotZ, spotX, spotY = where((voxelNormalized >= threshold) & (maskNucleus))
    ^

During: resolving callee type: type(CPUDispatcher(<function getSpotsNucleus at 0x0000029DBD4CCC10>))
During: typing of call at my_numbaUtils.py (69)

During: resolving callee type: type(CPUDispatcher(<function getSpotsNucleus at 0x0000029DBD4CCC10>))
During: typing of call at my_numbaUtils.py (69)

During: resolving callee type: type(CPUDispatcher(<function getSpotsNucleus at 0x0000029DBD4CCC10>))
During: typing of call at my_numbaUtils.py (69)

Dear @kuroiraven

I was not able to reproduce your error with the following code.

import numba as nb 
import numpy as np
np.random.seed(42)

threshold = 0.5
mask = np.random.randint(0, 2, (82, 512, 512), dtype=np.bool_)
voxelNormalized = np.random.rand(*mask.shape)

@nb.njit
def foo(voxelNormalized, threshold, mask):
    return np.where((voxelNormalized >= threshold) & (mask))

foo(voxelNormalized, threshold, mask)

Can you share a minimal executable examples?

1 Like

Dear @sschaer ,

Thanks a lot for your quick reply !
I tried to create a minimal reproducible example but couldn’t do it quite yet.

But as you showed me right now, it seems that the problem is on my side.
I didn’t even think about testing it like this before… Sorry.
One of my parameter must have been corrupted during the refactor to use numba.

I’ll give it an eye and post a minimal reproductible example if I manage to isolate the problematic parameter.

Thanks again and best regards,
Damien

@sschaer

Finally got why it was not working properly.
Looks like, when I was trying to work on this code, I encountered another numba limitation.
You cannot do a matrixA[mask] when you’re having 3D array.

I tried to workaround this problem by creating my own function but seems like I did it wrong and the result was not the one I expected there.
Therefore, that’s why my call was not functional at all.

I know I should create another topic for this question but if you have a workaround for the matrixA[mask] thingy, I would be very glad.

Thank you again for your time and help.