Numba failure with np.zeros in a static method of a class

Numba Support: What is this error message?
1

Hi everyone,

I am new to Numba and I am trying to speed up a static method in a class using the Numba library.
The method is implemented in this way:

@staticmethod
@numba.njit(cache=True, nogil=True)

And the function inside the method which is causing the Numba TypingError is:

rr_s = np.zeros(len(active_channels), dtype=[('data',np.int,records_required*rr_samples),('time',np.int), ('search',('U', 5)),('channel',np.int), ('event',np.int),('dt',np.int),('baseline',np.float)])

The output of the error says:

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<built-in function zeros>) found for signature:
>>> zeros(int64, dtype=LiteralList((Tuple(Literal[str](data), Function(<class 'int'>), int64), Tuple(Literal[str](time), Function(<class 'int'>)), Tuple(Literal[str](search), Tuple(unicode_type, int64)), Tuple(Literal[str](channel), Function(<class 'int'>)), Tuple(Literal[str](event), Function(<class 'int'>)), Tuple(Literal[str](dt), Function(<class 'int'>)), Tuple(Literal[str](baseline), Function(<class 'float'>)))))

There are 2 candidate implementations:
  - Of which 1 did not match due to:
  Overload in function 'zeros': File: numba/core/typing/npydecl.py: Line 507.
    With argument(s): '(int64, dtype=Poison<LiteralList((Tuple(Literal[str](data), Function(<class 'int'>), int64), Tuple(Literal[str](time), Function(<class 'int'>)), Tuple(Literal[str](search), Tuple(unicode_type, int64)), Tuple(Literal[str](channel), Function(<class 'int'>)), Tuple(Literal[str](event), Function(<class 'int'>)), Tuple(Literal[str](dt), Function(<class 'int'>)), Tuple(Literal[str](baseline), Function(<class 'float'>))))>)':
   Rejected as the implementation raised a specific error:
     TypingError: Poison type used in arguments; got Poison<LiteralList((Tuple(Literal[str](data), Function(<class 'int'>), int64), Tuple(Literal[str](time), Function(<class 'int'>)), Tuple(Literal[str](search), Tuple(unicode_type, int64)), Tuple(Literal[str](channel), Function(<class 'int'>)), Tuple(Literal[str](event), Function(<class 'int'>)), Tuple(Literal[str](dt), Function(<class 'int'>)), Tuple(Literal[str](baseline), Function(<class 'float'>))))>
  raised from /dali/lgrandi/strax/miniconda3/envs/strax/lib/python3.8/site-packages/numba/core/types/functions.py:235
  - Of which 1 did not match due to:
  Overload of function 'zeros': File: numba/core/typing/npydecl.py: Line 507.
    With argument(s): '(int64, dtype=LiteralList((Tuple(Literal[str](data), Function(<class 'int'>), int64), Tuple(Literal[str](time), Function(<class 'int'>)), Tuple(Literal[str](search), Tuple(unicode_type, int64)), Tuple(Literal[str](channel), Function(<class 'int'>)), Tuple(Literal[str](event), Function(<class 'int'>)), Tuple(Literal[str](dt), Function(<class 'int'>)), Tuple(Literal[str](baseline), Function(<class 'float'>)))))':
   No match.
During: resolving callee type: Function(<built-in function zeros>)
During: typing of call at /home/sdipede/Noise_Analysis/concatinate_rr.py (157)
File "concatinate_rr.py", line 157:
    def concatinate(rr_old, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels):
        <source elided>
        
            rr_s = np.zeros(len(active_channels), dtype=[('data',np.int,records_required*rr_samples),('time',np.int),
            ^

I think I am not writing the dtype in the correct way to let Numba interpret the zeros numpy function as one of the candidates Numba provides.

Maybe my question is more basic: which kind of np.zeros function does Numba provide?

Thank you very much to anyone can help me understand and solve this issue!

2

hi @SerenaDiPede, inside a jitted function, np.zeros does not accept a list as dtype input. You need to create a dtype object explicitly outside the function, and then use it inside, like in this example:

import numpy as np
from numba import njit

rec_dtype = np.dtype([('a', 'f8')])

@njit
def foo():
    return np.zeros(1, rec_dtype)
3

Hi @luk-f-a,

thank you very much for your help and explication! Now it accepts the argument :slight_smile:

But I now have other two Numpy functions (np.median and np.concatenate). For clarity, I will report here my entire numba function:

 @staticmethod
@numba.njit(cache=True, nogil=True)
def concatinate(rr_old, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):

    # Concatinate waveforms from raw_records: this is the core of the function 
    # Create an array of evenly spaced values in a given interval
    events = np.arange(0,num_events_available,1) # we use the num available because it is the maximum
            
    rr = [] #list of events, may need to convert to numpy array for faster process
    #rr_wf = []
    # Let's loop now on the events we want to process
    for i in range(num_events_processed):
        event = events[i]
      
        if event == 0:
            rr_start = 0
            rr_stop = records_required - 1
        else:
            rr_start = np.int(event*rr_per_wf + events[i-1])
            rr_stop = np.int((rr_start + records_required))
        #print(f'event {np.int(event)}: starting on record {rr_start} and stopping on record {rr_stop}')
      
        rr_s = np.zeros(len(active_channels), rr_dtype)
         
        for ix,ch in enumerate(active_channels):
            single_channel = rr_old[rr_old['channel'] == ch] 
            if event == 0:
                baseline = np.median(single_channel[rr_start:records_required]['data']) 
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - single_channel[rr_start:records_required]['data'])
            else:
                baseline = np.median(single_channel[rr_start:rr_stop]['data'])
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - single_channel[rr_start:rr_stop]['data'])
            
            rr_s[ix]['baseline'] = np.median(baseline)
            rr_s[ix]['time'] = single_channel[rr_start]['time']
            rr_s[ix]['data'] = long_wf
            rr_s[ix]['event'] = event 
            rr_s[ix]['channel'] = ch
            rr_s[ix]['dt'] = 10   
            
            if ch <= 493:
                rr_s[ix]['search'] = "dm"
            else:
                rr_s[ix]['search'] = "0vbb"
        rr.append(rr_s)
    return rr

And I received this error message:

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Internal error at <numba.core.typeinfer.StaticGetItemConstraint object at 0x7eff956fa3d0>.
Buffer dtype cannot be buffer, have dtype: nestedarray(int16, (500,))
During: typing of static-get-item at /home/sdipede/Noise_Analysis/concatinate_rr.py (171)
Enable logging at debug level for details. 
File "concatinate_rr.py", line 171:
def concatinate(rr_old, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):
    <source elided>
            if event == 0:
                baseline = np.median(single_channel[rr_start:records_required]['data'])

Do you know what does it mean that a “buffer dtype cannot be buffer”?

Thank you very much again.

Serena

4

hi, that line contains three operations: np.median and two getitem operations (those with []).

The line During: typing of static-get-item tells you that the problem happened with the getitem, not with median. In particular, the name static only applies to getitem with a literal key. So the problem is most likely when retrieving the 'data' element out of single_channel. It’s complaining about the nestedarray.

Nested arrays have some gaps in functionality at the moment. Not everything that you can do with a normal array you can also do with a nested array.

If I were you I would try to confirm the error by splitting the line like this

temp1 = single_channel[rr_start:rr_stop]
temp2 = temp1['data']
baseline = np.median(temp2)

That will then confirm which operation is the problem. If I’m right, the problem lies in temp1['data']. In that case, could you provide a smaller, self-contained example with only this problem? By self-contained I mean something that someone else could execute on their computer. You would need to provide an example of what rr_old contains, and then try to extract the data element of that, in order to get the same error message.

Cheers,
Luk

5

Hi,

thank you again, indeed the error is where you mentioned.

I saved a tiny part of the data so that the below code is reproducible for another person as well. In a Jupyter notebook, I executed this code:

import numpy as np
import numba
import math

# Channels
active_channel_small = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=np.int16)

# Data
rec_type = np.dtype([
    (('Start time', 'time'), '<i8'),
    (('Length of interval ', 'length'), '<i4'),
    (('Width ', 'dt'), '<i2'),
    (('Channel', 'channel'), '<i2'),
    (('Length of pulse ', 'pulse_length'), '<i4'),
    (('Fragment', 'record_i'), '<i2'),
    (('Baseline', 'baseline'), '<i2'),
    (('Waveform data', 'data'), '<i2', (500,))
])

rr_old_sample = np.array(
    [
      (1596395008015038110, 500, 10, 0, 98304, 0, 0, [13812, 13813, 13813, 13814, 13812, 13815, 13809, 13815, 13812, 13811, 13810, 13815, 13812, 13815, 13812, 13812, 13812, 13811, 13808, 13811, 13812, 13813, 13813, 13813, 13811, 13813, 13812, 13814, 13811, 13811, 13809, 13811, 13810, 13812, 13809, 13815, 13813, 13813, 13810, 13810, 13811, 13814, 13812, 13812, 13809, 13811, 13809, 13812, 13811, 13812, 13810, 13814, 13813, 13815, 13810, 13814, 13813, 13811, 13813, 13813, 13811, 13812, 13812, 13812, 13812, 13813, 13814, 13813, 13811, 13811, 13809, 13811, 13808, 13811, 13810, 13813, 13809, 13811, 13809, 13812, 13813, 13817, 13811, 13815, 13812, 13815, 13809, 13813, 13810, 13813, 13811, 13811, 13810, 13811, 13810, 13812, 13810, 13813, 13810, 13813, 13811, 13812, 13811, 13815, 13814, 13814, 13810, 13814, 13814, 13816, 13811, 13816, 13813, 13814, 13816, 13816, 13813, 13815, 13813, 13809, 13810, 13812, 13813, 13814, 13813, 13811, 13809, 13812, 13814, 13811, 13811, 13816, 13811, 13813, 13809, 13808, 13809, 13811, 13810, 13811, 13809, 13809, 13808, 13814, 13810, 13812, 13813, 13813, 13813, 13814, 13812, 13816, 13812, 13816, 13811, 13814, 13812, 13813, 13815, 13814, 13812, 13814, 13812, 13814, 13813, 13817, 13814, 13813, 13811, 13812, 13810, 13813, 13809, 13815, 13811, 13813, 13811, 13812, 13810, 13811, 13812, 13811, 13809, 13813, 13809, 13811, 13810, 13811, 13811, 13812, 13811, 13815, 13809, 13815, 13814, 13811, 13811, 13814, 13811, 13811, 13813, 13814, 13810, 13813, 13809, 13814, 13810, 13810, 13811, 13812, 13809, 13813, 13809, 13812, 13811, 13812, 13812, 13810, 13809, 13811, 13813, 13814, 13813, 13814, 13812, 13814, 13810, 13813, 13813, 13813, 13813, 13814, 13810, 13813, 13811, 13814, 13811, 13814, 13811, 13815, 13813, 13813, 13811, 13813, 13812, 13815, 13811, 13813, 13812, 13811, 13809, 13812, 13811, 13812, 13811, 13815, 13809, 13813, 13811, 13811, 13810, 13814, 13809, 13812, 13809, 13811, 13810, 13812, 13809, 13813, 13812, 13810, 13810, 13812, 13812, 13812, 13812, 13814, 13811, 13814, 13813, 13812, 13810, 13813, 13812, 13814, 13812, 13812, 13813, 13816, 13811, 13812, 13813, 13817, 13811, 13812, 13812, 13813, 13813, 13816, 13814, 13814, 13813, 13814, 13811, 13813, 13812, 13814, 13813, 13814, 13812, 13813, 13812, 13812, 13812, 13811, 13809, 13813, 13811, 13812, 13810, 13810, 13810, 13813, 13813, 13812, 13810, 13810, 13814, 13812, 13810, 13814, 13811, 13813, 13811, 13812, 13809, 13813, 13809, 13812, 13812, 13811, 13811, 13814, 13810, 13812, 13811, 13814, 13813, 13816, 13811, 13812, 13808, 13814, 13810, 13815, 13810, 13812, 13809, 13813, 13811, 13812, 13810, 13814, 13810, 13812, 13812, 13812, 13809, 13812, 13812, 13813, 13812, 13812, 13810, 13811, 13809, 13813, 13812, 13813, 13811, 13815, 13811, 13813, 13811, 13815, 13811, 13811, 13810, 13813, 13810, 13811, 13810, 13814, 13809, 13811, 13810, 13814, 13810, 13813, 13813, 13812, 13808, 13813, 13811, 13812, 13811, 13812, 13813, 13815, 13812, 13813, 13812, 13814, 13809, 13813, 13807, 13810, 13808, 13811, 13812, 13813, 13811, 13813, 13812, 13814, 13809, 13812, 13814, 13814, 13810, 13813, 13808, 13812, 13810, 13813, 13814, 13813, 13811, 13812, 13810, 13812, 13812, 13814, 13812, 13814, 13810, 13810, 13810, 13812, 13810, 13812, 13810, 13814, 13809, 13814, 13813, 13813, 13811, 13811, 13810, 13811, 13811, 13814, 13813, 13813, 13810, 13813, 13813, 13814, 13811, 13811, 13811, 13813, 13809, 13810, 13812, 13812, 13811, 13811, 13810, 13814, 13812, 13813, 13813, 13811, 13809, 13812, 13810, 13814, 13814, 13815, 13810, 13813, 13811, 13813, 13811, 13811, 13812, 13812]),
      (1596395008015038110, 500, 10, 1, 98304, 0, 0, [13621, 13622, 13621, 13621, 13625, 13620, 13625, 13624, 13623, 13624, 13624, 13626, 13624, 13623, 13624, 13622, 13625, 13622, 13624, 13623, 13622, 13623, 13623, 13621, 13624, 13622, 13624, 13621, 13622, 13624, 13623, 13623, 13624, 13623, 13626, 13623, 13625, 13623, 13625, 13624, 13625, 13621, 13622, 13624, 13621, 13622, 13622, 13621, 13623, 13621, 13624, 13621, 13627, 13626, 13626, 13624, 13626, 13624, 13623, 13622, 13622, 13624, 13622, 13621, 13622, 13621, 13623, 13622, 13625, 13623, 13623, 13620, 13623, 13623, 13623, 13619, 13623, 13623, 13623, 13622, 13623, 13621, 13623, 13622, 13624, 13622, 13623, 13623, 13621, 13623, 13620, 13620, 13623, 13621, 13622, 13622, 13623, 13625, 13622, 13622, 13624, 13623, 13622, 13623, 13627, 13622, 13623, 13624, 13625, 13623, 13627, 13623, 13624, 13623, 13627, 13624, 13622, 13622, 13623, 13621, 13622, 13621, 13623, 13623, 13623, 13623, 13621, 13621, 13623, 13624, 13624, 13622, 13625, 13624, 13625, 13622, 13627, 13622, 13622, 13622, 13622, 13621, 13624, 13622, 13622, 13620, 13620, 13622, 13623, 13621, 13622, 13621, 13622, 13623, 13623, 13624, 13625, 13622, 13623, 13621, 13622, 13625, 13624, 13623, 13624, 13624, 13625, 13624, 13625, 13622, 13623, 13621, 13620, 13621, 13621, 13623, 13624, 13622, 13624, 13624, 13623, 13621, 13623, 13623, 13624, 13623, 13623, 13622, 13625, 13623, 13623, 13621, 13621, 13623, 13623, 13621, 13623, 13624, 13622, 13619, 13621, 13622, 13624, 13623, 13624, 13623, 13622, 13625, 13626, 13621, 13624, 13623, 13624, 13623, 13624, 13622, 13620, 13620, 13623, 13618, 13622, 13623, 13623, 13621, 13621, 13620, 13625, 13623, 13624, 13624, 13620, 13622, 13622, 13621, 13621, 13622, 13623, 13623, 13621, 13619, 13622, 13620, 13622, 13622, 13623, 13623, 13624, 13623, 13625, 13620, 13622, 13622, 13623, 13621, 13624, 13624, 13622, 13622, 13622, 13621, 13619, 13621, 13622, 13622, 13620, 13622, 13621, 13618, 13622, 13623, 13621, 13624, 13623, 13620, 13621, 13622, 13623, 13623, 13624, 13625, 13625, 13624, 13621, 13621, 13623, 13621, 13622, 13619, 13623, 13625, 13624, 13621, 13622, 13624, 13624, 13622, 13625, 13624, 13622, 13623, 13623, 13625, 13625, 13623, 13624, 13623, 13623, 13624, 13623, 13622, 13622, 13621, 13620, 13620, 13621, 13620, 13621, 13622, 13622, 13624, 13621, 13621, 13624, 13623, 13621, 13620, 13622, 13620, 13626, 13622, 13622, 13624, 13625, 13625, 13624, 13620, 13621, 13621, 13624, 13620, 13619, 13625, 13621, 13621, 13624, 13621, 13623, 13622, 13623, 13625, 13624, 13624, 13624, 13624, 13624, 13623, 13624, 13622, 13624, 13620, 13625, 13624, 13624, 13621, 13621, 13621, 13623, 13621, 13622, 13620, 13623, 13624, 13624, 13623, 13624, 13622, 13624, 13622, 13624, 13623, 13622, 13623, 13626, 13622, 13623, 13622, 13622, 13622, 13621, 13623, 13623, 13621, 13623, 13621, 13621, 13624, 13624, 13624, 13624, 13622, 13625, 13624, 13627, 13623, 13624, 13622, 13623, 13623, 13623, 13623, 13623, 13624, 13624, 13621, 13624, 13622, 13625, 13623, 13625, 13622, 13622, 13622, 13624, 13623, 13625, 13622, 13623, 13625, 13625, 13621, 13623, 13622, 13623, 13621, 13620, 13622, 13621, 13624, 13623, 13622, 13623, 13623, 13624, 13621, 13625, 13622, 13625, 13625, 13625, 13622, 13622, 13622, 13624, 13625, 13622, 13619, 13625, 13623, 13622, 13622, 13622, 13621, 13625, 13622, 13621, 13623, 13624, 13622, 13621, 13622, 13623, 13622, 13624, 13622, 13625, 13620, 13624, 13622, 13624, 13622, 13623, 13621, 13621, 13621, 13619, 13622, 13623, 13624, 13624, 13621, 13621, 13621, 13623, 13624, 13625, 13623, 13623, 13620, 13622, 13622]),
      (1596395008015038110, 500, 10, 2, 98304, 0, 0, [13608, 13609, 13606, 13611, 13609, 13608, 13606, 13608, 13607, 13609, 13606, 13605, 13608, 13606, 13606, 13607, 13603, 13606, 13607, 13607, 13608, 13607, 13607, 13610, 13607, 13609, 13606, 13607, 13606, 13607, 13605, 13606, 13603, 13604, 13606, 13608, 13603, 13607, 13605, 13607, 13603, 13608, 13607, 13606, 13607, 13608, 13608, 13606, 13608, 13609, 13610, 13609, 13606, 13607, 13607, 13606, 13604, 13606, 13605, 13605, 13606, 13607, 13605, 13605, 13606, 13606, 13607, 13605, 13605, 13608, 13607, 13608, 13608, 13607, 13606, 13607, 13608, 13605, 13604, 13604, 13607, 13606, 13607, 13605, 13602, 13605, 13604, 13604, 13604, 13607, 13606, 13604, 13606, 13609, 13606, 13607, 13606, 13607, 13608, 13607, 13604, 13607, 13608, 13605, 13604, 13608, 13607, 13607, 13608, 13609, 13606, 13605, 13607, 13608, 13606, 13606, 13607, 13607, 13609, 13607, 13606, 13608, 13605, 13604, 13601, 13603, 13605, 13606, 13607, 13607, 13604, 13609, 13607, 13606, 13606, 13608, 13605, 13606, 13605, 13604, 13602, 13606, 13608, 13606, 13605, 13606, 13608, 13606, 13608, 13605, 13605, 13606, 13606, 13607, 13606, 13606, 13606, 13607, 13608, 13605, 13607, 13609, 13607, 13606, 13605, 13606, 13608, 13607, 13604, 13608, 13608, 13609, 13607, 13606, 13608, 13608, 13604, 13605, 13607, 13607, 13605, 13607, 13605, 13607, 13605, 13605, 13606, 13606, 13606, 13606, 13605, 13604, 13606, 13607, 13606, 13606, 13606, 13606, 13608, 13608, 13605, 13610, 13604, 13604, 13605, 13609, 13605, 13607, 13605, 13605, 13605, 13605, 13605, 13604, 13605, 13608, 13608, 13607, 13607, 13608, 13609, 13609, 13606, 13605, 13608, 13608, 13606, 13607, 13606, 13606, 13605, 13605, 13603, 13607, 13607, 13604, 13607, 13609, 13605, 13609, 13607, 13609, 13609, 13610, 13606, 13605, 13607, 13609, 13606, 13606, 13605, 13607, 13605, 13606, 13605, 13606, 13605, 13605, 13602, 13605, 13605, 13607, 13605, 13609, 13605, 13606, 13607, 13606, 13606, 13608, 13607, 13607, 13605, 13606, 13606, 13607, 13607, 13606, 13608, 13606, 13606, 13607, 13605, 13605, 13608, 13608, 13607, 13608, 13609, 13608, 13607, 13608, 13605, 13606, 13609, 13608, 13606, 13609, 13606, 13607, 13608, 13608, 13606, 13607, 13606, 13606, 13607, 13607, 13608, 13605, 13608, 13606, 13609, 13609, 13606, 13606, 13610, 13608, 13606, 13610, 13607, 13608, 13606, 13607, 13608, 13606, 13607, 13608, 13606, 13608, 13606, 13605, 13608, 13608, 13607, 13607, 13606, 13607, 13605, 13608, 13605, 13607, 13605, 13608, 13606, 13607, 13608, 13609, 13607, 13607, 13607, 13605, 13606, 13608, 13606, 13608, 13608, 13606, 13605, 13607, 13606, 13608, 13608, 13606, 13606, 13608, 13605, 13607, 13608, 13607, 13605, 13606, 13608, 13609, 13607, 13608, 13605, 13605, 13606, 13606, 13607, 13607, 13607, 13606, 13605, 13606, 13606, 13607, 13607, 13607, 13607, 13607, 13606, 13606, 13606, 13608, 13606, 13607, 13608, 13607, 13605, 13607, 13606, 13605, 13606, 13606, 13606, 13605, 13604, 13608, 13608, 13605, 13607, 13605, 13607, 13608, 13607, 13606, 13605, 13606, 13607, 13607, 13604, 13608, 13608, 13608, 13606, 13605, 13604, 13608, 13607, 13604, 13606, 13607, 13604, 13605, 13608, 13608, 13605, 13606, 13607, 13605, 13604, 13608, 13607, 13606, 13608, 13606, 13605, 13605, 13609, 13605, 13607, 13607, 13604, 13606, 13609, 13606, 13607, 13606, 13606, 13607, 13604, 13608, 13607, 13608, 13603, 13607, 13606, 13607, 13604, 13607, 13606, 13607, 13605, 13605, 13608, 13608, 13606, 13607, 13608, 13609, 13608, 13608, 13607, 13608, 13605, 13609, 13605, 13607, 13610, 13607, 13605, 13607, 13609, 13610, 13608, 13609, 13606, 13607]),
    ], dtype=rec_type
)

@numba.njit(cache=True, nogil=True)
def concatinate(rr_old, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):

    # Concatinate waveforms from raw_records: this is the core of the function 
    # Create an array of evenly spaced values in a given interval
    events = np.arange(0,num_events_available,1) # we use the num available because it is the maximum
                
    rr = [] #list of events, may need to convert to numpy array for faster process
    #rr_wf = []
    # Let's loop now on the events we want to process
    for i in range(num_events_processed):
        event = events[i]
          
        if event == 0:
            rr_start = 0
            rr_stop = records_required - 1
        else:
            rr_start = np.int(event*rr_per_wf + events[i-1])
            rr_stop = np.int((rr_start + records_required))
          
        rr_s = np.zeros(len(active_channels), rr_dtype)
             
        for ix,ch in enumerate(active_channels):
            single_channel = rr_old[rr_old['channel'] == ch] 
            if event == 0:
                temp1 = single_channel[rr_start:records_required]
                temp2 = temp1['data']
                baseline = np.median(temp2) 
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - single_channel[rr_start:records_required]['data'])
            else:
                temp3 = single_channel[rr_start:rr_stop]
                temp4 = temp3['data']
                baseline = np.median(temp4)
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - single_channel[rr_start:rr_stop]['data'])
                
            rr_s[ix]['baseline'] = np.median(baseline)
            rr_s[ix]['time'] = single_channel[rr_start]['time']
            rr_s[ix]['data'] = long_wf
            rr_s[ix]['event'] = event 
            rr_s[ix]['channel'] = ch
            rr_s[ix]['dt'] = 10   
                
            if ch <= 493:
                rr_s[ix]['search'] = "dm"
            else:
                rr_s[ix]['search'] = "0vbb"
        rr.append(rr_s)
    return rr

dt = rr_old_sample['dt'][0]*1e-9
rr_samples = len(rr_old_sample['data'][0]) 
num_rr_available  = len(rr_old_sample[rr_old_sample['channel'] == 0])
rr_per_wf = np.int(math.ceil(98304/rr_samples))
num_events_available = np.int(num_rr_available/rr_per_wf)
last_rr_omitted =  5
wf_max_len = (rr_per_wf-last_rr_omitted)*rr_samples*dt
num_events_processed = num_events_available
wf_len = wf_max_len
wf_min_len = rr_samples*10e-9
records_required = np.int(math.ceil(wf_len/(rr_samples*dt)))
rr_dtype = np.dtype([('data',np.int,records_required*rr_samples),
                             ('time',np.int),
                             ('search',('U', 5)),
                             ('channel',np.int),
                             ('event',np.int),
                             ('dt',np.int),
                             ('baseline',np.float)
                            ])
active_channel_small = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=np.int16)
wf_c = concatinate(rr_old_sample,rr_samples,rr_per_wf,num_events_processed,num_events_available,records_required,active_channel_small, rr_dtype)

And I obtained the same error message:

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Internal error at <numba.core.typeinfer.StaticGetItemConstraint object at 0x7f859dc02ee0>.
Buffer dtype cannot be buffer, have dtype: nestedarray(int16, (500,))
During: typing of static-get-item at <ipython-input-1-4aa8dfc00e9d> (54)
Enable logging at debug level for details.

File "<ipython-input-1-4aa8dfc00e9d>", line 54:
def concatinate(rr_old, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):
    <source elided>
                temp1 = single_channel[rr_start:records_required]
                temp2 = temp1['data']

Thank you very much if you can help me understand!

Serena

6

thanks, I was able to create a shorter version which throws the same error:

import numpy as np
import numba
import math

# Data
rec_type = np.dtype([
    (('Start time', 'time'), '<i8'),
    (('Baseline', 'baseline'), '<i2'),
    (('Waveform data', 'data'), '<i2', (2,))
])

rr_old_sample = np.array(
    [
      ( 0, 0,  [13812, 13813,]),
      ( 0, 0, [13621, 13622, ]),
      ( 0, 0, [13608, 13609, ]),
    ], dtype=rec_type
)

@numba.njit
def foo(rr_old_sample):
    return rr_old_sample['data']

foo(rr_old_sample)

This is a case of missing functionality. The nested array case is not working when trying to perform a “static getitem”. If you want, you can open a request here: Issues · numba/numba · GitHub
Please attach the short example above to explain what is missing.

In the meantime, since you probably cannot wait for this to be fixed, I suggest a workaround like the following below. If you extract the waveform data into its own array, you should be to write something like np.median(waveform_data[rr_old['channel'] == ch][rr_start:records_required]). It’s not as nice as what you have, but it’s not a big change and it should work.

import numpy as np
import numba
import math

# Data
rec_type = np.dtype([
    (('Start time', 'time'), '<i8'),
    (('Baseline', 'baseline'), '<i2'),
    (('Waveform data', 'data'), '<i2', (2,))
])

rr_old_sample = np.array(
    [
      ( 0, 0,  [13812, 13813,]),
      ( 0, 0, [13621, 13622, ]),
      ( 0, 0, [13608, 13609, ]),
    ], dtype=rec_type
)

rr_old_sample_short = rr_old_sample[['time', 'baseline']]
waveform_data = rr_old_sample['data']


@numba.njit
def foo(rr_old_sample_short, waveform_data):
    return waveform_data

foo(rr_old_sample, waveform_data)

cheers,
Luk

7

Hi,

Thank you very much for the workaround, and I will indeed open an issue for this nestedarray functionality.

This is getting a bit frustrating. Now the problem seems to be the np.concatenate function. I tried to convert the ndarray argument int(baseline) - waveform_data[rr_old_short['channel'] == ch][rr_start:records_required] into a tuple (before passing it to the np.concatenate), but then Numba has also some problem with the function tuple() of the conversion. I also tried to convert it into a list but it does not work (using the tolist() method).

This is the code I am using now (similar to that of before) but now it gives a TypeError related to the np.concatenate:

import numpy as np
import numba
import math

# Channels
active_channel_small = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=np.int16)

# Data
rec_type = np.dtype([
    (('Start time', 'time'), '<i8'),
    (('Length of interval ', 'length'), '<i4'),
    (('Width ', 'dt'), '<i2'),
    (('Channel', 'channel'), '<i2'),
    (('Length of pulse ', 'pulse_length'), '<i4'),
    (('Fragment', 'record_i'), '<i2'),
    (('Baseline', 'baseline'), '<i2'),
    (('Waveform data', 'data'), '<i2', (500,))
])

rr_old_sample = np.array(
    [
      (1596395008015038110, 500, 10, 0, 98304, 0, 0, [13812, 13813, 13813, 13814, 13812, 13815, 13809, 13815, 13812, 13811, 13810, 13815, 13812, 13815, 13812, 13812, 13812, 13811, 13808, 13811, 13812, 13813, 13813, 13813, 13811, 13813, 13812, 13814, 13811, 13811, 13809, 13811, 13810, 13812, 13809, 13815, 13813, 13813, 13810, 13810, 13811, 13814, 13812, 13812, 13809, 13811, 13809, 13812, 13811, 13812, 13810, 13814, 13813, 13815, 13810, 13814, 13813, 13811, 13813, 13813, 13811, 13812, 13812, 13812, 13812, 13813, 13814, 13813, 13811, 13811, 13809, 13811, 13808, 13811, 13810, 13813, 13809, 13811, 13809, 13812, 13813, 13817, 13811, 13815, 13812, 13815, 13809, 13813, 13810, 13813, 13811, 13811, 13810, 13811, 13810, 13812, 13810, 13813, 13810, 13813, 13811, 13812, 13811, 13815, 13814, 13814, 13810, 13814, 13814, 13816, 13811, 13816, 13813, 13814, 13816, 13816, 13813, 13815, 13813, 13809, 13810, 13812, 13813, 13814, 13813, 13811, 13809, 13812, 13814, 13811, 13811, 13816, 13811, 13813, 13809, 13808, 13809, 13811, 13810, 13811, 13809, 13809, 13808, 13814, 13810, 13812, 13813, 13813, 13813, 13814, 13812, 13816, 13812, 13816, 13811, 13814, 13812, 13813, 13815, 13814, 13812, 13814, 13812, 13814, 13813, 13817, 13814, 13813, 13811, 13812, 13810, 13813, 13809, 13815, 13811, 13813, 13811, 13812, 13810, 13811, 13812, 13811, 13809, 13813, 13809, 13811, 13810, 13811, 13811, 13812, 13811, 13815, 13809, 13815, 13814, 13811, 13811, 13814, 13811, 13811, 13813, 13814, 13810, 13813, 13809, 13814, 13810, 13810, 13811, 13812, 13809, 13813, 13809, 13812, 13811, 13812, 13812, 13810, 13809, 13811, 13813, 13814, 13813, 13814, 13812, 13814, 13810, 13813, 13813, 13813, 13813, 13814, 13810, 13813, 13811, 13814, 13811, 13814, 13811, 13815, 13813, 13813, 13811, 13813, 13812, 13815, 13811, 13813, 13812, 13811, 13809, 13812, 13811, 13812, 13811, 13815, 13809, 13813, 13811, 13811, 13810, 13814, 13809, 13812, 13809, 13811, 13810, 13812, 13809, 13813, 13812, 13810, 13810, 13812, 13812, 13812, 13812, 13814, 13811, 13814, 13813, 13812, 13810, 13813, 13812, 13814, 13812, 13812, 13813, 13816, 13811, 13812, 13813, 13817, 13811, 13812, 13812, 13813, 13813, 13816, 13814, 13814, 13813, 13814, 13811, 13813, 13812, 13814, 13813, 13814, 13812, 13813, 13812, 13812, 13812, 13811, 13809, 13813, 13811, 13812, 13810, 13810, 13810, 13813, 13813, 13812, 13810, 13810, 13814, 13812, 13810, 13814, 13811, 13813, 13811, 13812, 13809, 13813, 13809, 13812, 13812, 13811, 13811, 13814, 13810, 13812, 13811, 13814, 13813, 13816, 13811, 13812, 13808, 13814, 13810, 13815, 13810, 13812, 13809, 13813, 13811, 13812, 13810, 13814, 13810, 13812, 13812, 13812, 13809, 13812, 13812, 13813, 13812, 13812, 13810, 13811, 13809, 13813, 13812, 13813, 13811, 13815, 13811, 13813, 13811, 13815, 13811, 13811, 13810, 13813, 13810, 13811, 13810, 13814, 13809, 13811, 13810, 13814, 13810, 13813, 13813, 13812, 13808, 13813, 13811, 13812, 13811, 13812, 13813, 13815, 13812, 13813, 13812, 13814, 13809, 13813, 13807, 13810, 13808, 13811, 13812, 13813, 13811, 13813, 13812, 13814, 13809, 13812, 13814, 13814, 13810, 13813, 13808, 13812, 13810, 13813, 13814, 13813, 13811, 13812, 13810, 13812, 13812, 13814, 13812, 13814, 13810, 13810, 13810, 13812, 13810, 13812, 13810, 13814, 13809, 13814, 13813, 13813, 13811, 13811, 13810, 13811, 13811, 13814, 13813, 13813, 13810, 13813, 13813, 13814, 13811, 13811, 13811, 13813, 13809, 13810, 13812, 13812, 13811, 13811, 13810, 13814, 13812, 13813, 13813, 13811, 13809, 13812, 13810, 13814, 13814, 13815, 13810, 13813, 13811, 13813, 13811, 13811, 13812, 13812]),
      (1596395008015038110, 500, 10, 1, 98304, 0, 0, [13621, 13622, 13621, 13621, 13625, 13620, 13625, 13624, 13623, 13624, 13624, 13626, 13624, 13623, 13624, 13622, 13625, 13622, 13624, 13623, 13622, 13623, 13623, 13621, 13624, 13622, 13624, 13621, 13622, 13624, 13623, 13623, 13624, 13623, 13626, 13623, 13625, 13623, 13625, 13624, 13625, 13621, 13622, 13624, 13621, 13622, 13622, 13621, 13623, 13621, 13624, 13621, 13627, 13626, 13626, 13624, 13626, 13624, 13623, 13622, 13622, 13624, 13622, 13621, 13622, 13621, 13623, 13622, 13625, 13623, 13623, 13620, 13623, 13623, 13623, 13619, 13623, 13623, 13623, 13622, 13623, 13621, 13623, 13622, 13624, 13622, 13623, 13623, 13621, 13623, 13620, 13620, 13623, 13621, 13622, 13622, 13623, 13625, 13622, 13622, 13624, 13623, 13622, 13623, 13627, 13622, 13623, 13624, 13625, 13623, 13627, 13623, 13624, 13623, 13627, 13624, 13622, 13622, 13623, 13621, 13622, 13621, 13623, 13623, 13623, 13623, 13621, 13621, 13623, 13624, 13624, 13622, 13625, 13624, 13625, 13622, 13627, 13622, 13622, 13622, 13622, 13621, 13624, 13622, 13622, 13620, 13620, 13622, 13623, 13621, 13622, 13621, 13622, 13623, 13623, 13624, 13625, 13622, 13623, 13621, 13622, 13625, 13624, 13623, 13624, 13624, 13625, 13624, 13625, 13622, 13623, 13621, 13620, 13621, 13621, 13623, 13624, 13622, 13624, 13624, 13623, 13621, 13623, 13623, 13624, 13623, 13623, 13622, 13625, 13623, 13623, 13621, 13621, 13623, 13623, 13621, 13623, 13624, 13622, 13619, 13621, 13622, 13624, 13623, 13624, 13623, 13622, 13625, 13626, 13621, 13624, 13623, 13624, 13623, 13624, 13622, 13620, 13620, 13623, 13618, 13622, 13623, 13623, 13621, 13621, 13620, 13625, 13623, 13624, 13624, 13620, 13622, 13622, 13621, 13621, 13622, 13623, 13623, 13621, 13619, 13622, 13620, 13622, 13622, 13623, 13623, 13624, 13623, 13625, 13620, 13622, 13622, 13623, 13621, 13624, 13624, 13622, 13622, 13622, 13621, 13619, 13621, 13622, 13622, 13620, 13622, 13621, 13618, 13622, 13623, 13621, 13624, 13623, 13620, 13621, 13622, 13623, 13623, 13624, 13625, 13625, 13624, 13621, 13621, 13623, 13621, 13622, 13619, 13623, 13625, 13624, 13621, 13622, 13624, 13624, 13622, 13625, 13624, 13622, 13623, 13623, 13625, 13625, 13623, 13624, 13623, 13623, 13624, 13623, 13622, 13622, 13621, 13620, 13620, 13621, 13620, 13621, 13622, 13622, 13624, 13621, 13621, 13624, 13623, 13621, 13620, 13622, 13620, 13626, 13622, 13622, 13624, 13625, 13625, 13624, 13620, 13621, 13621, 13624, 13620, 13619, 13625, 13621, 13621, 13624, 13621, 13623, 13622, 13623, 13625, 13624, 13624, 13624, 13624, 13624, 13623, 13624, 13622, 13624, 13620, 13625, 13624, 13624, 13621, 13621, 13621, 13623, 13621, 13622, 13620, 13623, 13624, 13624, 13623, 13624, 13622, 13624, 13622, 13624, 13623, 13622, 13623, 13626, 13622, 13623, 13622, 13622, 13622, 13621, 13623, 13623, 13621, 13623, 13621, 13621, 13624, 13624, 13624, 13624, 13622, 13625, 13624, 13627, 13623, 13624, 13622, 13623, 13623, 13623, 13623, 13623, 13624, 13624, 13621, 13624, 13622, 13625, 13623, 13625, 13622, 13622, 13622, 13624, 13623, 13625, 13622, 13623, 13625, 13625, 13621, 13623, 13622, 13623, 13621, 13620, 13622, 13621, 13624, 13623, 13622, 13623, 13623, 13624, 13621, 13625, 13622, 13625, 13625, 13625, 13622, 13622, 13622, 13624, 13625, 13622, 13619, 13625, 13623, 13622, 13622, 13622, 13621, 13625, 13622, 13621, 13623, 13624, 13622, 13621, 13622, 13623, 13622, 13624, 13622, 13625, 13620, 13624, 13622, 13624, 13622, 13623, 13621, 13621, 13621, 13619, 13622, 13623, 13624, 13624, 13621, 13621, 13621, 13623, 13624, 13625, 13623, 13623, 13620, 13622, 13622]),
      (1596395008015038110, 500, 10, 2, 98304, 0, 0, [13608, 13609, 13606, 13611, 13609, 13608, 13606, 13608, 13607, 13609, 13606, 13605, 13608, 13606, 13606, 13607, 13603, 13606, 13607, 13607, 13608, 13607, 13607, 13610, 13607, 13609, 13606, 13607, 13606, 13607, 13605, 13606, 13603, 13604, 13606, 13608, 13603, 13607, 13605, 13607, 13603, 13608, 13607, 13606, 13607, 13608, 13608, 13606, 13608, 13609, 13610, 13609, 13606, 13607, 13607, 13606, 13604, 13606, 13605, 13605, 13606, 13607, 13605, 13605, 13606, 13606, 13607, 13605, 13605, 13608, 13607, 13608, 13608, 13607, 13606, 13607, 13608, 13605, 13604, 13604, 13607, 13606, 13607, 13605, 13602, 13605, 13604, 13604, 13604, 13607, 13606, 13604, 13606, 13609, 13606, 13607, 13606, 13607, 13608, 13607, 13604, 13607, 13608, 13605, 13604, 13608, 13607, 13607, 13608, 13609, 13606, 13605, 13607, 13608, 13606, 13606, 13607, 13607, 13609, 13607, 13606, 13608, 13605, 13604, 13601, 13603, 13605, 13606, 13607, 13607, 13604, 13609, 13607, 13606, 13606, 13608, 13605, 13606, 13605, 13604, 13602, 13606, 13608, 13606, 13605, 13606, 13608, 13606, 13608, 13605, 13605, 13606, 13606, 13607, 13606, 13606, 13606, 13607, 13608, 13605, 13607, 13609, 13607, 13606, 13605, 13606, 13608, 13607, 13604, 13608, 13608, 13609, 13607, 13606, 13608, 13608, 13604, 13605, 13607, 13607, 13605, 13607, 13605, 13607, 13605, 13605, 13606, 13606, 13606, 13606, 13605, 13604, 13606, 13607, 13606, 13606, 13606, 13606, 13608, 13608, 13605, 13610, 13604, 13604, 13605, 13609, 13605, 13607, 13605, 13605, 13605, 13605, 13605, 13604, 13605, 13608, 13608, 13607, 13607, 13608, 13609, 13609, 13606, 13605, 13608, 13608, 13606, 13607, 13606, 13606, 13605, 13605, 13603, 13607, 13607, 13604, 13607, 13609, 13605, 13609, 13607, 13609, 13609, 13610, 13606, 13605, 13607, 13609, 13606, 13606, 13605, 13607, 13605, 13606, 13605, 13606, 13605, 13605, 13602, 13605, 13605, 13607, 13605, 13609, 13605, 13606, 13607, 13606, 13606, 13608, 13607, 13607, 13605, 13606, 13606, 13607, 13607, 13606, 13608, 13606, 13606, 13607, 13605, 13605, 13608, 13608, 13607, 13608, 13609, 13608, 13607, 13608, 13605, 13606, 13609, 13608, 13606, 13609, 13606, 13607, 13608, 13608, 13606, 13607, 13606, 13606, 13607, 13607, 13608, 13605, 13608, 13606, 13609, 13609, 13606, 13606, 13610, 13608, 13606, 13610, 13607, 13608, 13606, 13607, 13608, 13606, 13607, 13608, 13606, 13608, 13606, 13605, 13608, 13608, 13607, 13607, 13606, 13607, 13605, 13608, 13605, 13607, 13605, 13608, 13606, 13607, 13608, 13609, 13607, 13607, 13607, 13605, 13606, 13608, 13606, 13608, 13608, 13606, 13605, 13607, 13606, 13608, 13608, 13606, 13606, 13608, 13605, 13607, 13608, 13607, 13605, 13606, 13608, 13609, 13607, 13608, 13605, 13605, 13606, 13606, 13607, 13607, 13607, 13606, 13605, 13606, 13606, 13607, 13607, 13607, 13607, 13607, 13606, 13606, 13606, 13608, 13606, 13607, 13608, 13607, 13605, 13607, 13606, 13605, 13606, 13606, 13606, 13605, 13604, 13608, 13608, 13605, 13607, 13605, 13607, 13608, 13607, 13606, 13605, 13606, 13607, 13607, 13604, 13608, 13608, 13608, 13606, 13605, 13604, 13608, 13607, 13604, 13606, 13607, 13604, 13605, 13608, 13608, 13605, 13606, 13607, 13605, 13604, 13608, 13607, 13606, 13608, 13606, 13605, 13605, 13609, 13605, 13607, 13607, 13604, 13606, 13609, 13606, 13607, 13606, 13606, 13607, 13604, 13608, 13607, 13608, 13603, 13607, 13606, 13607, 13604, 13607, 13606, 13607, 13605, 13605, 13608, 13608, 13606, 13607, 13608, 13609, 13608, 13608, 13607, 13608, 13605, 13609, 13605, 13607, 13610, 13607, 13605, 13607, 13609, 13610, 13608, 13609, 13606, 13607]),
    ], dtype=rec_type
)

dt = rr_old_sample['dt'][0]*1e-9
rr_samples = len(rr_old_sample['data'][0]) 
num_rr_available  = len(rr_old_sample[rr_old_sample['channel'] == 0])
rr_per_wf = np.int(math.ceil(98304/rr_samples))
num_events_available = np.int(num_rr_available/rr_per_wf)
last_rr_omitted =  5
wf_max_len = (rr_per_wf-last_rr_omitted)*rr_samples*dt
num_events_processed = num_events_available
wf_len = wf_max_len
wf_min_len = rr_samples*10e-9
records_required = np.int(math.ceil(wf_len/(rr_samples*dt)))

@numba.njit(cache=True, nogil=True)
def concatinate(rr_old_short, waveform_data, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):

    # Concatinate waveforms from raw_records: this is the core of the function 
    # Create an array of evenly spaced values in a given interval
    events = np.arange(0,num_events_available,1) # we use the num available because it is the maximum
                
    rr = [] #list of events, may need to convert to numpy array for faster process
    #rr_wf = []
    # Let's loop now on the events we want to process
    for i in range(num_events_processed):
        event = events[i]
          
        if event == 0:
            rr_start = 0
            rr_stop = records_required - 1
        else:
            rr_start = np.int(event*rr_per_wf + events[i-1])
            rr_stop = np.int((rr_start + records_required))
          
        rr_s = np.zeros(len(active_channels), rr_dtype)
             
        for ix,ch in enumerate(active_channels):
            single_channel = rr_old_short[rr_old_short['channel'] == ch] 
            if event == 0:
                baseline = np.median(waveform_data[rr_old_short['channel'] == ch][rr_start:records_required])
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - waveform_data[rr_old_short['channel'] == ch][rr_start:records_required])
            else:
                baseline = np.median(waveform_data[rr_old_short['channel'] == ch][rr_start:records_required])
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - waveform_data[rr_old_short['channel'] == ch][rr_start:rr_stop])
                
            rr_s[ix]['baseline'] = np.median(baseline)
            rr_s[ix]['time'] = single_channel[rr_start]['time']
            rr_s[ix]['data'] = long_wf
            rr_s[ix]['event'] = event 
            rr_s[ix]['channel'] = ch
            rr_s[ix]['dt'] = 10   
                
            if ch <= 493:
                rr_s[ix]['search'] = "dm"
            else:
                rr_s[ix]['search'] = "0vbb"
        rr.append(rr_s)
    return rr

rr_old_short = rr_old_sample[['time', 'length', 'dt', 'channel', 'pulse_length', 'record_i','baseline']]
waveform_data = rr_old_sample['data']

rr_dtype = np.dtype([('data',np.int,records_required*rr_samples),
                             ('time',np.int),
                             ('search',('U', 5)),
                             ('channel',np.int),
                             ('event',np.int),
                             ('dt',np.int),
                             ('baseline',np.float)
                            ])

active_channel_small = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=np.int16)

wf_c = concatinate(rr_old_short,waveform_data,rr_samples,rr_per_wf,num_events_processed,num_events_available,records_required,active_channel_small, rr_dtype)

The error is:

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<function concatenate at 0x7f60fc6ac700>) found for signature:
 
 >>> concatenate(array(int64, 2d, C))
 
There are 2 candidate implementations:
      - Of which 2 did not match due to:
      Overload in function 'concatenate': File: numba/core/typing/npydecl.py: Line 792.
        With argument(s): '(array(int64, 2d, C))':
       Rejected as the implementation raised a specific error:
         TypeError: np.concatenate(): expecting a non-empty tuple of arrays, got array(int64, 2d, C)
  raised from /dali/lgrandi/strax/miniconda3/envs/strax/lib/python3.8/site-packages/numba/core/typing/npydecl.py:768

During: resolving callee type: Function(<function concatenate at 0x7f60fc6ac700>)
During: typing of call at <ipython-input-2-6a75c6aa7a9f> (67)


File "<ipython-input-2-6a75c6aa7a9f>", line 67:
def concatinate(rr_old_short, waveform_data, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):
    <source elided>
                #baseline + flip waveform
                long_wf = np.concatenate(int(baseline) - waveform_data[rr_old_short['channel'] == ch][rr_start:records_required])
                ^

I am sorry, would you be able to help me again?

Cheers,
Serena

8

The error messages like "Buffer dtype cannot be buffer, have dtype: nestedarray(int16, (500,))"

since Improve the error message for unsupported Buffer in Buffer situation. by stuartarchibald · Pull Request #6489 · numba/numba · GitHub, reads:

Internal error at <numba.core.typeinfer.StaticGetItemConstraint object at 0x7fc06fb93510>.
The dtype of a Buffer type cannot itself be a Buffer type, this is unsupported behaviour.
The dtype requested for the unsupported Buffer was: nestedarray(int16, (2,)).
During: typing of static-get-item at di45.py (22)
Enable logging at debug level for details.

File "di45.py", line 22:
def foo(rr_old_sample):
    return rr_old_sample['data']
    ^

whilst the problem isn’t fixed, it hopefully explains that it is not supported and is a bit clearer about the issue.

9

Hi @SerenaDiPede,

The issue here is that concatenate needs a tuple of arrays. The reason for this (in general) is that Numba has to work out the types of all the variables in your code before it can compile it, this process is called “type inference”. Numba does this through static analysis, it doesn’t run your program, it just looks at the structure of the program and applies rules for each operation. If concatenate takes a “list of arrays” or similar, that list could be any length, and so it’s not possible to work out the shape (type) of the returned array. I think in your case you may have concatenate taking a single array, which I think Numba could support, please do open a feature request for this if it is important to you.

In general, to help Numba out with type inference, it’s a good idea to make your code as simple as possible and consider what’s being asked of Numba’s type inference algorithms. Another approach some people use is to build functions incrementally so that they start from a working state and each extra part is a small change that’s more easy to debug. Numba also doesn’t support every piece of syntax, argument and data type, if there’s something missing that would be useful, please do open a feature request.

Hope this helps.

PS: This seems to compile, I removed the concatenate and a np.median call (looks like baseline is scalar if it can be cast to an int so np.median has no effect?).

@numba.njit(cache=True, nogil=True)
def concatinate(rr_old_short, waveform_data, rr_samples, rr_per_wf, num_events_processed, num_events_available, records_required, active_channels, rr_dtype):

    # Concatinate waveforms from raw_records: this is the core of the function 
    # Create an array of evenly spaced values in a given interval
    events = np.arange(0,num_events_available,1) # we use the num available because it is the maximum

    rr = [] #list of events, may need to convert to numpy array for faster process
    #rr_wf = []
    # Let's loop now on the events we want to process
    for i in range(num_events_processed):
        event = events[i]

        if event == 0:
            rr_start = 0
            rr_stop = records_required - 1
        else:
            rr_start = np.int(event*rr_per_wf + events[i-1])
            rr_stop = np.int((rr_start + records_required))

        rr_s = np.zeros(len(active_channels), rr_dtype)

        for ix, ch in enumerate(active_channels):
            single_channel = rr_old_short[rr_old_short['channel'] == ch] 
            if event == 0:
                baseline = np.median(waveform_data[rr_old_short['channel'] == ch][rr_start:records_required])
                #baseline + flip waveform
                long_wf = int(baseline) - waveform_data[rr_old_short['channel'] == ch][rr_start:records_required]
            else:
                baseline = np.median(waveform_data[rr_old_short['channel'] == ch][rr_start:records_required])
                #baseline + flip waveform
                long_wf = int(baseline) - waveform_data[rr_old_short['channel'] == ch][rr_start:rr_stop]

            rr_s[ix]['baseline'] = baseline
            rr_s[ix]['time'] = single_channel[rr_start]['time']
            rr_s[ix]['data'][:] = long_wf[:]
            rr_s[ix]['event'] = event
            rr_s[ix]['channel'] = ch
            rr_s[ix]['dt'] = 10

            if ch <= 493:
                rr_s[ix]['search'] = "dm"
            else:
                rr_s[ix]['search'] = "0vbb"
        rr.append(rr_s)
    return rr