Help with error message

Just discovered this discourse group.

Figured I would cross-post my SO question which contains the code and stack trace.

A little more context:
My dataset is a pandas dataframe with a single column. That column is an array of values, having some irregular length. I need to run union commands for all elements, as shown in the example code. I’m going to pass in the .values attribute of that column into my process_values function.

a) is this a good use case for numba? I’m trying to improve on pandas .apply method.
b) can you help be decrypt the error message?

Thanks so much,


hi @jrjames83, I have replied on SO. Let me know if it’s clear how to solve it using lists.


Nice! Taking a look @luk-f-a. Appreciate the quick reply.

from numba import jit
from numba.typed import List

indices = np.arange(8806806, dtype=np.int64)
sizes = np.ones(8806806, dtype=np.int64)
connected_components = 8806806

def root(p: int) -> int:
    while p != indices[p]:
        indices[p] = indices[indices[p]]
        p = indices[p]
    return p

def connected( p: int, q: int) -> bool: 
    return root(p) == root(q)

def union( p: int, q: int) -> None:
    root1 = root(p)
    root2 = root(q)
    if root1 == root2:

    if (sizes[root1] < sizes[root2]):
        indices[root1] = root2
        sizes[root2] += sizes[root1]
        indices[root2] = root1
        sizes[root1] += sizes[root2]

    connected_components -= 1
def process_values(arr):
    for row in arr:
        for first, second in zip(arr, arr[1:]):
            union(first, second)

mylist = List()
for x in [[8018361, 4645960], [8018361, 4645960]]:
    new_list = List()


mylist evidently gets created, but now:

TypingError                               Traceback (most recent call last)
<ipython-input-64-6e08ff92ded5> in <module>
---> 50 process_values(mylist)

/opt/conda/lib/python3.7/site-packages/numba/core/ in _compile_for_args(self, *args, **kws)
    399                 e.patch_message(msg)
--> 401             error_rewrite(e, 'typing')
    402         except errors.UnsupportedError as e:
    403             # Something unsupported is present in the user code, add help info

/opt/conda/lib/python3.7/site-packages/numba/core/ in error_rewrite(e, issue_type)
    342                 raise e
    343             else:
--> 344                 reraise(type(e), e, None)
    346         argtypes = []

/opt/conda/lib/python3.7/site-packages/numba/core/ in reraise(tp, value, tb)
     78         value = tp()
     79     if value.__traceback__ is not tb:
---> 80         raise value.with_traceback(tb)
     81     raise value

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Internal error at <numba.core.typeinfer.CallConstraint object at 0x7f2127283ed0>.
"<class 'numba.core.cpu.CPUTargetOptions'> does not support option: 'npython'"
[1] During: resolving callee type: type(CPUDispatcher(<function union at 0x7f21272b2320>))
[2] During: typing of call at <ipython-input-64-6e08ff92ded5> (39)

Enable logging at debug level for details.

File "<ipython-input-64-6e08ff92ded5>", line 39:
def process_values(arr):
    <source elided>
        for first, second in zip(arr, arr[1:]):
            union(first, second)

What are the patterns then for getting my original array of arrays into this typed list format?

you just had a typo, @jit(npython=True) should be @jit(nopython=True)

Thanks @luk-f-a - good catch. I’ll report back with any updates and what the code ends up looking like.

@jrjames83 Bodo supports accelerating Pandas apply functions directly (usually great speedups). For example, you can just pass a Series with array of array input to Bodo:

import numpy as np
import pandas as pd
import bodo

def process_values(arr):

        [np.array([8018361, 4645960]),
            np.array([1137555, 7763897]),
            np.array([7532943, 2248813]),
            np.array([5352737,   71466, 3590473, 5352738, 2712260])], dtype='object')) 

Could you post your original Pandas code with apply so I can suggest how to use Bodo?