Pandas: source of "old_style" error-capturing warning

kartiksubbarao · September 28, 2023, 11:29pm

I reported a numba-related issue to the pandas project because I thought they might be using old_style error-capturing as the warning indicates. However, the response indicated that this might be an issue with numba itself. Could someone here help me better understand the source of this issue?

Here is the sample code:

import pandas as pd
import numpy as np

df = pd.DataFrame({'col1': [1, 2, 3, 4, 5]})
df.col1.rolling(2).apply(np.prod, raw=True, engine='numba')

Running this code with numba 0.58.0 results in the following warning:

/srv/temp/p311/lib64/python3.11/site-packages/pandas/core/window/numba_.py:72: NumbaPendingDeprecationWarning: Code using Numba extension API maybe depending on ‘old_style’ error-capturing, which is deprecated and will be replaced by ‘new_style’ in a future release. See details at Deprecation Notices — Numba 0.59.0dev0+179.ga4664180.dirty documentation
Exception origin:
File “/srv/temp/p311/lib64/python3.11/site-packages/numba/core/typing/templates.py”, line 807, in _build_impl
raise AssertionError(msg.format(pyfunc))
result[i] = numba_func(window, *args)

esc · September 29, 2023, 9:40am

Which version of Pandas is this? Asking because I am seeing a different error:

 💣 zsh» NUMBA_CAPTURED_ERRORS='new_style' python issue_55247.py
/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/decorators.py:262: NumbaDeprecationWarning: numba.generated_jit is deprecated. Please see the documentation at: https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-generated-jit for more information and advice on a suitable replacement.
  warnings.warn(msg, NumbaDeprecationWarning)

esc · September 29, 2023, 9:42am

Ah, this was pandas 2.0.3 – you seem to be on 2.1.1?

esc · September 29, 2023, 9:44am

I was able to pip install pandas into my conda env and then got:

esc@cyberdeck [test_pandas] /tmp
 💣 zsh» NUMBA_CAPTURED_ERRORS='old_style' python issue_55247.py
/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/numba_.py:72: NumbaPendingDeprecationWarning: Code using Numba extension API maybe depending on 'old_style' error-capturing, which is deprecated and will be replaced by 'new_style' in a future release. See details at https://numba.readthedocs.io/en/latest/reference/deprecation.html#deprecation-of-old-style-numba-captured-errors
Exception origin:
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/templates.py", line 807, in _build_impl
    raise AssertionError(msg.format(pyfunc))

  result[i] = numba_func(window, *args)
esc@cyberdeck [test_pandas] /tmp
 💣 zsh» NUMBA_CAPTURED_ERRORS='new_style' python issue_55247.py
Traceback (most recent call last):
  File "/private/tmp/issue_55247.py", line 5, in <module>
    df.col1.rolling(2).apply(np.prod, raw=True, engine='numba')
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 2035, in apply
    return super().apply(
           ^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 1500, in apply
    return self._apply(
           ^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 614, in _apply
    return self._apply_blockwise(homogeneous_func, name, numeric_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 467, in _apply_blockwise
    return self._apply_series(homogeneous_func, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 451, in _apply_series
    result = homogeneous_func(values)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 609, in homogeneous_func
    result = calc(values)
             ^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/pandas/core/window/rolling.py", line 606, in calc
    return func(x, start, end, min_periods, *numba_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/dispatcher.py", line 487, in _compile_for_args
    raise e
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/dispatcher.py", line 420, in _compile_for_args
    return_val = self.compile(tuple(argtypes))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/dispatcher.py", line 965, in compile
    cres = self._compiler.compile(args, return_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/dispatcher.py", line 125, in compile
    status, retval = self._compile_cached(args, return_type)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/dispatcher.py", line 139, in _compile_cached
    retval = self._compile_core(args, return_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/dispatcher.py", line 152, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler.py", line 770, in compile_extra
    return pipeline.compile_extra(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler.py", line 461, in compile_extra
    return self._compile_bytecode()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler.py", line 529, in _compile_bytecode
    return self._compile_core()
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler.py", line 504, in _compile_core
    raise e
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler.py", line 495, in _compile_core
    pm.run(self.state)
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler_machinery.py", line 364, in run
    raise e
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler_machinery.py", line 356, in run
    self._runPass(idx, pass_inst, state)
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass
    mutated |= check(pss.run_pass, internal_state)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/compiler_machinery.py", line 273, in check
    mangled = func(compiler_state)
              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typed_passes.py", line 110, in run_pass
    typemap, return_type, calltypes, errs = type_inference_stage(
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typed_passes.py", line 91, in type_inference_stage
    errs = infer.propagate(raise_errors=raise_errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typeinfer.py", line 1078, in propagate
    errors = self.constraints.propagate(self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typeinfer.py", line 177, in propagate
    raise e
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typeinfer.py", line 155, in propagate
    constraint(typeinfer)
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typeinfer.py", line 578, in __call__
    self.resolve(typeinfer, typevars, fnty)
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typeinfer.py", line 601, in resolve
    sig = typeinfer.resolve_call(fnty, pos_args, kw_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typeinfer.py", line 1557, in resolve_call
    return self.context.resolve_function_type(fnty, pos_args, kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/context.py", line 196, in resolve_function_type
    res = self._resolve_user_function_type(func, args, kws)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/context.py", line 248, in _resolve_user_function_type
    return func.get_call_type(self, args, kws)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/types/functions.py", line 312, in get_call_type
    raise e
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/types/functions.py", line 308, in get_call_type
    sig = temp.apply(nolitargs, nolitkws)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/templates.py", line 351, in apply
    sig = generic(args, kws)
          ^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/templates.py", line 614, in generic
    disp, new_args = self._get_impl(args, kws)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/templates.py", line 713, in _get_impl
    impl, args = self._build_impl(cache_key, args, kws)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esc/miniconda3-arm64/envs/test_pandas/lib/python3.11/site-packages/numba/core/typing/templates.py", line 807, in _build_impl
    raise AssertionError(msg.format(pyfunc))
AssertionError: Implementation function returned by `@overload` has an unexpected type.  Got <function prod at 0x1037efec0>

Does this help?

esc · September 29, 2023, 10:00am

The following variant does seem to work:

import pandas as pd
import numpy as np

from numba import njit

@njit
def foo(*args):
    return np.prod(*args)


df = pd.DataFrame({'col1': [1, 2, 3, 4, 5]})
print(df)
print("*---*")
print(df.col1.rolling(2).apply(foo, raw=True, engine='numba'))

I think there is smth. wrong with the way in which pandas hands off the np.prod function to be compiled by Numba.

lithomas1 · September 29, 2023, 11:35am

Thanks for the help.

In pandas, we basically do this to the function,

    if numba.extending.is_jitted(func):
        # Don't jit a user passed jitted function
        numba_func = func
    else:
        numba_func = numba.extending.register_jitable(func)

off the top of your head, do you know if there is anything wrong with the way now?

The code from the OP is proably going through the register_jitable path so that might be the problem.
(I haven’t checked though, I need to get numba 0.58.0 installed)

IIRC, we don’t use overload in the window code for pandas, maybe this error is coming from np.prod being implemented internally with overload?

esc · September 29, 2023, 11:59am

Off the top of my head I do not know what could be wrong here.

I have a feeling this is a followup regression from:

github.com/pandas-dev/pandas

Review by stuartarchibald in ENH: Add numba engine for rolling apply

pandas-dev:master ← mroeschke:numba_rolling_apply

@mroeschke no problem, I can try and help with this. I think it needs to look a …bit like this (for reference, this is untested, I am just guessing from the context! Also, the `pandas` variant is obviously wrapped to close over some configuration which I've omitted, so consider this as the function body of `make_rolling_apply`. I've left comments inline to try and explain what's going on): ```python import types import numpy as np from numba.extending import overload, is_jitted from numba import njit import numba # this provides a local definition to overload def overload_target(window, *_args): # If JIT is disabled, this function will run, so write the implementation here! pass nopython = True nogil = True parallel = False # pretend this is an arg to `make_rolling_apply` def func(window, *args): return window * 2 + args[0] @overload(overload_target, jit_options={'nopython':nopython, 'nogil':nogil, 'parallel':parallel}) def ol_overload_target(window, *_args): # This function "overloads" `overload_target`, whenever the Numba compiler # "sees" `overload_target` it will use this function. # Using `is_jitted` to avoid `isinstance` on # `numba.targets.registry.CPUDispatcher` as that may be considered an # internal Numba detail. if is_jitted(func): # it's already JIT compiled so just reference it overload_target_impl = func elif getattr(np, func.__name__, False) is func or isinstance( func, types.BuiltinFunctionType ): # it's a NumPy function or builtin so just reference it overload_target_impl = func else: # it's a Python function, so register it as JIT compilable and reference # that overload_target_impl = numba.jit(func, nopython=nopython, nogil=nogil) # This is the Numba implementation of the overload, it will just be JIT # compiled whenever the compiler "sees" a reference to "overload_target" in # code it is compiling. def impl(window, *_args): return overload_target_impl(window, *_args) return impl # demo @njit def roll_apply(window, *_args): return overload_target(window, *_args) print(roll_apply(np.arange(10.), 1.23)) ``` `@overload` is basically saying to Numba "when you see this specific python function (the one in the first argument in the `@overload` decorator) use this implementation". The concept about there being a "typing" part that can be used to dispatch different variants based on type is exactly the same as in `@generated_jit`. The largest difference is what happens if the JIT compiler is turned off. In the case of `@overload` the python function being overloaded will run, i.e., the code just executes as would be expected in the interpreter. Whereas in the case of `@generated_jit`, because the pure python implementation and the Numba implementation are the same function, if you turn the JIT compiler off it will just break (the value returned when calling a `@generated_jit` function is a function implementing the Numba specialisation). Essentially, `@generated_jit` is like doing `@overload` but the function being decorated is also the function being overloaded. Hope this helps?

Perhaps register_jitable isn’t the right thing to do. But I’m not sure what the alternative is here.

It’ll probably be one of these places:

 💣 zsh» git --no-pager grep -C 5 register_jitable
pandas/core/_numba/executor.py-def generate_apply_looper(func, nopython=True, nogil=True, parallel=False):
pandas/core/_numba/executor.py-    if TYPE_CHECKING:
pandas/core/_numba/executor.py-        import numba
pandas/core/_numba/executor.py-    else:
pandas/core/_numba/executor.py-        numba = import_optional_dependency("numba")
pandas/core/_numba/executor.py:    nb_compat_func = numba.extending.register_jitable(func)
pandas/core/_numba/executor.py-
pandas/core/_numba/executor.py-    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)
pandas/core/_numba/executor.py-    def nb_looper(values, axis):
pandas/core/_numba/executor.py-        # Operate on the first row/col in order to get
pandas/core/_numba/executor.py-        # the output shape
--
pandas/core/_numba/kernels/sum_.py-    TYPE_CHECKING,
pandas/core/_numba/kernels/sum_.py-    Any,
pandas/core/_numba/kernels/sum_.py-)
pandas/core/_numba/kernels/sum_.py-
pandas/core/_numba/kernels/sum_.py-import numba
pandas/core/_numba/kernels/sum_.py:from numba.extending import register_jitable
pandas/core/_numba/kernels/sum_.py-import numpy as np
pandas/core/_numba/kernels/sum_.py-
pandas/core/_numba/kernels/sum_.py-if TYPE_CHECKING:
pandas/core/_numba/kernels/sum_.py-    from pandas._typing import npt
pandas/core/_numba/kernels/sum_.py-
--
pandas/core/_numba/kernels/sum_.py-
pandas/core/_numba/kernels/sum_.py-    return output, na_pos
pandas/core/_numba/kernels/sum_.py-
pandas/core/_numba/kernels/sum_.py-
pandas/core/_numba/kernels/sum_.py-# Mypy/pyright don't like the fact that the decorator is untyped
pandas/core/_numba/kernels/sum_.py:@register_jitable  # type: ignore[misc]
pandas/core/_numba/kernels/sum_.py-def grouped_kahan_sum(
pandas/core/_numba/kernels/sum_.py-    values: np.ndarray,
pandas/core/_numba/kernels/sum_.py-    result_dtype: np.dtype,
pandas/core/_numba/kernels/sum_.py-    labels: npt.NDArray[np.intp],
pandas/core/_numba/kernels/sum_.py-    ngroups: int,
--
pandas/core/util/numba_.py-
pandas/core/util/numba_.py-    if numba.extending.is_jitted(func):
pandas/core/util/numba_.py-        # Don't jit a user passed jitted function
pandas/core/util/numba_.py-        numba_func = func
pandas/core/util/numba_.py-    else:
pandas/core/util/numba_.py:        numba_func = numba.extending.register_jitable(func)
pandas/core/util/numba_.py-
pandas/core/util/numba_.py-    return numba_func

If you set some breakpoint()s there I’m fairly convinced you’d end up in one of those places.

kartiksubbarao · September 29, 2023, 1:12pm

@esc Yes I am running pandas 2.1.1. Thanks for your help in identifying the source of this issue!

lithomas1 · September 29, 2023, 7:15pm

Interestingly enough, this problem didn’t reproduce with numpy 1.24. I upgraded to 1.25 and got the warning, same as OP.

Maybe something about the numpy ufuncs changed in the meantime to make numba raise the warning?

In the meantime, I think a good solution for pandas is just to not jit builtins and np functions as mentioned in the PR you linked.

Thanks for all the help by the way!

Topic		Replies	Views
Numba upgrade from 0.54.1 to 0.57 - differing behavior Support: What is this error message?	0	522	June 14, 2023
Calling Numba function on pandas Series Community Support	2	2653	November 28, 2021
Is this intended behaviour Community Support	1	548	March 13, 2021
Hard to debug new errors under 0.59rc1 Support: What is this error message?	5	212	January 31, 2024
Helping test Numba 0.53.0 RC Announcements	36	2402	March 11, 2021

Pandas: source of "old_style" error-capturing warning

Related topics