A new repo for "extra" features

Following up on a topic from the Sept 8 public meeting and the continuation of the discussion in the Sept 15 developer meeting, we are planning to setup an “extras” repository. Initially, the new repo is for features that need time to mature and stabilize, but it may grow into something more.

The new repo is aim to:

  • Reduce the barrier to contribute.
    Compiler features can be hard to design and test. Designs may need to be change after some time and usage in the wild. Features may interfere with others in ways that are hard to anticipate in code-review. The new repo can provide a ground for new features to mature.
  • Isolate unstable features.
    The new repo will provide a new Python package, which becomes an opt-in mechanic for these features. The core Numba repo can provide a higher stability guarantee without impeding the growth of new features.
  • Help Numba to become more extensible.
    The new repo can highlight the weaknesses of the extension API.
  • Increase community involvement.
    e.g. community code owners/managers for the new repo.

There will be coding conventions or technical restrictions for the new repo. So far, we have considered the followings:

  • It will be a pure-python repo. Any feature that requires native code is likely too complicated and should consider contributing to the core repo or start a new project.
  • Code is limited to use only the Numba extension API and not any internal Numba code. This will help Numba to better define the externsion API.
  • No additional required dependencies can be added, but optional dependencies are allowed.

At this time, we are still figuring out the details and the Numba developer is planning to bootstrap the repo with a few unstable features in Numba.

Lastly, we are considering the following names for the new repo:

  • numba-universe, numba-multiverse
  • numba-unstable, numba-experimental
  • numba-addons, numba-extras, numba-contrib, numba-boltons, numba-xtra

(I grouped the names into 3 categories based on their similarities.)

Please let us know your thoughts and any feedback for the plan and the names.

1 Like

Potential candidate - This jitclass annotations hack

@jitclass
class Foo:
    x: numba.int32 = 42
    y: numba.double

    def __init__(self):
        self.y = 4.2

@njit
def f():
    foo = Foo()
    return foo.x, foo.y

assert f() == (42, 4.2)

import inspect
import typing

import numba
from numba import njit
from numba.experimental import jitclass as _jitclass
from numba.experimental.jitclass.base import JitClassType

_annotations = {}


def jitclass(cls: typing.Type) -> typing.Type:
    if numba.config.DISABLE_JIT:
        return cls

    # print()
    # print(cls)

    try:
        annos = cls.__annotations__
    except AttributeError:
        spec = {}
    else:
        spec = dict(annos)
        del cls.__annotations__
        _annotations[cls.__qualname__] = spec

    values = {}
    for name, typ in spec.items():
        try:
            value = getattr(cls, name)
        except AttributeError:
            pass
        else:
            values[name] = value
            delattr(cls, name)

        if isinstance(typ, JitClassType):
            spec[name] = typ.class_type.instance_type

    if values:
        sep = "\n "

        body = ""
        init_locals = {}

        for name, value in values.items():
            newname = f"__init__local__{name}"
            body += sep + f"self.{name} = {newname}"
            init_locals[newname] = value

        if isinstance(cls.__init__, type(object.__init__)):
            cls__init__sig = "(self)"
        else:
            init_locals["__cls__init__"] = njit(cls.__init__)

            signature = inspect.signature(cls.__init__)
            param_names = ", ".join(signature.parameters)
            cls__init__sig = f"({param_names})"

            body += sep + f"__cls__init__{cls__init__sig}"

        code = f"def __jitcls__init__{cls__init__sig}:{body}"

        # print(code)
        # print()

        exec(code, init_locals)

        cls.__init__ = init_locals["__jitcls__init__"]

    return _jitclass(spec)(cls)