A new repo for "extra" features

Announcements
1

Following up on a topic from the Sept 8 public meeting and the continuation of the discussion in the Sept 15 developer meeting, we are planning to setup an “extras” repository. Initially, the new repo is for features that need time to mature and stabilize, but it may grow into something more.

The new repo is aim to:

  • Reduce the barrier to contribute.
    Compiler features can be hard to design and test. Designs may need to be change after some time and usage in the wild. Features may interfere with others in ways that are hard to anticipate in code-review. The new repo can provide a ground for new features to mature.
  • Isolate unstable features.
    The new repo will provide a new Python package, which becomes an opt-in mechanic for these features. The core Numba repo can provide a higher stability guarantee without impeding the growth of new features.
  • Help Numba to become more extensible.
    The new repo can highlight the weaknesses of the extension API.
  • Increase community involvement.
    e.g. community code owners/managers for the new repo.

There will be coding conventions or technical restrictions for the new repo. So far, we have considered the followings:

  • It will be a pure-python repo. Any feature that requires native code is likely too complicated and should consider contributing to the core repo or start a new project.
  • Code is limited to use only the Numba extension API and not any internal Numba code. This will help Numba to better define the externsion API.
  • No additional required dependencies can be added, but optional dependencies are allowed.

At this time, we are still figuring out the details and the Numba developer is planning to bootstrap the repo with a few unstable features in Numba.

Lastly, we are considering the following names for the new repo:

  • numba-universe, numba-multiverse
  • numba-unstable, numba-experimental
  • numba-addons, numba-extras, numba-contrib, numba-boltons, numba-xtra

(I grouped the names into 3 categories based on their similarities.)

Please let us know your thoughts and any feedback for the plan and the names.

1 Like
2

Potential candidate - This jitclass annotations hack

@jitclass
class Foo:
    x: numba.int32 = 42
    y: numba.double

    def __init__(self):
        self.y = 4.2

@njit
def f():
    foo = Foo()
    return foo.x, foo.y

assert f() == (42, 4.2)


import inspect
import typing

import numba
from numba import njit
from numba.experimental import jitclass as _jitclass
from numba.experimental.jitclass.base import JitClassType

_annotations = {}


def jitclass(cls: typing.Type) -> typing.Type:
    if numba.config.DISABLE_JIT:
        return cls

    # print()
    # print(cls)

    try:
        annos = cls.__annotations__
    except AttributeError:
        spec = {}
    else:
        spec = dict(annos)
        del cls.__annotations__
        _annotations[cls.__qualname__] = spec

    values = {}
    for name, typ in spec.items():
        try:
            value = getattr(cls, name)
        except AttributeError:
            pass
        else:
            values[name] = value
            delattr(cls, name)

        if isinstance(typ, JitClassType):
            spec[name] = typ.class_type.instance_type

    if values:
        sep = "\n "

        body = ""
        init_locals = {}

        for name, value in values.items():
            newname = f"__init__local__{name}"
            body += sep + f"self.{name} = {newname}"
            init_locals[newname] = value

        if isinstance(cls.__init__, type(object.__init__)):
            cls__init__sig = "(self)"
        else:
            init_locals["__cls__init__"] = njit(cls.__init__)

            signature = inspect.signature(cls.__init__)
            param_names = ", ".join(signature.parameters)
            cls__init__sig = f"({param_names})"

            body += sep + f"__cls__init__{cls__init__sig}"

        code = f"def __jitcls__init__{cls__init__sig}:{body}"

        # print(code)
        # print()

        exec(code, init_locals)

        cls.__init__ = init_locals["__jitcls__init__"]

    return _jitclass(spec)(cls)
3

We need to start making the new repo. But first, it needs a name so we don’t have to go through the rename progress later.

Please pick your favourite name for the new repo
  • numba-universe
  • numba-multiverse
  • numba-unstable
  • numba-experimental
  • numba-addons
  • numba-extras
  • numba-contrib
  • numba-boltons
  • numba-xtra

0 voters

4

The poll will be closed in a week just before the next numba dev meeting.

5

It’s a shame, that numba-bazaar, numba-jukyard and numba-trashcan are missing :kissing_heart:

1 Like
6

The poll will be closed at 10:25am US central time. So less than 1 hour left.

(This is so exciting because admin/mods can’t see the result either until then)

1 Like
7

my preferred option didn’t win, so I demand a recount. It was clearly rigged in favour of “extra” given the name of the thread :slight_smile:

1 Like
8

Hey everybody. I think it is a good idea to have a repository for code beside the main numba repository. So when are you going to create the repo?

I could contribute “prefetching” issues/#3769 and code for fast file reading (using the operating system’s open/read functions).

9

both sound very interesting, looking forward to seeing them in the new repo!

10

Quick update. This is being worked on, just setting up infrastructure etc. Should be available soon.

11

The repo is at https://github.com/numba/numba-extras.

It’s still very early. What is has now:

  • CI based on GH actions to build and test conda-packages.
  • coding-style enforced with black in pre-commit and in CI.
  • pytest
  • a “helloworld” example to show the basic structure for new features

The repo will need:

  • documentation and contributing guidelines
  • some type-annotation testing (hoping for contributions to help us with this)
  • (probably more things as we go)