Temporary typing context / data model?

Numba Development
1

Hi, so the official way to add support for types is numba.extending.register_model, which is irreversible.

We would like to have an extension active only for our @njited functions, but not all other packages’ code that uses numba.

How would we do that?

2

Would it be possible to see a minimal example of what you are trying to accomplish? Thanks.

3

@flying-sheep, I can understand changing data-model for different targets but it is unusual to change for a single function. Can you share more details about your use-case?

e.g.

  • Do you need different calling-convention (different machine representation at the function boundary)?
  • Do you need the equivalent of C-union?
4

Again, I want an extension to be active for my library’s numba functions, but not for anyone else’s.

Specifically, I’m developing fast-array-utils, a library designed to be used by many other projects. fast-array-utils works with scipy.sparse.cs{rc}_{matrix,array}s, so it contains a rudimentary numba extension that mostly allows to access the classes’ fields, and not more.

Projects depending on fast-array-utils therefore shouldn’t get access to that extension. If they want to use a more complete extension for these types, they should be able to do so.

5

As I understand it, your goal is to prevent projects from outside fast-array-utils from using an extension that you already have working inside fast-array-utils. Is that accurate?

6

OK, let’s start over!

Numba supports extensions. These extensions allow for numba to handle Python classes that an unextended numba has no support for. (I.e. without an exception, passing a scipy.sparse.csr_array into a numba function raises a TypingError)

I’m developing a library called fast-array-utils. A lot of its functions take scipy.sparse.cs{cr}_{array,matrix}s, just to access these classes’ common attributes (which are numpy arrays), nothing else. For convenience, I therefore developed a rudimentary numba extension that allows fast-array-utils’ functions to have objects of these types passed to them.

Since it’s so rudimentary and handles 3rd party types, I don’t want other projects to have the extension active by default. When someone tries to pass in these types to a numba function not from fast-array-utils, that numba function should behave exactly as if fast-array-utils’ extension had never been registered: if someone activates another extension for the scipy.sparse types, that one should be used! If none is active, passing one of these types to a numba function not from fast-array-utils should result in

TypingError: […] Cannot determine Numba type of <class ‘scipy.sparse._csr.csr_matrix’>

7

Have you tried rolling out and pointing at a custom registry, in which you can in turn intercept the lookup of what had been registered for the numba’s context?

8

That might be what I’m searching for. How would I do that?

I see

  • numba.core.registry:cpu_target
  • numba.core.datamodel.registry:default_manager

How are they related? How would I inherit from whatever I need to? How would I point njit at my derived registry so it’d have access to all the default stuff plus my extension?