Helping test Numba 0.53.0 RC

The experiment undertaken during the last Numba release of getting sign-offs from open source projects was very successful (thanks to all those who contributed). We are therefore again inviting the broader community and downstream projects to help us test the Numba 0.53.0 and llvmlite 0.36.0 release candidates.

The following projects have previously reported regressions and so we’d appreciate it if the maintainers of these projects could please test the RCs and report successes into this thread and regressions as issues on github.

  • Awkward
  • UMAP
  • librosa
  • RAPIDS
  • SDC
  • pydata sparse
  • clifford

For the wider community, following testing the RCs, please could you report the project you are working on (if you can) and success on this thread and regressions as issues on github.

We are currently at rc1 for conda packages and rc1.post1 for wheels (had to fix an issue in wheel builds). Conda packages are in the numba channel. Wheels are available on PyPI.

Installation commands:

Conda:


$ conda install -c numba numba=0.53.0rc1

Pip:


$ pip install numba==0.53.0rc1.post1

Thanks in advance to everyone for helping us test Numba.

I am happy to report that all tests for umap (and related pynndescent) PASS using Python 3.9 and numba==0.53.0rc1

Here there is the detailed log for UMAP: #572

Here is the log for pynndescent:

================================================================================= test session starts =================================================================================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/valerio/Research/UoB/repos/umap/pynndescent
collected 69 items

test_distances.py .........................................                                                                                                                     [ 59%]
test_pynndescent_.py .....................                                                                                                                                      [ 89%]
test_rank.py .......                                                                                                                                                            [100%]
===================================================================== 69 passed, 2 warnings in 115.16s (0:01:55) ======================================================================

Thank you @leriomaggio – can you say which versions of UMAP and pynndescent this was?

Oh sure, sorry! Forgot to say they were both tested considering the latest stable (i.e. umap|pynndescent==0.5.1) and the master branch (latest dev).

All works with no error!

@stuartarchibald following up to Numba 0.53.0 and llvmlite 0.36.0 Release Candidates - #6 by ryanchien.

Hopefully this illustrates the break.

#Numba 0.52.0, llvmlite 0.35.0

@nb.njit('Tuple((f8, f8)) ()', fastmath = True, nogil = True)
def foo():
    increment = 0.0
    foo_sum = 0.0
    i = 2
    
    while foo_sum < 100:
        a = [np.random.normal(1) for _ in range(i)]
        a = np.array(a)
        increment = np.sum(a)
        increment = np.abs(increment)
        foo_sum += increment
        i += 1
    
    return (foo_sum, i)

foo()
> (119.8389803524032, 16.0)
# Numba 0.53.0rc1, llvmlite 0.36.0rc1

@nb.njit('Tuple((f8, f8)) ()', fastmath = True, nogil = True)
def foo():
    increment = 0.0
    foo_sum = 0.0
    i = 2
    
    while foo_sum < 100:
        a = [np.random.normal(1) for _ in range(i)]
        a = np.array(a)
        increment = np.sum(a)
        increment = np.abs(increment)
        foo_sum += increment
        i += 1
    
    return (foo_sum, i)

foo()

> ---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
<ipython-input-16-9ae2843261d2> in <module>
      1 @nb.njit('Tuple((f8, f8)) ()', fastmath = True, nogil = True)
----> 2 def foo():
      3     increment = 0.0
      4     foo_sum = 0.0
      5     i = 2

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/decorators.py in wrapper(func)
    219             with typeinfer.register_dispatcher(disp):
    220                 for sig in sigs:
--> 221                     disp.compile(sig)
    222                 disp.disable_compile()
    223         return disp

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/dispatcher.py in compile(self, sig)
    907                 with ev.trigger_event("numba:compile", data=ev_details):
    908                     try:
--> 909                         cres = self._compiler.compile(args, return_type)
    910                     except errors.ForceLiteralArg as e:
    911                         def folded(args, kws):

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/dispatcher.py in compile(self, args, return_type)
     81             return retval
     82         else:
---> 83             raise retval
     84 
     85     def _compile_cached(self, args, return_type):

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/dispatcher.py in _compile_cached(self, args, return_type)
     91 
     92         try:
---> 93             retval = self._compile_core(args, return_type)
     94         except errors.TypingError as e:
     95             self._failed_cache[key] = e

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/dispatcher.py in _compile_core(self, args, return_type)
    104 
    105         impl = self._get_implementation(args, {})
--> 106         cres = compiler.compile_extra(self.targetdescr.typing_context,
    107                                       self.targetdescr.target_context,
    108                                       impl,

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler.py in compile_extra(typingctx, targetctx, func, args, return_type, flags, locals, library, pipeline_class)
    604     pipeline = pipeline_class(typingctx, targetctx, library,
    605                               args, return_type, flags, locals)
--> 606     return pipeline.compile_extra(func)
    607 
    608 

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler.py in compile_extra(self, func)
    351         self.state.lifted = ()
    352         self.state.lifted_from = None
--> 353         return self._compile_bytecode()
    354 
    355     def compile_ir(self, func_ir, lifted=(), lifted_from=None):

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler.py in _compile_bytecode(self)
    413         """
    414         assert self.state.func_ir is None
--> 415         return self._compile_core()
    416 
    417     def _compile_ir(self):

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler.py in _compile_core(self)
    393                 self.state.status.fail_reason = e
    394                 if is_final_pipeline:
--> 395                     raise e
    396         else:
    397             raise CompilerError("All available pipelines exhausted")

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler.py in _compile_core(self)
    384             res = None
    385             try:
--> 386                 pm.run(self.state)
    387                 if self.state.cr is not None:
    388                     break

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler_machinery.py in run(self, state)
    337                     (self.pipeline_name, pass_desc)
    338                 patched_exception = self._patch_error(msg, e)
--> 339                 raise patched_exception
    340 
    341     def dependency_analysis(self):

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler_machinery.py in run(self, state)
    328                 pass_inst = _pass_registry.get(pss).pass_inst
    329                 if isinstance(pass_inst, CompilerPass):
--> 330                     self._runPass(idx, pass_inst, state)
    331                 else:
    332                     raise BaseException("Legacy pass in use")

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     33         def _acquire_compile_lock(*args, **kwargs):
     34             with self:
---> 35                 return func(*args, **kwargs)
     36         return _acquire_compile_lock
     37 

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler_machinery.py in _runPass(self, index, pss, internal_state)
    287             mutated |= check(pss.run_initialization, internal_state)
    288         with SimpleTimer() as pass_time:
--> 289             mutated |= check(pss.run_pass, internal_state)
    290         with SimpleTimer() as finalize_time:
    291             mutated |= check(pss.run_finalizer, internal_state)

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/compiler_machinery.py in check(func, compiler_state)
    260 
    261         def check(func, compiler_state):
--> 262             mangled = func(compiler_state)
    263             if mangled not in (True, False):
    264                 msg = ("CompilerPass implementations should return True/False. "

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/typed_passes.py in run_pass(self, state)
    102                               % (state.func_id.func_name,)):
    103             # Type inference
--> 104             typemap, return_type, calltypes, errs = type_inference_stage(
    105                 state.typingctx,
    106                 state.func_ir,

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/typed_passes.py in type_inference_stage(typingctx, interp, args, return_type, locals, raise_errors)
     81         # return errors in case of partial typing
     82         errs = infer.propagate(raise_errors=raise_errors)
---> 83         typemap, restype, calltypes = infer.unify(raise_errors=raise_errors)
     84 
     85     # Output all Numba warnings

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/typeinfer.py in unify(self, raise_errors)
   1202         others = set(self.typevars) - temps
   1203         for var in sorted(others):
-> 1204             check_var(var)
   1205         for var in sorted(temps):
   1206             check_var(var)

/opt/conda/envs/numbastats/lib/python3.8/site-packages/numba/core/typeinfer.py in check_var(name)
   1172                     msg = ("Type of variable '%s' cannot be determined, "
   1173                            "operation: %s, location: %s")
-> 1174                     raise TypingError(msg % (var, val, loc), loc)
   1175                 else:
   1176                     typdict[var] = types.unknown

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Type of variable 'a' cannot be determined, operation: call $42load_method.9(a, func=$42load_method.9, args=[Var(a, <ipython-input-16-9ae2843261d2>:8)], kws=(), vararg=None), location: <ipython-input-16-9ae2843261d2> (9)

File "<ipython-input-16-9ae2843261d2>", line 9:
def foo():
    <source elided>
        a = [np.random.normal(1) for _ in range(i)]
        a = np.array(a)
        ^

Both images are otherwise identical: CentOS7, icc_rt 2020.2, conda 4.8.3, running on Docker 19.03.13.

Hope this helps!

The solution:

# Numba 0.53.0rc1, llvmlite 0.36.0rc1

@nb.njit('Tuple((f8, f8)) ()', fastmath = True, nogil = True)
def foo():
    increment = 0.0
    foo_sum = 0.0
    i = 2
    
    while foo_sum < 100:
        a = [np.random.normal(1) for _ in range(i)]
        a = np.array(a).astype(np.float64)    # <- solution
        increment = np.sum(a)
        increment = np.abs(increment)
        foo_sum += increment
        i += 1
    
    return (foo_sum, i)

foo()
> (102.77047920529394, 16.0)

@ryanchien many thanks for testing and also creating a reproducer for the regression, I can replicate locally. Regression in 0.53.0rc1 due to issue in temp variable assignment removal · Issue #6715 · numba/numba · GitHub is tracking, I think I’ve found the patch that caused it.

Excellent! Thank you.

I’d like to report that all 1K+ tests passed for stumpy=1.8.0 (latest) using python=3.7.9, python=3.8.5, and python=3.9.1 along with numba==0.53.0rc1. In case it matters, STUMPY has 4 cuda.jit functions as well as 25 njit functions.

Great, thanks for testing @seanlaw.

1 Like

All poliastro tests passed on Python 3.9 and the latest RC :+1:

Brilliant, thanks for confirming @astrojuanlu

1 Like

RC2 is now available and it is ready for testing.

All tests pass for my internal stats library using numba/llvmlite rc2. Thanks!

I am a little late to RC2 testing and I’m getting a:

PackagesNotFoundError: The following packages are not available from current channels:

  - numba=0.53.0rc2

Has it already been removed or am I doing something wrong here? This was from:

conda install -y -c numba numba=0.53.0rc2

Has it already been removed or am I doing something wrong here?

Numba 0.53.0rc3 is being built at the moment, and packages for some platforms are available - is 0.53.0rc3 installable for you?

1 Like

We haven’t removed any packages. If this problem continues, please let us know which OS, python version and numpy version.

1 Like

RC3 is available now:

Oops! It was totally my fault. I was trying to install RC2 from my Mac ARM64 machine and explains why it could not be found. I switched over to my Mac x86_64 machine and all is well! Sorry for the unnecessary noise

I’d like to report that all tests passed for:

stumpy=1.8.0 (latest), python=3.9.2, and numba=0.53.0rc2
stumpy=1.8.0 (latest), python=3.9.2, and numba=0.53.0rc3