# 50% performance drop from py 3.8/numba 0.51.2 to py 3.9/numba 0.55.1

**URL:** <https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526>\
**Category:** Community Support\
**Created:** [August 26, 2022, 8:46am UTC](https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526 "2022-08-26T08:46:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jordannickerson](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/jordannickerson/32/686_2.png) [@jordannickerson](https://numba.discourse.group/u/jordannickerson)\
**Post date:** [August 26, 2022, 8:46am UTC](https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526/1 "2022-08-26T08:46:57Z")

</div>

I have noticed a very large performance drop when executing the same code on identical hardware under two versions of Numba. The meat of the code performs a modified version of a dot product (implemented via loops, not external functions) on subsets of a very large array.

Setup 1:  
Anaconda 3.8.5  
numba 0.51.2  
numpy 1.19.2  
llvmlite 0.34.0  
mkl 2020.2  
Runtime: 248 seconds

Setup 2:  
Anaconda 3.9.12  
numba 0.55.1  
numpy 1.21.5  
llvmlite 0.38.0  
mkl 2021.4.0  
Runtime: 361 seconds

I was very shocked to see such a large decrease in performance. The code is being executed on identical hardware across two systems. I verified performance using CPU-Z, which yielded almost identical benchmark results.

Is there a way to identify potential library conflicts, etc. which might be leading to such a big performance drop?

---

<div class="post-metadata">

**Author:** ![max9111](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/max9111/32/538_2.png) [@max9111](https://numba.discourse.group/u/max9111)\
**Post date:** [August 26, 2022, 9:19am UTC](https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526/2 "2022-08-26T09:19:44Z")

</div>

According to your description it could be related to the following bug.

> <https://github.com/numba/numba/issues/8172#issue-1275299445>
>
> This bug first occurs in version 0.54 (0.49-0.53 are OK). This results in a perf…ormance decrease of a factor of 4 in a bit optimized summation and a factor of 2 in a naive implementation.
> 
> \*\*Example\*\*
> 
> \`\`\`python
> import numpy as np
> import numba as nb
> import time
> 
> 
> f = np.random.rand(32,33,2000,64)
> b = np.random.rand(32,64)
> 
> @nb.njit(fastmath=True)
> def test\_numba\_1(f,b):
> fnew = np.zeros((f.shape\[1\],f.shape\[2\]))
>     
> for i in range(f.shape\[0\]):
> for j in range(f.shape\[1\]):
> for k in range(f.shape\[2\]):
> for l in range(f.shape\[3\]):
> fnew\[j,k\] += f\[i,j,k,l\] \* b\[i,l\]
> return fnew
> 
> @nb.njit(fastmath=True)
> def test\_numba\_2(f,b):
> fnew = np.zeros((f.shape\[1\],f.shape\[2\])) 
>     
> for i in range(f.shape\[0\]):
> for j in range(f.shape\[1\]):
> for k in range(f.shape\[2\]):
> acc=fnew\[j,k\]
> for l in range(f.shape\[3\]):
> acc += f\[i,j,k,l\] \* b\[i,l\]
> fnew\[j,k\]=acc
> return fnew
> 
> %timeit test\_numba\_1(f,b)
> %timeit test\_numba\_2(f,b)
> 
> \#'0.53.1' Last version without bug
> \#108 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> \#53.9 ms ± 809 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
> 
> \#'0.54.0' First version with bug
> \#216 ms ± 4.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> \#200 ms ± 2.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> 
> \#'0.55.0'
> \#218 ms ± 5.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> \#201 ms ± 1.73 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> \`\`\`
> 
> Another question would be if, a rewrite pass is possible in Numba to optimize \`test\_numba\_1\` automatically to \`test\_numba\_2\`. Clang shows exactly the same behavior as Numba before version 0.54, therefore this likely has to be fixed before generating LLVM-IR.
> 
> \`\`\`c
> //clang v11
> //clang -shared -O3 -march=native -ffast-math -o summation.dll summation.c
> \_\_declspec(dllexport) void eq\_nb\_1(size\_t s0,size\_t s1,size\_t s2,size\_t s3,double f\[\]\[s1\]\[s2\]\[s3\],double b\[\]\[s3\],double fnew\[\]\[s2\])
> {
> for (size\_t i=0;i\<s0;i++){
> for (size\_t j=0;j\<s1;j++){
> for (size\_t k=0;k\<s2;k++){
> for (size\_t l=0;l\<s3;l++){
> fnew\[j\]\[k\]+=f\[i\]\[j\]\[k\]\[l\]\*b\[i\]\[l\];
> }
> }
> }
> }
> }
> 
> \_\_declspec(dllexport) void eq\_nb\_2(size\_t s0,size\_t s1,size\_t s2,size\_t s3,double f\[\]\[s1\]\[s2\]\[s3\],double b\[\]\[s3\],double fnew\[\]\[s2\])
> {
> for (size\_t i=0;i\<s0;i++){
> for (size\_t j=0;j\<s1;j++){
> for (size\_t k=0;k\<s2;k++){
> double acc=fnew\[j\]\[k\];
> for (size\_t l=0;l\<s3;l++){
> acc+=f\[i\]\[j\]\[k\]\[l\]\*b\[i\]\[l\];
> }
> fnew\[j\]\[k\]=acc;
> }
> }
> }
> }
> \`\`\`
> \`\`\`python
> import numpy as np
> import ctypes
> 
> lib = ctypes.cdll.LoadLibrary("summation.dll")
> 
> def eq\_nb\_1(f,b):
> fnew = np.zeros((f.shape\[1\],f.shape\[2\]))
> s\_0=ctypes.c\_size\_t(f.shape\[0\])
> s\_1=ctypes.c\_size\_t(f.shape\[1\])
> s\_2=ctypes.c\_size\_t(f.shape\[2\])
> s\_3=ctypes.c\_size\_t(f.shape\[3\])
>     
> f\_p =f.ctypes
> b\_p =b.ctypes
> fnew\_p=fnew.ctypes
>     
> lib.eq\_nb\_1(s\_0,s\_1,s\_2,s\_3,f\_p,b\_p,fnew\_p)
> return fnew
> 
> def eq\_nb\_2(f,b):
> fnew = np.zeros((f.shape\[1\],f.shape\[2\]))
> s\_0=ctypes.c\_size\_t(f.shape\[0\])
> s\_1=ctypes.c\_size\_t(f.shape\[1\])
> s\_2=ctypes.c\_size\_t(f.shape\[2\])
> s\_3=ctypes.c\_size\_t(f.shape\[3\])
>     
> f\_p =f.ctypes
> b\_p =b.ctypes
> fnew\_p=fnew.ctypes
>     
> lib.eq\_nb\_2(s\_0,s\_1,s\_2,s\_3,f\_p,b\_p,fnew\_p)
> return fnew
> 
> %timeit eq\_nb\_1(f,b)
> %timeit eq\_nb\_2(f,b)
> \#100 ms ± 311 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
> \#50.5 ms ± 1.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
> \`\`\`

---

<div class="post-metadata">

**Author:** ![jordannickerson](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/jordannickerson/32/686_2.png) [@jordannickerson](https://numba.discourse.group/u/jordannickerson)\
**Post date:** [August 26, 2022, 9:59pm UTC](https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526/3 "2022-08-26T21:59:58Z")

</div>

Thanks! I did a little more investigating following your thought. To try and pinpoint the specific version where performance dropped, i created a series of new virtual environments with different numba versions to see where the slow down first occurs. Each environment had numpy, scipy, and numba (along with dependencies).

The specific performance hit occurs when going from numba 0.51 → 0.52, and persists in all versions after that. Hopefully this helps!

---

<div class="post-metadata">

**Author:** ![gmarkall](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/gmarkall/32/42_2.png) [@gmarkall](https://numba.discourse.group/u/gmarkall)\
**Post date:** [August 30, 2022, 8:52am UTC](https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526/4 "2022-08-30T08:52:03Z")

</div>

As it seems like it might be a separate issue, could you open an issue with the code that demonstrates the performance regression please? You can open an issue here: [Issues · numba/numba · GitHub](https://github.com/numba/numba/issues)

---

<div class="post-metadata">

**Author:** ![gmarkall](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/gmarkall/32/42_2.png) [@gmarkall](https://numba.discourse.group/u/gmarkall)\
**Post date:** [August 30, 2022, 2:13pm UTC](https://numba.discourse.group/t/50-performance-drop-from-py-3-8-numba-0-51-2-to-py-3-9-numba-0-55-1/1526/5 "2022-08-30T14:13:59Z")

</div>

Ah, I see there is an issue already: [50% performance drop from py3.8/numba 0.51.2 to py3.9/numba 0.55.1 · Issue #8398 · numba/numba · GitHub](https://github.com/numba/numba/issues/8398) - I must have missed the mention of this earlier in the thread.