50% performance drop from py 3.8/numba 0.51.2 to py 3.9/numba 0.55.1

I have noticed a very large performance drop when executing the same code on identical hardware under two versions of Numba. The meat of the code performs a modified version of a dot product (implemented via loops, not external functions) on subsets of a very large array.

Setup 1:
Anaconda 3.8.5
numba 0.51.2
numpy 1.19.2
llvmlite 0.34.0
mkl 2020.2
Runtime: 248 seconds

Setup 2:
Anaconda 3.9.12
numba 0.55.1
numpy 1.21.5
llvmlite 0.38.0
mkl 2021.4.0
Runtime: 361 seconds

I was very shocked to see such a large decrease in performance. The code is being executed on identical hardware across two systems. I verified performance using CPU-Z, which yielded almost identical benchmark results.

Is there a way to identify potential library conflicts, etc. which might be leading to such a big performance drop?

According to your description it could be related to the following bug.

Thanks! I did a little more investigating following your thought. To try and pinpoint the specific version where performance dropped, i created a series of new virtual environments with different numba versions to see where the slow down first occurs. Each environment had numpy, scipy, and numba (along with dependencies).

The specific performance hit occurs when going from numba 0.51 → 0.52, and persists in all versions after that. Hopefully this helps!

As it seems like it might be a separate issue, could you open an issue with the code that demonstrates the performance regression please? You can open an issue here: Issues · numba/numba · GitHub

Ah, I see there is an issue already: 50% performance drop from py3.8/numba 0.51.2 to py3.9/numba 0.55.1 · Issue #8398 · numba/numba · GitHub - I must have missed the mention of this earlier in the thread.