# Performance discrepancy between jit/generated\_jit/overload

**URL:** <https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671>\
**Category:** Community Support\
**Created:** [April 14, 2021, 9:12am UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671 "2021-04-14T09:12:52Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![JSKenyon](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/jskenyon/32/301_2.png) [@JSKenyon](https://numba.discourse.group/u/JSKenyon)\
**Post date:** [April 14, 2021, 9:12am UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/1 "2021-04-14T09:12:52Z")

</div>

Hi! I am busy trying to compose a relatively complicated piece of numba code using generated\_jit/overload. However, whilst moving stuff to generated jit I noticed that performance was getting steadily worse. I then coded up the following example which demonstrates a fairly large discrepancy between jitted/generated\_jit/overload code. I am not quite sure if this qualifies as a bug and wanted to be sure that I am not doing something obviously wrong. The example is as follows:

```python
from numba.extending import overload
from numba import jit, types, generated_jit, literally
import numpy as np
import time

def dot2x2_impl(v1, v2, md):
    v100, v101, v110, v111 = v1[0], v1[1], v1[2], v1[3]
    v200, v201, v210, v211 = v2[0], v2[1], v2[2], v2[3]

    v300 = (v100*v200 + v101*v210)
    v301 = (v100*v201 + v101*v211)
    v310 = (v110*v200 + v111*v210)
    v311 = (v110*v201 + v111*v211)

    return v300, v301, v310, v311

@generated_jit(nogil=True, nopython=True, cache=True, inline="always")
def gen_f(v1, v2, md):

    if not isinstance(md, types.Literal):
        return lambda v1, v2, md: literally(md)

    if md.literal_value == "full":
        return dot2x2_impl

@jit(nogil=True, nopython=True, cache=True)
def ovr_f(v1, v2, md):

    return __v1_mul_v2(v1, v2, md)

def __v1_mul_v2(v1, v2, md):
    return

@overload(__v1_mul_v2, inline="always")
def __v1_mul_v2_impl(v1, v2, md):

    if not isinstance(md, types.Literal):
        return lambda v1, v2, md: literally(md)

    if md.literal_value == "full":
        return dot2x2_impl

@jit(nogil=True, nopython=True, cache=True)
def jit_f(v1, v2, md):
    v100, v101, v110, v111 = v1[0], v1[1], v1[2], v1[3]
    v200, v201, v210, v211 = v2[0], v2[1], v2[2], v2[3]

    v300 = (v100*v200 + v101*v210)
    v301 = (v100*v201 + v101*v211)
    v310 = (v110*v200 + v111*v210)
    v311 = (v110*v201 + v111*v211)

    return v300, v301, v310, v311

if __name__ == " __main__":

    n_run = 1000

    v1 = np.ones((4,))
    v2 = np.ones((4,))

    print(jit_f(v1, v2, "full"))
    print(gen_f(v1, v2, "full"))
    print(ovr_f(v1, v2, "full"))

    print("--------jit--------")
    t0 = time.time()
    for i in range(n_run):
        jit_f(v1, v2, "full")
    t1 = time.time()
    print(f"{(t1-t0)/n_run}")

    print("---generated_jit---")
    t0 = time.time()
    for i in range(n_run):
        gen_f(v1, v2, "full")
    t1 = time.time()
    print(f"{(t1-t0)/n_run}")

    print("------overload-----")
    t0 = time.time()
    for i in range(n_run):
        ovr_f(v1, v2, "full")
    t1 = time.time()
    print(f"{(t1-t0)/n_run}")

```

On my laptop, this produces:

```auto
(2.0, 2.0, 2.0, 2.0)
(2.0, 2.0, 2.0, 2.0)
(2.0, 2.0, 2.0, 2.0)
--------jit--------
2.2418498992919923e-06
---generated_jit---
0.0018690543174743651
------overload-----
0.010494077205657959

```

I am trying to understand the reason for this discrepancy and options to circumvent it. My instinct is that somehow the evaluation of the `md.literal_value == something` becomes expensive, but I have no evidence to support this claim.

---

<div class="post-metadata">

**Author:** ![Hannes](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/hannes/32/188_2.png) [@Hannes](https://numba.discourse.group/u/Hannes)\
**Post date:** [April 14, 2021, 1:09pm UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/2 "2021-04-14T13:09:27Z")

</div>

Hi @JSKenyon,

welcome to the board! 🙂

It feels to me as if what you are observing here are not performance problems of the compiled code, but different call overheads for different methods.

I have tried to modify your tests (thanks for the MWE!) a little:  
I have defined 3 functions that call the actual implementations with the right arguments, and then njitted them such that calling the top level function from Python should have the same complexity in all cases.

For reasons I cannot seem to figure out, this does not work for the `gen_f` case, the compilation always fails, so the timings of that are not really comparable to those of `jit_f` and `ovr_f`. I wonder if I don’t know how to do this correctly or if it may be a bug that’s conjured up when combining `jit`, `generated_jit` and `literally` ? Might be worth investigating.

Without further ado, here are my timings:

```auto
(2.0, 2.0, 2.0, 2.0)
(2.0, 2.0, 2.0, 2.0)
(2.0, 2.0, 2.0, 2.0)
--------jit--------
1.3805700291413814e-07
---generated_jit---
0.0024459507330029735
------overload-----
2.2707700554747134e-07

```

And here is the modified code I used (I also took the liberty to remove `jit_f` code duplication).

```python
from numba.extending import overload
from numba import jit, types, generated_jit, literally
import numpy as np
from timeit import timeit

def dot2x2_impl(v1, v2, md):
    v100, v101, v110, v111 = v1[0], v1[1], v1[2], v1[3]
    v200, v201, v210, v211 = v2[0], v2[1], v2[2], v2[3]

    v300 = (v100*v200 + v101*v210)
    v301 = (v100*v201 + v101*v211)
    v310 = (v110*v200 + v111*v210)
    v311 = (v110*v201 + v111*v211)

    return v300, v301, v310, v311

jit_f = jit(nogil=True, nopython=True, cache=True)(dot2x2_impl)

@generated_jit(nogil=True, nopython=True, cache=True, inline="always")
def gen_f(v1, v2, md):
    if not isinstance(md, types.Literal):
        return lambda v1, v2, md: literally(md)

    if md.literal_value == "full":
        return dot2x2_impl

@jit(nogil=True, nopython=True, cache=True)
def ovr_f(v1, v2, md):

    return __v1_mul_v2(v1, v2, md)

def __v1_mul_v2(v1, v2, md):
    return

@overload(__v1_mul_v2, inline="always")
def __v1_mul_v2_impl(v1, v2, md):

    if not isinstance(md, types.Literal):
        return lambda v1, v2, md: literally(md)

    if md.literal_value == "full":
        return dot2x2_impl

@jit(nogil=True, nopython=True, cache=True)
def test_jit_f():
    return jit_f(v1, v2, "full")

# @jit(nogil=True, nopython=True, cache=True)
def test_gen_f():
    return gen_f(v1, v2, "full")

@jit(nogil=True, nopython=True, cache=True)
def test_ovr_f():
    return ovr_f(v1, v2, "full")

if __name__ == " __main__":

    n_run = 1000

    v1 = np.ones((4,))
    v2 = np.ones((4,))

    print(test_jit_f()) #Also triggers first time compilation :-)
    print(test_gen_f())
    print(test_ovr_f())

    print("--------jit--------")
    t = timeit(test_jit_f, number=n_run)
    print(f"{t/n_run}")

    print("---generated_jit---")
    t = timeit(test_gen_f, number=n_run)
    print(f"{t/n_run}")

    print("------overload-----")
    t = timeit(test_ovr_f, number=n_run)
    print(f"{t/n_run}")

```

PS: Another point that I found strange while print debugging your examples was that the body of gen\_f is executed on every call from python. While I don’t know how exactly generated\_jit is designed I found that a bit odd. I have a suspicion this has something to do with the literal typing. Maybe someone else can explain whats going on there.

---

<div class="post-metadata">

**Author:** ![Hannes](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/hannes/32/188_2.png) [@Hannes](https://numba.discourse.group/u/Hannes)\
**Post date:** [April 14, 2021, 1:20pm UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/3 "2021-04-14T13:20:26Z")

</div>

As additional info concerning my confusion about `generated_jit`, here is the output I get when adding a `print(md)` statement on the first line of `gen_f` (and reducing n\_run)

```auto
(2.0, 2.0, 2.0, 2.0)
unicode_type
Literal[str](full)
(2.0, 2.0, 2.0, 2.0)
(2.0, 2.0, 2.0, 2.0)
--------jit--------
5.23999915458262e-07
---generated_jit---
unicode_type
unicode_type
unicode_type
unicode_type
unicode_type
unicode_type
unicode_type
unicode_type
unicode_type
unicode_type
0.003825946699362248
------overload-----
4.282002919353545e-07

```

For some reason the body of `gen_f` is entered on every call, despite the fact that the specialisation should already exist. (If I do the same for the `generated_jit` example in the docs [https://numba.readthedocs.io/en/stable/user/generated-jit.html](https://numba.readthedocs.io/en/stable/user/generated-jit.html), the body is only executed the first time a certain signature is called)

---

<div class="post-metadata">

**Author:** ![JSKenyon](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/jskenyon/32/301_2.png) [@JSKenyon](https://numba.discourse.group/u/JSKenyon)\
**Post date:** [April 14, 2021, 1:29pm UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/4 "2021-04-14T13:29:38Z")

</div>

Thanks @Hannes!

I can get the `generated_jit` expression to perform in much the same way by removing the `inline=="always"` from the `generated_jit` decorator. I think it is not supported properly, particularly when called from another jitted function.

---

<div class="post-metadata">

**Author:** ![JSKenyon](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/jskenyon/32/301_2.png) [@JSKenyon](https://numba.discourse.group/u/JSKenyon)\
**Post date:** [April 14, 2021, 1:32pm UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/5 "2021-04-14T13:32:37Z")

</div>

Overall I think I am happier now. I think some of the problems I was seeing in the more complicated code I am actually implementing stemmed from excessive use of `generated_jit` - I am now moving towards a factory function based implementation that seems to perform better.

---

<div class="post-metadata">

**Author:** ![Hannes](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/hannes/32/188_2.png) [@Hannes](https://numba.discourse.group/u/Hannes)\
**Post date:** [April 14, 2021, 1:55pm UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/6 "2021-04-14T13:55:23Z")

</div>

Hi,

ah, yes that does the trick and all the timings are similar then - great 🙂

/cc @stuartarchibald Is the behaviour observed here a known limitation or an issue that should be raised?

Cheers  
Hannes

---

<div class="post-metadata">

**Author:** ![stuartarchibald](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/stuartarchibald/32/10_2.png) [@stuartarchibald](https://numba.discourse.group/u/stuartarchibald)\
**Post date:** [April 21, 2021, 9:33am UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/7 "2021-04-21T09:33:24Z")

</div>

Hi @JSKenyon,

Thanks for sharing this, it’s an unusual problem 🙂

Thanks for debugging this so far @Hannes, I can observe similar behaviours to those you have encountered, which is reassuring 🙂

Augmenting the script as follows:

```python
from numba.extending import overload
from numba import jit, types, generated_jit, literally
import numpy as np
import time

def dot2x2_impl(v1, v2, md):
    v100, v101, v110, v111 = v1[0], v1[1], v1[2], v1[3]
    v200, v201, v210, v211 = v2[0], v2[1], v2[2], v2[3]

    v300 = (v100*v200 + v101*v210)
    v301 = (v100*v201 + v101*v211)
    v310 = (v110*v200 + v111*v210)
    v311 = (v110*v201 + v111*v211)

    return v300, v301, v310, v311

@generated_jit(nopython=True)
def gen_f(v1, v2, md):

    print("GEN F")
    if not isinstance(md, types.Literal):
        return lambda v1, v2, md: literally(md)

    if md.literal_value == "full":
        return dot2x2_impl

@jit(nopython=True)
def ovr_f(v1, v2, md):

    return __v1_mul_v2(v1, v2, md)

def __v1_mul_v2(v1, v2, md):
    return

@overload(__v1_mul_v2, inline="always")
def __v1_mul_v2_impl(v1, v2, md):
    print('ol', v1, v2, md)
    if not isinstance(md, types.Literal):
        print("literal escape")
        return lambda v1, v2, md: literally(md)

    if md.literal_value == "full":
        print("correct")
        return dot2x2_impl
    print("fall through")

@jit(nopython=True)
def jit_f(v1, v2, md):
    v100, v101, v110, v111 = v1[0], v1[1], v1[2], v1[3]
    v200, v201, v210, v211 = v2[0], v2[1], v2[2], v2[3]

    v300 = (v100*v200 + v101*v210)
    v301 = (v100*v201 + v101*v211)
    v310 = (v110*v200 + v111*v210)
    v311 = (v110*v201 + v111*v211)

    return v300, v301, v310, v311

if __name__ == " __main__":

    n_run = 100

    v1 = np.ones((4,))
    v2 = np.ones((4,))

    print(jit_f(v1, v2, "full"))
    print(gen_f(v1, v2, "full"))
    print(ovr_f(v1, v2, "full"))

    # Look at the LLVM IR control flow graphs
    jit_f.inspect_cfg(jit_f.signatures[0]).display(view=True)
    gen_f.inspect_cfg(gen_f.signatures[0]).display(view=True)
    ovr_f.inspect_cfg(ovr_f.signatures[0]).display(view=True)

    def show(x, item='inspect_asm'):
        print(str(x).center(80, '-'))
        fn = getattr(x, item)
        print(fn(x.signatures[0]))

    # Look at the disassembly control flow graphs
    show(jit_f, 'inspect_disasm_cfg')
    show(gen_f, 'inspect_disasm_cfg')
    show(ovr_f, 'inspect_disasm_cfg')

    print("--------jit--------")
    t0 = time.time()
    for i in range(n_run):
        jit_f(v1, v2, "full")
    t1 = time.time()
    print(f"{(t1-t0)/n_run}")

    # In this case it's evident that a new version is being compiled each time
    print("---generated_jit---")
    t0 = time.time()
    for i in range(n_run):
        gen_f(v1, v2, "full")
    t1 = time.time()
    print(f"{(t1-t0)/n_run}")

    # In this case it's evident that a new version is being compiled each time
    # and that the compilation is slower than with generated_jit (expected,
    # there's more to do to resolve overloads).
    print("------overload-----")
    t0 = time.time()
    for i in range(n_run):
        ovr_f(v1, v2, "full")
    t1 = time.time()
    print(f"{(t1-t0)/n_run}")

```

This shows a few things.

1. That the LLVM generated for the function is the same no matter which method is used to generated it.

2. That the machine code generated for the function is the same no matter which method is used to generated it (dump is below):

3. That the `generated_jit` and `overload` cases are recompiling every time. I think it’s a bad interaction with `literally` and the in memory compilation cache that’s causing it. Probably that there is caching on non-literal types but the overload is forcing literal. I am fairly convinced that this is the root cause of the performance difference, it’s not the execution time of the function that’s causing it, but that new ones keep on being compiled for each execution.

Essentially, this is a bug 🙂 [Excessive recompilation due to use of `literally` and potentially unaware in memory cache. · Issue #6956 · numba/numba · GitHub](https://github.com/numba/numba/issues/6956)

---

<div class="post-metadata">

**Author:** ![Hannes](https://yyz2.discourse-cdn.com/free1/user_avatar/numba.discourse.group/hannes/32/188_2.png) [@Hannes](https://numba.discourse.group/u/Hannes)\
**Post date:** [April 21, 2021, 10:42am UTC](https://numba.discourse.group/t/performance-discrepancy-between-jit-generated-jit-overload/671/8 "2021-04-21T10:42:24Z")

</div>

Thanks for the follow up @stuartarchibald ! 🙂