How to use ahead-of-time compilation with PyPI?

Numba Support: How do I do ...?
1

I have integrated ahead-of-time compilation as described in https://numba.pydata.org/numba-doc/dev/user/pycc.html#distutils-integration but when building the package like python3 setup.py sdist bdist_wheel and uploading the module to https://test.pypi.org/ with twine upload --repository-url https://test.pypi.org/legacy/ dist/*, I get the error message:

HTTPError: 400 Bad Request from https://test.pypi.org/legacy/
Binary wheel 'MyModule-1.0.0-cp37-cp37m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.

Instead, I could build a source-only package with python3 setup.py sdist, but then the module won’t be aot-compiled and import will fail. A solution might be to aot-compile the module on first import, but I am not sure where to put the resulting shared library file for each possible platform.

2

You will probably need to use one of the manylinux platform tags to upload to PyPI: https://packaging.python.org/specifications/platform-compatibility-tags/#platform-tags-for-common-linux-distributions

There is some more info about the manylinux platform here: https://github.com/pypa/manylinux

This repo contains an example of building manylinux wheels: https://github.com/pypa/python-manylinux-demo

3

Thank you for the links.

It seems like this would make the build process overly complex and I would still have to deal with Windows and Linux.

Instead, I’ll try to find out where to put the files when aot-compiling on import.

4

Would using the cache kwarg for the JIT decorator address your use case? https://numba.readthedocs.io/en/stable/user/jit.html#cache

Using the kwarg would result in compilation on users’ machines on the first use for each set of argument types - on subsequent runs the compiled code will be loaded from the cache.

5

We were using caching until a few days ago, but import still took several seconds. With aot, it only takes slightly more than one second. A previous version was using C with ffi, which took well under a second to import, but it was harder to maintain for different platforms.

In a perfect world, caching would also be near-instant, but as long as that is not the case, aot-compilation seems like the next-best option.

6

We are now compiling on first import: https://github.com/pymatting/pymatting/blob/838171bdf13ddd474c1e81b7a4f1427bf6da6703/pymatting_aot/cc.py#L1

This way, we do not have to deal with manylinux and import will be reasonably fast.