Which python build version?

I tried a quick cursory search here and on stack exchange and the answers I did find were a little ambiguous so I was hoping someone in the Numba community can clear this up.

For the record, i’m not a programmer/computerScientist by training so I mainly use Numba, other packages, and various languages as a tool and to be honest usually dont concern myself too much with whats going on under the hood such that everything works. My undergrad was in physics & mathematics and I’m currently a grad physics student.

Recently, I was troubleshooting an issue I was having using the basic function speed up:
@jit(nopython=True, parallel=True, nogil=True)

From reading the errors during the debugging I eventually found that I had a 32bit python for Intel installed on my machine and i’m using 64bit windows on an AMD 3950x, plus I had upgraded to python3.9 which i later found was not fully supported yet. I uninstalled that python and installed a 64bit AMD python 3.8.6 build and everything works fine for me now.


In this screenshot, towards the end of the 3rd line my problematic build was 32 bit (Intel), but as you can see it is now 64 bit (AMD), which is working flawlessly for me.

I am helping someone get their pycharm environment set up and running the same research code that I had issues with and eventually figured out as mentioned above. However, the only difference being theyre on an Intel machine and their build denotes the same that is shown in the screenshot above for my machine.

I found this python.org source to download builds from but I am looking for clarity on how to determine what is a 64 vs 32 bit build and AMD vs Intel build.

My first thought is that x86-64 corresponds to 64bit builds and x86 is 32bit builds and the build should automatically detect Intel vs AMD, but given that my previous build was 32bit and detected Intel on this pc (custom built with the 3950x from the beginning of its creation few months ago) I am not so sure about this anymore.

Any help would be greatly appreciated, thank you very much.
evan

Also, one more point of reference.


I found this and it states that the AMD64 build should work on Intel 64, but if all things being equal except for the CPU i’m curious if theres something more stringent under the hood with Numba that supercedes this cross cpu functionality.

thanks.

Hi @evanhowington,

I’m not quite sure what you are asking, but I’ll add some information that might help.

  1. The string [MSC v.1927 64 bit (AMD64)] is from the compiler used to compile the python interpreter. I think this is the same as that available from doing:
    In [73]: import platform
    
    In [74]: platform.python_compiler()
    Out[74]: 'GCC 7.3.0'
    
    (I’m on linux and my python executable is compiled with GCC).
  2. Given 1. the AMD64 string is just part of the compiler identification and I think is probably just synonymous with x86_64 i.e. 64 bit variant of the x86 instruction set. Whereas Intel 32 is the compiler identification string for the 32 bit variant of the x86 instruction set (I’ve just gone and checked this against Anaconda distribution Python).
  3. You can run 32bit programs on 64bit windows, but not vice versa.
  4. I don’t think this has anything to do with the vendor (Intel or AMD) of the underlying processor, the executable will just “see” the instruction set. That said, it is common, particularly in the high performance computing space, to have variants of binaries/libraries that are tuned to specific chips etc.

My first thought is that x86-64 corresponds to 64bit builds and x86 is 32bit builds and the build should automatically detect Intel vs AMD, but given that my previous build was 32bit and detected Intel on this pc (custom built with the 3950x from the beginning of its creation few months ago) I am not so sure about this anymore.

I don’t think there any detecting going on, if you installed a X bit Python interpreter, when you conda/pip install Numba, you’ll get a X bit Numba (X=32 or X=64). With the parallel=True issue in the OP, if the Python interpreter was 32 bit, then the Numba installation would also be 32 bit and as a result parallel=True transforms do not work (they aren’t implemented and Numba catches and reports this).

I hope the above helps?

Hello @stuartarchibald, thank you so much for the quick response.

What you said does help, and definitely clarifies things for me. I am not sure where I originally received the 32bit python installation but I suspect it came with the visual studio install that was suggested when I was setting up my c/c++ compiler “stuff”… again not a programmer so please forgive my terminology. I think this was similar to when I was on a mac and in order to compile c/c++/fortran code i had to install xcode for various libraries/dependencies… it appears this was just the windows version of requiring a certain IDE as it came with all the “goods” so to speak.

I was not aware of the parallel parallel=True requiring 62bit so that is a good piece of information to have, thank you. I just thought that all of Numba required it since it started working after I made the change[faulty logic on my part, definitely would have failed the truth table test there :joy:], but at least now I know for future endeavors.

After reading your response I did some further troubleshooting of my friend’s system and applied information you provided and found some other items that appeared to be causing issues. They had Anaconda and Pycharm Community installed with multiple iterations of python installed on their computer. I think there were 2- python 3.8.6 installations, 1-python 3.7.x installation, and a generic python under the anaconda directory. They decided to go with Pycharm Pro because we get it for free under student licensing. What I ended up doing was completely removing all of the python variants and Anaconda and things seem to be working now.

With your help and some more troubleshooting we were able to get everything up and running so we greatly appreciate your help. Also, this is my first interaction with this community and you have made a wonderful first impression.

#######

As a side note not directly related to Numba, but did have an impact on me using numba. Previously, when Jetbrains released their Pycharm Pro with Anaconda plugin(s) or whatever it was I had that and a separate Anaconda installation on my computer and I was having serious issues with installing packages into my environments inside Pycharm Pro(i was on a Mac at the time). I think there was some conflict between using pip or conda behind the scenes and I reported it to JetBrains and posted a writeup somewhere but I never received any responses. Just wanted to throw that out there.

I’m getting better at not making noob errors but its still a work in progress! Thanks again for all your help, it was greatly appreciated!

@evanhowington :slight_smile: glad to hear you’ve figured this out, and thanks for your kind words. I’d guess from your description that some of the problems have arisen from having numerous competing “default” Python interpreters. This is definitely something to keep a watch out for as it, along with cross installing package (i.e. having a Python 3.x package installed for a 3.y interpreter), are challenging issues.