Possible reason for prange slowdown

Numba Community Support
1

Hello everyone,

Introduction
I’m trying to parallelize a for loop in a large codebase called PyReason so that each iteration runs on a separate CPU core. I have 50 CPU cores at my disposal. The size of the loop depends on the number of nodes from a graph that I input to the program. Currently this value is in the neighbourhood of 300,000. I’m experiencing significant slowdowns, around 2x, 3x when running with parallel=True. I’m trying to understand the possible reasons for this slowdown.

Questions

  1. When using prange on a for loop, does this assign every loop iteration to different available CPU cores? If not, how can I do this?
  2. This is a link to the for loop in the codebase that I’m trying to parallelize. I’m calling different functions and adding elements to lists (threadsafe) in each iteration. I’m observing significant slowdown when running prange and I’m not sure why. I would assume that each iteration is on a separate CPU core so I should see speedups. (my knowledge about how prange works is limited)
  3. Is there a better way of parallelizing the above linked for loop on my CPU cores?

Thank you so much for your time,
Dyuman