We use Spark through Spark API with pyspark. I know Numba can make python code very fast so it is a good thing to use it on our udfs (user-defined function) but I’m not sure Numba can help me with map or mapPartion. I’m mapping with spark over all our data but the function itself doesn’t have a for…

Edit:. The link in my message is incorrect, Graham’s link below is the intended link. It’s not entirely clear to me what your question is… Do you have a minimal example of what you’re trying to do? There’s another related discussion here Python UDFs are generally relatively slow in Spark 2.x, tho…

Use Numba with PySpark

Numba Community Support

gmarkall June 27, 2022, 10:16am 3

See also: Numba and PySpark users?

Topic		Replies	Views
Numba and PySpark users? Community Support	3	1878	May 11, 2022
Pandas dataframes with numba Numba	2	713	October 19, 2022
Tutorial on supporting Python User-Defined Functions in CUDA-accelerated Applications with Numba Showcase	0	468	March 25, 2022
DFS implementation with Numba Showcase	5	1009	November 2, 2021
My numba code is slower than my original code Numba	1	96	February 11, 2025

Use Numba with PySpark

Related topics