Stl container for numba

Introducing nbstl: Fast and Simple Containers for Python

Hey everyone! :wave:

I’ve been working on a small library called nbstl—a collection of high-performance containers for Python, built with numba and numpy. If you’re working on numerical computing or need fast data structures, this might be useful!

What’s in the box?

  • Common Containers: Hash, Heap, Red-Black Tree, AVL Tree, and more.
  • Array-Based: Everything is built on ndarray, so it plays nicely with NumPy.
  • Fast: Hash and Red-Black Tree are as fast as C++ STL (yes, really!).
  • Flexible: Supports key-value pairs and shared memory modes for complex workflows.

Why use it?

  • Need a fast heap or hash table for your numerical data? Done.
  • Working with large, complex data structures? Shared memory mode has you covered.
  • Just want something simple and efficient? nbstl is designed to be easy to use.

Check it out and let me know what you think! Feedback and contributions are welcome. :blush:

Image-Py/nbstl: stl container for numba

Convex Hull

Algorithm Implementation:

  1. Sort the points by their X-coordinate.
  2. Build the upper half: Start from the leftmost point and push it onto the stack. For each new point, check the last two points on the stack. If they form a right turn with the new point, push the new point onto the stack. Otherwise, pop the top element from the stack until a right turn is obtained.
  3. Build the lower half: Start from the rightmost point and push it onto the stack. For each new point, check the last two points on the stack. If they form a right turn with the new point, push the new point onto the stack. Otherwise, pop the top element from the stack until a right turn is obtained.
  4. Combine the upper and lower halves to obtain the convex hull.
import numpy as np
import numba as nb
import nbstl

# build Point dtype and PointStack
t_point = np.dtype([('x', np.float32), ('y', np.float32)])
PointStack = nbstl.TypedStack(t_point)

# push to stack one by one, if not turn right, pop
@nb.njit
def convex_line(pts, idx):
    hull = PointStack(128)
    for i in idx:
        p2 = pts[i]
        while hull.size>1:
            p1 = hull.top(0)
            p0 = hull.top(1)
            s = p0.x*p1.y - p0.y*p1.x
            s += p1.x*p2.y - p1.y*p2.x
            s += p2.x*p0.y - p2.y*p0.x
            if s<-1e-6: break
            hull.pop()
        hull.push((p2.x, p2.y))
    return hull.body[:hull.size]

# get up line and down line, then concat the hull
@nb.njit
def convexhull(pts):
    idx = np.argsort(pts['x'])
    up = convex_line(pts, idx)
    down = convex_line(pts, idx[::-1])
    return np.concatenate((up, down[1:]))

if __name__ == '__main__':
    from time import time
    # use randn to make 102400 random points
    pts = np.random.randn(102400, 2).astype(np.float32)
    pts = pts.ravel().view(t_point)

    hull = convexhull(pts)
    start = time()
    hull = convexhull(pts)
    print('convex hull of 102400 point cost:', time()-start)

Then, we perform performance comparison using Shapely for convex hull computation on datasets of the same.

from shapely import geometry as geom

pts = np.random.randn(102400, 2).astype(np.float32)
mpoints = geom.MultiPoint(pts)
start = time()
cvxh = mpoints.convex_hull
print('the same points by shapely cost:', time()-start)

we got the result below:

convex hull of 102400 point cost: 0.01891
the same points by shapely cost: 0.04986

convex hull of 1024000 point cost: 0.23035
the same points by shapely cost: 1.08539
6 Likes

This is awesome! We’ve done some simple stl abstractions in-house, I think this is a perfect example of the power of numba to bring c++ performance to the python world

What a great a example of the utility of Numba, thank you for sharing!

Here are the performance test results at the 1e7 scale (STL was optimized with -O3):

Test Object Link Insertion Time Deletion Time
My AVL Tree nbstl/test/avlrbtreetest.py at main · Image-Py/nbstl · GitHub 6.4s 6.4s
My RedBlack Tree same as up 6.7s 6.9s
STL Set nbstl/test/StlSetPerformanceTest.cpp at main · Image-Py/nbstl · GitHub 7.6s 8.2s
  1. Nbstl is 20% faster than stl:set.
  2. Why doesn’t my implementation of a red-black tree demonstrate the commonly cited advantages over AVL trees in terms of insertion and deletion?

Hello, it’s a great honor to meet you. I believe we are working on similar things and have encountered similar challenges. I am currently facing issues with excessively long compilation times and the inability to cache. While searching for related information, I noticed your extensive involvement. I think you have now shifted to using structref , which I have not yet started to learn or understand. Would you mind if I privately seek your advice, or if you have the time, could you help me assess the feasibility of rewritting structron with structref ?