I have some data that is awkwardly structured. In Fortran, I can easily contain this data with a 2-D array of derived types (i.e. structs in C).
program main
use iso_fortran_env, only: dp => real64
implicit none
type :: ContainerType
real(dp), allocatable :: arr1(:)
real(dp), allocatable :: arr2(:)
end type
type(ContainerType), allocatable :: dat(:,:)
! `dat` can easily contain all my data
end program
My goal is to create a similar data structure in Numba. My current solution is using nested lists of jitclasses
spec = [
('arr1', types.double[:]),
('arr2', types.double[:]),
]
@nb.experimental.jitclass(spec)
class Container_0():
def __init__(self, arr1, arr2):
self.arr1 = arr1
self.arr2 = arr2
spec = [
("c0", nb.types.ListType(Container_0.class_type.instance_type)),
]
@nb.experimental.jitclass(spec)
class Container_1():
def __init__(self, c0):
self.c0 = c0
spec = [
("c1", nb.types.ListType(Container_1.class_type.instance_type)),
]
@nb.experimental.jitclass(spec)
class Container_2():
def __init__(self, c1):
self.c1 = c1
While this approach seems to work in numba, it takes up massive amounts of memory. For example, If I store 60,000 double precision numbers in this data structure, then it takes ~ 1 Gb of memory. In Fortran or C, it should only take at most 1 Mb to store the data.
MY QUESTION: Is there an alternative way to store my data in Numba which doesnāt use so much memory?