Underlying object has vanished

luk-f-a · March 15, 2023, 1:50pm

hi there, I have a problem, but unfortunately without reproducer so far. I was wondering if anyone can help me guess what might be causing this issue, or suggest how to further investigate.

In this code there is a closure function, that uses literal_unroll. When running single-process this works well. But when running in multiprocessing, I get this error.

Is this due to literal_unroll using weak references?

Thanks

  File "/opt/conda/lib/python3.9/site-packages/numba/core/dispatcher.py", line 487, in _compile_for_args
    raise e
  File "/opt/conda/lib/python3.9/site-packages/numba/core/dispatcher.py", line 420, in _compile_for_args
    return_val = self.compile(tuple(argtypes))
  File "/opt/conda/lib/python3.9/site-packages/numba/core/dispatcher.py", line 965, in compile
    cres = self._compiler.compile(args, return_type)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/dispatcher.py", line 125, in compile
    status, retval = self._compile_cached(args, return_type)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/dispatcher.py", line 139, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/dispatcher.py", line 152, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler.py", line 716, in compile_extra
    return pipeline.compile_extra(func)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler.py", line 452, in compile_extra
    return self._compile_bytecode()
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler.py", line 520, in _compile_bytecode
    return self._compile_core()
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler.py", line 499, in _compile_core
    raise e
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler.py", line 486, in _compile_core
    pm.run(self.state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 368, in run
    raise patched_exception
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 356, in run
    self._runPass(idx, pass_inst, state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 273, in check
    mangled = func(compiler_state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/untyped_passes.py", line 1660, in run_pass
    pm.run(state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 368, in run
    raise patched_exception
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 356, in run
    self._runPass(idx, pass_inst, state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 273, in check
    mangled = func(compiler_state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/untyped_passes.py", line 1285, in run_pass
    stat = self.apply_transform(state)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/untyped_passes.py", line 1149, in apply_transform
    self.unroll_loop(state, info)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/untyped_passes.py", line 1248, in unroll_loop
    unrolled_body = self.inject_loop_body(
  File "/opt/conda/lib/python3.9/site-packages/numba/core/untyped_passes.py", line 910, in inject_loop_body
    switch_ir.blocks[lbl] = deepcopy(loop_blocks[loop_start_lbl])
  File "/opt/conda/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/opt/conda/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/opt/conda/lib/python3.9/copy.py", line 263, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/opt/conda/lib/python3.9/copy.py", line 263, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 210, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/opt/conda/lib/python3.9/copy.py", line 210, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/opt/conda/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/opt/conda/lib/python3.9/copy.py", line 263, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/opt/conda/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.9/copy.py", line 161, in deepcopy
    rv = reductor(4)
  File "/opt/conda/lib/python3.9/site-packages/numba/core/types/functions.py", line 486, in __getnewargs__
    raise ReferenceError("underlying object has vanished")
ReferenceError: Failed in nopython mode pipeline (step: handles literal_unroll)
Failed in literal_unroll_subpipeline mode pipeline (step: performs mixed container unroll)
underlying object has vanished

stuartarchibald · March 17, 2023, 12:01pm

Hi @luk-f-a,

I don’t recall seeing this specific problem before. I’m reasonably sure that literal_unroll itself doesn’t use weak references, so it might be something that e.g. the Numba IR is referencing that’s causing the problem. If you can find a reproducer please could you open an issue on github for it? Might be worth looking at #6373 (Allow Dispatchers to be cached) for some hints?

Many thanks!

gmarkall · December 12, 2023, 12:06pm

There’s another mention here on the issue tracker: Random cache corruption · Issue #9348 · numba/numba · GitHub

The traceback in that case:

Traceback (most recent call last):
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/caching.py", line 469, in save
    data_name = overloads[key]
KeyError: ((type(CPUDispatcher(<function f at 0x7f7cdab2beb0>)), omitted(default=-1000.0), omitted(default=1000.0), omitted(default=24)), ('x86_64-unknown-linux-gnu', 'skylake', '+64bit,+adx,+aes,-amx-bf16,-amx-int8,-amx-tile,+avx,+avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,+bmi,+bmi2,-cldemote,+clflushopt,-clwb,-clzero,+cmov,+cx16,+cx8,-enqcmd,+f16c,+fma,-fma4,+fsgsbase,+fxsr,-gfni,+invpcid,-lwp,+lzcnt,+mmx,+movbe,-movdir64b,-movdiri,-mwaitx,+pclmul,-pconfig,-pku,+popcnt,-prefetchwt1,+prfchw,-ptwrite,-rdpid,+rdrnd,+rdseed,-rtm,+sahf,-serialize,+sgx,-sha,-shstk,+sse,+sse2,+sse3,+sse4.1,+sse4.2,-sse4a,+ssse3,-tbm,-tsxldtrk,-vaes,-vpclmulqdq,-waitpkg,-wbnoinvd,-xop,+xsave,+xsavec,+xsaveopt,+xsaves'), ('f25bd95d90667ac5e800efcc2c1e8e9c8de752dad7af5924e53678927f5a3b0d', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/helmer72/xyzcad-examples/fuzzbrick2.py", line 99, in <module>
    render.renderAndSave(f, f'fuzzbrick.stl', 0.125)
  File "/home/helmer72/.local/lib/python3.10/site-packages/xyzcad/render.py", line 457, in renderAndSave
    p = findSurfacePnt(func)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 487, in _compile_for_args
    raise e
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 420, in _compile_for_args
    return_val = self.compile(tuple(argtypes))
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 972, in compile
    self._cache.save_overload(sig, cres)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/caching.py", line 652, in save_overload
    self._save_overload(sig, data)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/caching.py", line 662, in _save_overload
    self._cache_file.save(key, data)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/caching.py", line 478, in save
    self._save_index(overloads)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/caching.py", line 522, in _save_index
    data = self._dump(data)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/caching.py", line 550, in _dump
    return dumps(obj)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/serialize.py", line 57, in dumps
    p.dump(obj)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/cloudpickle/cloudpickle_fast.py", line 568, in dump
    return Pickler.dump(self, obj)
  File "/home/helmer72/.local/lib/python3.10/site-packages/numba/core/types/functions.py", line 486, in __getnewargs__
    raise ReferenceError("underlying object has vanished")
ReferenceError: underlying object has vanished

Topic		Replies	Views
Using the Cache Support: What is this error message?	8	1567	April 30, 2021
Cacheing specific Numba function causes SIGABRT Support: What is this error message?	5	522	September 4, 2022
LLVM has deprecated and intends to remove lots of pass manager APIs llvmlite	2	387	October 27, 2022
Compiling function in fortran then calling in numba Numba	3	192	September 17, 2023
Applying cfunc decorator to numba extension type Support: How do I do ...?	0	111	November 21, 2023

Underlying object has vanished

Related Topics