How to prevent Numba cfunc failures with NumPy arrays and ctypes: pass arr.ctypes, not a raw pointer

Learn why Numba cfunc factories crash with NumPy and ctypes (unsupported PEP 3118) and how to fix it: avoid capturing raw pointers and call with arr.ctypes.

When wiring up a factory that returns a Numba cfunc to operate on NumPy arrays, a tempting approach is to pass a raw pointer obtained via ctypes. In practice this easily crashes with an unsupported PEP 3118 format error, because the closure captures a ctypes pointer that Numba cannot type in nopython mode. The fix is simpler than it looks: use the array’s .ctypes attribute directly when calling the compiled function.

Reproducing the issue

The following minimal example demonstrates a factory that compiles a no-arg C function, which internally calls another cfunc to compute the sum of a NumPy array. All logic is preserved; only names differ for readability.

import numpy as np
from numba import cfunc, carray
from numba.types import intc, CPointer, float64
import ctypes

class CalcBox:
    def __init__(self, buf):
        self.buf = buf
        self.length = buf.size

    # C signature: double func(double* input, int n)
    c_sig = float64(CPointer(float64), intc)
    c_sig_void = float64()

    @staticmethod
    @cfunc(c_sig)
    def c_sum(ptr, n):
        view = carray(ptr, n)
        return np.sum(view)

    def make_sum(self):
        arr = self.buf
        size = self.length
        compiled = type(self).c_sum
        raw_ptr = arr.ctypes.data_as(ctypes.POINTER(ctypes.c_double))
        @cfunc(self.c_sig_void)
        def thunk():
            return compiled(raw_ptr, size)
        return thunk.address

vec = np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float64)
box = CalcBox(vec)
addr = box.make_sum()
C0 = ctypes.CFUNCTYPE(ctypes.c_double)
fn = C0(addr)
print("Sum:", fn())

This pattern often fails with a traceback that includes an unsupported PEP 3118 format and an untyped global name related to the captured pointer. The core problem is not the summation itself but the way the ctypes pointer is captured inside a compiled closure.

Why it fails

The nested compiled function closes over a Python object that is a ctypes pointer. Inside a cfunc, this becomes a typing problem: Numba does not recognize the ctypes pointer object as a valid nopython value and emits a decoding error tied to PEP 3118 formats. In short, the pointer captured by the closure is not typable by Numba’s frontend, and the name ends up untyped.

There is more context that explains adjacent pitfalls. Closures are supported using a pretty-inefficient mechanism where referenced variables are treated in a way that makes them effectively constant for the generated function. Numba may generate a fresh C function in its own LLVM module, storing referenced variables as globals. Changing a referenced variable after compilation is UB, and while it can appear to work, it can also silently break, particularly when optimizers do not fully propagate constants through pointers. This approach can also lead to heavy compilation time and large generated modules, depending on the array and toolchain version. Finally, although closure capture can feel harmless for ints or floats, it becomes problematic for pointers, and the ctypes pointer object is not recognized by the type system in this scenario.

The fix

The workaround is straightforward: don’t materialize a ctypes pointer and capture it; instead, pass the array’s .ctypes attribute directly to the compiled function. This avoids the untyped ctypes pointer object in the closure.

import numpy as np
from numba import cfunc, carray
from numba.types import intc, CPointer, float64
import ctypes

class CalcBox:
    def __init__(self, buf):
        self.buf = buf
        self.length = buf.size

    c_sig = float64(CPointer(float64), intc)
    c_sig_void = float64()

    @staticmethod
    @cfunc(c_sig)
    def c_sum(ptr, n):
        view = carray(ptr, n)
        return np.sum(view)

    def make_sum(self):
        arr = self.buf
        size = np.intc(self.length)  # ensure intc if needed
        compiled = type(self).c_sum
        @cfunc(self.c_sig_void)
        def thunk():
            return compiled(arr.ctypes, size)
        return thunk.address

vec = np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float64)
box = CalcBox(vec)
addr = box.make_sum()
C0 = ctypes.CFUNCTYPE(ctypes.c_double)
fn = C0(addr)
print("Sum:", fn())

The only functional change is replacing the explicit ctypes pointer with arr.ctypes at the call site. If the signature expects intc for the size, make sure to pass np.intc to match it.

Why this matters

Bridging NumPy, Numba, and C interfaces is powerful, but tiny details—like how a pointer enters a compiled closure—can decide whether you get a clean C-callable function or an opaque typing error. Understanding that closures are supported in a way that treats captured values as constants helps avoid undefined behavior from mutating referenced data later. It also sets expectations about potential compile-time overhead and module size growth in some scenarios.

Conclusion

If you need a factory that returns a Numba cfunc consuming a NumPy array, avoid capturing raw ctypes pointers inside the compiled closure. Use arr.ctypes in the call to the compiled function and ensure your integer widths align with the declared signature, for example by passing np.intc when the signature uses intc. This keeps the function typable in nopython mode, sidesteps unsupported PEP 3118 formats, and reduces surprises from how closures are lowered under the hood.