Understanding pybind11 shared_ptr, Python wrappers, and id(): stable C++ lifetime vs. changing Python identity

Learn why pybind11 shared_ptr results show changing Python id() until you keep a reference. Understand C++ wrapper lifetime, return value policy, and usage.

When binding C++ classes to Python with pybind11, it’s easy to conflate the lifetime and identity of the underlying C++ instance with the Python wrapper that represents it. A common surprise follows: calling a method that returns a shared_ptr-backed object appears to yield a new Python object each time, so id() changes between calls—until you store the result. Let’s unpack what’s actually happening and why nothing is wrong on the C++ side.

Minimal setup that reproduces the behavior

The following C++ module exposes a simple class, plus a holder that stores a std::shared_ptr. Names are arbitrary; the important part is the ownership model.

#include <memory>
#include <iostream>

#include <pybind11/pybind11.h>

class Leaf {

};

class Box {
public:
    Box(std::shared_ptr<Leaf> leaf) : leaf_ref{leaf} {
        show_leaf_addr();
    }

    std::shared_ptr<Leaf> acquire_leaf() {
        return leaf_ref;
    }

    void show_leaf_addr() {
        std::cout << leaf_ref << std::endl;
    }

private:
    std::shared_ptr<Leaf> leaf_ref;
};

namespace py = pybind11;

PYBIND11_MODULE(ptrs, m) {
    py::class_<Leaf, std::shared_ptr<Leaf>>(m, "Leaf")
        .def(py::init<>());

    py::class_<Box>(m, "Box")
        .def(py::init<const std::shared_ptr<Leaf>&>(), py::arg("leaf"))
        .def("acquire_leaf", &Box::acquire_leaf)
        .def("show_leaf_addr", &Box::show_leaf_addr);
}

CMakeLists.txt stays straightforward:

cmake_minimum_required(VERSION 3.21)
project(ptrs VERSION 1.0 LANGUAGES CXX)

list(APPEND CMAKE_PREFIX_PATH "${CMAKE_BINARY_DIR}")
find_package(pybind11 REQUIRED)

pybind11_add_module(ptrs module.cpp)

What you’ll see from Python

Creating the object in two steps keeps the wrapper stable. The numeric values below are representative; they will differ on your machine.

>>> from ptrs import *
>>> leaf = Leaf()
>>> box = Box(leaf)
0x607d93adc800
>>> hex(id(leaf))
'0x7c9f96460a70'
>>> hex(id(box.acquire_leaf()))
'0x7c9f96460a70'

Constructing inline, however, leads to changing Python ids across calls until you hold onto the result:

>>> box = Box(Leaf())
0x6304fb8a8800
>>> hex(id(box.acquire_leaf()))
'0x7dd87bcc4eb0'
>>> hex(id(box.acquire_leaf()))
'0x7dd87bcc4c70'
>>> held = box.acquire_leaf()
>>> hex(id(box.acquire_leaf()))
'0x7dd87bcc4eb0'
>>> hex(id(box.acquire_leaf()))
'0x7dd87bcc4eb0'

In some interactive environments, the id can look stable even without explicit assignment. That happens when the environment itself keeps a reference to the last result (for example, stored in a special variable accessible as __), which prevents immediate destruction.

What’s actually happening and why the ids change

When you do box = Box(Leaf()), the temporary Python Leaf wrapper created for the constructor has no references after the call completes and is destroyed. This does not invalidate the C++ instance; the std::shared_ptr retained by Box still owns and keeps the C++ Leaf alive.

Each subsequent call to box.acquire_leaf() returns a shared_ptr to the same C++ Leaf. pybind11 must provide a Python-side wrapper for that C++ instance. If there isn’t an existing live Python wrapper for the exact C++ address, pybind11 creates a new Python object to wrap it. Because that temporary wrapper isn’t stored anywhere, it has a reference count of zero right after the expression completes and gets destroyed. That is why id(box.acquire_leaf()) appears to change between calls: you are observing the identity of short-lived Python wrappers, not the underlying C++ object.

Once you assign the result to a variable, Python keeps the wrapper alive. With a live wrapper in place, future calls return that same wrapper rather than creating a fresh one. The behavior is documented by pybind11:

One important aspect of the above policies is that they only apply to instances which pybind11 has not seen before, in which case the policy clarifies essential questions about the return value’s lifetime and ownership. When pybind11 knows the instance already (as identified by its type and address in memory), it will return the existing Python object wrapper rather than creating a new copy.

Reference: https://pybind11.readthedocs.io/en/stable/advanced/functions.html#return-value-policies

If a reference exists on the Python side, the wrapper will not be destroyed. Without a reference, it can be destroyed right away, and in environments that retain recent results the wrapper may persist longer, making id() appear stable.

Solution in practice

The C++ code is fine as-is; the std::shared_ptr guarantees the C++ object’s lifetime. On the Python side, keep a reference to the returned object whenever you want stable wrapper identity. The following minimal usage makes the effect explicit:

>>> from ptrs import *
>>> box = Box(Leaf())
0x6304fb8a8800
>>> # ephemeral wrappers; ids vary
>>> id(box.acquire_leaf()), id(box.acquire_leaf())
(12345678, 23456789)
>>> # hold a reference; ids become stable
>>> leaf_ref = box.acquire_leaf()
>>> id(box.acquire_leaf()) == id(leaf_ref)
True

No change is required to the binding code to achieve correctness. The key is understanding that id() reflects the Python wrapper’s identity, not the C++ pointer’s value printed from C++.

Why this matters

Misinterpreting id() can lead to chasing non-bugs. The underlying C++ object remains stable and usable; only the Python-side wrapper may be recreated when not referenced. Recognizing this separation helps avoid unnecessary changes to bindings and prevents incorrect assumptions about object lifetime or ownership.

Closing advice

Think of two layers: the C++ instance controlled by std::shared_ptr on one side, and the Python wrapper on the other. The C++ layer is stable in this setup. The Python wrapper is ephemeral unless you keep a reference. When you care about identity on the Python side, assign the result of a returning method to a variable. If you revisit pybind11’s documentation on return value handling, the behavior aligns with the guarantee that already-known instances return the existing wrapper, which is exactly what you observe after you keep one alive.