How to serialize Python Enums containing callables: fix functools.partial unpickling errors with pickle_by_enum_name or member

Learn why pickling Python Enums that hold functools.partial fails and how to fix it. Use pickle_by_enum_name or the member decorator to serialize enum members

Serializing enums that hold callables can be handy when configurations need to select an implementation and pass it across process boundaries. A common pattern is to store functions directly in a Python Enum and reference them from a config system like Hydra. But if you wrap those functions in functools.partial to avoid them being treated as methods, unpickling can fail with a ValueError stating that the partial is not a valid member. Below is a minimal, reproducible walkthrough and two focused, production-friendly fixes.

Reproducing the issue

The following example defines an enum whose values are functools.partial objects and then tries to pickle and unpickle a member. The unpickle step raises an error that the partial is not a valid member.

import pickle
from enum import Enum
from functools import partial
def op_alpha():
    pass
class FuncRefSet(Enum):
    ALPHA = partial(op_alpha)
if __name__ == "__main__":
    with open("demo.bin", "wb") as fh:
        pickle.dump(FuncRefSet.ALPHA, fh)
    with open("demo.bin", "rb") as fh:
        pickle.load(fh)

What is going on

The member being unpickled does not map back to a known enum member during loading, and Python raises a ValueError that the functools.partial object is not a valid member of the enum. This is exactly the failure scenario you will hit when the enum value itself is a partial, even though using partial is attractive here to prevent a function value from being interpreted as a method.

In practical terms, this often appears in configuration-driven code. With Hydra, for instance, you might want a YAML field to select a function through an enum. Wrapping with partial avoids method semantics, but then passing that enum member between processes by pickling breaks on load.

Two targeted fixes

The first fix is to instruct the enum to be pickled by name. This sidesteps the value matching problem entirely and makes the serialized form stable. The second fix removes functools.partial from the equation by using the member decorator to store the function directly as the value, and optionally adding __call__ so that the enum member itself is invokable.

Fix 1: pickle by name

Enable name-based pickling for the enum using pickle_by_enum_name. The value can remain a functools.partial.

import pickle
from enum import Enum, pickle_by_enum_name
from functools import partial
def op_alpha():
    print("alpha")
class FuncRefSet(Enum):
    __reduce_ex__ = pickle_by_enum_name
    ALPHA = partial(op_alpha)
if __name__ == "__main__":
    with open("demo.bin", "wb") as fh:
        pickle.dump(FuncRefSet.ALPHA, fh)
    with open("demo.bin", "rb") as fh:
        item = pickle.load(fh)
        print(item)

Fix 2: store functions directly with member and make members callable

Instead of wrapping with functools.partial, use the member decorator to set the function as the enum value. Adding __call__ makes the enum member itself behave like the underlying function. Name-based pickling is still recommended.

import pickle
from enum import Enum, member, pickle_by_enum_name
def op_alpha():
    print("alpha run")
class FuncCatalog(Enum):
    __reduce_ex__ = pickle_by_enum_name
    def __call__(self, *args, **kwargs):
        return self._value_(*args, **kwargs)
    ALPHA = member(op_alpha)
if __name__ == "__main__":
    with open("demo.bin", "wb") as fh:
        pickle.dump(FuncCatalog.ALPHA, fh)
    with open("demo.bin", "rb") as fh:
        fn = pickle.load(fh)
        print(fn)
        fn()

If you prefer to define the function inline within the enum, the same decorator works:

from enum import Enum, member, pickle_by_enum_name
class FuncCatalog(Enum):
    __reduce_ex__ = pickle_by_enum_name
    def __call__(self, *args, **kwargs):
        return self._value_(*args, **kwargs)
    @member
    def op_alpha():
        print("alpha run")

Why this matters

When enums are part of your configuration boundary, the ability to serialize them cleanly underpins interprocess communication, reproducible runs, and caching of configuration payloads. In setups where a YAML or similar config selects a behavior via an enum member, you want the enum to round-trip through pickle without surprising coupling to the representation of its value. Name-based pickling ensures stable identity, and storing functions directly with member avoids the edge cases that functools.partial introduces in this context while preserving convenient call semantics via __call__.

Takeaways

If an enum holds callables and needs to be pickled, switch to name-based pickling with pickle_by_enum_name. If you used functools.partial only to prevent functions from being treated as methods, replace it with the member decorator and, if needed, implement __call__ so members can be invoked directly. Both approaches keep the configuration ergonomics while making serialization straightforward.