How to prevent Python generator-of-generators from sharing state: capture variables correctly in generator expressions

Learn why Python generator-of-generators can mutate inner streams via closure capture, and how to fix it with generator expressions, map, partial, accumulate.

Building a generator that yields other generators sounds straightforward in Python, right up until the inner streams start mysteriously changing their behavior. If you have a generator factory that should produce independent sequences, but advancing the outer generator mutates earlier inner generators, you’ve hit a classic closure trap in generator expressions.

Problem demo

Consider a compact generator factory designed to yield infinite power sequences for growing bases:

import itertools as itx
stream_maker = ((pow(base, exp) for exp in itx.count(1)) for base in itx.count(10, 10))

The intended conceptual output looks like a stream of independent streams: (10, 100, 1000, ...), (20, 400, 8000, ...), (30, 900, 27000, ...), ...

But watch what happens when we split off a couple of inner generators and iterate them:

s0 = next(stream_maker)
next(s0)  # 10
next(s0)  # 100
s1 = next(stream_maker)
next(s1)  # 20
next(s0)  # 8000

The last line produces pow(20, 3), while the expectation was pow(10, 3). The exponent state inside s0 is intact, but the base value has changed as the outer generator advanced.

With a finite version, materializing the inner generators as lists appears to behave “correctly”:

finite_stream_maker = ((pow(base, exp) for exp in (1, 2, 3)) for base in (10, 20, 30))
[list(chunk) for chunk in finite_stream_maker]  # [[10, 100, 1000], [20, 400, 8000], [30, 900, 27000]]

But keeping the inner generators separate leads to the same issue:

finite_stream_maker = ((pow(base, exp) for exp in (1, 2, 3)) for base in (10, 20, 30))
s0 = next(finite_stream_maker)
s1 = next(finite_stream_maker)
next(s0)  # 20, should be 10

What’s going on

The inner generator expression closes over the outer loop variable. As the outer generator progresses, its loop variable gets rebound, and the inner generator—still referencing the same variable—observes the new value. That explains why the exponent state doesn’t jump, but the base does.

There’s a related detail in Python’s generator expression semantics that makes the fix possible. As documented:

the iterable expression in the leftmost for clause is immediately evaluated, so that an error produced by it will be emitted at the point where the generator expression is defined, rather than at the point where the first value is retrieved.

If we can bind the current value of the base into the inner generator’s local scope during creation, the inner stream won’t depend on the outer variable afterwards.

Fix: capture the current base per inner generator

The standard technique is to force a one-item iterable in the inner generator and loop over it first. That assigns the current outer value to a distinct local variable inside each inner generator, breaking the shared reference.

import itertools as itx
stream_maker = (
    (pow(base_local, exp) for base_local in [base] for exp in itx.count(1))
    for base in itx.count(10, 10)
)

When an inner generator is created, the list [base] is evaluated immediately and stored for that generator instance. The comprehension then assigns base_local to that captured value, so subsequent advances of the outer generator can’t affect previously produced inner generators.

Alternative constructions that avoid the pitfall

You can also express the same idea using tools that don’t rely on the inner generator closing over the outer variable.

import itertools as itx
stream_maker = (
    map(pow, itx.repeat(base), itx.count(1))
    for base in itx.count(10, 10)
)

import itertools as itx
stream_maker = (
    map(base.__pow__, itx.count(1))
    for base in itx.count(10, 10)
)

import itertools as itx
import functools as ftools
stream_maker = (
    map(ftools.partial(pow, base), itx.count(1))
    for base in itx.count(10, 10)
)

import itertools as itx
import operator as oper
stream_maker = (
    itx.accumulate(itx.repeat(base), oper.mul)
    for base in itx.count(10, 10)
)

Each of these binds the current base at the moment the inner iterator is created, so earlier streams stay stable as new ones are produced.

Why this matters

Generator factories are an elegant way to build layered, lazy pipelines in Python. But without guarding against shared closures, you can end up with subtle cross-stream coupling that’s hard to diagnose. Understanding how generator expressions capture variables—and how to intentionally capture a value at creation time—prevents data from drifting as the outer iterator advances.

Takeaways

If an inner generator depends on an outer loop variable, assume that variable will be shared unless you explicitly capture it. A simple pattern such as for base_local in [base] binds the value into the inner generator’s local scope, preserving independence. Where it reads better or fits the use case, consider map, repeat, partial, or accumulate to construct the inner stream without relying on the outer closure. With these patterns, your generator-of-generators will produce stable, independent sequences as intended.