Python multiprocessing.Queue on Windows: why processes hang after exceptions and how to exit cleanly

Why Python multiprocessing.Queue on Windows hangs after an exception: feeder thread and pipe buffer explained, plus fixes via draining or cancel_join_thread.

Python multiprocessing.Queue on Windows: when a simple script hangs after an exception

If you push a few hundred items into a Python multiprocessing.Queue on Windows and immediately raise an exception, you may see a puzzling hang instead of a clean exit. With small batches the process exits with code 1 as expected; bump the size slightly and the interpreter stalls until you kill it, often leaving an exit code of -1. The behavior was observed on Windows 10 across Python 3.13.1, 3.12.7, and 3.10.14, while the same minimal scenario didn’t reproduce on Linux Red Hat.

Minimal reproducer

The following snippet fills a queue and then raises a ValueError. With certain sizes on Windows (for example, 717 and above), it raises the error and then hangs.

from multiprocessing import Queue
payloads = list(range(717))
outbox: Queue = Queue()
for val in payloads:
    outbox.put(val)
raise ValueError(f"my len: {len(payloads)}")

What actually happens

Putting an item into multiprocessing.Queue appends it to an internal list. A background worker thread then feeds that list into an IPC pipe. The process will not exit until the worker has emptied the internal list into the pipe. The pipe itself has a small buffer, which is why small batches exit fine: everything fits quickly and the process can terminate. This design keeps queue.put non-blocking and generally faster.

When the internal list is empty, data may still sit in the OS pipe buffers. Buffer size varies but is typically 8192 bytes in Python, which explains why sizes below 716 work fine while 717 and above can trigger the hang on Windows.

The observed difference between Windows and Linux in this specific setup reflects that behavior; the minimal hang did not reproduce on Linux Red Hat for the scenario described.

How to make the process exit reliably

There are two pragmatic ways forward, depending on whether anyone will read from the queue.

If the data should be consumed, ensure the queue is drained before the process exits. You can read pending items in the main process while coordinating shutdown. The important bit is to pull items until the queue is empty so the feeder thread has nothing left to write.

from multiprocessing import Queue
from queue import Empty
items = list(range(2000))
msg_queue: Queue = Queue()
for x in items:
    msg_queue.put(x)
try:
    while True:
        pending = msg_queue.get_nowait()
        # process pending if needed
except Empty:
    pass
raise ValueError(f"my len: {len(items)}")

If no one will read the queue and you just want to discard what was enqueued, allow the process to exit without draining by calling cancel_join_thread on that Queue. This can leave the queue in a broken state and should be invoked only from the main process.

from multiprocessing import Queue
bulk = list(range(2000))
q: Queue = Queue()
for item in bulk:
    q.put(item)
q.cancel_join_thread()
raise ValueError(f"my len: {len(bulk)}")

With cancel_join_thread, the example exits with code 1 after raising the error even for large batches.

Why this matters

Silent hangs on process shutdown are difficult to diagnose and can lead to inconsistent exit codes, stalled CI pipelines, and timeouts in orchestrators. The threshold-like behavior is especially confusing: a script runs fine with a small number of puts and then locks up when the batch crosses what looks like an arbitrary limit. Knowing that a background feeder thread and the pipe buffer are involved helps explain the pattern and points to solutions that are simple and explicit.

Takeaways

When working with multiprocessing.Queue on Windows, treat the queue as having a feeder that must flush internal state before the process can exit. If your process is about to terminate, either consume everything from the queue or explicitly signal that you don’t care about draining by calling cancel_join_thread from the main process. If you need deterministic shutdown under error conditions, drain or cancel before raising, and test with larger batches to surface the behavior early.