Control Python ThreadPoolExecutor subprocess batches with UNIX signals: pause with SIGUSR1, exit cleanly with SIGTERM in WSL

Learn how to pause scheduling and shut down gracefully in Python ThreadPoolExecutor workloads with UNIX signals, setproctitle, SIGUSR1, SIGTERM, pkill on WSL.

Controlling a pool of long-running subprocess tasks after launch is a common need: pause scheduling, let in-flight work finish, and resume later or exit cleanly. When a Python workload uses ThreadPoolExecutor to dispatch subprocess calls, the first instinct is often to wire a keyboard hotkey. Under WSL terminals, however, hook-based libraries like pynput or keyboard may not receive key events as expected, and blocking input is not acceptable. The approach below replaces hotkeys with UNIX signals for robust, scriptable runtime control.

Minimal example that illustrates the control problem

The outline is straightforward: a pool dispatches subprocess tasks, but there is no reliable way to pause scheduling once the run has started.

import concurrent.futures
import subprocess
import sys
import time
def do_work(arg):
    # Simulate a slow subprocess call
    return subprocess.run([
        sys.executable,
        "-c",
        f"import time; time.sleep({arg})"
    ])
def main():
    durations = [5, 4, 6, 3, 5, 4]
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
        future_list = [pool.submit(do_work, d) for d in durations]
        for fut in concurrent.futures.as_completed(future_list):
            _ = fut.result()
if __name__ == "__main__":
    main()

Once the pool starts submitting tasks, there is no non-blocking way to pause or stop scheduling more work using a simple keystroke in the WSL terminal. Libraries that listen for global hotkeys may not see the keystrokes inside that environment, and using input would block the main flow. File-based signaling is also undesirable in this scenario.

What is actually going on

The crux of the issue is not the executor itself but the control channel. Hotkey hooks may not deliver events to a Python process running in a WSL terminal, leaving no reliable way to toggle state without blocking reads or polling files. The workload remains fire-and-forget once submitted.

Solution: UNIX signals, process title, and explicit handlers

The practical path is to use UNIX signals to control the Python process externally. First, assign a distinct process title using setproctitle so the process can be targeted by name. Then add handlers for SIGTERM and SIGUSR1. With SIGUSR1 you can instruct the scheduler to stop queuing new tasks while allowing running subprocesses to finish. With SIGTERM you can exit gracefully. Because the process has a recognizable title, pkill -USR1 -f [exe_name] is enough to signal it without manually hunting for PIDs.

End-to-end implementation

The following code uses a named process, reacts to SIGUSR1 by stopping further scheduling, and reacts to SIGTERM by performing a graceful shutdown. Running subprocesses are left to finish. The logic mirrors the approach above while making control explicit and observable.

import concurrent.futures
import signal
import subprocess
import sys
import threading
import time
from setproctitle import setproctitle
# Global control flags manipulated by signals
halt_queue = threading.Event()
terminate_run = threading.Event()
def on_usr1(sig, frame):
    # Stop scheduling new tasks; let in-flight tasks finish
    halt_queue.set()
def on_term(sig, frame):
    # Request a graceful stop: halt new tasks and proceed to shutdown
    halt_queue.set()
    terminate_run.set()
def worker_job(delay_s):
    # Simulate a slow subprocess call without killing it on control events
    return subprocess.run([
        sys.executable,
        "-c",
        f"import time; time.sleep({delay_s})"
    ])
def orchestrate():
    # Give the process an identifiable name for pkill -f targeting
    setproctitle("batch-runner-example")
    # Wire signal handlers
    signal.signal(signal.SIGUSR1, on_usr1)
    signal.signal(signal.SIGTERM, on_term)
    tasks = [5, 4, 6, 3, 5, 4, 7, 2]
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as exe:
        submitted = []
        for item in tasks:
            if halt_queue.is_set():
                break
            fut = exe.submit(worker_job, item)
            submitted.append(fut)
        for fut in concurrent.futures.as_completed(submitted):
            _ = fut.result()
    # Optional post-run logic after all submitted tasks complete
    if terminate_run.is_set():
        # Gracious shutdown already done
        pass
if __name__ == "__main__":
    orchestrate()

With this in place, the control flow is outside-in and scriptable. Set the process title once and send signals by name, without blocking stdin or relying on key listeners in the terminal.

How to send the signals by name

Signals can be dispatched without chasing the PID. Use the process title assigned in the script. Sending SIGUSR1 stops scheduling more tasks while letting current subprocesses finish. Sending SIGTERM performs a graceful stop using the same mechanism by default.

# Pause scheduling further tasks (let running ones finish)
pkill -USR1 -f batch-runner-example
# Ask for graceful termination (default signal is SIGTERM)
pkill -f batch-runner-example

It is also possible to look up the process with ps and use kill directly if preferred.

On Linux you can use ps to find the process_id and later kill process_id to send a signal (by default SIGTERM). Application code needs a signal handler to react appropriately.

Generally you handle signals sent to a Python process by implementing a signal handler (signal standard library module).

As an alternative for parallel execution itself, some practitioners recommend GNU Parallel for running scripts in parallel. The approach above focuses on Python-side control via signals.

Why this matters

Unix signals give you predictable, low-friction control in environments where keyboard hooks are unreliable or undesirable. You avoid Ctrl+C, which would kill subprocesses immediately, and you skip blocking input or file polling. The result is a workload that can be paused from the outside and allowed to drain safely, without invasive changes to the execution model.

Takeaways

If you need to pause scheduling or terminate gracefully in a ThreadPoolExecutor that launches subprocesses, favor signals over hotkey listeners in WSL/Linux terminals. Set a distinct process title with setproctitle, add handlers for SIGUSR1 and SIGTERM, and gate task submission on a shared event. Drive control with pkill -USR1 -f [exe_name] to stop queuing new tasks, and rely on the default SIGTERM for an orderly exit. This keeps the control plane simple, scriptable, and robust for long-running batches.