Why Salt minions do not stream stdout to the Python client and how to handle long-running jobs

Learn why Salt's Python client can't stream stdout from long-running jobs on minions, what get_cli_returns and find_job offer, and how to design polling UIs.

When you execute a long-running shell loop on a Salt minion and try to read its output from a Python client, it is tempting to expect a steady stream of lines in real time. In practice, nothing shows up until the job completes. This guide explains why that happens, what Salt actually returns, and how to structure your code and expectations accordingly.

Reproducing the expectation gap

The following Python example kicks off a command that prints a line per second for 50 seconds. The code then immediately asks for results and tries to print them as they arrive.

import salt.client

target = "test"
api = salt.client.LocalClient()


def launch_task(target_id):
    return api.run_job(
        target_id,
        "cmd.run",
        arg=['''for i in {1..50}; do echo "Log line $i at $(date '+%T')"; sleep 1; done''']
    )


def read_results(job_info):
    payload = api.get_cli_returns(
        job_info["jid"],
        job_info["minions"],
        timeout=30,
        verbose=True,
        show_jid=True,
    )
    print(list(payload))
    for item in payload:
        print(f"Response: {item}")


if __name__ == "__main__":
    print("Immediate check -- fails")
    job_meta = launch_task(target)
    print(job_meta)
    read_results(job_meta)

The expectation is continuous output like this:

Response: Log line 1 at 11:47:08
Response: Log line 2 at 11:47:09
Response: Log line 3 at 11:47:10
Response: Log line 4 at 11:47:11
Response: Log line 5 at 11:47:12
...

What actually happens and why

Salt is an async system. The command you send runs on the minion. That command writes to the minion’s console, not to your Salt client. A minion will not send anything back to the master until it is done running whatever you asked it to do. In this example, the loop sleeps one second for each of 50 iterations, so you will not see any return until the loop finishes. Only then will the minion send a result back, and only then will your client code have anything to print.

Salt doesn’t provide a live stream of stdout for running jobs via the Python client API used above. You can ask whether a job is still running using find_job, but that’s as far as runtime visibility goes in this context. Once the job completes, the return is sent and only then does get_cli_returns have data for you.

What to change in your client code

The fix is to adjust expectations and flow. Do not try to tail output in real time. Either wait long enough for the job to complete before collecting results, or track whether it is still running and retrieve the return after completion. The example below simply waits longer than the loop, then fetches the return. There is still no streaming: the result appears once the job is done.

import time
import salt.client

endpoint = "test"
session = salt.client.LocalClient()


def fire_job(match):
    return session.run_job(
        match,
        "cmd.run",
        arg=['''for i in {1..50}; do echo "Log line $i at $(date '+%T')"; sleep 1; done''']
    )


def collect_when_finished(job_meta):
    # Wait longer than the job's runtime; no streaming is available here
    time.sleep(60)
    result = session.get_cli_returns(
        job_meta["jid"],
        job_meta["minions"],
        timeout=90,
        verbose=True,
        show_jid=True,
    )
    for entry in result:
        print(f"Response: {entry}")


if __name__ == "__main__":
    info = fire_job(endpoint)
    collect_when_finished(info)

If you need to know whether a job is still running, use find_job to check its state, and only attempt to fetch the return after it finishes. This confirms liveness but still does not provide incremental stdout.

Why this matters for web UIs

If you plan to surface execution progress in a web page, you cannot rely on the Python client API to stream stdout from a minion. Design the UI to reflect this model: indicate that a job is running, optionally check with find_job whether it is still in progress, and display the result once the job completes. Expect the first output to appear only after the task is finished.

Takeaways

Salt runs commands asynchronously on minions. The stdout produced by a running command is local to the minion until completion. The Python client returns data only after the job finishes, so there is no streaming of output lines. If you need runtime visibility, you can check whether a job is still running with find_job, then retrieve and display the final output at the end. For long-running tasks, make sure your client waits appropriately or signals progress without attempting to tail the output.