How to redirect standard output and error of a Python program and its subprocesses to another program

published Mar 02, 2024

We'll exploit POSIX compatibility to get our way.

How to redirect standard output and error of a Python program and its subprocesses to another program

I recently had the need to redirect a Python program's output to the system journal.  But not just that program's — all its subprocesses too.  None of the tricks involving redirect_stdout() work for that set of requirements.  Fortunately, there is a very easy way to do exactly this.  Here's what you need to put at the very top of your Python program:

#!/usr/bin/python3

import os, subprocess, sys

# Close all stdio except stdin.
os.close(1)
os.close(2)

# Spawn the program we want to log to. This program must read our output
# via stdin. Spawn the program with its stdin as a pipe to that effect.
# In this example, we're using the standard logger utility that logs
# all its input to the journal or to syslog, but this could very well be
# a program that outputs to a network, or to a file, or something else.
logger = subprocess.Popen(["logger", "-t", "my-logged-program"], stdin=subprocess.PIPE)

# By the magic of POSIX stdio, that pipe just got assigned file descriptor
# number 1. This means our program now has standard output connected to
# the logger process above. But standard error is still closed, so we
# will fix that right in this very line.
os.dup(logger.stdin.fileno())

# Now we have stdout and stderr. Exercise them! print("Stdout") print("Stderr", file=sys.stderr)
# You should see no output on the console. However, that output should
# still appear on your journal or syslog.

With this trick, stuff like sys.stdout.write() will work as well.  And, most importantly, any subprocesses started by your Python program will inherit the stdio of your Python program now — so long as the subprocess handling our stdio is still running, it will receive all child processes' output in its standard input.

Needless to say, the trick above only works on POSIX systems.  It will not work on Windows.