The logging module

The logging module is one small file that configures Python's logger for the whole project, so every component writes dated lines carrying a level, a name and a message to the terminal and to a file at the same time.

Last updated: 19 Oct, 2026 · MLOps

Packaging with setup.py made sp importable. This is the first thing worth putting in it. A print statement disappears the moment a pipeline runs somewhere you are not watching, and a model pipeline spends most of its life exactly there: in a container, under a scheduler, inside a CI job. A log line survives, with the time it happened.

The clip writes every run into one running_logs.log; this project writes one file per day under logs/, and the format string and the file-plus-stream handler pair are the same.

Choosing where the log file goes

A file per day keeps a run findable without the folder turning into thousands of files. The folder is created on import, so no component has to remember to make it.

import logging, os, sys
from datetime import datetime

LOG_DIR = "logs"
os.makedirs(LOG_DIR, exist_ok=True)

# one file per day, so a run is easy to find later
LOG_FILE = os.path.join(LOG_DIR, f"{datetime.now():%Y-%m-%d}.log")

Configuring the format and the two handlers

basicConfig sets up the root logger once. The format decides what each line carries, and the handler list decides where the line goes. Two handlers means one line reaches two places.

logging.basicConfig(
    level=logging.INFO,
    format="[%(asctime)s] %(levelname)s %(name)s - %(message)s",
    handlers=[logging.FileHandler(LOG_FILE), logging.StreamHandler(sys.stdout)],
)

logger = logging.getLogger("sp")

• level=logging.INFO drops everything below INFO, so a logger.debug call left in the code costs nothing in production and comes back by changing one word.
• %(asctime)s is the timestamp, down to milliseconds. It is the field that makes two log lines comparable when a stage is slow.
• %(levelname)s is INFO, WARNING or ERROR, which is what a grep looks for first when something went wrong.
• %(name)s is the logger's name, sp here, so project lines stand apart from the lines libraries write into the same stream.
• FileHandler and StreamHandler are the two destinations: the dated file, and standard output, which is where a container's logs are read from.

Using it from a component

Every component imports the one logger and calls it. There is no second configuration step anywhere in the project.

from sp.logger import logger

logger.info(f"split into train={len(train)} test={len(test)}")

A real line from the training pipeline looks like this, which is what makes a log worth writing: the message carries the numbers, so the record answers a question later without the run being repeated.

[2026-10-19 11:03:05,838] INFO sp - split into train=800 test=200
[2026-10-19 11:03:05,876] INFO sp - transformed shapes train=(800, 19) test=(200, 19)

Watching the level filter and the file copy

This run uses the same handler pair and the same logger name, with a short format so the fields line up on the page. One call is below the level and never appears:

import logging, os, sys

os.makedirs("logs", exist_ok=True)
logging.basicConfig(
    level=logging.INFO,
    format="%(levelname)s %(name)s - %(message)s",
    handlers=[logging.FileHandler("logs/demo.log", mode="w"),
              logging.StreamHandler(sys.stdout)],
)

logger = logging.getLogger("sp")
logger.debug("reading params.yaml")        # below INFO, so it is dropped
logger.info("downloaded 1000 rows")
logger.warning("validation found a missing column")
print("--- logs/demo.log now holds ---")
print(open("logs/demo.log").read(), end="")

INFO sp - downloaded 1000 rows
WARNING sp - validation found a missing column
--- logs/demo.log now holds ---
INFO sp - downloaded 1000 rows
WARNING sp - validation found a missing column

• The DEBUG line is absent from both the terminal and the file. The level is a filter on the logger, not on the handlers, so one setting covers every destination.
• WARNING arrives without being asked for. Anything at INFO or above passes, which is why a project logs normal progress at INFO and keeps WARNING for the lines worth noticing.
• The file holds the same two lines as the terminal, byte for byte. That is the point of two handlers on one logger: nothing is written twice by hand and the copies cannot drift.
• The real module's format puts the timestamp first, as the quoted pipeline lines above show. It is left out here so each line fits on one row.

print against a configured logger

Where the log file is the only record

• A pipeline under a scheduler, where the run happened at three in the morning and the file is all there is to read.
• A service in a container, where the platform collects standard output and the handler is what puts the line there.
• A metric that moved, where the logged row counts and shapes say which stage changed before anyone opens the data.

Related

• Previous: Packaging with setup.py
• Next: Custom exceptions
• Reference: logging in the Python documentation