Python Logging to File: A Comprehensive Guide

While printing logs to the console (stdout or stderr) is useful during development, directing logs to a file is mandatory for production environments. File logging provides a persistent record of application events, crucial for debugging, auditing, and long-term monitoring.

The Python logging module manages file output through File Handlers. This guide covers the simplest setup, advanced rotation, and common configuration patterns.

1. Basic File Logging Setup

The simplest way to send all log messages to a single file is by using logging.basicConfig().

Code Example: basicConfig

import logging
import time

# 1. Set the root logger configuration
logging.basicConfig(
    level=logging.INFO,             # Minimum level to capture
    filename='app.log',             # The file to write logs to
    filemode='a',                   # Mode: 'a' (append) or 'w' (overwrite)
    format='%(asctime)s - %(levelname)s - %(name)s - %(message)s'
)

# 2. Get a logger instance and log messages
logger = logging.getLogger("MyApp")

logger.info("Application starting up...")
time.sleep(0.01)
logger.warning("Configuration file not found, using defaults.")
time.sleep(0.01)
logger.error("Database connection failed.")

Output in app.log:

2025-12-14 06:45:33,123 - INFO - MyApp - Application starting up...
2025-12-14 06:45:33,133 - WARNING - MyApp - Configuration file not found, using defaults.
2025-12-14 06:45:33,143 - ERROR - MyApp - Database connection failed.

---

2. The File Handler: Fine-Grained Control

For more complex setups-such as separating console output from file output, or using multiple files-you must manually configure and attach a FileHandler.

Code Example: Manual File Handler

import logging
import sys

# 1. Create a logger instance and set its level
app_logger = logging.getLogger('my_service')
app_logger.setLevel(logging.DEBUG) 

# 2. Create the File Handler
file_handler = logging.FileHandler('debug_full.log', mode='w') # Overwrite the file on restart
file_handler.setLevel(logging.DEBUG) # Send everything (DEBUG and up) to the file

# 3. Create a Formatter for the file output
file_formatter = logging.Formatter('%(asctime)s - %(module)s - %(funcName)s - %(message)s')
file_handler.setFormatter(file_formatter)

# 4. Attach the handler to the logger
app_logger.addHandler(file_handler)

# 5. Add a separate console handler for visibility
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(logging.WARNING) # Console only shows WARNING and up
app_logger.addHandler(console_handler)

app_logger.debug("This is ONLY in the file.")
app_logger.error("This is in BOTH the console and the file.")

---

3. Log Rotation: Mandatory for Production

In a long-running application, a single log file will grow indefinitely, eventually consuming all disk space and leading to performance issues. Log rotation automatically closes the current log file, renames it, and starts a new file when a certain size or time limit is reached.

A. Size-Based Rotation: RotatingFileHandler

The RotatingFileHandler rotates the file when it exceeds a specified size.

from logging.handlers import RotatingFileHandler
import logging

LOG_FILE = 'rotating_app.log'
MAX_BYTES = 1024 * 1024 * 5  # 5 MB
BACKUP_COUNT = 5             # Keep 5 backup files (app.log.1, app.log.2, etc.)

# 1. Create the Rotating Handler
rotating_handler = RotatingFileHandler(
    LOG_FILE, 
    maxBytes=MAX_BYTES, 
    backupCount=BACKUP_COUNT,
    encoding='utf-8'
)
rotating_handler.setLevel(logging.INFO)

# 2. Attach to a logger
rotation_logger = logging.getLogger('RotationService')
rotation_logger.setLevel(logging.INFO)
rotation_logger.addHandler(rotating_handler)

rotation_logger.info("Log messages will rotate when file size exceeds 5MB.")

B. Time-Based Rotation: TimedRotatingFileHandler

The TimedRotatingFileHandler rotates the file based on a specific time interval (daily, hourly, etc.).

from logging.handlers import TimedRotatingFileHandler
import logging

# Rotate daily at midnight, keeping 7 days of logs
daily_handler = TimedRotatingFileHandler(
    'daily_app.log', 
    when='midnight', # Rotate daily at midnight
    interval=1,      # Every 1 period (i.e., every day)
    backupCount=7    # Keep 7 days of historical logs
)

time_logger = logging.getLogger('DailyService')
time_logger.setLevel(logging.INFO)
time_logger.addHandler(daily_handler)

time_logger.info("This log file will automatically rotate every night.")

when options for time-based rotation: 'h' (hourly), 'm' (minute), 's' (second), 'midnight' (daily rotation).

---

4. Best Practice: Configuration via Dictionary

For complex production applications, manually configuring loggers and handlers can become verbose and messy. Python supports loading the entire logging configuration from a dictionary, often loaded from a YAML or JSON file.

This centralizes all settings-levels, formatters, handlers, and loggers-in a single place, making configuration management much cleaner.

import logging.config
import yaml # Requires PyYAML

# Assume this config is loaded from logging_config.yaml
LOGGING_CONFIG = {
    'version': 1,
    'handlers': {
        'file': {
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': 'prod.log',
            'maxBytes': 1048576, # 1MB
            'backupCount': 3,
            'formatter': 'standard',
        },
    },
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        }
    },
    'loggers': {
        '': {  # Root logger
            'handlers': ['file'],
            'level': 'WARNING',
            'propagate': True
        }
    }
}

# Load the configuration
logging.config.dictConfig(LOGGING_CONFIG)

# All loggers now follow the rules defined in LOGGING_CONFIG
logging.getLogger().warning("System is fully configured.")

---

Sources and Further Reading

1. Python Documentation - FileHandler and basicConfig
2. Python Documentation - RotatingFileHandler
3. Python Documentation - TimedRotatingFileHandler
4. Python Documentation - dictConfig (Configuration from a dictionary)