Setting Up Logrotate for Custom Linux Server Logs

Setting Up Logrotate for Custom Linux Server Logs

What You’ll Need

  • A Linux server hosted on Hetzner VPS or Contabo VPS running Ubuntu, Debian, Rocky Linux, or Ubuntu Server
  • A DigitalOcean droplet as an alternative hosting environment
  • System administrative access with standard sudo privileges
  • A domain name managed via Namecheap if you are routing web traffic to your server application

Table of Contents


Understanding Logrotate Architecture on Modern Linux Systems

Running microservices, custom APIs, or background workers on a cloud server like a Hetzner VPS requires proper log maintenance. Custom applications that emit unstructured standard output or structured JSON files can consume available disk space within days if left unchecked. Unbounded log files lead to filesystem exhaustion, kernel panics, and unexpected application failures.

Logrotate is the standard system utility designed to parse, compress, rename, and remove aging system log files. Rather than running continuously as a heavy background daemon, logrotate is typically invoked on a schedule by standard system automation utilities like cron or systemd timers.

The master configuration file lives at /etc/logrotate.conf. This file sets global defaults such as default rotation frequency, file creation permissions, and compression rules. At the bottom of /etc/logrotate.conf, an inclusion directive reads external configuration files:

include /etc/logrotate.d

This modular architecture allows system administrators to isolate application settings. Instead of modifying the global configuration file, you create modular definition files inside the /etc/logrotate.d/ directory. When logrotate executes, it aggregates every file inside this folder, evaluates their matching paths against filesystem state, and executes rotation actions according to your specific rules.

Understanding how logrotate interacts with open file descriptors is essential. Linux systems track open files using inodes rather than path strings. If logrotate simply renames /var/log/custom-app/api.log to /var/log/custom-app/api.log.1, your application process will keep holding the file descriptor for inode pointing to api.log.1. The application continues writing data to the renamed file, frustrating your cleanup efforts. Logrotate offers two strategies to solve this: process signaling via postrotate hooks, or copy-truncation via the copytruncate directive.


Step 1: Preparing Custom Application Directories and Log Output

To demonstrate log management, let us set up a dedicated workspace for a multi-threaded web application. We will create a non-root system user, build dedicated logging directories, set appropriate access control permissions, and construct a Python application script that streams high-volume logs.

Executing custom worker scripts is common when designing idempotent webhook receivers for distributed systems. High throughput endpoints generate massive event trails that require structured log rotation rules.

First, create the dedicated system user and log directories on your server:

sudo useradd --system --shell /bin/false sys3app
sudo mkdir -p /var/log/sys3app
sudo chown -R sys3app:sys3app /var/log/sys3app
sudo chmod 0750 /var/log/sys3app

Now create a Python script located at /usr/local/bin/sys3_worker.py that continuously generates simulated API log entries into two separate files: access.log and error.log.

#!/usr/bin/env python3
import time
import os
import sys
from datetime import datetime

LOG_DIR = "/var/log/sys3app"
ACCESS_LOG = os.path.join(LOG_DIR, "access.log")
ERROR_LOG = os.path.join(LOG_DIR, "error.log")

def write_log(filepath, message):
    timestamp = datetime.utcnow().isoformat()
    with open(filepath, "a") as f:
        f.write(f"[{timestamp}] {message}\n")

def main():
    if not os.path.exists(LOG_DIR):
        sys.exit(1)
        
    counter = 0
    while True:
        counter += 1
        write_log(ACCESS_LOG, f"INFO processing request status=200 path=/api/v1/resource id={counter}")
        if counter % 5 == 0:
            write_log(ERROR_LOG, f"ERROR database connection timeout query_id={counter}")
        time.sleep(0.1)

if __name__ == "__main__":
    main()

Make the script executable and set up a systemd service file at /etc/systemd/system/sys3-worker.service so the application runs continuously in the background:

[Unit]
Description=Sys3 Custom Application Worker
After=network.target

[Service]
Type=simple
User=sys3app
Group=sys3app
ExecStart=/usr/usr/bin/python3 /usr/local/bin/sys3_worker.py
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Enable and start your newly created service using systemctl:

sudo systemctl daemon-reload
sudo systemctl enable --now sys3-worker.service

Verify that log streams are accumulating inside your log directory:

ls -la /var/log/sys3app/
tail -n 5 /var/log/sys3app/access.log
tail -n 5 /var/log/sys3app/error.log

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.


Step 2: Writing a Production Logrotate Configuration File

Now that our application streams data into /var/log/sys3app/access.log and /var/log/sys3app/error.log, we must define explicit rotation policies. Create a new logrotate configuration file located at /etc/logrotate.d/sys3-app.

/var/log/sys3app/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    dateext
    dateformat -%Y%m%d-%s
    create 0640 sys3app sys3app
    sharedscripts
    copytruncate
}

Let us analyze each directive used in this production configuration:

  1. /var/log/sys3app/*.log: Target path pattern. Applying wildcards ensures all files ending in .log within this directory are evaluated.
  2. daily: Triggers log rotation once per calendar day. Alternative interval directives include hourly, weekly, or monthly.
  3. rotate 14: Keeps 14 rotated log files before purging the oldest archives. Combined with daily, this maintains a two-week historical rolling audit window.
  4. missingok: Prevents system errors. If the log directory is missing or empty during execution, logrotate skips processing without raising fatal error codes.
  5. notifempty: Skips file processing if the log file is zero bytes. This avoids generating empty archive files.
  6. compress: Compresses rotated log files using standard gzip compression to reduce disk space overhead.
  7. delaycompress: Postpones compression of the most recent rotated file until the next rotation cycle. The active log becomes access.log.1 as plain text, while access.log.2.gz is compressed. This accommodates applications that continue writing to an old file handle immediately after rotation.
  8. dateext: Appends a date string extension to rotated log files instead of using sequential numbers like .1 or .2.
  9. dateformat -%Y%m%d-%s: Defines the timestamp pattern appended by dateext. Using standard Year-Month-Day followed by Unix epoch timestamp ensures unique filenames during manual or sub-daily force rotations.
  10. create 0640 sys3app sys3app: Instructs logrotate to immediately create a fresh, empty log file after rotation, applying Unix permissions 0640 owned by user sys3app and group sys3app.
  11. sharedscripts: Ensures that pre-rotation or post-rotation hook scripts execute only once per cycle, rather than once for every file matching the target glob pattern.
  12. copytruncate: Copies the active log file content to a backup location, then truncates the original file in-place to zero bytes. This strategy allows long-running daemon processes to continue writing to their open file handle without requiring a process restart or signal handling logic.

Step 3: Managing File Descriptors and Signal Handling

While copytruncate works for simple applications, high-performance production services like web servers, proxy layers, and system daemons require direct signal handling to rotate files cleanly without IO race conditions.

During heavy log output, copytruncate can lose log lines between the copying phase and the truncation phase. To solve this, servers emit a POSIX signal (typically SIGHUP or SIGUSR1) instructing the running application to close its current file descriptor and re-open the file path on disk.

When implementing SSL setups covered in our Nginx SSL certificate renewal automation guide, process signaling ensures web servers drop old log handles cleanly without interrupting active HTTPS traffic.

Below is a configuration file demonstrating process signaling via postrotate hooks instead of copytruncate. Create this configuration file at /etc/logrotate.d/sys3-signaled:

/var/log/sys3app/access.log
/var/log/sys3app/error.log {
    weekly
    size 100M
    rotate 8
    missingok
    notifempty
    compress
    delaycompress
    su sys3app sys3app
    create 0640 sys3app sys3app
    sharedscripts
    postrotate
        /usr/bin/systemctl reload sys3-worker.service > /dev/null 2>&1 || true
    endscript
}

Key features introduced in this configuration block:

  • Explicit block target lists: Path definitions listed across separate lines scope rules specifically to listed paths.
  • size 100M: Overrides standard schedule rules. Rotation triggers whenever a log file reaches 100 Megabytes in size, regardless of whether a week has passed.
  • su sys3app sys3app: Explicitly instructs logrotate to switch user and group privileges to sys3app before touching log files. This avoids permission mismatches when accessing paths secured by tight Access Control Lists (ACLs).
  • postrotate ... endscript: Execution block containing custom shell commands executed immediately after log files are moved. The line /usr/bin/systemctl reload sys3-worker.service sends an instruction to your system daemon to flush buffers and re-open file descriptors.

Background scheduling tools, such as scripts created when executing scheduled Python tasks using APScheduler, benefit from signal hooks by releasing file handles cleanly during automated jobs.


Step 4: Testing, Debugging, and Systemd Automation

Before leaving logrotate configs in production, you must validate syntax and test execution mechanics using built-in debug commands.

Validating Configuration Syntax

Run logrotate in debug mode using the -d (or --debug) flag. Debug mode prints execution steps without modifying files, renaming logs, or executing script blocks:

sudo logrotate -d /etc/logrotate.d/sys3-app

Output should show matching target files and planned rotation steps:

reading config file /etc/logrotate.d/sys3-app
Reading state from /var/lib/logrotate/status

Allocating hash table for state file, 1000 entries

Handling 1 logs

rotating pattern: /var/log/sys3app/*.log  after 1 days (14 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/sys3app/access.log
  Now: 2026-03-30 12:00
  Last rotated at 2026-03-30 00:00
  log does not need rotating (log has already been rotated)
considering log /var/log/sys3app/error.log
  Now: 2026-03-30 12:00
  Last rotated at 2026-03-30 00:00
  log does not need rotating (log has already been rotated)

Executing a Force Rotation

To test actual file system modifications, compression creation, and postrotate script execution, trigger a forced rotation using the -f (or --force) flag alongside the verbose -v flag:

sudo logrotate -fv /etc/logrotate.d/sys3-app

Verify that rotated log files exist in your target log directory:

ls -la /var/log/sys3app/

Expected filesystem output:

total 24
drwxr-xr-0 2 sys3app sys3app 4096 Mar 30 12:05 .
drwxr-xr-x 8 root    root    4096 Mar 30 12:00 ..
-rw-r----- 1 sys3app sys3app 1240 Mar 30 12:05 access.log
-rw-r----- 1 sys3app sys3app 9812 Mar 30 12:05 access.log-20260330-1743336300
-rw-r----- 1 sys3app sys3app  320 Mar 30 12:05 error.log
-rw-r----- 1 sys3app sys3app 1940 Mar 30 12:05 error.log-20260330-1743336300

Checking Systemd Timers and Status Persistence

Modern Linux distributions manage standard logrotate scheduling using systemd timers rather than legacy crontab files. Check the status of your logrotate timer using systemctl:

sudo systemctl status logrotate.timer

To view next scheduled execution windows across system tasks, query systemd timer listings:

sudo systemctl list-timers --all | grep logrotate

Logrotate tracks rotation history inside a state tracking file located at /var/lib/logrotate/status (or /var/lib/logrotate/logrotate.status on older Debian releases). Inspect state records for your custom application using grep:

grep "sys3app" /var/lib/logrotate/status

Output:

"/var/log/sys3app/access.log" 2026-3-30-12:05:0
"/var/log/sys3app/error.log" 2026-3-30-12:05:0

Getting Started

Deploying custom applications on standard server infrastructure requires proactive log management. By setting up logrotate on your cloud instances hosted with Hetzner VPS or DigitalOcean, you prevent unexpected disk exhaustion, maintain reliable system logs, and ensure consistent application performance. If you need domain routing for your public-facing apps, configure DNS management via Namecheap.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3
system online