Managing Application Logs with logrotate

When an application on Linux continuously writes to its own log file, the log can grow indefinitely and consume disk space if left unmanaged. With logrotate, you can define rotation conditions, the number of retained generations, compression, and post-rotation processing for each log.

On the other hand, if the process writing to the log keeps the file open, simply renaming the file does not switch writing to a new log file. This makes it important to decide whether to use copytruncate or use postrotate to make the application reopen the log or restart.

In this article, we will prepare a test log-writing service on an RHEL-compatible Linux system and verify the following:

configure rotation conditions, retention generations, and compression
perform a dry-run of the configuration with logrotate -d
force rotation with logrotate -f
verify that writing continues when copytruncate is used
verify the behavior when copytruncate is not used
use postrotate to make the process open a new log file
verify that old logs exceeding the retention generations are deleted

Step 1: Install logrotate

First, install logrotate.

sudo dnf install -y logrotate

Step 2: Prepare a sample service that continuously writes logs

To verify the difference between copytruncate and the normal rename method, create a service that opens the log file once and then continues writing through the same file descriptor.

sudo tee /usr/local/bin/example-log-writer.sh >/dev/null <<'EOF'
#!/bin/bash
exec >>/var/log/example-app.log 2>&1
while true; do
    printf '%s example application heartbeat\n' "$(date -Is)"
    sleep 1
done
EOF
sudo chmod 0755 /usr/local/bin/example-log-writer.sh

sudo tee /etc/systemd/system/example-log-writer.service >/dev/null <<'EOF'
[Unit]
Description=Example application log writer

[Service]
Type=simple
ExecStart=/usr/local/bin/example-log-writer.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo touch /var/log/example-app.log
sudo chmod 0640 /var/log/example-app.log
sudo systemctl daemon-reload
sudo systemctl enable --now example-log-writer.service
sudo systemctl is-active example-log-writer.service

If active is displayed, the service is running.

Also verify that the log file has been created and that data is being written to it.

sudo test -s /var/log/example-app.log
sudo stat -c '%n %s bytes inode=%i' /var/log/example-app.log

Step 3: Create a configuration that uses copytruncate

First, try the copytruncate method. In this example, two generations are retained and older logs are compressed with gzip.

sudo tee /etc/logrotate.d/example-app-copytruncate >/dev/null <<'EOF'
/var/log/example-app.log {
    size 1
    rotate 2
    compress
    missingok
    notifempty
    copytruncate
}
EOF

sudo grep -E 'size|rotate|compress|copytruncate' /etc/logrotate.d/example-app-copytruncate

The main settings have the following meanings:

Directive	Role
`size 1`	Uses the log size as the condition for rotation
`rotate 2`	Retains two generations of rotated logs
`compress`	Compresses older logs with gzip
`missingok`	Does not produce an error if the log does not exist
`notifempty`	Does not rotate an empty log
`copytruncate`	Copies the current log and then truncates the original file

With copytruncate, the file itself that the application has open can be retained. This means it can also be used when the application does not provide a log reopen function.

However, copying the log and truncating the original file do not happen at exactly the same time. Data written during that interval may be lost, so when possible, prefer a method in which the application itself reopens the log.

Step 4: Check the configuration with a dry-run

Before actually modifying the log, you can use -d to check configuration parsing and the rotation decision.

sudo logrotate -d /etc/logrotate.d/example-app-copytruncate && printf '%s\n' 'dry-run-ok'

In debug mode, logs are not rotated and the state file is not updated. This can be used to check syntax immediately after changing the configuration and to verify which logs are selected for rotation.

Step 5: Force rotation with copytruncate

Using -f forces execution regardless of the normal rotation schedule. After execution, verify that writing to the current log continues.

sudo logrotate -f /etc/logrotate.d/example-app-copytruncate
before=$(sudo stat -c %s /var/log/example-app.log)
after=$before
for i in 1 2 3 4 5; do
    sleep 1
    after=$(sudo stat -c %s /var/log/example-app.log)
    if [ "$after" -gt "$before" ]; then
        break
    fi
done
sudo test "$after" -gt "$before"
sudo test -f /var/log/example-app.log.1.gz
printf '%s\n' 'copytruncate-ok'

With copytruncate, the original example-app.log remains at the same path and is truncated, so the process writing the log can continue writing to the same file.

Step 6: Check a configuration without copytruncate

Next, remove copytruncate. With normal rotation, the current log is moved to another name, and create creates a new current log file.

At this stage, do not configure postrotate yet so that the difference can be observed.

sudo rm -f /etc/logrotate.d/example-app-copytruncate
sudo rm -f /var/log/example-app.log.*
sudo systemctl restart example-log-writer.service

sudo tee /etc/logrotate.d/example-app >/dev/null <<'EOF'
/var/log/example-app.log {
    size 1
    rotate 2
    nocompress
    missingok
    notifempty
    create 0640 root root
}
EOF

sudo grep -E 'size|rotate|nocompress|create' /etc/logrotate.d/example-app

Step 7: Verify that writing continues to the old file after rename

When normal rotation is performed on a process that still has the log file open, the file descriptor held by the process continues to point to the old renamed file.

sudo test -s /var/log/example-app.log
sudo logrotate -f /etc/logrotate.d/example-app
old_before=$(sudo stat -c %s /var/log/example-app.log.1)
old_after=$old_before
for i in 1 2 3 4 5; do
    sleep 1
    old_after=$(sudo stat -c %s /var/log/example-app.log.1)
    if [ "$old_after" -gt "$old_before" ]; then
        break
    fi
done
sudo test "$old_after" -gt "$old_before"
sudo test ! -s /var/log/example-app.log
printf '%s\n' 'non-copytruncate-ok'

sudo systemctl restart example-log-writer.service
current_before=$(sudo stat -c %s /var/log/example-app.log)
current_after=$current_before
for i in 1 2 3 4 5; do
    sleep 1
    current_after=$(sudo stat -c %s /var/log/example-app.log)
    if [ "$current_after" -gt "$current_before" ]; then
        break
    fi
done
sudo test "$current_after" -gt "$current_before"
printf '%s\n' 'reopen-ok'

Immediately after rotation, even though a new example-app.log is created, the process continues writing to the renamed example-app.log.1.

When the service is restarted, it reopens the file and begins writing to the new example-app.log.

If the actual application provides a log reopen function using SIGHUP or a similar mechanism, that function can be used instead of restarting the entire service.

Step 8: Reopen the log with postrotate

In the final configuration, enable compression and restart the sample service with postrotate after rotation. This causes the service to open the newly created current log.

sudo rm -f /var/log/example-app.log.*

sudo tee /etc/logrotate.d/example-app >/dev/null <<'EOF'
/var/log/example-app.log {
    size 1
    rotate 2
    compress
    missingok
    notifempty
    create 0640 root root
    postrotate
        /usr/bin/systemctl restart example-log-writer.service
    endscript
}
EOF

sudo grep -E 'size|rotate|compress|create|postrotate|systemctl' /etc/logrotate.d/example-app

In a production environment, if the target application provides a signal or dedicated command for reopening logs, calling it from postrotate can reduce service interruption compared with restarting the entire service.

Step 9: Verify that postrotate is executed

Compare the service PID before and after rotation. If the PID changes and writing to the new current log begins, the restart performed by postrotate is working.

before_pid=$(sudo systemctl show -p MainPID --value example-log-writer.service)
sudo logrotate -f /etc/logrotate.d/example-app
after_pid=$(sudo systemctl show -p MainPID --value example-log-writer.service)
sudo test "$before_pid" != "$after_pid"

current_size=0
for i in 1 2 3 4 5; do
    current_size=$(sudo stat -c %s /var/log/example-app.log)
    if [ "$current_size" -gt 0 ]; then
        break
    fi
    sleep 1
done
sudo test "$current_size" -gt 0
printf '%s\n' 'postrotate-restart-ok'

postrotate is executed when rotation actually occurs. It is not executed during a simple configuration check.

Step 10: Verify retention generations and compression

Finally, perform several forced rotations and verify that rotate 2 and compress work as expected.

Before each rotation, add a sample line so the file is not skipped as empty.

sudo rm -f /var/log/example-app.log.*
for i in 1 2 3 4; do
    printf 'manual rotation sample %s\n' "$i" | sudo tee -a /var/log/example-app.log >/dev/null
    sudo logrotate -f /etc/logrotate.d/example-app
done

sudo test -f /var/log/example-app.log.1.gz
sudo test -f /var/log/example-app.log.2.gz
sudo test ! -e /var/log/example-app.log.3.gz
sudo ls -l /var/log/example-app.log /var/log/example-app.log.1.gz /var/log/example-app.log.2.gz
printf '%s\n' 'retention-ok'

With rotate 2, up to two generations of rotated files are retained, and older generations are deleted. Because compress is specified, the retained rotated logs are stored in gzip format.

How to choose between copytruncate and postrotate

copytruncate is useful when the application writing the log cannot be stopped or made to reopen the log file. However, there is a possibility of losing log data during the short interval between copying and truncate.

The basic decision can be made as follows:

Condition	Choice
The application supports reopening the log file	Prefer normal rename with `postrotate`
The log can be reopened with SIGHUP or a similar mechanism	Perform the reopen operation from `postrotate`
Restarting is the only way to reopen the log	Restart the service from `postrotate` if acceptable
The application cannot be stopped or made to reopen the log	Consider `copytruncate`

After changing the configuration, first verify it with logrotate -d, then run logrotate -f against a log that can be safely tested to confirm the behavior.