systemd-journald Persistence, Capacity Limits, and Rotation

1. Back to home
2. Linux
3. systemd-journald Persistence, Capacity Limits, and Rotation

systemd-journald Persistence, Capacity Limits, and Rotation

When using systemd-journald logs for troubleshooting, simply being able to search them with journalctl is not enough. Persistence is required to retain logs from before a reboot, and for long-term operation you also need to define an upper limit for disk usage and a method for removing old journals.

This article targets RHEL 8-compatible Linux systems and configures and verifies the following:

• persist journals with Storage=persistent
• set an overall capacity limit for persistent journals with SystemMaxUse=64M
• limit the size of each file with SystemMaxFileSize=8M to make rotation units smaller
• explicitly rotate the current journal with journalctl --rotate
• remove old archives with journalctl --vacuum-size
• view logs recorded before a reboot after the system has restarted

When Storage=persistent is specified and persistent storage is available, journals are stored under /var/log/journal. SystemMaxUse controls the amount of space used by persistent journals.

vacuum is an operation that removes old logs. Before using it in a production environment, check your audit requirements and required retention period, then determine the cleanup size accordingly.

Step 1: Check the current journald configuration

Before changing the configuration, check the current settings, including the main configuration file and drop-ins.

sudo systemd-analyze cat-config systemd/journald.conf

systemd-analyze cat-config lets you review journald.conf together with the drop-ins that are loaded.

Step 2: Configure persistence and capacity limits

Settings for local administrators are placed as drop-ins under /etc/systemd/journald.conf.d/.

Create the directory for persistent journals, apply the appropriate attributes according to the tmpfiles definition, and then configure persistence and capacity limits.

sudo mkdir -p /var/log/journal /etc/systemd/journald.conf.d
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo tee /etc/systemd/journald.conf.d/60-persistent.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=64M
SystemMaxFileSize=8M
EOF

Each setting has the following meaning:

Setting	Meaning
Storage=persistent	Stores journals in persistent storage
SystemMaxUse=64M	Limits the amount of space used by journals under /var/log/journal
SystemMaxFileSize=8M	Limits the maximum size of each journal file and increases the granularity of rotation and deletion

With capacity control using SystemMaxUse, old archived journals become candidates for deletion. Active files are not deleted, so instantaneous usage does not necessarily match the configured limit exactly.

Step 3: Apply the configuration to journald

Restart systemd-journald to reload the configuration, and if an existing volatile journal is present, flush it to persistent storage.

sudo systemctl restart systemd-journald
sudo journalctl --flush

When persistent storage is enabled, journalctl --flush moves data from /run/log/journal to /var/log/journal.

Step 4: Check the loaded configuration

Confirm that the drop-in is included in the configuration and that systemd-journald is running.

sudo systemd-analyze cat-config systemd/journald.conf
sudo systemctl is-active systemd-journald

If Storage=persistent, SystemMaxUse=64M, and SystemMaxFileSize=8M appear in the output and the service is active, you can confirm both the configuration and service state.

Step 5: Verify that logs are stored in the persistent journal

Record one sample message, synchronize it to disk, and then read it directly from the journal files under /var/log/journal.

sudo logger -t journald-persistence-example "persistent journal sample"
sudo journalctl --sync
sudo journalctl --file='/var/log/journal/*/*.journal' -t journald-persistence-example -n 1 --no-pager

If persistent journal sample is displayed, the verification log has been read successfully from a persistent journal file.

Step 6: Generate logs for rotation verification

To make it easier to verify rotation and vacuum behavior, write random data to the journal for testing.

The following command intentionally generates a large volume of logs. Do not run it on a production server.

sudo sh -c 'head -c 16777216 /dev/urandom | base64 | fold -w 32000 | systemd-cat -t journald-rotation-example'
sudo journalctl --sync
sudo journalctl --disk-usage

journalctl --disk-usage displays the current combined usage of active and archived journals.

Step 7: Explicitly rotate the journal

Archive the journal that is currently being written to and switch to a new journal.

sudo journalctl --rotate
sudo find /var/log/journal -type f -name '*@*.journal' -print

When you run journalctl --rotate, the current active journal file is archived and writing switches to a new file.

If find displays a .journal file containing @ in its name, you can confirm that an archived journal exists.

Step 8: Check usage before vacuum

Record the journal usage before deletion.

sudo journalctl --disk-usage

Compare this value with the result after running vacuum.

Step 9: Vacuum old archived journals

Delete old files from archived journals until the retained size is below 8 MiB.

sudo journalctl --vacuum-size=8M

--vacuum-size removes archived journals only. The active file currently being written to is not affected.

In an operational environment, do not use the value specified here merely as a temporary test value. Define it based on the required log retention period, log volume, and available disk capacity.

Step 10: Check usage after vacuum

Check the usage again after vacuum.

sudo journalctl --disk-usage

If usage is lower than in Step 8, the deletion of archived journals has taken effect.

SystemMaxUse controls capacity during continuous operation, while --vacuum-size is an explicit deletion operation performed by an administrator at a specific point in time.

Step 11: Record a log for retention verification after reboot

Record a new sample log and synchronize it to disk to verify retention across a reboot.

sudo logger -t journald-reboot-example "persistent journal sample before reboot"
sudo journalctl --sync
sudo journalctl --file='/var/log/journal/*/*.journal' -t journald-reboot-example -n 1 --no-pager

Before rebooting, confirm that persistent journal sample before reboot is displayed.

Step 12: Reboot the server

Reboot the server to verify persistence. When --no-block is specified, systemd validates the reboot request, queues it, and then exits systemctl without waiting for the reboot process to complete.

sudo systemctl --no-block reboot

The connection will be interrupted during the reboot. Once the reboot is complete, reconnect to the server.

Step 13: Verify that the journald configuration is retained after reboot

After rebooting, check the drop-in configuration, service state, and persistent journal files again.

sudo systemd-analyze cat-config systemd/journald.conf
sudo systemctl is-active systemd-journald
sudo find /var/log/journal -type f -name '*.journal' -print

Confirm that Storage=persistent, SystemMaxUse=64M, and SystemMaxFileSize=8M are still displayed, that systemd-journald is active, and that journal files exist under /var/log/journal/.

Step 14: View the log from before the reboot

Finally, limit the query to the immediately preceding boot session and search for the message recorded before the reboot.

sudo journalctl -b -1 -t journald-reboot-example --no-pager

If persistent journal sample before reboot is displayed, the journal from before the reboot was retained in persistent storage and can be viewed after rebooting.

journalctl -b -1 displays logs belonging to the immediately preceding boot. If persistent journals are retained, you can search not only the current boot but previous boots as well.

Capacity planning considerations

Even if SystemMaxUse is configured, it is safer not to assume that usage will always remain strictly below the specified value. systemd-journald removes archived files when adjusting capacity, while active journal files remain. In addition, if other capacity-related conditions such as SystemKeepFree are configured, they also affect the amount of space that can actually be used for storage.

For long-term operation, it is easier to manage the system if the following three aspects are designed separately:

• SystemMaxUse: controls the total disk usage of the journals
• SystemMaxFileSize: adjusts the granularity of rotation and deletion
• journalctl --vacuum-size or --vacuum-time: lets an administrator explicitly clean up existing archives

If persistence is enabled without setting a capacity limit, more historical data will be available for troubleshooting, but disk consumption will also increase. In practice, persistence and capacity limits should be designed together.

Related articles

• Linux Log Investigation and Retention Management with journalctl and systemd-journaldThis guide systematically explains how to search and investigate Linux logs and manage persistence, capacity control, and retention with journalctl and systemd-journald.
• Managing Application Logs with logrotateThis guide explains how to use logrotate to define application log rotation policies, decide when to use copytruncate, configure postrotate, perform dry-runs and forced tests, and safely prevent custom log files from growing excessively.
• Disk cleanup and maintenance automationThis article explains how to automate WinSxS optimization, Temp folder cleanup, log rotation, and event log maintenance using purpose-specific PowerShell scripts and a PSD1 configuration file.

Category: Linux