Linux Log Investigation and Retention Management with journalctl and systemd-journald

In Linux environments that use systemd, systemd-journald collects logs from sources such as the kernel, services, standard output, and standard error. These logs can be searched with journalctl.

When investigating failures, it is important not to simply display all logs, but to narrow the scope by combining service, boot, time, and priority. Persistence is also required to retain logs from before a reboot, while capacity limits and retention settings are needed to control disk usage.

This article explains, step by step, how to configure log searching, persistence, capacity control, and retention management on RHEL-compatible Linux systems.

Step 1: Check the status of systemd-journald

First, verify that systemd-journald is running, check the current journal disk usage, and review the saved boot history.

sudo systemctl status systemd-journald --no-pager
sudo journalctl --disk-usage
sudo journalctl --list-boots --no-pager

With journalctl --list-boots, the current boot is represented by 0, and the immediately preceding boot by -1. In environments where persistent journals are not stored, previous boot history may not be available.

Step 2: Filter logs by service, boot, time, and priority

When investigating failures, it is efficient to first identify the target service and then add search conditions.

In the following example, systemd-journald.service is restarted, and then up to 50 log entries from the current boot, within the last 10 minutes, with priority info or higher are displayed.

sudo systemctl restart systemd-journald
sleep 1
sudo journalctl -u systemd-journald.service -b 0 --since "10 minutes ago" -p info --no-pager -n 50

The main conditions are as follows.

Option	Purpose
`-u UNIT`	Limit output to the specified systemd unit
`-b 0`	Limit output to the current boot
`-b -1`	Limit output to the immediately preceding boot
`--since`	Limit output to entries after the specified time
`--until`	Limit output to entries before the specified time
`-p`	Filter by syslog priority
`-n`	Limit the number of displayed entries
`--no-pager`	Display output directly on standard output without using a pager

When -p info is specified, the results include not only info entries but also entries with higher severity.

When investigating a service failure, first narrow the target with -u. For problems immediately after a reboot, use -b. If the time of occurrence is known, add --since and --until. If you want to review only more serious events, add -p.

Step 3: Make the journal persistent

To retain journals after a reboot, configure Storage=persistent.

Instead of editing the main configuration file directly, create a drop-in file under /etc/systemd/journald.conf.d/.

sudo mkdir -p /etc/systemd/journald.conf.d
cat <<'EOF' | sudo tee /etc/systemd/journald.conf.d/10-persistent.conf >/dev/null
[Journal]
Storage=persistent
EOF
sudo systemctl restart systemd-journald
sudo journalctl --flush
sudo test -d /var/log/journal

When Storage=persistent is used, persistent journals are stored under /var/log/journal. journalctl --flush moves data remaining in the runtime journal to persistent storage.

To test whether a log entry can be retrieved after a reboot, write an easily identifiable sample message.

printf '%s\n' 'example-persistence-before-reboot' | sudo systemd-cat -t example-persistence
sudo journalctl --sync
sudo journalctl -t example-persistence -b 0 --no-pager -n 5

The string specified with -t in systemd-cat can later be used for searches with journalctl -t.

Step 4: Check previous logs after a reboot

To verify persistence, you must actually reboot the operating system.

If this command is executed over a remote connection, the connection will be interrupted. Immediately after the reboot is requested, the SSH service stops temporarily, so an immediate reconnection attempt may fail with Connection refused.

sudo systemctl reboot

After the reboot completes, reconnect to the same server over SSH. Search for the sample message recorded before the reboot and verify that its boot ID differs from the current boot ID.

current_boot=$(tr -d '-' < /proc/sys/kernel/random/boot_id)
marker_boot=$(sudo journalctl -t example-persistence -o export --no-pager | awk -F= '$1 == "_BOOT_ID" { id=$2 } END { print id }')

test -n "$marker_boot" && test "$marker_boot" != "$current_boot" &&
  sudo journalctl --list-boots --no-pager &&
  sudo journalctl "_BOOT_ID=$marker_boot" -t example-persistence --no-pager | grep -F 'example-persistence-before-reboot' &&
  printf '%s\n' 'persistence-ok'

If the message recorded before the reboot can be retrieved and its _BOOT_ID differs from the current boot ID, the journal has been stored persistently across the operating system reboot.

Step 5: Set the journal capacity limit and retention period

If persistent journals are stored without limits, disk usage will increase during long-term operation. Explicitly setting the available capacity and retention period makes them easier to manage.

In the following example, the total journal limit is set to 512 MiB, the maximum size of each journal file to 64 MiB, and the retention period to 30 days.

cat <<'EOF' | sudo tee /etc/systemd/journald.conf.d/20-retention.conf >/dev/null
[Journal]
SystemMaxUse=512M
SystemMaxFileSize=64M
MaxRetentionSec=30day
EOF
sudo systemctl restart systemd-journald
sudo grep -E '^(SystemMaxUse|SystemMaxFileSize|MaxRetentionSec)=' /etc/systemd/journald.conf.d/20-retention.conf

Each setting has the following meaning.

Setting	Description
`SystemMaxUse`	Maximum amount of disk space used by journals under `/var/log/journal`
`SystemMaxFileSize`	Maximum size of a single journal file
`MaxRetentionSec`	Maximum retention period for journal entries

When SystemMaxFileSize is reached, the file is rotated. When reducing disk usage, archived journals are deleted while the active file currently being written is retained. As a result, actual disk usage may temporarily exceed SystemMaxUse by some amount.

512 MiB and 30 days are not universal recommended values. Choose values based on the amount of logs generated, the period required for failure investigation, and the available disk capacity.

Step 6: Verify that capacity control works

To verify capacity control in practice, temporarily configure a smaller limit than would be used during normal operation and write a sufficient amount of sample log data.

The following verification generates a large amount of journal data and may delete older archived journals. Do not run it in a production environment where existing logs must be preserved. Use it only in a test environment.

To make the verification complete quickly, the temporary configuration also disables rate limiting and compression. The temporary configuration is removed after the verification.

set -euo pipefail

cat <<'EOF' | sudo tee /etc/systemd/journald.conf.d/90-capacity-test.conf >/dev/null
[Journal]
SystemMaxUse=32M
SystemMaxFileSize=8M
RateLimitIntervalSec=0
Compression=no
EOF

sudo systemctl restart systemd-journald
sudo journalctl --flush
sudo sh -c 'base64 -w 4096 /dev/urandom | head -c 67108864 | systemd-cat -t example-capacity'
sudo journalctl --sync

count=$(sudo find /var/log/journal -type f \( -name '*.journal' -o -name '*.journal~' \) | wc -l)
bytes=$(sudo find /var/log/journal -type f \( -name '*.journal' -o -name '*.journal~' \) -printf '%b\n' | awk '{s += $1} END {printf "%.0f", s * 512}')

test "$count" -ge 2
test -n "$bytes"
test "$bytes" -le 50331648
printf 'capacity-ok: %s bytes\n' "$bytes"

sudo rm -f /etc/systemd/journald.conf.d/90-capacity-test.conf
sudo systemctl restart systemd-journald

This verification writes 64 MiB of sample data while SystemMaxFileSize=8M triggers rotation and SystemMaxUse=32M enforces capacity control.

Because of the active file and other factors, actual usage does not necessarily remain strictly below the configured value. For this reason, the verification uses an upper limit with some margin.

After the temporary file is removed, the normal capacity and retention policies configured in Step 5 apply again.

Step 7: Delete old journals with vacuum

journalctl provides a vacuum function that reduces stored journals based on size or age.

The main options are as follows.

--vacuum-size=: Delete archived journals until their size is below the specified value
--vacuum-time=: Delete archived journals older than the specified period
--vacuum-files=: Limit the number of archived journal files

Vacuum operates on archived files. If necessary, run --rotate first to rotate the current active file and make it eligible for deletion.

The following example immediately verifies deletion based on the retention period. Because --vacuum-time=1s may also delete existing older archived journals, do not run this test in a production environment.

set -euo pipefail

printf '%s\n' 'example-retention-expire' | sudo systemd-cat -t example-retention
sudo journalctl --sync
sudo journalctl --rotate
sleep 2
sudo journalctl --vacuum-time=1s

if sudo journalctl -t example-retention --no-pager | grep -Fq 'example-retention-expire'; then
  printf '%s\n' 'retention-check-failed' >&2
  exit 1
fi

printf '%s\n' 'retention-ok'

If the sample message written before rotation is deleted, time-based vacuum processing is working.

In normal operation, do not use an extremely short value such as 1s. Specify a period according to your organization’s log retention requirements. Use MaxRetentionSec for an ongoing retention policy, and use journalctl --vacuum-* when you need to urgently recover disk space or clean up existing journals.

Troubleshooting

SSH connection is refused immediately after reboot

During an operating system reboot, the SSH service also stops. Therefore, Connection refused can occur normally while the system is rebooting.

After requesting the reboot, wait for the system to finish booting, then reconnect over SSH before checking the journal. Even immediately after a successful reconnection, journal or systemd startup may not yet be complete.

Logs from before the reboot are not displayed

Compare the current boot ID with the _BOOT_ID of the message recorded before the reboot.

If the two IDs are the same, the operating system reboot has not yet completed. If they differ but the previous message still cannot be retrieved, the persistent journal may not have been saved.

Disk usage does not exactly match the configured capacity limit

With SystemMaxUse, archived journals are deleted. The active file currently being written is not deleted, so usage may temporarily exceed the configured value.

In addition, files that are not part of the journal are not managed by SystemMaxUse.

Vacuum barely reduces disk usage

Vacuum deletes archived journal files. If the current active journal is large, rotate it first to increase the amount of archived data that can be deleted.

However, rotation and vacuum actually delete log history. Before running them, confirm whether logs required for failure investigation or audits must be retained.

Summary

When investigating failures with journalctl, you can narrow down large volumes of logs by combining -u, -b, --since, and -p.

If logs must be retained across a reboot, configure Storage=persistent and, after the reboot completes, verify that logs can be retrieved using a previous _BOOT_ID.

For long-term operation, SystemMaxUse, SystemMaxFileSize, and MaxRetentionSec control disk usage and the retention period.

To temporarily clean up older logs, you can use journalctl --vacuum-*. Because this operation actually deletes archived logs, check the retention requirements before running it.