Practical Guide to systemd Service Management and Unit Files

systemd is the mechanism used in many Linux environments, including RHEL-compatible distributions, to manage service startup, shutdown, automatic startup, dependencies, and restarts after failures.

For operating existing services, the basic systemctl commands are usually sufficient. However, when running your own applications or persistent processes, you also need to understand Service Unit design, dependency definitions, restart policies, and configuration changes using drop-ins.

In this article, you will create a Service Unit that runs a simple HTTP service on 8080/TCP and then verify startup and shutdown, automatic startup, failure recovery, and configuration overrides using a drop-in.

The options available in Python’s http.server may differ depending on the environment. Here, the published directory is specified with systemd’s WorkingDirectory, while ExecStart specifies only the port number and bind address.

Immediately after a service starts, systemd may temporarily report the state as activating while the TCP port is not yet listening. Therefore, instead of assuming a LISTEN state immediately after startup, use a bounded wait to confirm that the service is active and that 8080/TCP is listening.

Variable notation

Values that vary by environment use the following variable notation.

Variable name	Example setting	Description
`<<SERVER_IP>>`	`192.0.2.10`	IP address of the destination Web server

Basic structure of a Unit file

A systemd Service Unit is mainly composed of the following three sections.

[Unit]: Defines the Unit description, dependencies, and startup order.
[Service]: Defines the process to execute, restart conditions, and runtime restrictions.
[Install]: Defines which target the Unit is associated with when enable is used.

In this example, network-online.target is specified as a dependency so that the HTTP service starts after the network is available. In addition, Restart=on-failure automatically restarts the service when it terminates abnormally.

Step 1: Install the required packages

Python 3 is used to run the sample HTTP service, firewalld is used for external access, and ss is used to verify the listening port.

sudo dnf install -y python3 firewalld iproute

Step 2: Create sample content to publish

Create the static file returned by the HTTP service. The service itself uses this directory as read-only content.

sudo install -d -m 0755 /srv/example-web
printf '%s\n' 'example systemd service' | sudo tee /srv/example-web/index.html >/dev/null
sudo chmod 0644 /srv/example-web/index.html

Step 3: Allow 8080/TCP with firewalld

With firewalld, zones can be assigned not only to interfaces but also based on source addresses. Therefore, checking only the zone associated with a specific interface may not be sufficient if incoming packets are actually handled by a different active zone.

In this example, 8080/TCP is allowed in the current default zone and all active zones for connectivity testing.

sudo systemctl enable --now firewalld

DEFAULT_ZONE="$(sudo firewall-cmd --get-default-zone)"
ACTIVE_ZONES="$(sudo firewall-cmd --get-active-zones | awk 'NF && $1 !~ /^(interfaces:|sources:)/ {print $1}')"
ZONES="$(printf '%s\n%s\n' "$DEFAULT_ZONE" "$ACTIVE_ZONES" | awk 'NF' | sort -u)"
test -n "$ZONES"

for zone in $ZONES; do
  sudo firewall-cmd --permanent --zone="$zone" --add-port=8080/tcp
done
sudo firewall-cmd --reload

for zone in $ZONES; do
  sudo firewall-cmd --zone="$zone" --query-port=8080/tcp
done

If the result for each zone is yes, the 8080/TCP permission has been applied.

In environments that use multiple zones, it is important not to expose the port in more zones than necessary. In production environments, identify the actual communication path and restrict the permission to only the required zones.

Step 4: Create a custom Service Unit

Create the Service Unit under /etc/systemd/system.

Wants=network-online.target includes the network-ready state as a dependency, while After=network-online.target specifies the startup order.

Restart=on-failure restarts the service only when it terminates abnormally. It does not automatically restart a service that an administrator explicitly stops.

The HTTP server uses --bind 0.0.0.0 so that it listens on 8080/TCP on all IPv4 interfaces.

The published directory /srv/example-web is specified as the working directory using systemd’s WorkingDirectory instead of Python’s --directory option. This allows the same published directory to be used even in Python environments where http.server does not support --directory.

For security, the configuration also includes DynamicUser=yes, which creates a runtime user without requiring a fixed user account, along with settings that restrict filesystem access and privileges.

sudo tee /etc/systemd/system/example-web.service >/dev/null <<'EOF'
[Unit]
Description=Example HTTP service managed by systemd
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/srv/example-web
ExecStart=/usr/bin/python3 -m http.server 8080 --bind 0.0.0.0
Restart=on-failure
RestartSec=1s
DynamicUser=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemd-analyze verify /etc/systemd/system/example-web.service
sudo systemctl daemon-reload
sudo systemctl cat example-web.service

After checking for syntax problems with systemd-analyze verify, run daemon-reload so that systemd loads the new Unit definition.

When you create a new Unit file or modify its contents, restarting the service alone is not sufficient. You also need to run daemon-reload.

WorkingDirectory specifies the current working directory of the service process. Because Python’s HTTP server publishes files relative to this directory, there is no need to add a directory-specific option to ExecStart.

Step 5: Enable automatic startup and start the service

Enable the Service Unit and start it immediately.

Immediately after startup, the state may temporarily be activating, so check up to 10 times at one-second intervals until the service is active and 8080/TCP is in the LISTEN state. If the conditions are not met within 10 checks, the final state check can fail.

sudo systemctl enable --now example-web.service

for _ in 1 2 3 4 5 6 7 8 9 10; do
  if [ "$(sudo systemctl is-active example-web.service)" = "active" ] && sudo ss -ltn | grep -Eq 'LISTEN[[:space:]].*:8080[[:space:]]'; then
    break
  fi
  sleep 1
done

sudo systemctl is-enabled example-web.service
sudo systemctl is-active example-web.service
sudo systemctl show example-web.service -p Wants -p After --no-pager
sudo ss -ltn | grep -E 'LISTEN[[:space:]].*:8080[[:space:]]'

enable configures the service to start automatically on subsequent operating system boots. This is separate from starting the current process, so combining it with --now lets you enable and start the service at the same time.

If the final checks show enabled and active, include network-online.target in the dependencies, and display a LISTEN line for 8080/TCP, then automatic startup and service startup are working correctly.

Step 6: Start, stop, and restart the service

In day-to-day operations, use systemctl to stop, start, and restart the service.

Immediately after a restart, the listening socket may still not be ready, so use another bounded wait of up to 10 seconds before checking the final state.

sudo systemctl stop example-web.service
sudo systemctl start example-web.service
sudo systemctl restart example-web.service

for _ in 1 2 3 4 5 6 7 8 9 10; do
  if [ "$(sudo systemctl is-active example-web.service)" = "active" ] && sudo ss -ltn | grep -Eq 'LISTEN[[:space:]].*:8080[[:space:]]'; then
    break
  fi
  sleep 1
done

sudo systemctl is-active example-web.service
sudo ss -ltn | grep -E 'LISTEN[[:space:]].*:8080[[:space:]]'

If you changed only a configuration file read by the application, you can recreate the process with restart. If you changed the Unit file itself, run daemon-reload first so that systemd reads the updated definition.

Step 7: Verify the HTTP connection from another system

After confirming on the server that the service is active and 8080/TCP is in the LISTEN state, test the HTTP connection from another system.

curl --fail --silent --show-error http://<<SERVER_IP>>:8080/

If the HTTP response contains example systemd service, the systemd-managed process is responding on 8080/TCP and is reachable externally.

If the connection fails, troubleshoot in the following order:

Is the service active according to systemctl is-active?
Does the output of ss contain a LISTEN line for 8080/TCP?
Is 8080/TCP allowed in the firewalld zone that is actually being used?
Is <<SERVER_IP>> reachable over the network from the client?

Checking the service state and listening state separately makes it easier to distinguish a systemd startup that is still in progress from a network path problem.

Step 8: Override existing Unit settings with a drop-in

When changing settings for a Unit that is already in use, it is preferable to separate overrides using a drop-in rather than repeatedly modifying the original Unit file directly.

In this example, only the delay before restart is changed from one second to three seconds.

sudo install -d -m 0755 /etc/systemd/system/example-web.service.d
sudo tee /etc/systemd/system/example-web.service.d/override.conf >/dev/null <<'EOF'
[Service]
RestartSec=3s
EOF

sudo systemctl daemon-reload
sudo systemctl restart example-web.service
sudo systemctl show example-web.service -p RestartUSec --no-pager

A drop-in lets you override only the necessary settings while preserving the original Unit definition. If RestartUSec=3s is displayed, the setting has been overridden without modifying the original Unit file.

Step 9: Verify automatic restart after a failure

To verify that Restart=on-failure works as intended, do not use a normal stop operation. Instead, terminate the running process abnormally.

After sending SIGKILL to the main process, check up to 10 times at one-second intervals for three conditions: the service is active, the restart count is at least one, and 8080/TCP is in the LISTEN state.

sudo systemctl kill --signal=KILL --kill-who=main example-web.service

for _ in 1 2 3 4 5 6 7 8 9 10; do
  if [ "$(sudo systemctl is-active example-web.service)" = "active" ] && [ "$(sudo systemctl show example-web.service -p NRestarts --value)" -ge 1 ] && sudo ss -ltn | grep -Eq 'LISTEN[[:space:]].*:8080[[:space:]]'; then
    break
  fi
  sleep 1
done

sudo systemctl is-active example-web.service
test "$(sudo systemctl show example-web.service -p NRestarts --value)" -ge 1
sudo ss -ltn | grep -E 'LISTEN[[:space:]].*:8080[[:space:]]'

An explicit stop and a failure-induced termination are handled differently. With Restart=on-failure, a service intentionally stopped by an administrator is not started again automatically, while a process that terminates abnormally can be restarted.

After the restart, it is important to confirm not only that systemd reports the service as active, but also that the service has actually resumed listening on its port.

Step 10: Verify connectivity after the automatic restart

After the process returns to active and 8080/TCP is again in the LISTEN state, verify that the service functionality has also recovered from the user’s perspective.

From another system, make another HTTP request.

curl --fail --silent --show-error http://<<SERVER_IP>>:8080/

If example systemd service is returned again, systemd has restarted the service after the abnormal termination and external availability has been restored.

Key points when designing Units

When designing a Service Unit, it is not enough to simply place a command in ExecStart.

When resources depend on one another, use Wants, Requires, After, and related directives according to their purpose, and explicitly define both dependencies and startup order. Dependencies and ordering are separate concepts: specifying only After does not cause the target Unit to be pulled into the startup process as a dependency.

If options for the executed program depend on the environment, consider whether that responsibility can be moved to settings provided by systemd itself. In this example, using WorkingDirectory for the published directory avoids depending on the HTTP server’s --directory option.

For failure recovery, automatic restart is not always appropriate. For typical persistent services that should not restart automatically after a normal stop, Restart=on-failure is a practical choice.

For network services, the systemd state and socket listening state do not necessarily become ready at the same time. Immediately after startup or restart, use a short bounded wait to confirm that both conditions are satisfied, and only treat the operation as failed if they are not.

When modifying Units provided by a distribution or baseline Units already in use, separate local changes with drop-ins whenever possible. Keeping the original file separate from local modifications makes it easier to assess the impact during updates and configuration reviews.

Summary

When managing services with systemd, it is important to consider the following elements together:

Define the process to execute and its runtime conditions in a Service Unit
Explicitly define the service execution directory with WorkingDirectory
Define dependencies and startup order
Configure automatic startup at operating system boot with enable
Manage the lifecycle with start, stop, and restart
Immediately after startup, use a bounded number of checks to wait for both the active state and LISTEN state
With firewalld, consider the zones actually applied to the traffic
Define failure recovery with Restart and RestartSec
Override settings with a drop-in without directly modifying the original Unit file
Verify not only the active state, but also actual service connectivity

By registering a custom service with systemd, you can move from manually starting processes to centralized management that covers startup order, automatic startup, and failure recovery.