systemd
defines itself as a system and service manager. The project was initiated in 2010 by Lennart Poettering and Kay Sievers to create an integrated set of tools for managing a Linux system including an init
daemon. It also includes device management (udev
) and logging, among other things. Some would say that it is not just an init
program, it is a way of life. It is state of the art, and still evolving rapidly. systemd
is common on desktop and server Linux distributions, and is becoming popular on embedded Linux systems too, especially on more complex devices. So, how is it better than System V init
for embedded systems?
init
, systemd
has unit configuration files to set parameterssystemd
can monitor services and restart them if neededystemd
itselfA complete description of systemd
is neither possible nor appropriate here. As with System V init
, I will focus on embedded use-cases, with examples based on the configuration produced by Yocto Fido, which has systemd
version 219. I will give a quick overview and then show you some specific examples.
The default init
in Yocto Fido is System V. To select systemd
, add these lines to your configuration, for example, in conf/local.conf
:
DISTRO_FEATURES_append = " systemd" VIRTUAL-RUNTIME_init_manager = "systemd"
Note that the leading space is important! Then rebuild.
Buildroot has systemd
as the third init
option. It requires glibc as the C library, and kernel version 3.7 or later with a particular set of configuration options enabled. There is a complete list of dependencies in the README
file in the top level of the systemd
source code.
Before I describe how systemd init
works, I need to introduce these three key concepts.
Firstly, a target is a group of services, similar to, but more general than, a SystemV runlevel
. There is a default target which is the group of services that are started at boot time.
Secondly, a service is a daemon that can be started and stopped, very much like a SystemV service
.
Finally, a unit is a configuration file that describes a target
, a service
, and several other things. Units are text files that contain properties and values.
You can change states and find out what is going on by using the systemctl
command.
The basic item of configuration is the unit file. Unit files are found in three different places:
/etc/systemd/system
: Local configuration/run/systemd/system
: Runtime configuration/lib/systemd/system
: Distribution-wide configurationWhen looking for a unit, systemd
searches the directories in that order, stopping as soon as it finds a match, allowing you to override the behavior of a distribution-wide unit by placing a unit of the same name in /etc/systemd/system
. You can disable a unit completely by creating a local file that is empty or linked to /dev/null
.
All unit files begin with a section marked [Unit]
which contains basic information and dependencies, for example:
[Unit] Description=D-Bus System Message Bus Documentation=man:dbus-daemon(1) Requires=dbus.socket
Unit dependencies are expressed though Requires
, Wants
, and Conflicts
:
Requires
: A list of units that this unit depends on, which is started when this unit is startedWants
: A weaker form of Requires
: the units listed are started but the current unit is not stopped if any of them failConflicts
: A negative dependency: the units listed are stopped when this one is started and, conversely, if one of them is started, this one is stoppedProcessing the dependencies produces a list of units that should be started (or stopped). The keywords Before
and After
determine the order in which they are started. The order of stopping is just the reverse of the start order:
Before
: This unit should be started before the units listedAfter
: This unit should be started after the units listedIn the following example, the After
directive makes sure that the web server is started after the network:
[Unit] Description=Lighttpd Web Server After=network.target
In the absence of Before
or After
directives, the units will be started or stopped in parallel with no particular ordering.
A service is a daemon that can be started and stopped, equivalent to a System V service
. A service is a type of unit file with a name ending in .service
, for example, lighttpd.service
.
A service unit has a [Service]
section that describes how it should be run. Here is the relevant section from lighttpd.service
:
[Service] ExecStart=/usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf -D ExecReload=/bin/kill -HUP $MAINPID
These are the commands to run when starting the service and restarting it. There are many more configuration points you can add in here, so refer to the man page for systemd.service
.
Now we can see how systemd
implements the bootstrap. systemd
is run by the kernel as a result of /sbin/init
being symbolically linked to /lib/systemd/systemd
. It runs the default target, default.target
, which is always a link to a desired target such as multi-user.target
for a text login or graphical.target
for a graphical environment. For example, if the default target is multi-user.target
, you will find this symbolic link:
/etc/systemd/system/default.target -> /lib/systemd/system/multi-user.target
The default target may be overridden by passing system.unit=<new target>
on the kernel command line. You can use systemctl
to find out the default target, as shown here:
# systemctl get-default multi-user.target
Starting a target such as multi-user.target
creates a tree of dependencies that bring the system into a working state. In a typical system, multi-user.target
depends on basic.target
, which depends on sysinit.target
, which depends on the services that need to be started early. You can print a graph using systemctl list-dependencies
.
You can also list all the services and their current state using systemctl list-units --type service
, and the same for targets using systemctl list-units --type target
.
Using the same simpleserver
example as before, here is a service unit:
[Unit] Description=Simple server [Service] Type=forking ExecStart=/usr/bin/simpleserver [Install] WantedBy=multi-user.target
The [Unit]
section only contains a description so that it shows up correctly when listed using systemctl
and other commands. There are no dependencies; as I said, it is very simple.
The [Service]
section points to the executable, and has a flag to indicate that it forks. If it were even simpler and ran in the foreground, systemd
would do the daemonizing for us and Type=forking
would not be needed.
The [Install]
section makes it dependent on multi-user.target
so that our server is started when the system goes into multi-user mode.
Once the unit is saved in /etc/systemd/system/simpleserver.service
, you can start and stop it using the systemctl start simpleserver
and systemctl stop simpleserver
commands. You can use this command to find its current status:
# systemctl status simpleserver simpleserver.service - Simple server Loaded: loaded (/etc/systemd/system/simpleserver.service; disabled) Active: active (running) since Thu 1970-01-01 02:20:50 UTC; 8s ago Main PID: 180 (simpleserver) CGroup: /system.slice/simpleserver.service └─180 /usr/bin/simpleserver -n Jan 01 02:20:50 qemuarm systemd[1]: Started Simple server.
At this point, it will only start and stop on command, as shown. To make it persistent, you need to add a permanent dependency to a target. That is the purpose of the [Install]
section in the unit, it says that when this service is enabled it will become dependent on multi-user.target
, and so will be started at boot time. You enable it using systemctl enable
, like this:
# systemctl enable simpleserver Created symlink from /etc/systemd/system/multi-user.target.wants/simpleserver.service to /etc/systemd/system/simpleserver.service.
Now you can see how dependencies are added at runtime without having to edit any unit files. A target can have a directory named <target_name>.target.wants
which can contain links to services. This is exactly the same as adding the dependent unit to the [Wants]
list in the target. In this case, you will find that this link has been created:
/etc/systemd/system/multi-user.target.wants/simpleserver.service /etc/systemd/system/simpleserver.service
If this is were an important service you might want to restart if it failed. You can accomplish that by adding this flag to the [Service]
section:
Restart=on-abort
Other options for Restart
are on-success
, on-failure
, on-abnormal
, on-watchdog
, on-abort
, or always
.
Watchdogs are a common requirement in embedded devices: you need to take action if a critical service stops working, usually by resetting the system. On most embedded SoCs, there is a hardware watchdog which can be accessed via the /dev/watchdog
device node. The watchdog is initialized with a timeout at boot and then must be reset within that period, otherwise the watchdog will be triggered and the system will reboot. The interface with the watchdog driver is described in the kernel source in Documentation/watchdog
, and the code for the drivers is in drivers/watchdog
.
A problem arises if there are two or more critical services that need to be protected by a watchdog. systemd
has a useful feature that distributes the watchdog between multiple services.
systemd
can be configured to expect a regular keepalive call from a service and take action if it is not received, in other words, a per-service software watchdog. For this to work, you have to add code to the daemon to send the keepalive messages. It needs to check for a non-zero value in the WATCHDOG_USEC
environment variable and then call sd_notify(false, "WATCHDOG=1")
within that time (a period of half of the watchdog timeout is recommended). There are examples in the systemd
source code.
To enable the watchdog in the service unit, add something like this to the [Service]
section:
WatchdogSec=30s Restart=on-watchdog StartLimitInterval=5min StartLimitBurst=4 StartLimitAction=reboot-force
In this example, the service expects a keepalive every 30 seconds. If it fails to be delivered, the service will be restarted, but if it is restarted more than four times in five minutes, systemd
will force an immediate reboot. Once again, there is a full description of these settings in the systemd
manual.
A watchdog like this takes care of individual services, but what if systemd
itself fails, or the kernel crashes, or the hardware locks up. In those cases, we need to tell systemd
to use the watchdog driver: just add RuntimeWatchdogSec=NN
to /etc/systemd/system.conf
. systemd
will reset the watchdog within that period, and so the system will reset if systemd
fails for some reason.
systemd
has a lot of features that are useful in embedded Linux, including many that I have not mentioned in this brief description such as resource control using slices (see the man page for systemd.slice(5)
and systemd.resource-control(5)
), device management (udev(7)
) and system logging facilities (journald(5)
).
You have to balance that with its size: even with a minimal build of just the core components, systemd
, udevd
, and journald
, it is approaching 10 MiB of storage, including the shared libraries.
You also have to keep in mind that systemd
development follows the kernel closely, so it will not work on a kernel more than a year or two older than the release of systemd
.