leah blogs: August 2025

Yesterday I announced a first public release of nitro, a tiny init system and process supervisor. This got a fair bit of attention, and to my delight was even boosted on the fediverse by both Laurent Bercot (of s6 fame) and djb himself.

One of the most requested things was a comparison to other init systems. Since I’m most familiar with runit, I shall compare nitro and runit here.

Comparison to runit

runit and nitro share the basic design of having a directory of services and using small scripts to spawn the processes.

Comparing nitro to runit, there are a few new features and some architectural differences.

From a design point of view, runit follows the daemontoools approach of multiple small tools: The runit-init process spawns runsvdir, which spawns a runsv process for each service.

nitro favors a monolithic approach, and keeps everything in a single process. This makes it also easier to install for containerization.

The new features are:

• nitro keeps all runtime state in RAM and provides an IPC interface to query it, whereas runit emits state to disk. This enables nitro to run on read-only file systems without special configuration. (However, you need a tmpfs to store the socket file. In theory, on Linux, you could even use /proc/1/fd/10 or an abstract Unix domain socket, but that requires adding permission checks.)

• support for one-shot “services”, i.e. running scripts on up/down without a process to supervise (e.g. persist audio volume, keep RNG state). For runit, you can fake this with a pause process, which has a little more overhead.

• parametrized services. One service directory can be run multiple times, e.g. agetty@ can be spawned multiple times to provide agetty processes for different terminals. This can be faked in runit with symlinks, but nitro also allows fully dynamic creation of service instances.

• log chains. runit supports only one logger per service, and log services can’t have loggers on their own.

Currently, nitro also lacks some features:

• service checks are not implemented (see below), a service that didn’t crash within 2 seconds is considered to be running currently.

• runsvchdir is not supported to change all services at once. However, under certain conditions, you can change the contents of /etc/nitro completely and rescan to pick them up. nitro opens the directory /etc/nitro once and just re-reads the contents on demand. (Proper reopening will be added at some point when posix_getdents is more widespread. opendir/readdir/closedir implies dynamic memory allocation.)

• You can’t override nitroctl operations with scripts as for sv.

nitro tracks service identity by name, not inode number of the service directory. This has benefits (parametrized services are possible) and drawbacks (you may need to restart more things explicitly if you fiddle with existing services, service lookup is a bit more work).

On the code side, nitro is written with modern(ish) POSIX.1-2008 systems in mind, whereas runit, being written in 2001 contains some quirks for obsolete Unix systems. It also uses a less familiar style of writing C code.

Do containers need a service supervisor?

It depends: if the container just hosts a simple server, probably not. However, sometimes containers also need to run other processes to provide scheduled commands, caches, etc. which benefit from supervision.

Finally, PID 1 needs to reap zombies, and not all processes used as PID 1 in containers do that. nitro is only half the size of dumb-init, and less than twice as big as tini.

Declarative dependencies

Both runit and nitro don’t support declaring dependencies between services. However, services can wait for other services to be up (and nitro has a special state for that, so only successfully started services are considered UP.)

Personally, I don’t believe service dependencies are of much use. My experiences with sysvinit, OpenRC, and systemd show that they are hard to get right and can have funny sideeffects such as unnecessary restarts of other services when something crashed, or long delays until the system can be brought down.

For system bringup, it can be useful to sequence operations (e.g. start udevd very early, then bring up the network, then mount things, etc.). nitro supports this by allowing the SYS/setup script to start and wait for services. Likewise, services can be shutdown in defined order.

Deferring to policies

nitro is a generic tool, but many features provided by other supervisors can be implemented as site policies using separate tools. For example, nothing stops you from writing a thing to infer service dependencies and do a “better” bringup. However, this code doesn’t need to be part of nitro itself, nor run inside PID 1.

Likewise, things like liveness checks can be implemented as separate tools. External programs can quite easily keep track of too many restarts and trigger alerts. An simple Prometheus exporter is included in contrib.

Lack of readiness checks

At some point I want to add readiness checks, i.e. having an explicit transition from STARTING to UP (as mentioned above, currently this happens after 2 seconds).

Unfortunately, the existing mechanisms for service readiness (e.g. systemd’s sd_notify or s6 notification fd) are incompatible to each other, and I don’t really like either. But I also don’t really want to add yet another standard.

Historical background

[This is mostly written down for future reference.]

I think my first exposure to daemontools-style supervision was back in 2005 when I had shared hosting at Aria’s old company theinternetco.net. There was a migration from Apache to Lighttpd, which meant .htaccess files weren’t supported anymore. So I got my own Lighttpd instance that was supervised by, if I remember correctly, freedt.

Later, I started the first musl-based Linux distribution sabotage and built busybox runit-based init scripts from scratch.

When Arch (which I used mostly back then) moved towards systemd, I wrote ignite, a set of runit scripts to boot Arch. (Fun fact: the last machine running ignite was decommissioned earlier this year.)

Finally, xtraeme discovered the project and invited me to help move Void to runit. En passant I became a Void maintainer.

Work on nitro started around 2020 with some experiments how a monolithic supervisor could look like. The current code base was started in 2023.

NP: EA80—Die Goldene Stadt

You may perhaps not recognize the name of Kevin S. Braunsdorf, or “ksb” (kay ess bee) as he was called, but you certainly used one tool he wrote, together with Matthew Bradburn, namely the implementation of test(1) in GNU coreutils.

Kevin S. Braunsdorf died last year, on July 24, 2024, after a long illness.

In this post, I try to remember his work and legacy.

He studied at Purdue University and worked there as a sysadmin from 1986 to 1994. Later, he joined FedEx and greatly influenced how IT is run there, from software deployments to the physical design of datacenters.

Kevin was a pioneer of what we today call “configuration engineering”, and he wrote a Unix toolkit called msrc_base to help with these tasks. (Quote: “This lets a team of less than 10 people run more than 3,200 instances without breaking themselves or production.”) Together with other tools that are useful in general, he built the “pundits tool-chain”. These tools deserve further investigation.

Now, back in those days, Unix systems were vastly heterogeneous and ridden with vendor-specific quirks and bugs. His tooling centers around a least common denominator; for example, m4 and make are used heavily as they were widely available (and later, Perl). C programs have to be compiled on their specific target hosts. Remote execution initially used rsh, file distribution was done with rdist. Everything had to be bootstrappable from simple shell scripts and standard Unix tools, porting to new platforms was common.

The idea behind msrc

The basic concept of how msrc works was already implemented in the first releases from 2000 we can find online: at its core, there is a two-stage Makefile, where one part runs on the distribution machine, and then the results get transferred to the target machine (say, with rdist), and then a second Makefile (Makefile.host) is run there.

This is a practical and very flexible approach. Configuration can be kept centralized, but if you need to run tasks on the target machine (say, compile software across your heterogeneous architecture), it is possible to do as well.

Over time, tools were added to parallelize this (xapply), make the deployment logs readable (xclate), or work around resource contention (ptbw). Likewise, tools for inventory management and host definitions were added (hxmd, efmd). Stateful operations on sets (oue) can be used for retrying on errors by keeping track of failed tasks….

All tools are fairly well documented, but documentation is spread among many files, so it takes some time to understand the core ideas.

Start here if you are curious.

Dicing and mixing

Unix systems contain a series of ad-hoc text formats, such as the format of /etc/passwd. ksb invented a tiny language to work with such file formats, implemented by the dicer. A sequence of field separators and field selectors can be used to drill down on formatted data:

% grep Leah /etc/passwd
leah:x:1000:1000:Leah Neukirchen:/home/leah:/bin/zsh
% grep Leah /etc/passwd | xapply -f 'echo %[1:5 $] %[1:$/$]' -
Neukirchen zsh

The first field (the whole line) is split on :, then we select the 5th field, split by space, then select the last field ($). For the basename of the shell, we split by /.

Using another feature, the mixer, we can build bigger strings from diced results. For example to format a phone number:

% echo 5555551234 | xapply -f 'echo %Q(1,"(",1-3,") ",4-6,"-",7-$)' -
(555) 555-1234

The %Q does shell-quoting here!

Since the dicer and the mixer are implemented as library routines, they appear in multiple tools.

“Business logic” in m4

One of the more controversial choices in the pundits tool-chain is that “business logic” (e.g. things like “this server runs this OS and has this purpose, therefore it should have this package installed”) is generally implemented using the notorious macro processor m4. But there were few other choices back then: awk would have been a possibility, but is a bit tricky to use due to its line-based semantics. perl wasn’t around when the tool-chain was started, though it was used later for some things. But m4 shines if you want to convert a text file into a text file with some pieces of logic.

One central tool is hxmd, which takes tabular data file that contain configuration data (such as, which hosts exist and what roles do they have), and can use m4 snippets to filter and compute custom command lines to deploy them, e.g.:

% hxmd -C site.cf -E "COMPUTONS(CPU,NPROC)>1000" ...

Later, another tool named efmd was added that does not spawn a new m4 instance for each configuration line.

m4 is also used as a templating language. There I learned the nice trick of quoting the entire document except for the parts where you want to expand macros:

`# $Id...
# Output a minimal /etc/hosts to install to get the network going.

'HXMD_CACHE_TARGET`:
	echo "# hxmd generated proto hosts file for 'HOST`"
	echo "127.0.0.1	localhost 'HOST ifdef(`SHORTHOST',` SHORTHOST')`"
	dig +short A 'HOST` |sed -n -e "s/^[0-9.:]*$$/&	'HOST ifdef(`SHORTHOST',` SHORTHOST')`/p"
'dnl

This example also shows that nested escaping was nothing ksb frowned upon.

Wrapper stacks

Since many tools of the pundits tool-chain are meant to be used together, they were written as so-called “wrappers”, i.e. programs calling each other. For example, above mentioned hxmd can spawn several commands in parallel using xapply, which themselves call xclate again to yield different output streams, or use ptbw for resource management.

The great thing about the design of all these tools is how nicely they fit together. You can easily see what need drove the creation of the tool, and how they still can be used in a very general way, also for unanticipated use cases.

Influences on my own work

Discovering these tools was important for my own Unix toolkit and some tools are directly inspired, e.g. xe, and arr.

I still ponder host configuration systems.

NP: Adrianne Lenker—Not a Lot, Just Forever