Systemd: The Biggest Fallacies

Systemd: The Biggest Fallacies

Jude’s Blog

Systemd: The Biggest Fallacies
Over this past year, I’ve seen a lot of frequently-used but logically invalid arguments for using systemd. This blog post is meant to serve as a repository of common but invalid arguments for using systemd that I and others have had to refute multiple times. This is meant to be a living document–I’ll update it with more fallacies as I encounter them, and I will direct people here who make these mistakes.

Please be informed that this post is not meant to be a criticism of systemd or its authors. For the record, I personally believe that the question “is systemd good or bad?” can only be answered in terms of a particular user’s requirements–systemd is good for the people who need it, but perhaps less desirable for people who do not.
Fallacy #1: “Systemd is multiple binaries, therefore it is not monolithic”
In his blog, Lennart Poettering claims that systemd is not monolithic by pointing out that it is made of upwards of 69 separate binaries. This is a non sequitur, because “modular” and “monolithic” are not mutually exclusive terms.

A piece of software is modular if it is decomposable into distinct functional units such that each unit addresses a specific concern. Systemd, the Linux kernel, and are all examples of modular software. Systemd addresses its concerns with its binaries, the Linux kernel with loadable kernel modules, and with its drivers and extensions.

Now, a piece of software is monolithic if its components (if it has any at all) are tightly coupled–that is, components logically depend on one another to the point where using them in different contexts requires re-implementing the missing ones. Examples include Linux and–in Linux’s case, you can’t use a kernel module without the kernel, the kernel can’t run without the requisite kernel modules to interface with the hardware, and you can’t use a Linux kernel module with other kernels or as a stand-alone program. Similarly, you can’t use an X video driver without the X server, you can’t use the X server without at least one video driver, and you can’t use X’s video drivers with other graphics managers or as stand-alone programs.

Under these definitions, systemd qualifies as both modular and monolithic. You cannot run journald without systemd, and cannot run systemd without journald (at least, not without losing logging for systemd-supervised programs). None of the *ctl programs work without systemd, nor do its collection of systemd-*d daemons. It used to be possible to run logind separately, but not anymore. According to the systemd developers, udev will likely be next to hard-depend on systemd. The point is, despite the fact that systemd is comprised of multiple binaries, the hierarchical logical coupling between them means that it is more accurate to think of them as extensions to systemd-PID-1 that just happen to run in separate address spaces. They are not truly independent, composable programs.

By contrast, examples of modular, non-monolithic software include GNU coreutils, the s6 supervisor system, DJB’s daemontools, and the GNU compiler collection. Not only are these applications broken into modules, but you can use the modules largely independently of one another in different contexts, composing them together to accomplish more complex tasks than the individual modules themselves can handle.

Now, “modular vs non-modular” and “monolithic vs non-monolithic” are continuous trade-offs. Systemd is less monolithic and more modular than, say, the “ls” program. But it is more monolithic than any of the non-monolithic examples above, due to the inter-binary logical coupling they exhibit.

Fallacy #1.1: “Systemd’s components have well-defined interfaces, so you can just replace the parts you don’t like.”
This line of thinking commits an error of omission. Not all of its interfaces are stable, so this isn’t a sustainable option unless you can commit to it full-time. Of particular importance is the interface to the main service daemon, which isn’t stable. Also, the authors themselves concede that parts of systemd cannot be re-implemented independently.
Fallacy #1.2: “The Linux kernel is monolithic, therefore it is okay for systemd to be monolithic”
This is also a non sequitur. Whether or not systemd should be monolithic has absolutely nothing to do with the Linux kernel.

Fallacy #2: “Lots of people use systemd, therefore you should too”
This is a classic example of the bandwagon fallacy. Recall that lots of people and distros used sysvinit at one point, but that did not make it the most superior init system either.
Fallacy #2.1: “Systemd earned widespread adoption through its technical merits”
This is an example of a self-serving bias. It remains to be seen how many people use systemd because they explicitly want systemd, versus how many people use it because their favorite upstream programs depend on it, their Linux distribution switched to it, or their workplace mandated its adoption. The latter cases–installing systemd as a means to use other programs–do not count as adoption through technical merit.

Why do distributions adopt systemd, if not for its technical merits? A big motivating factor is that other application suites (e.g. GNOME) increasingly depend on it to work. If the goal of the distribution is to provide support for these end-user application suites, then pulling in systemd as a dependency is only a means to that end: they’re not adopting systemd because it’s technically superior; they’re adopting it because it’s less costly than maintaining patches to keep systemd decoupled.

Speaking of adoption, how does one measure “widespread?” Android is probably the most widely-used Linux-powered OS by far, and it doesn’t use systemd. Perhaps the most widely-deployed consumer GNU/Linux distribution by sales is ChromeOS, but ChromeOS uses Upstart, and that probably won’t change. So, arguably systemd isn’t that widespread across the Linux landscape in the first place.

Adoption in GNU/Linux distributions isn’t universal either. Gentoo and Slackware–distributions for technically-inclined users–have not adopted it as the default init system. Also, have a look at Debian’s popcon numbers for systemd-sysv vs sysvinit-core. At the time of this writing there are roughly 3x as many installs as systemd-sysv (which conflicts with sysvinit-core) than there are of sysvinit-core, which is about the same disparity as that between the meta-packages for Xfce 4 and GNOME 3. So, despite being available to all major GNU/Linux distributions, it has by no means achieved universal uptake.

Fallacy #3: “People who don’t like systemd just don’t like change”
This is an ad hominem fallacy. Just because systemd is popular these days doesn’t mean it will solve everyone’s problems better than the status quo. The people who do not use systemd often have very good technical reasons for avoiding it, and there are plenty of alternative init systems that are better suited for the task.

It is tempting to think that there is no technical reason to keep using an existing init system, because systemd is backwards-compatible with initscripts. However, this line of thinking makes a lot of implicit assumptions about the needs of the user, which may not hold true. For example:
It assumes that the user is comfortable adding all of systemd’s code to his/her trusted computing base.
It assumes that the user isn’t already doing advanced things with namespaces and cgroups which systemd would break.
It assumes that the user is unhappy with his/her current service management setup, which systemd would replace.
It assumes that the user has sufficient RAM, CPU, storage, and power to add systemd to his/her running computer. This is not always the case for low-power embedded Linux, where “init” might just be the embedded application itself (NB: I’m referring to really low-powered devices such as this, not smartphones or equivalent hardware).
It assumes that the user has a sufficiently recent Linux kernel. This is also not always the case with more exotic hardware, where proprietary kernel modules can keep you locked into an old version.
It assumes that the user will have a completely bug-free experience and will not have to sink unjustifiable amounts of time into setting systemd up in their environment.
It assumes that the user will easily be able to migrate away from systemd if it turns out to be a bad match down the road. If the user is setting a computer up that he/she won’t be able to easily administer (like an Internet-of-Things device), it will need to be as bug and quirk free as possible on the first try. A provably-correct init system would be ideal, but a tried-and-true well-understood init system would be preferable to an unproven one.
Dismissing other peoples’ init system requirements as illegitimate is a great way to get written off as a troll. So is trying to tell them they’re wrong about their own requirements.

Fallacy #4: “Unit files are better than shell scripts”
This is a faulty generalization that ignores a lot of important realities. Now, I don’t think anyone disagrees that less code is better when you can get away with it. However, the devil is in the details, and there are subtle but important engineering details regarding service management that often get overlooked in conversations about unit files.
Fallacy #4.1: “Unit files reduce complexity”
This is misleading vividness, and a mischaracterization of the problem. While it is true that unit files look simpler than init scripts and can reduce the complexity of defining services, they don’t reduce the complexity of the init system, nor do they reduce the complexity of debugging the init system when it misbehaves (see Fallacy #4.2). In fact, all the complexity that used to be in shell scripts under sysvinit (as well as a few daemons and tools) has simply moved into systemd, which currently has over 275,000 lines of code. Moreover, it’s 245,000 lines of C according to this analysis, of which 81,000 are required to have a minimal working instance (this is charitably excluding systemd-udevd, which adds an additional 13,000 lines). By way of comparison, the unmodified initscripts on my Debian laptop weigh in at just over 10,000 lines of shell script (according to sloccount(1)).

It’s not clear that systemd makes booting and service management less error-prone than other systems, but the overall complexity of systemd is certainly higher than sysvinit.

Fallacy #4.2: “Shell scripts are buggy”

This is a red herring often used to argue against using shell scripts. Bugs aren’t intrinsic to shell scripts–buggy software is buggy, no matter what language it’s written in.

For those of you who have never had to debug a C program (the language systemd is written in), take my word for it that C is a lot harder to debug than a shell script, since a lot more can go wrong in C. This is because the programmer must not only deal with variables and flow control, but also memory management, thread management, thread synchronization, file descriptor management, low-level IPC, CPU-specific memory models, C’s undefined behaviors (see Appendix J), and a bunch of other things that aren’t applicable to reasoning about the behavior of shell scripts. Moreover, if something goes wrong during boot, the C debugging tools aren’t guaranteed to be available–they could be installed on media that gets mounted late in the boot process (such as on /usr or /usr/local). This problem does not exist when using shell scripts to boot the system.

While it is true that the programs a shell script invokes are (usually) also written in C, this does not invalidate the aforementioned claim. In practice, if the shell script fails to execute, the bug is almost always in the script, not the underlying shell or program it invokes. If systemd fails to act on a valid parameter of a unit file, however, the user must debug systemd’s C code to find out why.

Fallacy #4.3: “Systemd is better because it gets rid of shell scripts!”

This is another red herring, and more of an opinion than a technical argument. However, shell scripts were never necessary to boot your system in the first place. The kernel and plenty of init systems (including sysvinit) can be configured to start an arbitrary program, written in arbitrary languages. For example, Pardus Linux’s init system uses Python programs instead of shell scripts.

Fallacy #4.4: “Systemd is better because it reduces shell script code duplication!”

This is also a red herring, which distracts from the fact that nothing stops the user from consolidating all of the init scripts’ common code into a file that individual scripts can source. Take a look at what the BSDs do with /etc/rc.subr, for example–service definitions with BSD’s rc are often shorter than unit files!

Fallacy #5: “Systemd enables you to use $LINUX_SPECIFIC_FEATURE”
This is yet another red herring. Nothing was stopping the user from leveraging those Linux-specific features before systemd. By definition, all these features had to predate systemd before systemd could be written to use them.

Fallacy #6: “Systemd is open source, so if you don’t like it, you can fork it!”
Some people are working on that, despite the fact that systemd is a large moving target.

However, this statement is an example of black-and-white thinking that usually misses the point. A non-systemd user is often more interested in having systemd not encroach on his/her systems. This fallacious argument tells them to fork systemd…to not have to use systemd.
Fallacy #6.1: “Those systemd detractors should stop whining and make their own alternative!”
This also misses the point, and is a mixture of black-and-white thinking and ad hominem. Most non-systemd users are perfectly happy to leave systemd alone and let it do its thing for the people who want to use it. Whatever init system they are currently using is perfectly fine for their needs.

The problem is that the systemd project doesn’t want to leave them alone. Lennart Poettering’s vision for systemd is for it to subsume the whole base OS. He’s also advocated for GNOME’s hard-dependency on logind, and is of course responsible for logind having a hard-dependency on systemd-as-PID-1. There are plenty of init systems, but only one of them seems to be actively trying to exclude the others by making unrelated software require it to function.

A lot of non-systemd users don’t need a systemd clone, since they’re happy with what they have already. So, there’s no point in telling them to go work on cloning systemd.

Fallacy #7: “You should use systemd, since it gives you socket activation!”
This is propaganda. Systemd isn’t needed for socket activation, since the kernel already gives it to you for free.

For Internet socket activation, all the user has to do is start the daemon, have it bind on the port, and let it sit there idly. If the kernel needs RAM, it will swap the (idle) daemon to disk. If the kernel receives an incoming connection, it will swap the daemon back into RAM so it can handle it. Systemd isn’t involved at all here, but it has the same effect. As an added bonus, the kernel’s virtual memory subsystem ensures the daemon’s runtime state gets preserved across swaps–something systemd does not allow with its form of socket activation.

Fallacy #7.1: “You should use systemd, since it lets you assign targets to be activated on receiving the connection!”
This is also propaganda. Xinetd has done this for decades. It can be configured to run whatever would be invoked by the unit files, or run “systemctl … enable” to achieve the same end.

Fallacy #7.2: “You should use systemd, since it supports more than network sockets! It supports pipes and UNIX sockets too!”
More propaganda. This tiny shell script achieves the same effect with pipes:

$ mkfifo /path/to/pipe
$ sh -c ‘while true; do cat /path/to/pipe >/dev/null; /path/to/program; done’ &

When a program writes to /path/to/pipe (i.e. echo “” > /path/to/pipe), /path/to/program will be executed. This script can be extended to do more complex things of course, like have the “while” loop take different actions depending on what you write to /path/to/pipe, but the point is, pipe activation isn’t hard to come by. A similar script can be written for using UNIX sockets with netcat and socat. If lifecycle management is a concern–for example, ensuring that the pipe or UNIX socket always gets unlinked once the program is done with it–I encourage you to take a look at runfs.

Fallacy #7.3: “You need to use systemd for socket activation, since otherwise you’ll encounter race conditions with parallel start-up!”
This is a moral panic. First, this has been a non-issue for decades, and continues to be a non-issue for modern event-driven systems (for example, Android gets along fine without systemd). Second, both parallel start-up and guaranteed absence of race conditions can be had using basic information from procfs.

The problem is formulated as follows. Suppose processes B depends on processes A[0], A[1], …, A[n] to be started and listening before B can start. We want to start B in parallel with A[0]…A[n], but ensure that B does not try to connect to any A[i] until all of A[0]…A[n] are listening.

The solution prior to systemd is to simply start A[0], A[1], …, A[n], and wait a few seconds to let them reach a listening state before starting B. The intuition is that A[0]…A[n] will reach listening state before B connects, since even one second is a very long time for a process A[i] to go from double-fork daemonizing to listening on a socket. While it is true that B could race ahead of A[0]…A[n] and try to connect before one of them is listening, the probability of this happening has proven extremely small over the decades.

Now, suppose for the sake of argument that B does manage to race ahead of some process A[i]. What happens? B’s connection attempt will fail with a “connection refused” error, even though all processes A[0]…A[n] are listening. B would fail to start, unless it is programmed to handle this case (e.g. to retry connecting until a retry threshold or timeout has been exceeded).

Ideally, B is smart enough to do this on its own. If not, the socket activation solution proposed by systemd is to modify processes A[0]…A[n] so that they receive a socket descriptor from systemd, which systemd used to buffer up messages from B while each A[i] was starting up (systemd’s “socket activation”). This lets B start in parallel with A[0]…A[n] while preventing it from making forward progress until A[0]…A[n] are all listening.

This solution is costly, however–it requires modifying O(n) programs, it will only work with systemd, and it will solve a problem that is rare in practice. A simpler strategy is to leverage information in procfs. We can write a script that first checks the socket descriptors in /proc/A[0]/fd/, /proc/A[1]/fd/, …, /proc/A[n]/fd/ to see if they are in a listening state by matching their inode numbers against open sockets in /proc/net/$PROTOCOL. Once all sockets are in a listening state, the script starts B. There is no need for systemd to get involved, there is no need to modify A[0]…A[n], and the solution works wherever procfs is available (and even if it isn’t, only the script–not the daemons–needs to be modified to port the “socket activation” functionality).

Fallacy #8: “$PROGRAM doesn’t depend on systemd, it just depends on something that implements systemd’s interface”
When talking about one of the components or interfaces that cannot be re-implemented independently (such as systemd-logind or the service bus API), this argument is a distinction without a difference. It’s like saying “Internet Explorer doesn’t depend on Microsoft Windows, it just depends on something that implements Microsoft Windows’s interface!” While it’s certainly possible to re-implement the requisite parts of the API, doing so effectively re-implements most of the platform as well. So, if $PROGRAM depends on one of these components or interfaces, then $PROGRAM either requires systemd or a mostly-complete re-implementation, making it systemd in all but name.

Fallacy #9: “Systemd is the KISS principle in action”
This is a misattribution. The KISS (“Keep It Simple, Stupid”) principle refers to a system’s design, not its user interface. KISS is the principle of avoiding unnecessary complexity. But, as we have established in Fallacy #4.1, the minimally-usable systemd is more complex than sysvinit and tries to do more.

Now, the systemd developers believe that the extra complexity in even the minimal systemd is necessary to meet their requirements. All I am pointing out is that these requirements are not universal, and for many, the complexity is unjustified.

Fallacy #10: “Binary log files aren’t a problem in practice”
This can be either faulty generalization or false induction, and is often made in the context of “I’ve used binary logging, therefore binary logging must be okay for everyone.”

The question we’re interested in answering is “Are the log format and logging facility always able to record the set of events and failures that the user requires, in the order in which they occur?” If the answer to this question is “yes,” then the choice of formatting is arbitrary, since the implementation details won’t matter. Otherwise, if there are cases where the logging facility can behave unexpectedly (such as a bug in the logger, a kernel panic, or hardware failure–things that happen somewhat regularly), the implementation details do matter, since the user will need to manually intervene to determine the sequence of events and failures leading up to the bad behavior. In this case, logging structured binary records is almost always a worse design choice than logging unstructured human-readable text, since it’s almost always easier to recover a plain-text log from unrecoverable logging failures (particularly corruption).

Log corruption is perhaps the hardest problem for a logging facility to address, since information is lost. However, human-readable text remains at least partially readable when corrupted, since the rules, grammar, and semantics of written language serve as naturally-occurring error-correcting measures. For example, a user can tell at a glance whether or not a string of text is truncated, or is missing one or more words, or contains data that shouldn’t be there (i.e. non-printable characters), or doesn’t fit any expected log message pattern, and so on, simply because logging only human-readable text lets the user deduce what the logger intended to write. However, the more binary data gets included with log records, the harder it becomes for the user to leverage written language to deduce intent (e.g. is a string of arbitrary bytes a sign of corruption, or is it part of the record, or is it a bit of both?).

At the time of this writing, journald can corrupt the log simply by crashing, putting the log into a state where it is impossible to tell whether or not it was corrupted or tampered with (the developers have known about this for years, and they don’t think this is a bug). Now, it is possible to manually parse a corrupt journald log if journalctl can’t help, but it requires more effort than parsing a corrupted plain-text log since the user must be aware of the journald log format in order to extract the human-readable strings from the binary records. So, when it comes to dealing with failure modes that involve log corruption (in journald’s case, these are all the failures that can lead to unclean shutdown–exactly the failures a logging facility should capture), a plain-text log will almost always be of more immediate use than journald’s log.

Now, journald’s logging could be more robust. It could improve resilience in the face of corruption by including forward error correction codes, by keeping its binary data in a separate file (or encoding it as human-readable text), and by implementing ACID semantics on log writes. However, it does not do any of these things at the time of this writing, which makes it all the more fragile in the face of failure.

Fallacy #10.1: “Journald’s binary format lets its perform authenticity and integrity checks”
This is propaganda. Most modern syslog implementations do this too, so this doesn’t compel journald’s adoption.

Fallacy #10.2: “Journald binary format lets it have an index that makes log access faster”
This is also propaganda. While no one will contest that indexing will speed up log access (at least in theory), there’s also no technical reason why journald can’t keep the index separate from the logfile, or can’t write the index into the logfile as human-readable text.

Fallacy #10.3: “All the problems you have with journald can be avoided by having it simply pass everything to syslog”
This assumes a faulty premise that there is a need for journald at all in this case. If all the user needs is syslog, then why require journald at all?

Fallacy #11: “People who don’t like systemd really just don’t like Lennart Poettering, and that’s not a valid excuse to avoid it”
First, this is putting words in the mouths of non-systemd users (ad hominem). Second, there are plenty of valid non-technical reasons to use or avoid software. Not trusting the lead developer to have your interests at heart is one of them.

The GNU/Linux ecosystem is a bazaar, not a cathedral. Software development in this space necessarily has a social dimension, because far-flung developers working on a common goal have to communicate to get things done. As such, developers cultivate reputations for their management style, coding practices, responsiveness to bug reports, and so on, and these rightfully influence the degree to which other developers trust them to do a good job on their projects (for example, Debian switched from GNU libc to EGLIBC for a time, citing friction with the GNU libc maintainer as a major reason).

Now, among some circles, Lennart Poettering has developed a negative reputation for too often breaking compatibility and dismissing constructive criticism of his projects (including bug reports). This is not an attack on his character; it is a critique of his project management style. For those users who are negatively affected by this, it makes complete sense to avoid systemd, since it does not appear to them that the project lead has their interests at heart. This is no different than a user leaving Windows for Linux, because Microsoft no longer appears to have their interests at heart.

Disliking how Lennart does his job and manages his projects is not the same as disliking him personally, but too often, it seems to me that both proponents and detractors conflate the two. It is prudent for proponents to determine which is being criticized before throwing around ad hominem accusations, and it is prudent for detractors to emphasize the target of their criticisms. My point is, skepticism towards systemd due to its management style is not ad hominem, and skepticism towards any software project due to its management style can be quite reasonable.

Fallacy #11.1: “You should judge systemd on its technical merits alone, otherwise your criticisms are wrong”
This is simply naive. Software does not exist in a vacuum–there are people, organizations, intentions, and reputations behind every line of code, and there are costs to buying into software and switching away from it even when it’s free (as in beer and as in freedom). As such, it’s perfectly reasonable to embrace or avoid software for reasons besides technical merits. For example, Free Software aficionados don’t use non-free software, even if there exist technically superior non-free offerings, since non-free software violates one or more of the Four Freedoms.

However, suppose for the sake of argument that systemd does exist in a vacuum, and we want to judge systemd on its technical merits alone. This means that we cannot permit any biases or prejudices we may harbor for or against systemd, its authors, its users, etc. to influence our evaluation. We only want to know whether or not systemd meets a user’s requirements.

To judge on technical merits alone, there first needs to be a formal specification that precisely and unambiguously defines what systemd does and does not do. Second, there needs to be a formal proof that systemd’s implementation meets this specification. Both are necessary to evaluate systemd on its technical merits alone; otherwise, there’s no way conclusively show that systemd meets the requirements.

Sadly, doing these two things is a huge undertaking that’s not feasible in practice (this isn’t specific to systemd; this applies to most non-trivial pieces of software). So, we’re left with less-than-perfect heuristics for the time being, like doing black-box tests, or reading the source code, or considering the intentions and trustworthiness of the developers. But that’s okay, since more often than not they still yield useful insights into whether or not software meets your requirements. There’s plenty of space for argument over what are the best heuristics, but my points are that (1) technical merits aren’t always enough to decide whether or not to use a piece of software, and (2) judging on technical merits alone is infeasible in practice anyway.

Fallacy #12: “The Linux kernel is complex, and you use the Linux kernel, therefore you are okay with systemd being complex too”
This is a non sequitur similar to Fallacy 1.2. Just because someone runs the Linux kernel does not mean that he/she wants more complexity than it offers. Suggesting otherwise is a false dilemma: accept all the complexity of Linux+systemd, or go without either. Complex vs simple is not a binary choice, and tolerance of software complexity depends on a wide variety of factors, including the context in which it is used, the degree of testing it has received, the degree of familiarity with its inner workings, etc.

Besides, a lot of Linux’s extra complexity can be factored out at build time, which some users put in the effort to do.

Fallacy #13: “Systemd has at least 574 contributors from many different backgrounds, therefore it is not pushed or controlled by a small group”
This is naive at best, and an outright non sequitur at worst. While it is true that the systemd project has many contributors, the vast majority of systemd code came from only a few people.

I checked out the latest systemd code using the links here, and ran gitstats on it (I charitably included udev in this analysis, even though udev existed before systemd). Here’s what I found:

The systemd project has 688,539 lines total (1,468,444 added, 779,905 removed). Note that this includes everything, not just the C code that compiles into systemd’s binaries.
The systemd project has 17,503 commits.
The top ten contributors by number of commits represent 82.73% of all commits.
Of the top 10 contributors by commit count, eight of them are also in the top ten list of contributors by lines of code added.
The top 10 contributors wrote 90.8% of systemd. By lines of code added, the top 10 are responsible for contributing a net total of 625,711 lines.
So, it is in fact more accurate to say that systemd is controlled by a handful of people. The fact that a few hundred people each sent them a small amount of code does not mitigate this handful’s influence.
The point of this article is to serve as a repository of commonly used but invalid arguments for using systemd. I will keep adding more fallacies as I encounter them, and update the text above to improve clarity. Again, this post is not meant to be a criticism of systemd; only a refutation of common but invalid arguments in favor of its adoption.
Comment Policy
I’m not a fan of censorship, so I don’t delete comments (but might employ anti-spam policies). However, I will move off-topic comments to a separate web page to keep the conversation here relevant. You’re free to have whatever off-topic conversations you want elsewhere 🙂
Posted 26th September 2014 by Jude C. Nelson
Labels: systemd

31 View comments

AnonymousSeptember 26, 2014 at 5:38 PM
Your popcon numbers are wrong, because sysvinit is a transitional package. Compare sysvinit-core with systemd-sysv


Jude C. NelsonSeptember 26, 2014 at 9:29 PM
Hi Joey,

Thank you for your comment, and thank you for pointing this out! I agree that it is more correct to compare using sysvinit-core, since it contains the sysvinit binary (and is the package that gets uninstalled and replaced by systemd-sysv when you upgrade to systemd).

I’ve updated the text to reflect this.


CamBamSeptember 26, 2014 at 7:16 PM
Regarding your 275k LOC comment:

I do agree for the most part with you, however.


Jude C. NelsonSeptember 26, 2014 at 9:48 PM
Hi CamBam,

Thank you for your comment! I’ve added some clarification to the LOC analysis to incorporate the data from your link.


KeithCuSeptember 26, 2014 at 10:02 PM
There are a number of fallacies in this document. Go through the comments for more:


Mike BirdSeptember 26, 2014 at 10:39 PM
Excellent analysis.

FWIW systemd claims to be a drop-in replacement for sysvinit and yet it fails to support sysvinit’s primary config file – inittab.

Systemd’s design is awful and its code is mediocre but the real problem is the leverage. When you drink the systemd kool-aid you allow RH to churn your distro at will – burning up your valuable volunteer hours – and you allow RH to claim that your distro is always playing catchup to RHEL.


saynteSeptember 27, 2014 at 12:06 AM
Contradictions, quoting:

4.1: C is a lot more error-prone than shell
4.2: Bugs aren’t intrinsic to shell scripts–buggy software is buggy, no matter what language it’s written in.


4.1: systemD is big and written in C, and we’ll have to debug it
4.1 (later): bash (to pick a common shell) is big and written in C too, but we’ll assume we never have to debug it

Fallacy #11: Ad hominem is a valid form of reasoning when applied to Lennart Poettering


Jude C. NelsonSeptember 27, 2014 at 10:12 AM
Hi saynte,

Thanks for your comment!

I don’t think 4.1 and 4.2 are contradictory. It is demonstrably the case that you have to do things in C that you don’t have to do in the shell to reach the same end. This is what is meant by “error-prone.” However, “buggy” refers to software, not the language. You can have software that is proven to be consistent with its specification (i.e. bug-free) regardless of what language it is written in.

Jude C. NelsonSeptember 27, 2014 at 10:33 AM
> Fallacy #11: Ad hominem is a valid form of reasoning when applied to Lennart Poettering

Please, let’s keep it civil.

saynteSeptember 27, 2014 at 11:01 AM
Haha, sorry, I was being a bit tongue-in-cheek there. I was trying to summarize and critique your point that not liking Lennart is a reason to not like systemD. This is an ad hominem reasoning. systemD is software and is either good or bad based on its technical merits, or lack there-of, not who wrote (part of) the code.

Of course one is free use this reasoning, but it doesn’t make it good reasoning, which I think is the point.

saynteSeptember 27, 2014 at 11:08 AM
Re: 4.1 and 4.2, then error-prone is the wrong word to use, because it clearly means more likely to have errors.

I would say “lower level of abstraction”. Although, even then, I find C has a relatively clean semantics and shell script is quite a bit more arcane, so even then this point is quite subjective (re: which is more complicated to debug).

ОпасностеSeptember 27, 2014 at 11:59 AM
Well, ad hominem is kind of valid, that’s what we do in real life anyway (e.g. elections are mostly about personalities, much less about specific fantasies and outright lies of each candidate — and this is how the future of country and nation is chosen, which affects daily lives of Linux users slightly more than systemd).

And yes, it’s mostly safe to assume we never have to debug bash.

Jude C. NelsonSeptember 27, 2014 at 10:45 PM
not liking Lennart is a reason to not like systemD. This is an ad hominem reasoning. systemD is software and is either good or bad based on its technical merits, or lack there-of, not who wrote (part of) the code.

Can you point out where I passed holistic judgement on systemd being “good” or “bad,” so I can remove the offending text? Passing holistic judgement on systemd is not the intent of this post.

While I do not disagree that ideally systemd should be judged on its technical merits alone, there are two practical barriers to doing so that cannot be overlooked. First, there is no formal specification that defines what systemd should and should not do. Second, there is no formal proof that the systemd implementation is consistent with the specification. We will need to rigorously address these to conclusively prove that systemd meets a given set of requirements (thereby “judging it on its technical merits alone”), but doing so with state-of-the-art formal verifiers would be exceptionally difficult (I would love to be proven wrong about this claim, by the way). Of course, these difficulties are not specific to systemd; they apply to most non-trivial pieces of software. The point is, “judging software on its technical merits alone” is a nice ideal, but infeasible in practice.

So, if it’s not feasible to formally prove that systemd will meet your requirements, what options do you have? One option is to read the source code and documentation. Another option is to try systemd out and see whether or not it meets your requirements under your tests. Both of these usually (but not always) work well enough, but they’re not sufficient to judge software on its technical merits *alone*, since you may still accept faulty software, or test software under conditions that don’t match your requirements. However, I think we can agree that these are usually good heuristics in practice.

Now, another heuristic when determining if software meets your requirements is to consider the intentions of the developers. Do they intend for their software to solve your problems? Do they intend to add/remove features later on that will make the software unsuitable for you? Do they know what your problems are, and do they know how to solve them? Do they have a history of producing software that addresses your requirements? Do they have a history of creating software that creates more problems while solving others?

I believe these are reasonable questions to ask when deciding whether or not to rely on unproven software (you’re free to disagree), since they tend to yield useful insights into whether or not it is likely to meet your requirements in the absence of a more rigorous evaluation. For example, if the NSA released a GPL’ed kernel that was backwards compatible with and is technically superior to Linux, answering the above questions might cause you to have reservations about using it because (1) backdoors are incompatible with your kernel requirements, and (2) you suspect that the NSA may have put one in that would be hard for you to find. I think we can agree that this isn’t a bad thing–the burden of proof is on the NSA to show that they have added no backdoors. However, for better or worse, being skeptical required you to factor in the authors’ reputations into your decision making process.

That said, pointing out that people apply this heuristic on systemd is not an attack on Lennart (who I have never interacted with and thus cannot even begin to judge), and applying this heuristic isn’t necessarily an act of malice (but to be fair, a lot of systemd detractors have gone on to say mean things about Lennart that I personally feel are undeserved). After all, if we didn’t apply this heuristic, we might find ourselves doing foolish things like completely trusting executables downloaded from unsolicited emails.

Jude C. NelsonSeptember 27, 2014 at 10:45 PM
I would say “lower level of abstraction”. Although, even then, I find C has a relatively clean semantics and shell script is quite a bit more arcane, so even then this point is quite subjective (re: which is more complicated to debug).

It’s more than abstraction, though. C programs are also prone to expecting implementation-defined behaviors. Unlike a shell script, a C program often implicitly assumes a particular set of architectures, a particular set of compiler implementations and versions, and/or a particular set of compiler-specific behaviors. This isn’t limited to simple things like 32-bit vs 64-bit–for example, the C language didn’t even specify a *memory model* until C11, meaning that prior to C11, multithreaded C programs are allowed by the language specification to have implementation-defined memory consistency on different architectures. There’s an illustrative example of how this can go wrong here:

I consider the fact that the shell shields you from a lot of (but not all of) C’s undefined behaviors to be a point in its favor, since it reduces ambiguity. Of course, you’re free to avoid shell scripts; substitude your favored (less ambiguous) langauge here 🙂

saynteSeptember 28, 2014 at 4:42 AM
I didn’t mean to imply that you were passing judgement on it being “good” or “bad”. The quote in Fallacy #11’s title is pointing out that an often-used argument against systemD is ad hominem line of reasoning. Then you’re essentially stating that using this ad hominem reasoning is OK (this was why I posted my initial cheeky response). Are you saying the quote in Fallacy #11 is not ad hom.?

But by focusing in on issues of project management (your heuristics) you’re actually adding more impartiality and sanity than the original heading has, namely that not “liking Lennart” is a good reason.

Jude C. NelsonSeptember 28, 2014 at 10:26 AM
The point I’m trying to make in #11 is that taking a developer’s reputation into account when deciding whether or not to use his/her software is not an act of malice or spite. Statements like “I don’t use systemd because I don’t like Lennart’s coding style”, “I don’t use systemd because I don’t like the way Lennart handles bug reports”, and “I don’t use systemd because I don’t like Lennart’s attitude towards the BSDs” aren’t ad hominem, since they’re not talking about his person. The fallacy I’m getting at is the dismissal of these statements as ad hominem, when they are not.

I will update the text of #11 to make this clearer. It is not my intention to claim that ad hominem is acceptable.

saynteOctober 1, 2014 at 12:19 PM
Hi Jude,

Firstly, thank you for taking the time to talk about these things in a calm way, I appreciate the chance to discuss what you’ve written even if I disagree with the argumentation and some of the points.

However, even your own example: “attitude toward BSD” is very much about his person (HIS attitude… that seems to be about his person…), and doesn’t directly affect the viability of systemD. As I said, such arguments can be extended into better-reasoned points, but that is not one of them. For example: “attitude toward BSD” could be changed into “systemD has limited portability” or “limits its test-base/use-cases by excluding BSD”, which could then be debated on their technical merits.

Also not “liking the way bug reports are handled” is too vague, a reasoned argument would show the negative affect on either the community (thus may affect the number of future bug reports) or the quality of the code.

Of course aspects of project management can be brought in, but should be kept as technical as they can be so they can be effectively argued.

Most times they are nearly trivial to pick apart if names enter the picture. Work in impersonal terms and the debate is much saner I find (just my experience, no concrete proof there).

Even if I disagree with your conclusions and argumentation, thank you again for taking the time to consider other perspectives, it is very refreshing.

Arne BabenhauserheideOctober 6, 2014 at 4:15 PM
@saynte: I would say that a negative effect on the community can also be seen in the very negative reactions systemd gets. Which other init system has seen boycott calls? How would you measure negative effects on the community?

Ingolf SchaeferOctober 7, 2014 at 1:18 AM
What community are you talking of? In the Mageia community there has been no controversy about systemd at all. Same goes for the Fedora and openSUSE folks. There has been discussion in Arch and Debian. But it is not THE community.

Yes, Lennart Poettering has done controversial statements in the past, but I have not seen anything uncivil in the context of systemd. But, I have seen many very impolite (modestly put) statements about systemd from many other people.

Last, I refuse to be part of any community that collects BitCoins to beat someone up.

Laurence PerkinsNovember 25, 2014 at 5:21 PM
To jump in on the “ad hominem” debate, I would point out that ad hominem is a fallacy because it is trying to discredit something or someone based on factors that are not relevant to the decision at hand. (Fallacies in general are such because they’re diverting the chain of reasoning away from the factors which actually impact the correctness of the conclusion.)

Arguing about systemd based on Lennart’s personal political leanings, hair color, choice of recreational intoxicants, et cetera would be ad hominem. Those factors have (in all but the most convoluted of scenarios) little to no impact on the current or future quality, reliability, and usefulness of his project.

Arguing about systemd based on (dis)agreement with his beliefs about what the scope of the project should be, history of ignoring user complaints, propensity to propagate changes that break things and expect other projects to clean up the mess, attitude about supporting non-linux kernels, et cetera is not ad hominem because all of those things have an impact on the future expected quality, usability, and reliability of the project.

In other words, the proper response to a perceived ad hominem attack is not, “That’s ad hominem! You suck! All your reasoning is crap!” The proper response is “How is that relevant?” Because it might be in some way that you just haven’t noticed yet.


Leonardo L.September 28, 2014 at 3:49 PM
Systemd goes very far from a modular init system: you can’t easily extend the unit files language, for instance.

Trying to say something like “start this service after network interface X is available” is not possible – and if you search in the FAQ, it says: “waiting for a particular interface is bad design, fix your code”. Sorry, dnsmasq is not my code, neither most of the daemons I need to bind to an specific interface. So I need this feature.

If there was some way to add plugins/modules to interpret additional keywords to unit files and extend the current behaviour, that would be a modular design – something very easy to do with current sysvinit and service files. This is not only good for the extensiveness of the project, but this would remove the need for the core developers to think about everything, and put the power on the hands of the users.


A_flj_April 5, 2015 at 12:39 PM
There’s a mantra about the interpreter pattern: don’t use the interpreter pattern unless you’re willing to implement a full-featured language. Config files for an init system, especially one that aims as high as systemd, need to be written in a powerful language. Which isn’t at all what unit files are. (Personally, for complex setups, I also find unit files less comprehensible than script – natural language consists of sentences, not bulleted lists.)

“… and put the power on the hands of the users” – and would also go against every bit of philosophy expressed by the whole systemd project. They want to be in control, they don’t want to give control to you. Their stated aim is to reduce variability among Linux distros, and to provide a one stop solution for configuring an entire Linux system. How do you think they can achieve this if they put distro maintainers, admins and end users in control?


BaylinkJune 24, 2015 at 9:43 PM
I have two problems with systemd in addition to the items you mention above to (correctly, IME) support your assertions about the fallacious defenses made in its behalf, and they are somewhat unobvious, but I think, very important:

1) It forces me — as a sysadmin who is prone to write, and more to the point, to *have written over time* lots of code that is buried in the initscripting system and similar paces — to revisit and update code that ought simply to continue to work, but may not now, since the assumptions it made about its environment are changing. It’s easy to say that should all be documented — and it’s probably even correct 🙂 — but the simple fact is that it’s not, and some of it’s going to die, and it will be *then* that I will be forced to try to figure out why. And that was not my time for Lennart Poettering to spend.

And figuring it out will be difficult, because what the systemd team alleges is ‘documentation’ does not meet the standard *I* have come to expect for that material since 1981.

2) Even if you don’t need to run gnome or udev, there’s an even more unobvious reason why it’s hard to stay on — or go back to — sysVinit:

Distributions appear no longer to be requiring that packages contain working initscripts in order to be valid for submission to repos, and the initscripts that are in there are either being removed or allowed to rot, or they already have been.

And having to repackage an entire repo for yourself is a non-starter.

It still escapes me how so *many* supposedly competent Release Configuration Managers at real, full-sized distributions were induced to drink the Flavor-Aid, especially since most of this stuff improves things servers are never supposed to do anyway — boot — and servers are 80+% of the Linux ecosystem, or somewhere near that. If not higher.


CapricornyMarch 30, 2016 at 2:55 AM
Maybe this, from the Wikipedia article on cgroups, is something to consider here?
In the figure, it reads (verbatim):
Cgroups are responsible for ressource management. It makes sense, to grant some daemon exclusive access to this functionality to avoid lots of problems. systemd-nspawn User ScotXW


TomGeoAugust 5, 2016 at 10:00 PM


AmaroqSeptember 15, 2016 at 12:54 AM
I just now found out that there’s a controversy over an init system. (Former linux nerd who went back to windows for gaming.) I gotta say, it’s really enlightening (and irritating) to read a fallacy list like this and find out that there’s a whole propaganda machine out there trying to push an init system.

The biggest fallacy they seem to be committing is the idea that I should immediately follow their preferences just because they have counter-arguments to problems I might have with it.

The biggest offender is “You don’t like systemd because you are resistant to change.” And to about the same extent, “You already use complex software so you should be okay with a complex init system.”

Their utilitarian arguments are not the point. The point is, linux supposed to be a thing where I can look under the hood and arrange/build it how -I- want it to be. If -I- want to be stubborn and stick to what I’m familiar with, that’s my choice. If I want to use complex software and an elegant init system alongside each other, that’s my choice.

Maybe some people don’t care about that choice. But some people do. Just a few days ago, I installed Gentoo for the first time (I’m glad that’s over…). When it came time to pick an init system, I had no idea what the instructions for systemd were trying to get at. It was so complicated and unfamiliar. I also installed it on a dead laptop someone gave me that I fixed, and it uses a Core 2 Solo, so I had no need for parallel booting anyway.

So you know what I did? I picked a simple init system that just uses an init script and runlevels, because that’s simple and familiar and because that’s what I wanted to do.

Same thing with Grub by the way. I had no idea what Grub2’s overcomplicated configuration was going on about, so I said “screw this” and went with Grub-Legacy and its simple, familiar menu.lst.

I don’t pretend to understand every part of my system when I chose it. But I decided I did want to understand my boot/init system (without having to take classes on it). So I picked according to my own personal preference.

A freedom that these systemd guys seem to want to take away from me.


WyattNovember 16, 2016 at 12:08 PM
Funny thing about grub 2: It doesn’t have to be that way! Seriously, look at this. First, a legacy grub.conf:

timeout 4
default 0

root (hd0,1)
kernel /vmlinuz root=/dev/sda2 zcache net.ifnames=0

Here, an equivalent grub.cfg that works for grub2:


menuentry ‘Linux’ {
# This blog doesn’t support indentation, so it’s a little ugly.
linux /vmlinuz root=/dev/sda2 zcache net.ifnames=0

…but I’ll bet you can guess how hard it was to find any simple documentation on this. (Hint: I searched for hours and only found it by accident after I gave up.)


HaakonLøtveitDecember 12, 2016 at 12:29 AM
I think that it’s really weird that the SystemD advocates compare it to SysV init.
Why not compare it to Upstart, which was maturing at the time?

It is completely irrelevant whether or not shell scripts are all evil buggy programs, since the competitors to SystemD weren’t SysV init.
It was SysV init, upstart and openrc, and perhaps some others I’ve forgotten.

Please stop letting the SystemD people say that it’s better than SysV-init. It’s not their only competitor.

I like Upstart. I specifically liked the part where if something better came along it could replace Upstart.


William StinnerJune 30, 2017 at 4:10 AM
That’s amazing. many points to understand. thanks for sharing.binary options strategy


kbmDecember 3, 2017 at 7:39 AM
Ah yes, every item in the laundry list should start with a fancy “bias”, preferably a Latin word. That makes it sound more convincing. Ah sorry, this is an ad hominem non sequitur bias.


AnonymousDecember 8, 2017 at 5:43 AM
Just installed Nagios XI … which insists on RHEL 6/7 or CentOS 6/7.

It comes with xinetd (I immediately pined for the manageable times of old) and MariaDB still hangs off the “safe_mysqld” monitoring script.


100’000 lines written in C … WHY! … A square wheel, reinvented by barbarians camping in the library. Prolog with appropriate side-effect predicates would be the correct language to write this in. Or maybe one could just bang out the functionality in Python for sanity’s sake.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.