After the cacaphony erupted over my columns in August and September, I thought I might take a break from the systemd wars for a while, but the battles I’ve seen in forums across the Internet seem to be escalating. As I predicted, the release of RHEL 7 with systemd as the only option for system and process management has reignited the debate.
I won't rehash all the pros and cons of systemd, nor will I argue whether binary logging is a good idea (it isn’t), but I have to share my observations in watching these discussions take shape, along with the positions and comments on both sides of the argument. They’re rather telling, and they highlight what I was discussing when I posited that we could really use a Linux distro that did not maintain any desktop packages.
In discussions around the Web in the past few months, I've seen an overwhelming level of support of systemd from Linux users who run Linux on their laptops and maybe a VPS or home server. I've also seen a large backlash against systemd from Linux system administrators who are responsible for dozens, hundreds, or thousands of Linux servers, physical and virtual. I noticed this divide when I wrote that first column, but it’s even clearer now. The release of RHEL 7 has brought the reality of systemd to a significant number of admins whose mantra is stability over all else and who perhaps had not waded into the choppier waters of Fedora or Debian unstable to work with systemd before it arrived in RHEL.
Systemd did not come to them as a surprise, but its reality as the only option in RHEL 7 and CentOS 7 has brought the message home, and veteran Linux server admins aren’t thrilled. They aren't casual users, and they aren't developers. This is their bread and butter.
The uselessd site has an interesting essay on systemd. The piece makes many good points, alongside a bizarre highlight of Lennart Poettering's complete misunderstanding of the "everything in Unix is a file or a pipe" concept, followed by Poettering's inexplicable declaration that his printer isn't a file. But the author misses one part of the debate completely.
The author (apparently choosing to remain anonymous) describes the two sides of the systemd debate as those who are “part of the modern Desktop Linux bandwagon” (for systemd) and those who “typically hail from more niche distributions like Slackware, Gentoo, Crux, and others” (against systemd). This completely overlooks the huge number admins that are neither -- they are the ones using RHEL, CentOS, Debian, and Ubuntu as server platforms for all kinds of production services, who don’t care a whit for desktop support, and who are not fans of systemd.
These are the people who don’t care how fast their servers boot because, as I’ve stated over and over, the POST on modern physical servers takes far longer than the OS boot. The fact that VMs may boot a second or two faster has no measurable impact. Of course, systemd isn't just about boot times, but many of the issues highlighted in discussions about systemd -- such as how it handles hotplug services and service management -- don’t matter for servers.
We’re talking about either big physical boxes that will run only one or two core services, or VMs that will run one or two small services. Systemd provides nothing that positively impacts these use cases in significant ways. Systemd thus appears to be a massive, fundamental change to core Linux administration for no perceivable gain. This is what is meant when people say that systemd was an answer to a question nobody asked.
Except some people did ask that question: desktop users. Here the uselessd piece does well to identify the major proponents of systemd.
Though some Linux server admins say they need systemd on their servers to prevent race conditions and service dependencies, I’ve never run into more than a few instances over decades of *nix server work (and thousands upon thousands of servers running sysvinit or Upstart) where a boot-time race condition or priority was a problem. It just isn’t a big deal. What do most Linux servers run? A mail service, a Web service, a database service, or a Memcached service. On those systems, we only care that they are available and log properly -- and systemd changes that too.
You might say that because systemd does process monitoring and respawns services it's therefore helpful on a server, but I don’t agree. If I have a server and a service keeps crashing, I don’t want systemd to keep respawning it; I want my monitoring system to tell me the server is unavailable. Then I will go and find the root of the problem and fix it. Generally, merely restarting the service won't fix the underlying problem. On a desktop or laptop, hotplug events and other peripheral changes can cause other issues that may in fact benefit from a service respawn, but those are desktops. We’re talking about servers here.
We have more capability now than ever to segment and silo services with little to no negative performance impact. This means we can build out service infrastructures with lots of little pieces that can be moved around as needed. Even infrastructures that aren’t very advanced use virtualization to split up services into separate VMs. We are beyond the days when we had large servers that ran an entire application stack and needed boot-time dependency tending and other considerations to be made -- and back in those days, we did it with sysvinit, not systemd.
When I read all about how “greybeards” don’t get systemd and won't let go of the past, and how terrible everything was before systemd because you had to know shell scripting and cron, it says more to me about the person making those statements than about the subject of discussion. These are desktop users with desktop mindsets and desktop experiences. They have every right to shape their desktop desires into their desktop distribution and to refuse to learn Bash or to know what run levels are.
It’s this mindset that came up with forkfedora.org, an unintentionally amusing spoof of debianfork.org. Server admins don’t want to fork Fedora. There’s no point, because it’s never been a server-centric distribution. Forking Debian to allow for a choice of init systems, however, makes sense from a server perspective. The sarcastic statement, “Maybe we’re not veteran enough,” is absolutely true. They literally don't understand the difference.
Those of us who build server infrastructures on Linux couldn’t care less about systemd and desktop concerns -- but don’t let your desktop desires cripple the server world. Unfortunately, this is where Ubuntu, Debian, and Red Hat appear to be heading. Perhaps now what I was discussing about server-only Linux distributions and a resurgence of FreeBSD may make a little more sense to those who don’t live here.
