Earlier today, as I was putting the finishing touches on a new Gentoo install on my wife's new ROG Flow X13 (review of that incoming), I asked her how fancy of a sound setup she wants. She replied that simple is key. She does want Bluetooth, but probably doesn't need multiple user, auto-routing output, or other fancy things. Fair enough. While getting all the audio libraries set up, with correct USE flags, I spotted that Pipewire has grown a new set of flags to keep it from fighting with JACK or Pulseaudio. Then, while doing a bit of research related to the new flags, and setting up multi-user audio (because I know she'll eventually want to run Steam, which gets its own locked down account), I came across an old Reddit thread asking why people are so enthusiastic for Pipewire while Systemd got so much hate. After verifying it wasn't a simple troll post, I paused to consider. The replies were mostly not wrong, but they were overly simplified. They also focused heavily on the common claims about what Systemd got wrong, when the much more interesting line of inquiry is what Pipewire has done right. This can be seen in stark contrast when comparing the migration to systemd vs the migration to pipewire. Having lived through both, I speak from personal, professional, experience.

Let's back up a minute.

For the past 6 years or so, I have been running JACK2 on just about every system, since they either need the ability to handle audio from multiple users (logical and physical), or sanely route audio to the correct output device. For example, Pandora, IHeart, or other streaming music needs to go out the house speakers. Youtube or movies need to output audio to the desktop speakers. And Discord or Slack or Zoom or whatever need to go to the headset, dynamically switching between Bluetooth and wired for the headset output.

Note, this cannot be done simply via a set of Patchbay rules. First, because Patchbay can only create connections, and alsa clients must have a default connection. This can be done using a dummy default output, but that is not ideal. Second, multiple sources (all the website based ones) appear the same in Patchbay. How they should be routed depends on the origin domain, which must be passed out-of-band to the supervisor that figures out connections (via a simple browser extension).

My History With Pipewire

First Encounter

I first came across Pipewire in late 2018 or early 2019, when I first needed to record video from KDE Wayland. At the time, Pipewire was not ready yet, but had some hype as being for video what JACK2 was for audio. Seeing it wasn't ready, I used ffmpeg to capture the scanout buffer instead, even playing with capturing the TTY. I then picked up a USB capture card and moved on with life.

First Surprise

Fast forward a couple years: KDE Wayland matured; Pipewire matured; Firefox on Wayland matured, and then broke. I next took note of Pipewire when screen-sharing in Firefox, which worked on Wayland for viewing other Xwayland windows, broke. Digging into why, I discovered that the latest KDE meta package now had a screencast USE-flag, and several related packages now had a pipewire USE flag. None of that was enabled, so I toggled them on, and rebuilt. I could now share specific windows, workspaces, and so on. Around the same time, OBS gained Pipewire support and I could directly capture screens without needing the capture card, or janky ffmpeg plugins. Life with pipewire was great.

First Breakage

I'm not sure how much time passed, a couple KDE releases at least, before pipewire hit a snag. A victim of it's own success, in fact. See, while I'd been ignoring Pipewire, quite happy with my JACK2 install, it had been eating Pulseaudio's lunch. At this point, it had a pretty much complete re-implementation of Puleaudio's client and server libraries, and with the latest KDE update, started launching its own pulseaudio server on login. So when my supervisor daemon tried to launch pulseaudio, it found the address already in use. Easy enough to find where Pipewire read it should do that and disable it. And then disable it again after the next update when it moved where that was set. And then again a few months later when it again changed how and why it decided it should launch itself. At this point, I was somewhat annoyed with pipewire, because there was no good reason for a new version to ignore previous attempts to disable it, but it was easy enough to add "disable pipewire-pulse.conf" to my update checklist and move on.

Today

And that's where it sat for over a year: Pipewire was great for video; was quietly replacing Pulseaudio most places; still tried to step on my toes every few updates; and was once again mostly forgotten. Eventually, I saw it had a USE flag to disable the integrated sound server, so I toggled that off, content to never mess with it again. Until today, when I was asked for a simple setup. So I toggled the USE flags the other way, and rebuilt. I tried playing audio, and it just worked. So I tried playing a Pulseaudio-requiring game. And it just worked. So I said "mission accomplished". But what about when she wants to use her headphones? One of the major problems with Pulseaudio, for the 6 months I tried to use it back-in-the-day, was the inability to sanely route audio. For Pipewire, I'd heard mention of Wireplumber, but it is command line only, so not great UX. It would maybe work for recreating my audio supervisor, but she doesn't want that. Poking around a bit, I discovered that media-video/pipewire now owns /usr/lib64/libjack.so, so I thought I'd try a JACK client. Not having a jack-native program handy to try, I fired up qjackctl. And it just worked. Not only did it connect, it could see falkon playing audio, and it could see my hardware audio sources and sinks. Intrigued, I did some more investigation.

Pipewire as JACK3

Pipewire does not use /dev/shm for message and data passing, so it is unlikely to work correctly with anything shipping its own copy of libjack.so, but that is the case between JACK1 and JACK2 anyway. This means Pipewire could just as easily have been called JACK3, just another implementation of the JACK protocol and libraries, with (impressive) extensions. Any application portable between media-sound/jack-audio-connection-kit and media-sound/jack2 will also be portable to pipewire-jack.

As an historical note, JACK2 has the ability to pass arbitrary data, which at one time was planned to give JACK2 video routing abilities that are remarkably similar to what Pipewire now does. I have not yet looked to see if Pipewire video channels are exposed through this JACK interface, but they likely could easily be.

JACK2's utilities (alsa_out, alsa_in and so on), also work just fine. As does Python's jack.py. My audio supervisor will need some rules updates (to handle Pulseaudio client sources) but, otherwise, just works.

Pipewire as Pulseaudio2

Using pactl, I can see the same list of sources and sinks as qjackctl shows. This is huge. Instead of Pulseaudio, alsa, and Jack coexisting on the same system, but in a hierarchy with only one ultimately in charge (and the rest presenting unified streams that can only coarsely be managed), they are each now first-class citizens on the system. pulseaudio client library users, just like jack client library users, cannot tell the difference between pulseaudio and pipewire.

Systemd: Disruption Done Wrong

The Systemd Troubles

I've written and spoken before about the myriad issues with (especially early) Systemd. So in-lieu of rehashing all that here, below is a recreation of one of the many conversations I had regarding Systemd starting around 2011.

Technician: Did you hear RedHat is working on a replacement for SysV Init? It's called systemd.

Sysadmin: No. Not sure why someone would do that. Another case of "not invented here I suppose".

Technician: Well, modern systems have multiple logical cores, and SysV doesn't take advantage of that when booting. systemd does.

Sysadmin: You mean like SysV's "Makefile-style concurrent boot"? Been using that for years. They don't recommend or support it, since you can end up with hard to diagnose race conditions. I assume systemd will make sure to have a complete dependency graph before they enable that?

Technician: It will at least have the default packages sorted, probably.

Sysadmin: Well, as long as it's a predictable order, or I can turn it off for production, it'll be fine.

Technician: No, the order will depend on how fast tasks complete, (and the phases of the moon), but it would be better to devote the resources to finding and fixing races rather than adding the ability to disable the major reason to switch. If you don't want parallel boot, just stay with SysV.

Sysadmin: That's fair enough, we can give systemd a few years to work the kinks out and then consider switching.

Technician: Exactly. Unless you want to use Gnome3, they're making it mandatory in September.

Sysadmin: Of this year? As in after less than 2 years of development? As in next update? Why the &*@# wasn't this on Gnome's release notes as an upcoming change?

Technician: Well, RedHat is switching to it, it'll be fine.

Sysadmin: Alright, guess we better start rewriting our SysV init scripts.

Technician: You don't have to. That's the best part. systemd is fully backwards compatible with SysV. You won't even notice the switch.

Sysadmin: Oh, that shouldn't be so bad. Worst case we can hammer out a script to fully serialize the boot order and move on.

Technician: If you want to use the advanced features, including how to set up dependency chains, they're on the wiki.

Sysadmin: Not in the man page? How am I supposed to look it up if the network is what fails?

Technician: Well, the wiki is easier to collaborate. So it should be reliably up to date with fixes and how-tos.

Sysadmin: Fine. I can use a second machine to check if needed. Worst case I can trace whatever the problem is locally.

Technician: Make sure you have a copy of the source code, it's in C, so you won't be able to just read it.

Sysadmin: …

— One Week Later —

Sysadmin: Hey, I noticed that the wiki lists some advanced features that don't appear to actually be implemented in the source code.

Technician: Yeah, those should get added, eventually. The wiki is a reflection of how we wish systemd works, but it's all a work in progress.

Sysadmin: … Tracing problems locally it is…

— Two Weeks Later —

Sysadmin: Well, it's a good thing we tried to roll out the update on the test machine first. It successfully boots 1 time in 20. Not the end of the world, production only reboots once a month at most and I can baby it if needed. Still, not a great first look. Hm… what's going on here.

Nope, that's no good either.

— Two Hours Later —

Sysadmin: Looks like none of the changes I make in /etc/init.d are reflected… anywhere.

Technician: The wiki says it's backward compatible with SysV.

Sysadmin: That would be the same wiki that is flat wrong about advanced features? I guess you can mark "compatible with SysV" an advanced feature… Alright, Executive decision, we're dumping Gnome3. LMDE has mate available, and gets us the updated software stack we need.

— End —

While this is a re-creation and dramatization of my first experience with systemd, it very much captures the essence of that period. I must confess I took some liberties to paint systemd in a more favorable light. For one thing, there was no technician, at least not before I hit IRC asking questions. And the test machine doubled as my daily dev machine. The first indication I had that anything was amiss was the morning I came in and started updates, only for the system to explode. I fought it for a the rest of the week before I did indeed switch everything to LMDE, and never looked back.

Systemd Today

It's been 12 years since I spent a week fighting systemd. By the time Debian (and LMDE, specifically) switched to systemd, and dropped support for SysV, I had already made the leap to Gentoo in order to get support for AVX2. It wasn't until 2016 that I next encountered systemd, in the form of a debootstrap-based VM. With the benefit of a fully functional machine to investigate, and 5 years worth of internet posts on systemd, I only spent the better part of 1 day fighting init scripts before discovering service files. I don't think I knew about the still-wrong freedesktop.org systemd wiki. Instead, I mistakenly relied on debian's wiki, which to this day claims systemd is a drop-in replacement for SysV init.

Overall, my opinion of systemd hasn't really changed. From a technical side, it's fine, but has relatively little as-an-init-system to recommend it over SysV, and the neat ideas were all implemented in OpenRC.

Pipewire Vs Pulseaudio

No discussion of systemd's poor reception can be complete without considering pulseaudio and its poor reception. Pulseaudio, when created in 2004, was flat terrible. It would grab exclusive lock of your sound card, start automatically when a program wanted to use it, and then require killing it before any other program could play sound. It added significant audio latency, and heavy CPU overhead (at least if you were running Linux on low-end hardware). And, much like systemd, it aimed to solve a non-issue. ALSA already had support for multiple applications playing audio, via the plug:dsp system. So the only advantage you got from pulseaudio was per-application volume control: a nice feature, but hardly worth the complexity and CPU overhead. Also, pulseaudio itself was opinionated, stepping on many toes. I remember in 2008 or 2009 when I ended up with a pulseaudio-based audio stack (for a short time). I discovered that pulseaudio was playing audio out every interface. This meant plugging in headphones would not prevent audio coming out the normal speakers. When I looked into it, I was told this was expected behavior and I shouldn't have any reason to change it. Why is this relevant to systemd outside of pipewire? Because Lennart Poettering was the mastermind behind both pulseaudio and systemd. So when my office heard that the same genius behind pulseaudio was working on the replacement for SysV, and RedHat would be pushing it into gnome, we knew it would be nothing but trouble for years. The first time I happily used pulseaudio was in 2013, when it finally behaved itself as a plug:dsp client. Notable, because the last contribution by Poettering to pulseaudio was 16 May 2012.

Pipewire Vs Systemd

If you've read this far, I think this section will be obvious to you. But if you want a succinct summary, one that can be used elsewhere, it is as follows.

systemd started in late March 2010. 18 months later, in September 2011, or very shortly thereafter, it was automatically applied to any system running Gnome3. It was not ready for production use. It was unstable, prone to locking, badly documented, and generally alpha-quality. It was an opinionated project, which stepped on many toes, culminating in 2014 with Torvalds saying he would no longer accept patches from Kay Sievers, one of systemd's original creators. Why? Because systemd repurposed the kernel command line argument debug to spew a massive number of messages into the kernel log, which often crashed systems. Kay claimed this was expected behavior. To this day, it claims to be a drop-in replacement for SysV init, but is fundamentally incompatible with SysV. Also, its wiki is wrong.

By comparison, pipewire started in 2016. Debian only switched to pipewire by default in Debian 12, 7 years after pipewire was created. Gentoo with KDE Plasma started wanting to use pipewire for audio last year, 6 years on. As Pipewire is actually a drop-in replacement for Pulseaudio and JACK, you can easily switch between pipewire and pulseaudio or jack2; you can even have all three playing nicely on the same system.

Further, systemd "solved" problems that weren't. It allowed parallel booting, but SysV already had played around with parallel booting and planned to make it the default, eventually (in fact, MX Linux still uses SysV init, and has concurrency enabled by default. 14 years on and the dependency tree is complete). systemd also allowed advanced resource management through logind, but the advantages of this could only really be appreciated years later with containerization (docker) and Wayland. Gnome saw no tangible benefit from it at the time (and a fracturing of their userbase as people fled to Mate and Cinnamon).

Thanks to the herculean effort of Andy Wingo, logind was extracted from systemd, in the form of elogind. elogind's code is considerably smaller (about half the lines of code), but not lacking any meaningful functionality. Why was it created integrated into systemd? Because Poettering's mindset is integrated-by-default, even when that causes additional unneeded disruption.

The end result for Systemd is a tightly integrated userspace "micro kernel". Each component runs in its own address space, but you either use all or nothing. Pipewire is, continuing the kernel analogy, a modular "monolithic kernel". Everything runs in the same address space (well, except some of the pulseaudio stuff), but you can mix and match modules as you like. This led to Pipewire first taking over the video routing space, and then Pulseaudio's share, as more people turned on audio routing and found it acceptable. Those that encountered issues simply turned off the broken bits until they were ready.