A rant in two parts. This is part one…
The discussion earlier today about systemd replacing /var/log with some dedicated facility and a special command, journalctl, to query it impelled me to write up just what I think is wrong with a lot of Linux. Basically, they've given up on the Unix philosophy of small, composable, generally usable but simple tools in favor of a mass of large, specialized tools. "They've paved Paradise and put up a parking lot.”
Why specify a pager instead of just piping the output to a pager? Being able to search on a specific field is nice, but grep can do that. Maybe grep should be enhanced to say “apply the RE to fields m..n” (which is easy enough with awk anyway), and arguably it could take a file giving mappings of a fieldname to a field number. That's a generally useful tool; why limit it to systemd log files? Searching by time is nice, but it's nice in other contexts, e.g., the output of 'ls -l' on a large directory. The same goes for json-style output: why limit it to this context? (I won't even rant about why there has to be a single-line vs. multiline json option—that could also be a pair of simple, general commands.)
Too many Linux subsystems (or rather, their authors) have decided that they are the world and have to provide lots of functionality specific to that subsystem, rather than building general tools. Steve Jobs once said of Windows, “The only problem with Microsoft is that they just have no taste. They have absolutely no taste.” That's what's going on here.
@SteveBellovin I broadly agree but that said I've yet to install a Linux system (at least that I administer) that actually uses systemd, even if that means staying with a less mainstream distro. Which suits me just fine. If that becomes impossible it'll be off to BSD-land for good.
@eaterofsnacks @SteveBellovin I'm glad there are so many Linux users/admins who prefer to avoid systemd. However, as a BSD-first user who keeps tabs on the efforts to port & maintain Wayland on #OpenBSD, I worry for the longevity of BSDs if they're just "Plan B" for when systemd becomes required. Wayland is just one example, but clearly a core one, where there are so many architectural decisions & Linux subsytems required that cross-platform is coming to just mean cross-Linux-distro. Concerning.

@SteveBellovin but one cannot deny the role systemd played in streamlining and securing the whole thing.

For applications it is hell to implement every specific flavor for all distros. Instead, you either target something widely used (systemd, no matter if we like it or not) or you do some custom stuff (e.g. put everything in ~ ...) and annoy your users.

and aren't the specialized tools closer to the philosophy of "one tool for one job"?

@brahms @SteveBellovin I don’t credit systemd for this.

Rather I think systemd is / was just the most popular init system when the security enhancements came along.

@drscriptt @brahms Systemd is a symptom, not the problem.
@drscriptt @brahms @SteveBellovin I think putting so many security controls in one place, where you can read the man page and enumerate them, helps promote adoption of the available features more than of you needed to use specialized custom tools for each one

@SteveBellovin @mms correct on all points. every single claim that systemd proponents make to its benefit is already an existing tool or command or method which has existed in Unix and BSDs (and to a lesser extent in pre-systemd Linux distros).

having been professionally involved with all of the major non-windows OS's for over a decade longer than systemd has existed, I can assuredly say that far too many of those claims exist simply due to the claimants vast inexperience with operating system design and engineering. they don't even know, and seemingly don't care, that we never needed systemd, and the vast majority never wanted it.

it's no coincidence that systemd was forced into the Linux communities at the same time that MSFT stopped trying to crush it from beyond -- instead that was the very same era in which they began their assaults on open source via "embrace extend extinguish".

@winterschon @SteveBellovin @mms and I'm sure it's totally not a coincidence that Lennart Pottering is a Microsoft employee now.
@nivex @winterschon @SteveBellovin @mms I had money on Microsoft hiring him. I won that bet but lost a couple of friends over it.

@winterschon @SteveBellovin @mms

> Systemd was forced

I call BS on that, systemd was *offered* to the Linux distro maintainer community and the current generation of UNIX implemetors chose it because it better aligns with their current style and values, which aren't the same style and values as the UNIX implemetors of the 1970s and 1980s, or the 90s and 2000s.

There is a difference between the technical tradeoffs between system designs and the subjective *style*

@SteveBellovin I agree that including a pager is stupid and annoying—in bad taste indeed. And there’s lots of systemd scope creep that I abhor.

But syslog style logs suck. And you can tell this because everybody who does mass log retention stores the logs in fields in some sort of database schema—weeping tears of blood throughout the process as they try and deal with what is largely a free text log format. grep/awk can handle matching specific fields in specific types of log messages, but beyond time/date, hostname, and SOMETIMES process name things get real weird. Is there a PID? Is it in square brackets or something else? And beyond that, here be dragons. And systemd logs carry much more useful/searchable log metadata.

Search by date in syslog logs? Hahaha. Do you remember the classic syslog date format? Months are text abbreviations rather than numeric and there’s not even a year in there. Also I’m sure one second granularity was great in the days of one MIP machines. Not so much today.

I suspect the multiple output formats in journalctl is actually there as a backwards compatibility mechanism. I’d be happy if journalctl only produced JSON output because I know how to use jq. The text output modes are there for the awk/sed/grep crowd.

Frankly it is a defect in Unix-like operating systems that command output is free-form text. Life would be better if all Unix/Linux commands at least had the option to emit JSON or some structured output format. Instead we’ve gotten used to using awk/sed/grep, but that doesn’t mean that’s a good thing.

@hal_pomeranz @SteveBellovin The problems with syslog mean it should be replaced, but there’s no reason for that to be tied to the startup system.
argv minus one (@[email protected])

@[email protected] If journald wasn't part of the systemd suite, then `systemctl status` wouldn't show the unit's last few log messages and the journal wouldn't record which systemd unit emitted each log message. Segregating things into separate projects makes this sort of two-way integration much harder, if not impossible, because it creates a dependency cycle. @[email protected] @[email protected]

Mastodon @ SDF
@hal_pomeranz I think a lot of the problems in the post you linked there could easily be sidestepped by giving each unit it's own log file. Dead simple and effective. @adamshostack @SteveBellovin

@aaron @adamshostack @SteveBellovin

Units don’t log. The process started by the unit file is what sends the logs. And those logs are identified by a shared syslog facility number like LOG_DAEMON or LOG_AUTHPRIV in the case of things like sshd, sudo, PAM, etc.

So you can’t just wave a magic wand and say “give each unit its own log file”. You can disagree architecturally with systemd’s decision to interpose itself in the traditional logging channel, but you don’t get being able to explicitly track individual log sources for free.

@hal_pomeranz Per the linked post: "the journal wouldn't record which systemd unit emitted each log message."

If it's the unit itself emitting the log, route each to a unique per unit file. Solaris SMF does this as an example; said logs are distinct from the application logs collected via syslog.

( Also, it isn't necessary to explain the syslog protocol to me, I implemented both RFC 3164 and 5424 parsers in Lua some years ago. )

@adamshostack @SteveBellovin
@adamshostack @hal_pomeranz @SteveBellovin the use cases that put it over the edge are early collection and mostly having a handoff for stdout/stderr from daemons that previously was logged to /dev/null. It doesn't have to be "journal" with the hashed and indexed file format, but that has its own advantages. What likely motivated the devs to tackle logging is that all programs will try to log so logging needs to be one of the first started services and init is the only thing recving stdout/err
@hal_pomeranz @SteveBellovin Also on the json output, check out jc, which adds that capability to lots of things, unix style, by adding a command.
@adamshostack @SteveBellovin You’re not seriously arguing that the miscellaneous text output formats of Unix-like commands is a feature, are you? jc is a band-aid on 50+ years of benign neglect.
@hal_pomeranz @SteveBellovin I’m arguing that text is a low common denominator that enables a great deal of flexibility and experimentation. Some of that experimentation deserves to be taken out and shot, for sure. But the fact that you can build jc is a feature and Steve can use awk is a feature.

@SteveBellovin The Steve Jobs quote is spot on. The Linux 'user land' developers do not want "Unix" architecture for free they want "Windows" architecture for free. They learned to program on windows, they grew up using windows, they "get" windows, they "want" windows but without the gatekeeping of Microsoft.

Once you understand *that*, you understand systemd and all of the other tasteless things that have become part of distributions.

@ChuckMcManis @SteveBellovin I would rather say, they're completing the work Sun started with Solaris. systemd isn't a clone of Windows' services, systemd is a clone of Solaris' svcs.
@hennichodernich @ChuckMcManis @SteveBellovin No it isn't. SMF was a dependency-based init system, nothing else. It didn't subsume syslog or DNS or NTP. It was a tool (albeit on with flaws), not an attempt to remake the universe.
@ChuckMcManis @SteveBellovin
Ist ein Interessanter Punkt und ich sehe mich auch dazu gezwungen irgendwann mit Systemd auseinander zu setzen. Oder direkt auf ein anderes init system zu wechseln...
@ChuckMcManis @SteveBellovin Whatever the reasoning it's half-arsed and stupid. They should invest the time in learning the benefits of what we had, not dragging them away from the rest of us by weight of numbers. The #MickeySoft steam roller continues on. And it's shit. We've lost all the simplicity and granularity we once enjoyed.

@SteveBellovin Yes. there's even an interesting tool (jc) that jsons up unix command output if you like using that toolchain. Just today I was struggling with how to pipe git output into a shell command...

https://blog.kellybrazil.com/category/jc-news/

JC News Archives - Brazil's Blog

Latest news on the JC JSON Conversion tool project

Brazil's Blog
@SteveBellovin Re: JSON, back in the days when XML was The Hot Thing, Juniper needed a way for existing normal FreeBSD command-line tools to generate XML output, so they created a library (libxo, for Xml Output) which is like "structured printf", and added standard command-line options to invoke it. Then when JSON became The New Hot Thing, it was just one change in the library to add it as a new format. FreeBSD has shipped this for eight years now, with additional utilities converted over time.
@SteveBellovin do you want PowerShell? Because this is how you get PowerShell

@SteveBellovin the most important thing about this is "taste" not technical tradeoffs. The new generation of developers have different opinions on how a "good" system behaves based on preference and the different environment they operate in. "They paved over paradise" is probably true but it's done because the paved space is more useful now to the operational needs of business which uses compute to extract money.

I don't think systemd is _wrong_ it's just aligned to current needs and style

@SteveBellovin
When I connect a monitor it takes this system about a second to wake up.

When I touch the mouse or the keyboard it takes about 20 seconds to stir.

What on earth is it doing?

@zl2tod If you could access the logs you would have known. @SteveBellovin

@SteveBellovin The job of the sysadmin is being industrialized. That is what is causing the discomfort.

The single operator having machines named after characters from Tolkien lore is disappearing or more precisely: becoming an edge case.

The „normal“ system is setup automagically via toolchain. The typical system never sees an operator logging in via ssh or console during its complete lifecycle. In modern ops, needing to do a thing by hand is worth a bug report by itself.

I am mot talking hyperscalers here, but the ops of small shops with a few thousand cores.

Yes, I have my own set of a dozen handcrafted machines (mostly VMs), but I am aware thar Linux is no longer primarily developed for my use case.

My (and probably your) use case make up only a small part of the Linux world and may not be the target for the development any more.

@masek @SteveBellovin even in that environment I prefer Journald over /var/log. Journald never filled the filesystem with logs. Got paged a lot in the past for missing or wrong log rotate though.

And it makes writing services so much easier. Just write your logs to STDOUT/STDERR. Just as in your local dev setup. No special handlers for releasing files if log rotate wants to rotate etc.

@hikhvar @masek @SteveBellovin Not sure what the root cause is, but I've had multiple systems fill a partition to 100% with an uncontrolled journal. Maybe it was a bug that was later fixed -- but it was a failure mode so fundamental to the use case that it only added to my general distrust (in addition to the multiple "no way to troubleshoot why the system won't boot because of a dependency loop in systemd" failure modes (which has also happened to me multiple times)).

@hikhvar @masek @SteveBellovin
Oh yes, journald can fill the disk, e.g. during error floods. And if it does, systemd stops the system hard. By design. Then you _need_ to mount your system disk(s) to an externally booted system and clean up from there, instead of doing that from inside the running system (with a degraded logging), restart syslog and you‘re done.

And don’t get me started with missing/buffered log forwarding…

@hikhvar @masek @SteveBellovin
Regarding easier service programming, just usr STDIN/STDOUT/STDERR with systemd .socket() ?

That’s what inetd was written for 40ish years ago.

@vampirdaddy @masek @SteveBellovin all the applications I wrote professionally in the last 7 years only write to STDOUT/STDERR file descriptors. I don't care which init system starts those processes. Worked so far very well with systemd/Journald, docker, nomad, Kubernetes, AWS Serverless stuff.

Only the old unit systems on Linux required the operator to build some wrappers in their init scripts.

@hikhvar @masek @SteveBellovin

The old Linux needed some wrapper scripts,
when on systemd you need .service and .socket files,
on Docker you need the Dockerfile and maybe a few mappers,
on AWS Lambda the configuration files

Automation is about HOW you use your tools, not so much about WHICH BRAND.
Unless of course the tools have flaws…

@masek @SteveBellovin Come to the OpenBSD side, we have pufferfish. ;-)

More seriously... i very much agree with your point on the slide between cattle and pets. I fear, though, that this loss of the care component will really hurt us long-term, together with a loss of end-to-end understanding due to abstractions taking over.

See also my / Doris' 13 propositions and the lack of care: https://doing-stupid-things.as59645.net/burning/world/resillience/2022/06/27/propositions-part-1.html / https://ripe85.ripe.net/archives/video/877/

13 Propositions on an Internet for a Burning World (Part 1)

OpenBSD version: Not relevant here... Arch: ? NSFP: Well...

Doing stupid things (with packets and OpenBSD)
@tfiebig @masek @SteveBellovin Tempora mutantur et nos mutamur in illis
@masek @SteveBellovin Die Frage ist eher, ob wir nicht leichtfertig die faehigkeit aufgeben, uns zu aendern.

@masek @SteveBellovin
Superstition.

The problem is that a feature-creeping and landgrabbing megablob is harder to control, debug, isolate and secure - and especially less resilient - than well defined components working in concert.

@vampirdaddy @masek @SteveBellovin You can still use libc DNS, NTPD / chrony, libvirt, custom scripts aplenty with systemd. You're not obligated to use the extra tools that are **offered**.

I am still yet to find am equivalent to `systemctl list-units --failed` in init-land.

What about `journalctl -b -p3` - can you pull up EVERY error from ANY service (and the kernel) since the system booted? Does the command work just as well on SLES, RHEL, Debian?

As a sysadmin by trade, it's a godsend.

@nroach44 @masek @SteveBellovin

> EVERY error from ANY service

_Only_, if the service is actually using journald, or is run from a .service or .socket definition.

If its running its own wrapper and/or logging (as proprietary business monstrosities like to do), you're back to hunting for and within the logfiles.

Syslog has/had the same problem: it only works as long as stuff stays within the ecosystem.

@vampirdaddy @masek @SteveBellovin The problem with how syslog tends to be configured is that there is /var/log/
acpid
audit/*
auth
chown
dmesg
lastlog
mail
messages
secure
spooler
sudo

So on top of your weird app not logging system errors to syslog, you now have an added ~11 log files to sift through.

At least as far as I'm concerned, the app having it's own logs isn't a problem because I'm not responsible for the app, just the OS. Syslog is for OS things, not app things.

@nroach44 @masek @SteveBellovin

Most of the system logs in /var/log actually are just excerpts. Kind of early/primitive SIEM.

In /etc/syslog.conf (or config files of siblings like syslog-ng or rsyslog) you can find the patterns how the syslog will be split up into files.

It’s just kinda statically configured "journald -u …" that’s writing into separate files for selected processes or systems.

1/2

@nroach44 @masek @SteveBellovin

2/2

You usually can find the complete log in /var/log/syslog or /var/log/messages (the naming tradition vary between Unix/Linux flavours)

@vampirdaddy @masek @SteveBellovin That is what I thought up until recently, I've seen some (either by default or by an individual's hand I can't say) where not all of the ssh logs go into messages, some of it goes to auth/secure etc.

At least with the sysd servers it's far less common to see that kind of "tuning" done and "all services log to the journal" is a far more reliable assumption

@nroach44 @masek @SteveBellovin

Look into the syslog conf file. Some philosophies/distros prefer to isolate auth messages away from "lesser" admins, some drop info/debug level to keep log volume small.

When coming from journald think multiple journalctl -u XX -p N jobs piped into different files at /var/log/ . If the historic / default parameters are not to your liking, just change them.

@masek @SteveBellovin

But to fair - you need to understand the concept and what you are glueing together with automation.

I have seen more junior people using blindly ansible roles not understanding what is done by them …

That can work but more successful when knowing what should happen and why …

@jalogisch @masek @SteveBellovin Yeah, I have seen the same in my youth with init scripts. I have rewritten their scripts for several vendors whose init scripts caused my eyes bleed.