A rant in two parts. This is part one…
The discussion earlier today about systemd replacing /var/log with some dedicated facility and a special command, journalctl, to query it impelled me to write up just what I think is wrong with a lot of Linux. Basically, they've given up on the Unix philosophy of small, composable, generally usable but simple tools in favor of a mass of large, specialized tools. "They've paved Paradise and put up a parking lot.”
Why specify a pager instead of just piping the output to a pager? Being able to search on a specific field is nice, but grep can do that. Maybe grep should be enhanced to say “apply the RE to fields m..n” (which is easy enough with awk anyway), and arguably it could take a file giving mappings of a fieldname to a field number. That's a generally useful tool; why limit it to systemd log files? Searching by time is nice, but it's nice in other contexts, e.g., the output of 'ls -l' on a large directory. The same goes for json-style output: why limit it to this context? (I won't even rant about why there has to be a single-line vs. multiline json option—that could also be a pair of simple, general commands.)
Too many Linux subsystems (or rather, their authors) have decided that they are the world and have to provide lots of functionality specific to that subsystem, rather than building general tools. Steve Jobs once said of Windows, “The only problem with Microsoft is that they just have no taste. They have absolutely no taste.” That's what's going on here.

@SteveBellovin The job of the sysadmin is being industrialized. That is what is causing the discomfort.

The single operator having machines named after characters from Tolkien lore is disappearing or more precisely: becoming an edge case.

The „normal“ system is setup automagically via toolchain. The typical system never sees an operator logging in via ssh or console during its complete lifecycle. In modern ops, needing to do a thing by hand is worth a bug report by itself.

I am mot talking hyperscalers here, but the ops of small shops with a few thousand cores.

Yes, I have my own set of a dozen handcrafted machines (mostly VMs), but I am aware thar Linux is no longer primarily developed for my use case.

My (and probably your) use case make up only a small part of the Linux world and may not be the target for the development any more.

@masek @SteveBellovin
Superstition.

The problem is that a feature-creeping and landgrabbing megablob is harder to control, debug, isolate and secure - and especially less resilient - than well defined components working in concert.

@vampirdaddy @masek @SteveBellovin You can still use libc DNS, NTPD / chrony, libvirt, custom scripts aplenty with systemd. You're not obligated to use the extra tools that are **offered**.

I am still yet to find am equivalent to `systemctl list-units --failed` in init-land.

What about `journalctl -b -p3` - can you pull up EVERY error from ANY service (and the kernel) since the system booted? Does the command work just as well on SLES, RHEL, Debian?

As a sysadmin by trade, it's a godsend.

@nroach44 @masek @SteveBellovin

> EVERY error from ANY service

_Only_, if the service is actually using journald, or is run from a .service or .socket definition.

If its running its own wrapper and/or logging (as proprietary business monstrosities like to do), you're back to hunting for and within the logfiles.

Syslog has/had the same problem: it only works as long as stuff stays within the ecosystem.

@vampirdaddy @masek @SteveBellovin The problem with how syslog tends to be configured is that there is /var/log/
acpid
audit/*
auth
chown
dmesg
lastlog
mail
messages
secure
spooler
sudo

So on top of your weird app not logging system errors to syslog, you now have an added ~11 log files to sift through.

At least as far as I'm concerned, the app having it's own logs isn't a problem because I'm not responsible for the app, just the OS. Syslog is for OS things, not app things.

@nroach44 @masek @SteveBellovin

Most of the system logs in /var/log actually are just excerpts. Kind of early/primitive SIEM.

In /etc/syslog.conf (or config files of siblings like syslog-ng or rsyslog) you can find the patterns how the syslog will be split up into files.

It’s just kinda statically configured "journald -u …" that’s writing into separate files for selected processes or systems.

1/2

@nroach44 @masek @SteveBellovin

2/2

You usually can find the complete log in /var/log/syslog or /var/log/messages (the naming tradition vary between Unix/Linux flavours)

@vampirdaddy @masek @SteveBellovin That is what I thought up until recently, I've seen some (either by default or by an individual's hand I can't say) where not all of the ssh logs go into messages, some of it goes to auth/secure etc.

At least with the sysd servers it's far less common to see that kind of "tuning" done and "all services log to the journal" is a far more reliable assumption

@nroach44 @masek @SteveBellovin

Look into the syslog conf file. Some philosophies/distros prefer to isolate auth messages away from "lesser" admins, some drop info/debug level to keep log volume small.

When coming from journald think multiple journalctl -u XX -p N jobs piped into different files at /var/log/ . If the historic / default parameters are not to your liking, just change them.