Steve BellovinDec 21, 2024
A rant in two parts. This is part one…
The discussion earlier today about systemd replacing /var/log with some dedicated facility and a special command, journalctl, to query it impelled me to write up just what I think is wrong with a lot of Linux. Basically, they've given up on the Unix philosophy of small, composable, generally usable but simple tools in favor of a mass of large, specialized tools. "They've paved Paradise and put up a parking lot.”
Why specify a pager instead of just piping the output to a pager? Being able to search on a specific field is nice, but grep can do that. Maybe grep should be enhanced to say “apply the RE to fields m..n” (which is easy enough with awk anyway), and arguably it could take a file giving mappings of a fieldname to a field number. That's a generally useful tool; why limit it to systemd log files? Searching by time is nice, but it's nice in other contexts, e.g., the output of 'ls -l' on a large directory. The same goes for json-style output: why limit it to this context? (I won't even rant about why there has to be a single-line vs. multiline json option—that could also be a pair of simple, general commands.)
Too many Linux subsystems (or rather, their authors) have decided that they are the world and have to provide lots of functionality specific to that subsystem, rather than building general tools. Steve Jobs once said of Windows, “The only problem with Microsoft is that they just have no taste. They have absolutely no taste.” That's what's going on here.
Show threadHal PomeranzDec 22, 2024

@SteveBellovin I agree that including a pager is stupid and annoying—in bad taste indeed. And there’s lots of systemd scope creep that I abhor.

But syslog style logs suck. And you can tell this because everybody who does mass log retention stores the logs in fields in some sort of database schema—weeping tears of blood throughout the process as they try and deal with what is largely a free text log format. grep/awk can handle matching specific fields in specific types of log messages, but beyond time/date, hostname, and SOMETIMES process name things get real weird. Is there a PID? Is it in square brackets or something else? And beyond that, here be dragons. And systemd logs carry much more useful/searchable log metadata.

Search by date in syslog logs? Hahaha. Do you remember the classic syslog date format? Months are text abbreviations rather than numeric and there’s not even a year in there. Also I’m sure one second granularity was great in the days of one MIP machines. Not so much today.

I suspect the multiple output formats in journalctl is actually there as a backwards compatibility mechanism. I’d be happy if journalctl only produced JSON output because I know how to use jq. The text output modes are there for the awk/sed/grep crowd.

Frankly it is a defect in Unix-like operating systems that command output is free-form text. Life would be better if all Unix/Linux commands at least had the option to emit JSON or some structured output format. Instead we’ve gotten used to using awk/sed/grep, but that doesn’t mean that’s a good thing.

Show threadAdam Shostack  
@hal_pomeranz @SteveBellovin The problems with syslog mean it should be replaced, but there’s no reason for that to be tied to the startup system.
Dec 22, 2024 at 2:05amWeb
Show threadSteve BellovinDec 22, 2024
@adamshostack @hal_pomeranz Bingo.
Show threadHal PomeranzDec 22, 2024
@adamshostack @SteveBellovin https://mastodon.sdf.org/@argv_minus_one/113693718059152160
argv minus one (@[email protected])

@[email protected] If journald wasn't part of the systemd suite, then `systemctl status` wouldn't show the unit's last few log messages and the journal wouldn't record which systemd unit emitted each log message. Segregating things into separate projects makes this sort of two-way integration much harder, if not impossible, because it creates a dependency cycle. @[email protected] @[email protected]

Mastodon @ SDF
Show threadAaronDec 22, 2024
@hal_pomeranz I think a lot of the problems in the post you linked there could easily be sidestepped by giving each unit it's own log file. Dead simple and effective. @adamshostack @SteveBellovin
Show threadHal PomeranzDec 22, 2024

@aaron @adamshostack @SteveBellovin

Units don’t log. The process started by the unit file is what sends the logs. And those logs are identified by a shared syslog facility number like LOG_DAEMON or LOG_AUTHPRIV in the case of things like sshd, sudo, PAM, etc.

So you can’t just wave a magic wand and say “give each unit its own log file”. You can disagree architecturally with systemd’s decision to interpose itself in the traditional logging channel, but you don’t get being able to explicitly track individual log sources for free.

Show threadAaronDec 22, 2024
@hal_pomeranz Per the linked post: "the journal wouldn't record which systemd unit emitted each log message."

If it's the unit itself emitting the log, route each to a unique per unit file. Solaris SMF does this as an example; said logs are distinct from the application logs collected via syslog.

( Also, it isn't necessary to explain the syslog protocol to me, I implemented both RFC 3164 and 5424 parsers in Lua some years ago. )

@adamshostack @SteveBellovin
Show threadRaven667Dec 22, 2024
@adamshostack @hal_pomeranz @SteveBellovin the use cases that put it over the edge are early collection and mostly having a handoff for stdout/stderr from daemons that previously was logged to /dev/null. It doesn't have to be "journal" with the hashed and indexed file format, but that has its own advantages. What likely motivated the devs to tackle logging is that all programs will try to log so logging needs to be one of the first started services and init is the only thing recving stdout/err