Mastodawn

ok systemd experts, how the heck do I get the output of a oneshot container with LogDriver=passthrough into both journald _and_ a regular log file that's rotated daily? Similar use case as described here https://github.com/systemd/systemd/issues/38684#issuecomment-3221956220

I need simple log files for long term archiving

In the old world I'd just do "./program | tee -a /logs/$(date +%Y%m%d.log)"

Rotate StandardOutput= file when rotating logs · Issue #38684 · systemd/systemd

Component journalctl, systemd Description The most efficient way to write daemon logs into a specific file is using: [Service] StandardOutput=append:/var/log/xxx.log StandardError=append:/var/log/x...

GitHub

Job Snijders 4d ago

I'm not the only one struggling with what at first glance would appear to be a simple requirement... https://serverfault.com/questions/1103320/systemd-standardoutput-to-a-file-and-to-the-journal

systemd StandardOutput to a file _and_ to the journal?

I am regularly doing a long running (~5 day) data processing programme. I'm using Ubuntu and running the command with a systemd transient task via systemd-run --unit data_import /path/to/my-script....

Server Fault

@job Yea one would think so. Or indeed pass it true rsyslog for it, or have a from job that dumps them daily orso.

Job Snijders 4d ago

@evey I find it somewhat unsatisfying to have to install another syslog daemon alongside journald … what is the purpose of journald then?

@job The idea of journald is to store it all in there own "efficent" format. And keep stuff in memory to only periodicly flush stuff to disk (if enabled). And offer better filtering methods to get the stuff out. Plus allowing also json and other log formats (Orso i personally don't like the design it breaks to often).

Job Snijders 4d ago

@evey store “efficiently” but it’s circular buffers and I wanna store forever… so it’s not durable materialisation it seems.

@job You can just configure journalctl to store it persistant to disk, in there own format. Altho that also has its issues to some degree.

With Storage=persistent

But agreed it all is quite a mess and really feels people with little operational experience designed it tbh.

@evey @job "But agreed it all is quite a mess and really feels people with little operational experience designed it tbh."

Perfect description of journald. Designed by developers who thought they know what operators need, but never considered asking an actual operator. Oh and they clearly knew the Windows event log and considered it "good".

Some of the ideas are really good, but it overall feels like a knife that is all blade no handle.

Job Snijders 3d ago

I solved it by adding the following to the .container file:

ExecStopPost=/bin/sh -c '/usr/bin/journalctl -u rrdp-sync.service --invocation 0 >> /data/logs/$$(date +%%Y%%m%%d).log'

Thanks @jaseg for the hint

@job I would install rsyslog. systemd-journald will forward logs to it and then you can process output files the traditional way e.g. with logrotate.

fedops 💙💛4d ago

@job could try this:

[Unit]
Description=Container Task

[Service]
Type=oneshot
ExecStart=/usr/bin/podman run --rm --log-driver=passthrough my-image
StandardOutput=append:/var/log/my-container/task.log
StandardError=inherit

fedops 💙💛4d ago

@job otherwise you can also tee into systemd-cat which adds its input to the journal. It has quite a few useful options.

@job you could put something in ExecStopPost that dumps the output from journald into a plain text file after the container is done executing

Job Snijders 4d ago

@jaseg thank you, that might be workable. Next challenge: figuring out how to extract only the last run :-)

@job probably —invocation -1

@job first you'll need some candles, a pentagram drawn with the blood of a virgin goat, a dagger dripping with the blood and tears of the innocent, and 7 people guilty of each of the cardinal sins to sacrifice

@job I'm by no means an expert. A few ideas in the order of personal preference:

Daily cronjob that dumps last day's journal entries for the unit into a file.

Have systemd unit run a .sh file that does the "program | tee" thing.

Use ForwardToSyslog=yes in journald.conf and configure syslog+logrotate/etc. to do the right thing