Pat13h ago
Larvitz  

Running Podman  in production for years now, and I don't miss the Docker daemon one bit.

I just published a deep dive on managing OCI containers the Unix way: daemonless, rootless, and natively integrated with systemd via Quadlets.

I cover:
- Real secrets management
- Auto-updates via systemd timers
- The Docker compatibility layer

This is the guide I wish I had when making the switch.

Read it here: https://blog.hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/

#Podman #Linux #DevOps #Systemd #Homelab #Sysadmin #Containers

Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility

An opinionated production-ops guide to Podman on Linux servers - why I prefer it over Docker, how Quadlets replace Compose files, and practical patterns from real deployments including secrets mana...

Larvitz Blog
Apr 5 at 1:23pmWeb
Show threadMarkus Lindenberg1d ago
@Larvitz I'm in a similar place with podman and I love that guide. You introduce everything I like about podman and quadlets in such a nice way, thank you!
Show threadmmu_man23h ago
@Larvitz been wondering about switching to either podman or libvirt for the plain LXC things I have on a server, because some other admins are not used to it and want GUI tools, but I suppose that means migrating… ?
Show threadLarvitz  23h ago
@mmu_man For GUIs, there's Podman Desktop (https://podman-desktop.io) and also the web-based Cockpit Client for Podman (https://github.com/cockpit-project/cockpit-podman)
Podman Desktop - Containers and Kubernetes | Podman Desktop

Podman Desktop - An open source graphical tool for developing on containers and Kubernetes

Show threadmmu_man23h ago
@Larvitz yeah but it won't keep containers as is I guess, so I won't be able to keep using lxc commands directly…
Show threadLarvitz  23h ago
@mmu_man Yeah for sure. LXC and Podman are different technologies.
Show threadoldsysops23h ago
@[email protected] #proxmox have a gui and can spin off lxc container no ?
Show threadmmu_man23h ago
@oldsysops not sure, I'll have to check that
Show threadshom23h ago
@Larvitz this is awesome, thanks for putting it together. I've been using podman for a few years and got started after generating the unit files from running containers. I wish I had a guide like this for getting started.
Quick question if you don't mind; I have a separate container running user and put the unit files in ~/.config/systemd/user/ instead. You suggest ~/.config/containers/systemd/ which seems to make sense as a path but I was hoping to understand the difference better. Could you please point me to a resource?
Show threadLarvitz  23h ago

@shom

~/.config/systemd/user/ is for systmd units (podman generate systemd). That was the old way to do it.

~/.config/containers/systemd/ is for Quadlet files, the modern way to describe containers declaratively:

https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html

Quadlets files are similar to Systemd units and describe a container with all it's attributes.

podman-systemd.unit — Podman documentation

Show threadshom23h ago
@Larvitz ahhhhh perfect, this made it click finally. I was just generically describing how to run an application (happened to be a container) and Quadlets use the unit file approach but describes the container itself (which I read in the unit file but didn't make the connection). Thanks so much!!
Show threadAndrew Furrow22h ago

@Larvitz another person of culture I see…/me tips hat

I’ve been operating with a mixture of quadlets and manual podman-compose containers for quite some time. I’ve found compatibility issues with some projects, but I decided those do not justify switching to docker. There’s also an annoying race condition with CNI coming up before networkmanager, but manual fix is easy enough for those times.

Great blog post! Thanks

Show threadLarvitz  21h ago
@andrew That blog article took me the longest of them all. A first draft had been lingering in my blog's git repo since November last year, and I went through numerous rewrites of various parts until I found them good enough. Today, I added the final paragraph about Ansible and decided to publish it before I end up waiting another 6 months 😂
Show threado/1MS\o ⌨️🐧 | #WeAreNatenom21h ago
@andrew @Larvitz I had a similar issue with NetworkManager and WLAN on my Laptop.
NetworkManager gets an IP-address after KDE login, but the quadlets starts directly after boot.
The dependency on podman-user-wait-network-online.service didn't helps.
So I added an "ExecStartPre" in the network quadlet, which helped for me.
```
[Service]
ExecStartPre=/bin/sh -c 'until hostname --ip-addresses | grep --quiet \"192.168.\"; do sleep 10; echo \"%n: Wait for IPv4 home network address ...\"; done'
```
Show threado/1MS\o ⌨️🐧 | #WeAreNatenom21h ago
@Larvitz Maybe you could add the hint, that automatic starting of rootless quadlets needs an user, where lingering is enabled.
It can be found at the examples.
https://docs.podman.io/en/latest/markdown/podman-system-service.1.html
```
loginctl enable-linger <USER>
```
podman-system-service — Podman documentation

Show threadwoodsb0217h ago
@Larvitz I love podman too, but recently I’ve been wondering about running rootless containers that aren’t tied to a specific host user. I’ve posted this as a discussion topic here - thoughts? https://github.com/containers/podman/discussions/28445
Rootless Podman with Ephemeral User · containers podman · Discussion #28445

I love the idea of rootless podman, running as quadlets, as I want to protect against container escape gaining root. That said, I also want to protect against container escape having access to my u...

GitHub
Show threadPat13h ago
@Larvitz Thanks for the article.
I never went beyond podman compose because I couldn't really find beginner-friendly examples on how to use Quadlets in production, so this is a great reference on how to get started.
Show threadIlkka Tengvall8h ago
@Larvitz very good starter pack for podman! I'd say all the normal operations described well and then some. Only thing more I use is exec for various debugging things or fixing something in storage. But if this is for docker users, there is no difference.
Show threadKayla Eilhart (en)8h ago
@Larvitz amazing guide, thanks for that! I started to write something similar last year, but never got far.

I quickly skimmed it and I have one practical tip though: It would be good to add how to create a service user, configure lingering, set XDG_RUNTIME_DIR and check if podman is working properly (including reboots, in some cases podman falls back to keep running data on file system which is persistent, not temporary, and refuses to start) - you'd never guess how I struggled with it a few years ago when I was starting with podman on a remote VM over ssh.

And I especially thank you for showing the traefik approach! I'm now manually configuring nginx, and it's quite a boring task.
Show threadrhempel8h ago

@Larvitz Now I know what I'll be reading tomorrow!

I made the switch about two years ago and use Podman for embedded systems development. It's much easier than spinning up a VM and combined with VSCodium makes a nice IDE that allows remote debugging.

Show threadpixelschubsi5h ago

@Larvitz So the post starts with how podman is great because it supports rootless containers and then goes at length to describe a full example setup of not using rootless containers.

I do struggle with a great setup of podman rootless in production setups. More documentation on that would be much appreciated - like having the full forgejo example just rootless.

Show threadLarvitz  5h ago

@pixelschubsi Even rootful, Podman has the advantage of cleanly forked, seperate processes and the absence of a central daemon. That's already a better security posture than Docker has and it simply feels "cleaner".

I did opt for rootful containers in my setup because of the networking. If I'd run containers in user-contexts, then my Traefik setup with automatic ingress routing based on labels wouldn't work so frictionless, because the networks are user-specific and countainers wouldn't be able to find each other.

I run some containers rootless ($HOME/.config/containers/systemd/*.container), but that's usually not web-applications.

Show threadpixelschubsi5h ago
@Larvitz
Yeah, that's what I meant with things get more complicated when running rootless. I have a hacky setup where I have one user per pod and each user/pod gets its own ip address (I happen to have a few available), but I don't really like it either, hence I was looking for alternatives.
Show threadArtemis5h ago
@Larvitz Interesting.
For me, the poor Compose support is one of the reasons I'm still sticking with Docker for development. Although, I think, the advantage of Compose files is simplicity (not having to commit to the whole systemd units system).
Show threadLarvitz  5h ago
@art_codesmith You can use the original docker-compose with Podman, by enabling the compatibility socket. I run one application in production like that, where the author only maintains a (rather complex) compose file.
Show threadArtemis5h ago
@Larvitz I might be wrong but, don't you lose basically all advantages of Podman by doing so? Having to give up daemonless and rootless.
Show threadLarvitz  5h ago
@art_codesmith Not really. Podman containers still spawn as their own forked processes, even when rootful. The socket isnt't a daemon, but just executing a fresh container context. In my opinion still a cleaner architecture than dockerd.
Show threadArtemis4h ago
@Larvitz Thank you. I might have to dig a bit further into this.
Show threadHans-Cees 🌳🌳🤢🦋🐈🐈🍋🍋🐝🐜5h ago
@Larvitz I have been planning tot migrate tot podman but life has many priorities
Will read
Show threadSlash909uk3h ago

@Larvitz nice!

I am halfway with podman; still have compose files launched from systemd units that I write myself - they are all basically identical except the home directory setting 🙂

I deliberately use compose start only, not run. I do not want restarts to be messing about pulling new images when I dont expect it!

Is there an equivalent to quadlets for alternative init tools? I would not want to lock myself into systemd right now 😁 seriousky looking at BSD.

Show threadLarvitz  3h ago

@Slash909uk I doin't know of any alternatives. Quadlets are transniently transformed into systemd units by a generator. That's all very systemd specific.

FreeBSD's Podman port ships with rc.d service scripts already. You enable them with:

sysrc podman_enable=YES
service podman start
sysrc podman_service_enable=YES
service podman_service start

Then, containers started with --restart=always will be automatically restarted after a host reboot. Podman's internal restart logic handles this, with the podman service acting as the supervisor. This is the closest equivalent to what quadlets do on Linux.

Show threadSlash909uk2h ago
@Larvitz thanks, good to know there is BSD support already 👍
Show threadLarvitz  2h ago

@Slash909uk

https://burningboard.net/@Larvitz/116357824557155636

🙂

Show threadtoniglandyl2h ago

@Larvitz Thanks for this great guide! I’m also a heavy user of
podman since years, and it's my number one solution for deploying services.

I had a question about the pod-in-pod deployment of forgejo / traefik,
giving access to the docker.socket allows thoses pods to create pods, but then
it can create privileged pods which mount the root volume of the host, right?
Even with the NoNewPrivileges arg?

Is there a way to control what a pod having access to the docker.socket can
create?

Show threadOliv'1h ago
@Larvitz thanks. I never took the time to explore Podman, I think I will do it in close future thanks to your nice article 👍