After quite some time, I finally have all the pieces in place. Over the last 30 minutes, I’ve set up one of my servers from scratch. Here are some key changes:
- Reverse Proxy: Nginx with Modsecurity (WAF)
- Container Isolation: Every container runs in a seperate linux user
- Podman Quadlet: I rewrote all my compose stacks into quadlet files - now all containers are starting probably after reboot 🥳
- Grafana: Grafana's configuration is no managed by Opentofu which provitions at the moment the datasources (Grafana Loki and Prometheus) as well as the dashboards.
- Server hardening: Improved ssh configuration, firewall, permissions in general on this host
- Ansible: Everything is powered by ansible
- Certbot: Use wildcard certificates for my domains / subdomains for easier renew process
- Backups: All those services have proper backups configured which are timed with systemd timer and are replicated into my local homelab.
- Services that are running at the moment
- Grafana
- Prometheus
- Grafana Loki
- Grafana Alloy
- GitLab Runner
- some other services that I wanna migrate to this server

#homelab #sysadmin #linux #ansible #automation #devsecops #selfhosting #declarative #gitops #monitoring

@niclas how did you manage to run each container as a separate user? I have a few containers in my stack that have to run as uid 1000 (like homepage for example), what do you do in this situation?

@hotsause If I take Grafana as an example:
1. Create service account user "grafana"
2. su grafana
3. Copy the podman quadlet file to the folder"~/.config/containers/systemd" so the podman quadlet runs rootless as the user "grafana"
4. systemctl --user daemon-reload
5. systemctl --user start grafana

One disadvantage is that "podman ps" no longer shows the status of all running containers of this host, since the containers are no longer running under the same user.

Edit: Also enable linger for the user, otherwise you get warnings and the user is not able to autostart the containers on boot

@niclas ah, thanks for the explanation! I'm currently running everything in docker but I'll have a closer look at podman now