I can’t remember what I was watching, but I remember watching something where they said Kubernetes is designed for something so large in scale that the only reason people have heard about it is because some product manager asked what Google use and then demanded that they use it to replicate the success of Google and subsequently, hobbyists also followed and now a bunch of people are using stuff that’s poorly optimized for such small scale systems.

You should try out all the options you listed and the other recommendatiins and find what works best for you.

I personally use Kubernetes. It can be overwhelming but if you’re willing to learn some new jargon then try a managed kubernetes cluster. Like AKS or digital ocean kubernetes. I would avoid managing a kubernetes cluster yourself.

Kubernetes gets a lot of flack for being overly complicated but what is being overlooked with that statement is all the things that kubernetes does for you.

If you can spin up kubernetes with cert-manager, external-dns, and an ingress controller like istio then you got a whole automated data center for your docker containers.

In my opinion trying to set up a highly available fault tolerant homelab adds a large amount of unnecessary complexity without an equivalent benefit. It’s good to have redundancy for essential services like DNS, but otherwise I think it’s better to focus on a robust backup and restore process so that if anything goes wrong you can just restore from a backup or start containers on another node.

I configure and deploy all my applications with Ansible. You can programmatically create config files, pass secrets, build or start containers, cycle containers automatically after config changes, basically everything you could need.

Sure it would be neat if services could fail over automatically but things only ever tend to break when I’m making changes anyway.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters DNS Domain Name Service/System Git Popular version control system, primarily for code NAS Network-Attached Storage k8s Kubernetes container management package

[Thread #417 for this sub, first seen 10th Jan 2024, 04:15] [FAQ] [Full list] [Contact] [Source code]

I was familiar with just organise my docker-compose containers without any frontend. But I discovered casaOS, which make things pretty simple. An AppStore and a SMB-Shared File manager gave me a really good workflow. Things that aren’t on the AppStore can be handled outside of Casa, too.

PS. But never make the mistake to integrate the outside handled containers, this mess things up.

Portainer + caddy + watchtower, this will give you the benefits of containers without the complexity of Kubernetes. As someone who professionally works with Kubernetes, I agree with what other people have said here: “only run it if you want to learn it for professional use”.

Portainer is a friendly UI for running containers. It supports docker compose as well. It helps with observability and ops. Caddy is an easy proxy with automatic Let’s Encrypt support. Watchtower will update and restart your containers if there’s an update.

First, hire a team of energetic full-time container bros. Half of them will help architect your setup, and other half will focus entirely on supporting the container cult.

I am happy with my simple docker-compose setup - one root folder with one subfolder per project containing the compose file and any configuration mounted into the container. Traefik automatically exposes all services I want under a well-known URL using a single line in each compose file. Watchtower updates the containers.

This has been running stable for over two years with probably 2-3 reboots in between. If my current NUC ever breaks I’ll set it up again using Podman instead of Docker, but aside from that I couldn’t be happier!

Podman pods + systemd units to manage pods lifecycle. Ansible to deploy the base OS requirements, the ancillary services (SSH, backups, monitoring…), and the pods/containers/services themselves.

I really enjoy these type of conversations, learn a lot.

Since you’ve gotten lots of good advice on container manager, I’ll encourage your desire for IaC/DevOps CM, etc.

I believe all the leading CM choices support what you’re wanting to do. I can’t guide you on which one to chose, but just browse through the options or functions your favorite does for the Kx container solution you go with.

I use SALT because of Security Onion, and open source IDS. I have all my nix systems being babysat by SALT, and can have a new x-arr media server, NGINX, blog, etc running in the amount of time to deploy the template (I use vSphere) and salt applies the desired state. Back up and restore a mount folder, np. IaC is only limited by your imagination. I have salt also specifying all the containers I have running, defining the config files, etc. Basically poor mans/simpleton kub.

I suspect you already know this, but if there isn’t a module that directly does what you want like running SQL specific functions, you can just have it run programmatic CLI files on the host, or in the container for you.

I am in the process of moving my IaC code from manager file system to Gitlab. I imagine you’d do this from jump street. Have fun.

I see no one else commented my stack, so I suggest:

Nomad for managing containers if you want something high availability. Essentially the same as k8s but much much much simpler to deploy, learn, and maintain. Perfect for homelabs imo. Most of the concepts of Nomad translate well to k8s if you do want to learn it later. It integrates really well with Terraform too if you are also hoping to learn that, but it’s not a requirement.

NixOS for managing the bare metal. It’s a lot more work to learn than say, Debian, but it is just as stable, and all configuration will be defined as code, down to the bootloader config (no bash scripts!). This makes it super robust. You can also deploy it remotely. Once you grow beyond a handful of nodes it’s important to use a confirmation management tool, and Nix has been by far my favourite so far.

If you really want everything to be infra-as-code, you can manage cloud providers via Terraform too.

For networking I use wireguard, and configure it with NixOS. Specifically, I have a mesh network where every node can reach every node without extra hops. This is a requirement if you don’t want a single point of failure (hub and spoke) to disconnect your entire cluster.

Everything in my setup is defined ‘as-code’, immutable, and multi-node (I have 7 machines) which seems to be what you want, from what you say in your post. I’ll leave my repo here, and I’m happy to answer questions!

–

My opinions on the alternatives:

Docker compose is great but doesn’t scale if you want high availability (ie, have a container be rescheduled on node failure). If you don’t want higher availability, anything more than docker might be overkill.

Ansible and Puppet are alright but are super stateful, and require scripting. If you want immutability you will love Nix/NixOS