halloween_spooksterHow do you manage your home server configuration?
Snapshots largely, most everything is VMs and docker containers. I have one VM set aside for dev work to test configs before updating the prod boxes as well.
reproducible
You tried writing bash scripts that set things up for you, haven’t you? It’s NixOS for you.
Terraform and Puppet. Not very simple to get into, but extremely powerful and reliable.
I’m just using Unraid for the server, after many iterations (PhotonOS, VMware, baremetal Windows Server, …). After many OSes, partial and complete hardware replacements, and general problems, I gave up trying to manage the base server too much. Backups are generally good enough if hardware fails or I break something.
The other side of this is that I’ve moved to having very, very little config on the server itself. Virtually everything of value is in a docker container with a single (admittedly way too large) docker compose file that describes all the services.
I think this is the ideal way for how I use a home server. Your mileage might vary, but I’ve learned the hard way that it’s really hard to maintain a server over the very long term and not also marry yourself to the specific hardware and OS configuration.
emeraldHow do you manage your home server configuration
Poorly, which is to say that I just let borgmatic back up all my compose files and hope for the best
🇵🇸antifa_ceoI got a bunch of docker compose files and the envs documented so its easy to spin things up again or rollback changes. It works well enough if I’m good about keeping everything all up to date and not making changes without noting it down for myself later.
Eager EagleI’m the only user of my setup, but I configure docker compose stacks, use configs as bind mounts, and track everything in a git repo synchronized every now and then.
I went the nuclear option and am using Talos with Flux to manage my homelab.
My source of truth is the git repo with all my cluster and application configs. With this setup, I can tear everything down and within 30 min have a working cluster with everything installed automatically.
I have a similar setup, and even though I am hosting git (forgejo), I use ssh as a git server for the source of truth that k8s reads.
This prevents an ouroboros dependency where flux is using the git repo from forgejo which is deployed by flux…
irmadladI use snapshots, once a month an image is made of the entire drive, and I have Duplicati that backs up to cloud. Whatever choice you make tho, remember 3,2,1, and backups are useless unless tested on a regular basis. The test portion always gives me anxiety.
I’d really like to know if there’s any practical guide on testing backups without requiring like, a crapton of backup-testing-only drives or something to keep from overwriting your current data.
Like I totally understand it in principle just not how it’s done. Especially on humble “I just wanna back up my stuff not replicate enterprise infrastructure” setups.
You can use qemu utilities to convert your Linux disk image to VDI which you can then import into VM Workstation or Virtualbox:
qemu-img convert -f qcow2 -O vdi your-image.qcow2 your-image.vdi
One thing you might run into is that Ubuntu server images often use VirtIO drivers, So you may have to make adjustments for that. Or you may run into the need for other drivers that VM Workstation or VirtualBox don’t provide.
documentation.ubuntu.com/server/how-to/…/qemu/#qe… systemadministration.net/converting-virtual-disk-…
Recently switched to ucore. While I cannot for the life of me get SELinux to let my containers run without Permissive mode (my server was previously Endeavour OS and either didn’t have it or I disabled it long ago), I’ve otherwise had great success.
The config is a single yaml file that gets converted into a json file for Ignition, which sets everything up on first boot. It’s an OCI-based immutable distro with automatic updating, so I can mostly just leave it to its own devices and everything has been smooth for the first week I’ve been using it.
My Docker root directory is on a separate drive with plenty of space, so setting up involves directing Docker to that new root directory and basically being done (which my Ignition config handles for me).
I use git and commit configs/setup/scripts/etc. to it. I at least have a road map for how to get everything back this way. Testing this can be difficult, but it really depends on what you care about really.
- Testing my kopia backups of important data? that I manually test every once n’ while.
- Testing if my ZFS setup script is 100% identical to my setup? that’s not that important, as long as I have a general idea I can figure out the gaps and improve the script for the next time around. Obviously, you can spend a lot more time ensuring scripts and what not stays consistent, but it depends on what you care about! For a lot of my service config, git has always worked well for me and I can go back to older configs if needed. You can get super specific here and save versions in git, then have something update the versions (e.g. WUD)
I used to have a fille with every cli command and notes on how each thing was set up. When I had to reinstall it from scratch it took all day going through lots of manual steps and remembering how it should all go.
Recently I converted the whole thing to Ansible. Now I could rebuild my entire system from a fresh install with one command that completes in minutes. If I ever break anything, it will reset everything to its intended state and leave it alone otherwise. And it’s pretty easy to learn and start using.
NixOS for configuration and restic for data
Terraform and ansible. Script service configuration and use source control. Containerize services where possible to make them system agnostic.
How do you decide what’s for Terraform and what’s for Ansible?
They’re good at different things.
Terraform is better at “here is a configuration file - make my infrastructure look like it” and Ansible is better at “do these things on these servers”.
In my case I use Terraform to create proxmox VMs and then Ansible provisions and configures software on those VMs.
Out of curiosity: Are you running nix-ops with nix-secrets or how did you cover orchestration & credentials?
I use flakes and all hosts are configured from a single flake, where each host has its own configuration. I have some custom modules and even custom package in the same flake. I also use home manager. I have 4 hosts managed in total: home server, laptop, gaming PC, and a cloud server. All hosts were provisioned using nixos-anywhere + disko, except for the first one which was installed manually. For secrets I use sops-nix, encrypted secrets are stored in the same flake/repo.
Not op, but using colmena for that
MicroOS is a decent choice, because it can cold boot off a configuration that uses ignition and combustion files. microos.opensuse.org
And they have this file configurator so you don’t have to manually type all the syntax for your configs.
giacomoGit controlled docker-compose files and backed up docker data volumes.pretty easy to go back to a point in time.
bigchungusThat’s actually a really good idea. From now on I will do the same. Thanks!
Yay i satisfied big chungus!
thirdBreakfastProxmox on the metal, then every service as a docker container inside an LXC or VM. Proxmox does nice snapshots (to my NAS) making it a breeze to move them from machine to machine or blow away the Proxmox install and reimport them. All the docker compose files are in git, and the things I apply to every LXC/VM (my monitoring endpoint, apt cache setup etc) are all applied with ansible playbooks also in git. All the LXC’s are cloned from a golden image that has my keys, tailscale setup etc.
eliThis is pretty much my setup as well. Proxmox on bare metal, then everything I do are in Ubuntu LXC containers, which have docker installed inside each of them running whatever docker stack.
I just installed Portainer and got the standalone agents installed on each LXC container, so it’s helped massively with managing each docker setup.
Of course you can do whatever base image you want for the LXC container, I just prefer Ubuntu for my homelab.
I do need to setup a golden image though to make stand-ups easier…one thing at a time though!
Yes, essentially I have:
Proxmox Baremetal ↪LXC1 ↪Docker Container1 ↪LXC2 ↪Docker Container2 ↪LXC3 ↪Docker Container 3Or using real services:
Proxmox Baremetal ↪Ubuntu LXC1 192.168.1.11 ↪Docker Stack ("Profana") ↪cadvisor grafana node_exporter prometheus ↪Ubuntu LXC2 192.168.1.12 ↪Docker Stack ("paperless-ngx") ↪paperless-ngx-webserver-1 apache/tika gotenberg postgresdb redis ↪Ubuntu LXC3 192.168.1.13 ↪Docker Stack ("teamspeak") ↪teamspeak mariadbI do have a AMP game server, which AMP is installed in the Ubuntu container directly, but AMP uses docker to create the game servers.
Doing it this way(individual Ubuntu containers with docker installed on each) allows me to stop and start individual services, take backups via proxmox, restore from backups, and also manage things a bit more directly with IP assignment.
I also have pfSense installed as a full VM on my Proxmox and pfSense handles all of my firewall rules and SSL cert management/renewals. So none of my ubuntu/docker containers need to configure SSL services, pfSense just does SSL offloading and injects my SSL certs as requests come in.
Incus and ansible
Packer builds the terraformable/openTofuable templates to launch into the hypervisor where chef (eventually mgmtConfig) will manage them from there until they die.
All that is launched by git. Fire and forget. Updates are cronned.
There are no containers. Don’t got time to fuck about. If Systemd wasn’t an absolute embarrassment I’d not worry about updates even as much as I do, which isn’t much aside the aforementioned cancer.
xcjsFleet from Rancher to deploy everything to k8s. Baremetal management with Tinkerbell and Metal3 to management my OS deployments to baremetal in k8s. Harvester is the OS/K8S platform and all of its configs can be delivered in install or as cloudinit k8s objects. Ansible for the switches (as KubeOVN gets better in Harvester default separate hardware might be removed), I’m not brave enough for cross planning that yet. For backups I use velero, and shoot that into the cloud encrypted plus some nodes that I leave offline most of the time except to do backups and updating them. I user hauler manifests and a kube cronjob to grab images, helm charts, rpms, and ISO to local store. I use SOPS to store the secrets I need to boot strap in git. OpenTofu for application configs that are painful in helm. Ansible for everything else.
For total rebuilds I take all of that config and load it into a cloudinit script that I stick on a Rocky or sles iso that, assuming the network is up enough to configure, rebuilds from scratch, then I have a manual step to restore lost data.
That covers everything infra but physical layout in a git repo. Just got a pikvm v4 on order along with a pikvm switch, so hopefully I can get more of the junk on Metal3 for proper power control too and less IPXE shenanigans.
Next steps for me are CI/CD pipelines for deploying a mock version of the lab into Harvester as VMs, running integrations tests, and if it passes merge the staged branch into prod. I do that manually a little already but would really like to automate it.
Definitely overkill lol. But I like it. Haven’t found a more complete solutions that doesn’t feel like a comp sci dissertation yet.
The goal is pretty simple. Make as much as possible, helm values, k8s manifests, tofu, ansible, cloud init as possible and in that order of preference because as you go up the stack you get more state management for “free”. Stick that in git and test and deploy from that source as much as possible. Everything else is just about getting to there as fast as possible, and keeping the 3-2-1 rule alive and well for it all (3 backups, 2 different media, 1 off-site).
Currently I’m migrating from compose.y(a)ml to terraform. I’m using proxmox -> 2x VM -> docker swarm.
I will soon try to engineer a solution to quickly scale up and down any service I want using the same terraform codebase with rundeck.
I have my configs as terraform templates and it gets deployed as a swarm config (or secret), then mapped to the container the same way.
Using Forgejo as a base is really nice.
heartfelthumburgerThe place I work at uses chef so I use cinc to manage my vms and proxmox host.