Mastodawn

Serge from Babka

I have a #devops (or what we used to call #sysadmin) question...

I like Docker Swarm for its simplicity and apparent "lightweight" nature. From a user standpoint, you can simply define a set of services and it's not that much of a leap to go from a docker-compose file to a full blown distributed system for a small number f nodes.

The problems are that Docker Swarm only appears to be offered by Docker (tm) and requires the real Docker (tm) stack, as opposed to the solution most distros use today, which is to use podman as a Docker replacement (for many good reasons).

And the fact that Docker is owned by Mirantis, Mirantis's future seems uncertain is good reason not to stay.

Is anyone still using Docker Swarm? If not, do you have a lightweight alternative (not Kubernetes)? I've heard not-great things about Nomad.

I feel like this is a huge missing area in the orchestration landscape.

#docker #orchestration #dockerswarm #kubernetes #nomad

𝙻𝚞𝚔𝚎 𝚆𝚑𝚒𝚝𝚒𝚗𝚐 Jul 26, 2023

@serge A few years back I would have said Rancher v1 (Cattle) was a good, simple option as an alternative but they have gone all in with K8S...

Would Hashicorp Nomad perhaps scratch the itch for you?

Serge from Babka Jul 26, 2023

I've heard not-great things about Nomad, like it gets into weird edge conditions and then just stops working. Have you used it yourself?

𝙻𝚞𝚔𝚎 𝚆𝚑𝚒𝚝𝚒𝚗𝚐 Jul 26, 2023

@serge I haven't no... I drank the K8S Koolaid after Rancher switched so not spent much time looking at alternatives.

Serge from Babka Jul 26, 2023

Kubernetes is a lot of complexity for my use case of 3-5 nodes.

𝙻𝚞𝚔𝚎 𝚆𝚑𝚒𝚝𝚒𝚗𝚐 Jul 26, 2023

@serge 100% agree. My rule for K8S is that if you don't have a 4+ person team to look after it 24/7 as their sole job, then you aren't big enough to need K8S 😅

For a case that small where you perhaps don't need things like reusable ingress or overlay networking, could you perhaps get away with Ansible or Terraform to control podman directly instead? Deal with the containers more like how we used to treat bare metal / VMs?

Serge from Babka Jul 26, 2023

It's funny; that's exactly where I'm leaning, essentially manual orchestration- and I know this well because 20 years ago my team was managing ~1000 bare metal hosts (with 62 distinct configurations) this way, including real time services, developer services, and others.

The host allocation part is easy, but what gives me more pause is some of the benefits that orchestration provides on the reconciliation phase, eg how if you change a container image version in a pod definition in k8s, it will first try to load that new container up, run it, and add it to the ingres, then wait to shut down the service, and then do the same across the nodes.

This is the part that was always very unfun. Doing this in ansible seems challenging, and would require writing a set of pre and post scripts to remove the service from the load balancer, etc. and it's not possible *AFAIK!* in ansible (without something like ansible tower) to do rolling deployment.

gester Jul 26, 2023

@serge Personally I find #nomad extremely easy to use. Simple configuration with lots of options if needed. Basic configs can be run with just Nomad, if you need something more advanced the interconnections with Vault and Consul are very straight forward.

Given the scale some shops are running nomad at I find it very hard to justify using other orchestrators.

csbchristopher Jul 26, 2023

@serge Maybe consider a lightweight k8s distrubution (like k3s https://k3s.io/)

I tried it for my 4 node homelab, with positive results.

K3s

Serge from Babka Jul 26, 2023

@csbchristopher

Thanks, I really appreciate the reply.

k3s and minikube are lighter weight and remove some of the abstractions. I'm less worried about the management of the kubernetes than just the every day complexity of adding services, adding operators, making new operators, etc.

For that reason, I've ruled k3s and minikube out for me, because while they're smaller, they don't collapse the problem enough for me right now.

csbchristopher Jul 26, 2023

@serge Hmm, I've been using k8s for several years, and I've never had the need to write a new operator.

Usually one can handle the basics with just the "Service" and the "Deployment" objects. (https://kubernetes.io/docs/tutorials/kubernetes-basics/)

In any case, good luck!

Learn Kubernetes Basics

<!DOCTYPE html> Kubernetes Basics This tutorial provides a walkthrough of the basics of the Kubernetes cluster orchestration system. Each module contains some background information on major Kubernetes features and concepts, and a tutorial for you to follow along. Using the tutorials, you can learn to: Deploy a containerized application on a cluster. Scale the deployment. Update the containerized application with a new software version. Debug the containerized application. What can Kubernetes do for you?

Kubernetes

blobfoxcomputer

@serge I'm in the same boat, wanting to do something for like 3 nodes (I say scales of up to a dozen nodes.)

Pretty much the only container orchestrators I know in that category are Swarm and Nomad. As far as I know, both of them have much less shiny toys with regard to storage and ingress.

I know https://dockerswarm.rocks/ exists, but it hasn't had any updates since 2020. The tools it references are in a mixed state--swarmpit is still going strong, but swarmprom is archived.

I was in the process of building a cluster to play with this, but I got stymied by hardware issues (choices were made), and I haven't gotten back to it.

Docker Swarm Rocks

Docker Swarm mode ideas and tools

@serge Highly recommend rancher's k3s (lightweight kubernetes).

@NotMyBub @serge I second this!

Johannes Kastl Jul 27, 2023

@NotMyBub @serge If #podman (Quadlet, or using docker-compose files with #podman) does not solve my problem, k3s usually does.

Yeah, it is kubernetes. But it is easy to setup and comes with everything you need.

And personally I would not touch #docker or docker swarm with a ten foot pole...

@johanneskastl This is my experience as well. k3s is easy to setup, it's just a single binary.

Caveat being you either need some basic kubernetes knowledge, or are willing to learn.

There are also some great tutorials and even some terraform.