Cees-Jan Kiewiet 
It came back up with the switch in its new raised position after I took the old raise down for redesigning. And you might notice that the #IKEA cabinet it was on is no longer standing, but is now lying on it's side providing double the space and a lot more height to work with:
Now I can put all the nodes directly on the cabinet instead of stacking them on unstable towers. There will be a better more epic #LEGO #MOC to it all once I've figured it all out. But for today I'm happy with the progress:
New #lego bricks came up so some new updates https://twitter.com/WyriHaximus/status/1555962791785385985
Made a minor tweak to keep the node better in place. Not sure if I'm happy with it but we'll see:
It can't move back or forward anymore.
Also experimented with a new rear closer for cable management but folder it behind it put a lot of strain on the cable. Might need to do a few varriations:
Another important part of this project is cable management. So I started creating a hole different groups of cables can go into the structure. This was the first attempt
This is the second attempt. (Got to love me some #glowinthedark #lego.) Not fully happy but getting somewhere with this. It needs two more whole like this for all 3 groups. (Downstream switch, nodes, and upstream switch.)
Nice little milestone, serving HTTPS from the cluster internally on the home network only. Without port forwarding 😎:
Part of getting this to work was to put all my DNS records for that domain in #digitalocean's nameserver service and let #certmanager do a DNS01 challenge.
So I made the call to put all my DNS records in #terraform making managing it easier and more atomic. This makes migrating
away in the future also easier. And I can have a set of defaults on all my domains.
Initial prototype to keep an RJ45 patch block in place with just plain and simple standard #LEGO_Group #lego 😎.
#LEGO_Group The locking in place in action on video:
#LEGO_Group This will make sure I can keep the cool external aesthetics of the coloured cables from the switch, without having to pull that throughout the MOC to the nodes (it just makes maintenance also a lot easier): https://toot-toot.wyrihaxim.us/@wyri/111193914559469567
Cees-Jan Kiewiet :rp: :wm: (@[email protected])
Attached: 1 image · Content warning: Kubernetes home lab using lego thread imported from Bird site
Also lowered the switch, these legs are both more solid and require 100 or 200 connector pegs less to keep it up. The previous design had a flaw where the top was turned/twisted a few degrees out of alignment with the bottom due to the tensions in the design.
Also with this new design I can put the fan you see to the right inside to cool the switch off extra when needed. (It should only be needed during heatwaves; plus temperature is monitored through several systems.)
s/right/left
New parts came in while I was for work on the other side of the pond. All for the base plate, and cable management at the front:
The front cable management looks like this and will divide different groups of cables into different sections:
The baseplates, all 18, will replace the current 48x48 grey baseplate and making it much more sturdy and expandable in the future:
Putting that together gives a nice solid base plate, covering the entire designated space:
While doing this I learned that #LEGO_Group 3x6 "BRICK 4/3, 16X16 W/ 4.85 HOLE" (element ID 6302092) matches the size of an #IKEANederland KALLAX cabinet (https://t.co/erKDPgpjLT) perfectly:
It took me a bit of timing, and cable patching, to get all cables through the cable management holes. The #Kubernetes cluster never went down as it was shifting workloads between nodes as the cables were patched:
Table of contents for the blog series of this project incoming in a week or two: https://twitter.com/WyriHaximus/status/1580295782783016960
Cees-Jan Kiewiet (@[email protected]) on X
Started writing the first blog post of this Twitter thread: https://t.co/iuUdBOHixy
Ok the ToC is up a bit earlier: https://blog.wyrihaximus.net/2022/10/building-a-kubernetes-homelab-with-raspberry-pies-and-lego-table-of-contents/
Partly because it was quick and easy to write. Onto a post with more insights
Building a kubernetes homelab with Raspberry Pi and Lego: Table of contents
For years I've been using Raspberry Pi's to do in home automation. One of the major constraints was maintenance, if one goes down I have to look at it while things are down. This made running Home Assistant less than ideal. After working with Kubernetes for a few years I decided to bring it into my home permanently. But with a challenge, by using Lego. During the Pandemic we got back into building Lego and this is a way for me to bring my Lego skills to the next level. The post is probably the most boring one as it is the table of contents. However, it will be updated every time a new post is published. Also, posts won't be written in a logical order but more in order of the things that are already done. As such the "Home Assistant" post will come before the "Node Software set up" post because the former can be done again without changes or data loss due to the "Terraform" and "Storage" posts.
PVC storage post is up: https://blog.wyrihaximus.net/2022/11/building-a-kubernetes-homelab-with-raspberry-pies-and-lego-persistent-data-storage-for-pvcs/
Building a kubernetes homelab with Raspberry Pi and Lego: Persistent Data Storage for PVC's
While most apps are stateless, the often do need to store state somewhere. In Kubernetes you can use a Persistent Volume Claim. That claims a piece of storage for the pod for now and in the future.
The #lego parts for the RJ45 plug holders came in today. Here is one of them I build to make sure the parts and idea fully worked. But since we put the parts in a personal advent calendar it's going to take up to christmas before all of them are assembled and on their spot.
My little junior OPS mentee is really excited about his role managing the #home #lego #kubernetes #raspberrypi #cluster, down to the sniffing
3/6 RJ45 holders pick up from the advent calendar, with 3 to go and 4 days left it should be straight forward getting the last 3 as well before I can start hooking them up at the rear of this MOC
Meet #time, the new node that will trial https://www.talos.dev/
#SSD and the #USB to #SATA adapter are coming later this week and early next week. If all goes well with https://www.talos.dev/ it will replace k3os. This is also the first time in 9 months I got my hands on a #raspberrypi 4 8GB, thanks to #kiwielectro
#kiwielectro Put the #RJ45 blocks in place, it's going to make managing cable a lot easier now. In the end, each node will have matching cables because both sides of the row are on the other side cables 🤣.
Cables are still somewhat messy but this is a good step in the right direction. Already have some ideas for the next steps, including a nice exit hole for the power cable
Had another experiment yesterday to be able to slide the #raspberrypi #kubernetes node enclosure in and out of the #lego MOC. The bottom is pretty solid. But the higher I got the worse it got. Sliding in and out is great, needs some tweaks but it will work. The network and #USB
cables I wanted to plug in behind it. But that would mean getting deeper and under the roof (the "roof" shown in the photos for sure won't make it into the final version). So considering making plugs in the side to plug the node into MOC.
#SideroLabs So far so good! Going to do some firewall tests to lock down the cluster vlan to only what is needed over the weekend.
#SideroLabs One of the things that is always important with computers is cooling them. So as such, I have a fan from #hema to cool the nodes. And right now it is always on, but I already connected a #shelly_IoT plug to be able to only turn it on when the nodes get hot enough that they need
#SideroLabs #HEMA #shelly_IoT some help cooling down. The automation isn't in place yet, but I already started preparing for it :).
If the experiments with https://www.talos.dev/ are positive, it will replace k3os. One of big plusses is that this is #k8s and not #k3s so I should be able to use the clusterautoscaler for autoscaling. Which needs some code to work with #unifi switches.
This #kubernetes #lego #raspberyrpi thread is now fully imported and can be found:
* On #Mastodon: https://toot-toot.wyrihaxim.us/@wyri/109858081844775179
* On Twatter: https://twitter.com/WyriHaximus/status/1404501201438261260
Regular updates will now resume 🎉
Cees-Jan Kiewiet :rp: :wm: (@[email protected])
Content warning: Kubernetes home lab using lego thread imported from Bird site
One of the biggest things is that since half a year ago the cluster is now fully on https://www.talos.dev/ and managed through #terraform. It's been a use success and a quick and easy move. (Well it took me a day tho.) It is also a lot easier to rebuild nodes:
* Write raw image to disk
* Apply TerraForm
* Write raw image to disk
* Apply TerraForm
The second biggest thing is that, since a couple of days ago, the cluster has not 1 but 3 control plane nodes. And for some reason hours before I started working on adding the other two the only one running then crapped out: https://toot-toot.wyrihaxim.us/@wyri/111165411454560918
Cees-Jan Kiewiet :rp: :wm: (@[email protected])
Attached: 1 image An hour before I wanted to add two more controlplane nodes to my home #k8s cluster the current single one stopped responding to API calls 🤦
Meet Deathwing, one of our #cats, he has a medical history and sometimes likes to retreat. So he started claiming the spot next to the switch tucked away in that corner. So we decided to turn it into a catbed for him. Including the roof, which needs to be added again in the current iteration but the places where needed to extend the cluster surface area.
Here is a reason why I love #kubernetes; just made #RabbitMQ replicas run in different zones. Downstairs where 2 of the control plane nodes are is a zone, my desk for the other control plane node is a zone, and the rest of the cluster behind me is a zone. So I now forced a broker in each zone, but also told it not to get on the same node as #homeassistant.
This will get me some more usage out of those nodes. Because honestly 3 nodes as control plane for a home cluster? That is overkill.
Today the long enough cables for the SFP modules came in (the other cames were 4cm shorter than advertised and about 3cm too short). So hooked it up. This now means there is a patch location to the side of the MOC. So now I have the last two ports I need for the last two nodes:
While yanking on some cables I took out the support tower for the cat bed on top. So no cat bed there for Deathwing tonight. Will do the cables better tomorrow and rebuild it. Might have to take part of the foundation loose, going to be fun.
Now that all colour matching cables are hooked up, it looks absolutely fabulous! Still needs a lot of work, but it is getting there.
Next up getting each node it's own fan instead of using those #hema that @erikaheidi showed cased a year ago on nodes to cool them down. They need some cleaning once in a while tho.
When Deathwing found out on this belated #caturday he wasn't amused. But accepted a temporary cat bad underneath the MOC.
Today was to rebuild that supporting pillar and restore Deathwing's cat bed in time for #caturday tomorrow.
Started with ensuring adding a 16x16 in the middle of the pillar for support, connecting it to the plates it supports.
Baroness Draka came by to inspect it and then put it back up after rearranging some cables that pulled it over in the first place.
Put in air flow directors for some extra cooling for the switch in place as well. It's currently running at a fine 57 degrees (inside the outside is always touchable but can feel hot), but can go up to 70 especially when it is hot up here.
Monitoring everything, currently building something that queries #Prometheus and tosses it over #MQTT to @homeassistant so #nodered can turn on the cooling fan, and in the worse case scenario cut the power.
The first attempt to put a roof on: failed.
The roof was technically on, and while it's not intended for the #cats to walk on, they will. Any light force would bend the walls. So going to reinforce those first and come up with a solid skeleton and then give it another go.
Finished building most of the blade in stud.io, only thing left is the front/rear node suspension (the side with all the USB and Network sockets) and be happy with a final design.
Got several designs in the cluster, from sloppy to tight but not happy with any of them honestly.
Dust sucks, but have to keep it in mind. This is the controlplane node on my desk, didn't clean it on purpose for a few months to see how bad it would be.
Took less than a minute to clean, and the dust barely affected the cooling. But it was still a lot more than expected.
