Cees-Jan Kiewiet 
Right, so the lego parts haven't arrived yet. But the next 3 PoE+ hats for the next 3 nodes did!
Another thing that came in are two USB extension cables. Mainly got those to experiment with less tension on the USB <=> SATA cable
And the big frames to the right might be come a key part in making #LEGO node blades:
Made scale model pillars with diagonal beams for sturdiness but still some movement possible (had bigger ones with the green bars but didn't take a photo), for connecting the blades to:
Had another go at this tonight, and managed to get the RPI and SSD into the cube. Not perfect but getting there 😎
It was interesting to see the amount of traffic flowing through the cluster: https://t.co/IqhxR1XEwE
But sadly had to declare it dead after ETCD decided it ran out of storage: https://t.co/LEmDZxPyGw
So preparing (with k3d) for a triple master cluster, S3 backup and snapshot and automatic recovery before starting it again. Will then do a series of chaos engineering tests to make sure it's resilient against power outages etc etc.
The triple master set up will have each master node on a different switch (!!!) (or access point). Mainly because one of the masters will replace my Raspberry Pi 1 that reads information from the smart meter in this house. The other extra master will have a special purpose.
The meterkast (utility closet) node casing and storage just came in. Never realised M.2 SSD's are so, tiny 😱
Yay! The fine folks at #LEGO_Group seem to have caught up with the custom bricks orders after the holidays zerg 🎉. So I can put together the node #lego enclosures in a few days and start the cluster back up.
Still going for the 3 master setup in the long term, but since I can already start it up with a single master going to focus on building and testing power outage and recovery testing. Hopefully including having a fresh node every time it starts, including the master 🤐
This is a first for me! #LEGO_Group put the custom bricks order in a box 😱. Building the node kube's tonight 🎉. Probably powering the cluster back up next week, and hopefully the more space between components should make it require less active cooling
Hah, progress! But seems I'm missing parts, connector pegs to be specifically so I can't do anything at the moment. Good thing I ordered 200 of those the other day. Might need a few more parts, so another #lego order will go out soon 🤣. But really loving the progress here 😍
Started prototyping a #Kubernetes node with a screen. Next steps are starting X and a browser in kiosk mode
Another big milestone. A new RPI 4 8GB came in, so I can start assembling the utility closet master node:
It's been a decade of two since I had to do configuration through jumpers
There is only one issue. Doing rapid rebuilding of the cluster to test things out is going to be a lot harder because this one won't be as easy to access as the others currently are...
Got it up and running at least https://t.co/0s4snsqtBh and while I love the #argon_forty casing I still think defaulting to using the USB3 instead of the USB2 port will leave plenty of users with I/O issues.
#argon_forty All the nodes in my #k8s cluster run on SSD's on the #USB v2 port because the v3 port has stability issues. Couldn't even write the image to the M.2 #SATA #SSD using the v3 port. Hence the extension cables to hook it to the v2 port. So it has a tail:
But at least I now have two leader nodes, and I'm getting errors like this during some #terraform calls:
Whoop whoop, the #LEGO parts I ordered 6 weeks ago just came in! Time to finish the current enclosures and start with building the tower to put them in
The end result of tonight's thinking with the new #lego parts that came in. Fully hanging node enclosure with room to make it more blade-like/slide it in and out for easy maintenance but that is for another time. Plus the black technic baseplates are amazingly solid <3
The end result for tonight, it is not perfect, and there should be another set of parts arrive in two weeks, but we're getting there. Going to have to order more baseplates tho, those things are awesome and will make the whole project stable af.
Had a small set of #LEGO pieces come in a few days ago. And I finally managed to get a good and sturdy solution for the SSD, and also the RPI, which was a bit tricky with the SSD already in it. Making it about 90% done, the final thing I really want is a way to slide it in and
the enclosure tower. Because maintenance with everything fixed in place will be messy.
And that is the 3rd TF apply done. The (almost) full cluster back up in 3 commands and about 30 minutes of waiting for everything to be installed and back to running:
Next up is rook as a potential replacement for longhorn to use NFS/ISCSI for storage instead. Longhorn is really cool, but with my current number of cluster reinstalls I'm looking for something easier to recover data with
The day ends with only two running because I don't need more, so why would I
Realised today that I could just stack all 3 into one tower above each other. Like the intended result:
Another win for the cluster today is the set up of NFS as PVC storage backend. The SSD's are useful and Longhorn did do what is intended to do. But NFS survives cluster reinstalls. And trying to KISS it here: https://t.co/AwqmBDit7w
Installed #linuxserverio's Heimdall on the cluster today. A bit sad the #Unifi app isn't an enhanced app, but I'm pretty sure I can muster up writing some #laravelphp to turn it into an enchanted app 🤔
#linuxserverio #laravelphp Had lots of fun this weekend building a platform above the switch. Starting out with these half perfect towers (the top still needs diagonal support before adding more nodes):
#linuxserverio #laravelphp The switch is placed in the middle on top of the middle blue statch for additional support:
#linuxserverio #laravelphp It looks big, and it is big but this finally means I can start working on a long term cabling plan and have plenty of space underneath the switch to do so and for cooling:
#linuxserverio #laravelphp During the placement (without anything going down FYI), I found a slight miscalculation:
#linuxserverio #laravelphp After fixing that it has been standing fine like this for a day now. Just need to make the space between the top of the switch and the node platform smaller:
Really happy with the results of this weekend. Swapping all of this when all the nodes where up and running and my wife was playing online games that went through this switch made it "fun" as at some point I had the live running nodes on my lap while trying to get the switch in
Had some fun yesterday prototyping an easy way to support maintenance/swap nodes out without having to take the rack apart:
Think I found a good way to try and keep the cats off the platform, or at least detect when they get up it:
Worked on a PoC to make adding and removing node enclosures easier by not directly making them part of the structure. But instead by putting them on a cart you can take out:
First "big" success of the cluster: https://twitter.com/WyriHaximus/status/1534893994731352066 (sorry no Toot)
Cees-Jan Kiewiet (@wyri@haxim.us) on X
Nearly 10 years ago my first @Raspberry_Pi arrived. Two years later we moved to a different house. And last weekend, 8 years after that I finally took it off DSMR P1 duty and offline (for now). As it has been succeeded by a #kubernetes node running @home_assistant:
Also, it looks like I missed the first birthday of the project (and thread): https://toot-toot.wyrihaxim.us/@wyri/109858081844775179
The #LEGO node enclosure for Reality came out pretty well last night. Been iterating over this design for months and it's come together really well. Both the #raspberrypi and the #SSD in it are well kept in place. All supervised by #deathwing_t_cat who just wanted a nap 🤣.
Don't try this at home:
Came upstairs this morning and found out a certain cat has been on the cluster during the night and made a mess. Now the cool part is that this shows the latest iteration of node enclosure kept the SSD in place for those two nodes:
Been doing maintenance on the cables in the home office today, and as such 3/4 of the cluster was down for a few hours today:
It came back up with the switch in its new raised position after I took the old raise down for redesigning. And you might notice that the #IKEA cabinet it was on is no longer standing, but is now lying on it's side providing double the space and a lot more height to work with:
Now I can put all the nodes directly on the cabinet instead of stacking them on unstable towers. There will be a better more epic #LEGO #MOC to it all once I've figured it all out. But for today I'm happy with the progress:
New #lego bricks came up so some new updates https://twitter.com/WyriHaximus/status/1555962791785385985
