Things I would love to have: a 32-port 400GbE switch. Things I cannot afford: a 32-port 400GbE switch.
| Website | https://blog.jamesthebard.net |
| Pronouns | he/him |
Jason Weatherly
@jamesthebard@social.linux.pizza
- 508 Followers
- 262 Following
- 2.5K Posts
DevOps Engineer, electronics tinkerer, Linux and Homelab enthusiast. Also a big fan of anime, a good book, and the occasional short circuit...
Multipathing: it makes fixing cabling issues so much easier. Swapped out the 2x2m fiber patches to the NAS with 2x5m ones because the old ones were just a smidge strained. Had absolutely no issues swapping them out one by one and no downtime.
Got `acme.sh` installed on all the Proxmox nodes and managed to get the certs issued and installed to the right location without too much hassle. All of the UIs now no longer anger the Firefox gods. Also all of the servers are updated and nothing is on fire. Time to relax and debate on what to eat for supper.
Did a smidge of cable management because the fiber runs to the QNAP needed a bit more slack. Things I love: being able to disconnect the connections to the iSCSI storage and have absolutely no loss of connectivity.
Eh, final home for the time being. Once the second nameserver finishes updating I can get the cables managed a bit better. Either way, officially done.
Things I enjoy: adding a new ring address across the cluster. Since I added the migration network post-cluster creation it couldn't being used by `corosync` in case something tragic happened. Also, it meant that if you added a new node to the cluster it wouldn't verify that the migration network was online on the new node when it brought it into the cluster. However, solved that pretty fast by just adding it as a `ringX` entry (in this case `ring3`) to all the nodes and enabled it as an interface at the bottom of the config.
Got a question about how it's setup because there are a lot of ports in use from a friend, figured I'd reply here:
- Port 1 (1GbE) goes to the Mikrotik PoE switch in the front of the case.
- Port 2 (10GbE) trunks VM and management traffic into the network.
- Ports 3, 5, 7 (2.5GbE) are VM and management traffic for the three nodes.
- Ports 4, 6, 8 (2.5GbE) are solely for VM migrations
- Ports 9, 11, 13, 15 (10GbE) are the primary iSCSI VLAN and are responsible for VM storage.
- Ports 10, 12, 14, 16 (10GbE) are the secondary iSCSI VLAN and are also responsible for VM storage.
All of the iSCSI stuff is setup in a round-robin configuration across the cluster which helps out with bandwidth. The migration VLAN is isolated from everything and only accessible internally.
Some thoughts:
- While the "mini" part of minirack might be debatable for this build, the footprint of my Proxmox cluster has become miniscule when compared to the size of three full-sized ATX builds.
- The Minisforum MS-A2s are very warm bois. Keeping the cables clear from behind the server nodes will pay off tons later as the hot air is unobstructed.
- Putting those massive power bricks into the bottom of the minirack is not going to work. I'm not sure exactly what I'm going to do with them, but for now they stay outside of the case
- The QNAP QSW-M3216R-8S8T switch was perfect for this setup. A little pricey, but has been pretty painless to get configured and running
- This thing got weighty _fast_
Also moved the migrations over to the previously unused 2.5GbE ports. Figured it would be better to be slower but 100% isolated from all of the other traffic as the migrations can max out the bandwidth on the interfaces.