Mastodawn

I now have a working #IEEE #1588 hardware-corrected #time sync setup using an #Intel #I226V card. No #Arista switch required. FreeBSD #ptpd slave sees #linuxptp master. Yet to test with the #NVidia #Mellanox #ConnectX-4 LX clients in #FreeBSD though. #BeagleBone Black support appears broken. Awaiting a #SiRFStar IV USB #GPS with alleged #PPS capability, and yet to recover my #GPSDO unit from my aunt's attic in Fife.

Mr. GM Mar 1

Ugggh this is taking *forever* and it's the first software upgrade of many...

#Homelab #Mellanox #SX6036

Mr. GM Mar 1

Fumbled through sparse documentation on disparate sources before I found that there is a command to enable a license for the "_shell" command for my new-to-me Mellanox SX6036 switch. Once I had that figured out, I was able to follow along with several guides and get Ethernet enabled instead of the proprietary Infiniband thing. With all of this done, I've changed the default login and then started uploading newer firmware via the web UI.

#homelab #Mellanox #SX6036 #40G

Show thread

Kevin Karhan

Dec 25

@c8h4 @dragonarchitect @demicus well, I've not encountered problems with #Ethernet.

OFC some folks will not understand that cheap #Realtek, #Mediatek, #Aquantia & #ASIX-NICs don't do #PacketFiltering and -Sorting in hardware thus never saturating the physical link width and also causing excessive CPU load compared to some #Intel or #Mellanox NICs, but that's jist the nature of it.

The biggest problems I had were garbage NICs from #Broadcom under #BSDs: These are just horrendously bad!

Andriy Utkin Nov 7

All this talk how SFP standard makes network equipment medium-agnostic, yet the network interface itself has a notion of supported link modes and it doesn't list the one I planned to use (10000baseLR/Full). Still waiting for cables delivery, so can't test, I don't know what to make of it but this is upsetting.
This is Mellanox ConnectX-3 Pro and by that maker's software tools it makes sense that they are part of Nvidia now.
#10GbE #sfp #FiberOpticNetwork #Mellanox #HomeLab

Klaus Frank Nov 6

Can you connect a #SAS disk array through a #Mellanox ConnectX card (aka a QSFP port)?

Or what exactly are these "QSFP to SFF-8644" cables used for???

#Server #Storage #Homelab #SFF8644

Eva Oct 29

It's been a long day. Tomorrow though, I really need this MikroTik CCR2004-1G-2XS-PCIe, running the home router, to be quiet. Therefore, lay in bed, figure out which Thermal Grizzly conductive pad is most appropriate for replacement between the NIC's heatsink.. and maybe there a STL file for making a carbon fitment plus copper core (taken from an old Broadwell system, heatsink with a couple of blah blah blah... yep. Go to sleep brain, stop this, maybe more tomorrow.

So GPT-Pro and Deep Research... tell me what to buy.

#homelab #tired #mikrotik #intel #mellanox #networking #sleepy #uuuuuuugh #gptpro

Eva Oct 2

PG&E outage for all of Sunnyvale, UPS resources kept the home office and homelab online most of the hours. In the interim via LTE, decided to resume efforts on spec'ing the latest network refresh.

Big decisions; mostly around scale model design principles, some blandness re: OFED drivers and SPDK version parity for specific NIC/DPU SKUs, and analysis paralysis when focusing on critical nerdatorium protocol support.

Otherwise to say... I'm leaning towards additional Bluefield DPUs, where offload options are more important than bandwidth parity for production port speeds (also lower total port count using 4x25 breakouts), and there's no sense in paying more per unit for a CX6 25G vs BF2 25G... so... yep.

#networking #homelab #mellanox #nvidia #dpu #zfs #distributedsystems #ai #hpc

lymenzies Oct 2

I managed to scavenge an ancient AMD Caicos GPU to replace the Nvidia in my Xeon workstation.

Configuring the Mellanox 40gb Ethernet cards was easy using nmtui and now i have a point-to-point connection.

I was able to get 26.6Gbps on a single thread. The Xeon is a PCIe 2.0 system so I should be able to get a little more bandwidth from the card. I did some sysctl perf tuning, but didn't set a large MTU yet.

Thankfully so much simpler than the Infiniband cards. More later.

#Homelab #mellanox

Eva Sep 24, 2025

Returning to the Valley from SF, feeling mostly quite tired (in a generally good way). Will soon depart, then work, then sometime in the evening decided whether to install a MikroTik CCR2004-1G-2XS-PCIe, or a Mellanox CX-5 2x25G (OCP v2 form factor) mounted on a PCIe converter card. 🤔

Ok, suppose it's time to get ready.

#homelab #SF #networking #linux #freebsd #engineering #nvidia #mellanox #mikrotik