✧✦Catherine✦✧the network card i'm poking at, which has two different CPU clusters with Linux running on it, apparently has eMMC and NVMe drives onboard
also it has docker^W containerd installed by default
the operating system is booted by grub??
somebody here was making a joke about the network card running kubernetes
i regret to inform you that the network card does in fact run kubernetes (there is kubelet in ps)
okay there's "kubelet" in
ps. it's definitely running it
medium rare bird@whitequark nooooooooo
Graham Sutherland / Polynomial@whitequark "Kubernetes running on a network card" is an entire new level of cursed.
@whitequark Dante better get inventing some new layers of hell
Rairii 
@gsuberland @whitequark i'm sure it's only in the six-seventh layers of hell
Pxl Phile@Rairii @gsuberland @whitequark I mean it's on a network card. You know, like
Ma can we have leaders of hell?
- we have layers of hell at home
Hell at home: 8 OSI layers
🇺🇦 haxadecimal 🚫👑@gsuberland @whitequark
(OK, I included "financial" and "political", which OSI didn't officially include. So sue me.)
(OK, I included "financial" and "political", which OSI didn't officially include. So sue me.)
@brouhaha @gsuberland that's the judicial layer! above the other two
artemist@gsuberland @whitequark despite fun problems, like "entire broadcast studio breaks because someone plugged in a misconfigured audio interface", aes67 was surprisingly tame for dante's inferno.
Palmer Dabbelt@gsuberland @whitequark at my first job I had to get PHP running on a network card
Fazal Majid@gsuberland @whitequark probably still one or two orders of magnitude less complex and more sensible/secure than a telco-mentality 5G networking stack in a Qualcomm or Apple C1X cellular modem.
4censord 
@whitequark oh then is it running a version of SONiC?
@4censord what is that
@whitequark https://sonicfoundation.dev/
its a linux based switch or router operating system
its quite good and comfy compared to e.g. cisco ios
its a linux based switch or router operating system
its quite good and comfy compared to e.g. cisco ios
Falk Stern@whitequark Then especially don’t look at the doca-hbn or doca-snap manifests. Or the hbn setup scripts.
@falk_ terrifying
Lona Theartlav@whitequark Hmm. That makes me wonder if it would be possible to get it to run on my Xeon Phi 7120A.
Tobias@whitequark Let me tell you about the *other* Mellanox product that comes with docker enabled by default.
Yes, it's their 100/200G Ethernet and 200/400/800G infiniband switches.
Joe Chilliams@whitequark jokes aside fwiw the way to run P4 on the ASIC is to load the compiled P4 using grpc through a service running on kube https://docs.nvidia.com/doca/sdk/loading-dpl-applications/index.html
@joew whats p4
@whitequark it's a weird language to do network packet manipulation in hardware. NICs like bluefield, AMD pensando and even some switches with the right ASIC (tofino) support loading up custom code this way. So you can do things like firewalls, load balancing and etc in hardware rather than in the OS.
Hugo Slabbert ⚠️@joew @whitequark or you can start caching your kv store in-network on your switches
Chris@whitequark i feel vindicated (and I wasn’t the only one joking about k8s)
Greg Bell@whitequark satire has become impossible
Echo Nar@whitequark at first I was surprised then remembered ISPs/carriers. Due to how modern networking systems work (think 5g), they run a lot of service stuff in containers and sticking it on the NIC is probably giving faster network connectivity by bypassing PCIe to CPU translations.
@lethedata the bf-3 is mostly for AI stuff as far as i know. also like. the CPU on this thing is connected to the NIC (the actual NIC part of NIC) over... maybe PCIe, maybe AMBA? not sure. some sort of bus. but you're definitely going to still have NIC to CPU translations, is my point
@whitequark I'd have to look more into that hardware (I was thinking it was a Broadcom BCM95750X or the like) but my hunch is it's hardware offloading the entire network stack to NIC PCIe cards that can fully act as nodes of the cluster. Pin those network service pods to those PCIe cards and offload the entire network speeding it up.
Raven667@lethedata @whitequark I was going to joke that kube was overkill and overhead for a dedicated packet-pushing device, some modern dev FOMO shit that makes everything more complicated and slower than it needs to be (for ~SCALABILITY~) but that makes sense, the more you can push down into the ASIC on the NIC the less CPU you have to burn, freeing it for workload.
Christoph Petrausch@whitequark The OS on the NIC is also pretty useful to build a Cloud as your cloud provisioning from the provider can run on the NIC, and the customer can own the complete other hardware, but still can't interfere with the NIC offloading stuff. So you can build something similar to AWS nitro engine, where a lot of magic like EBS and stuff is implemented. Like you can mount remote NVMes and let them appear local to the customer on demand, when they click it in your API/Cloud UI.
@hikhvar @whitequark @lethedata and networking too. You can do all the fancy vxlan/evpn multipathing stuff in the NIC in hardware and the host just sees a single interface. A single BF3 can push up to wirespeed (~400G) in that config. The software bugs are … interesting though.
Anna@whitequark I wasn't joking! Bluefield Kubernetes is a real thing and can hurt you
@annathyst can confirm, am hurt by this knowledge
clara "parasocialist" sparks@annathyst @whitequark what in tarnation
@mxsparks @whitequark it is absolutely real and can hurt you
rj@whitequark kuberNICes
FxChiP@whitequark the WHAT is running kubernetes?????????? (/hyperbole for comedic effect)
@whitequark reminds me of the Microsoft Azure senior dev who was pointing out they were planning on running 173 agents on the wimpy ARM cloud controller/SmartNIC they used to offload cloud node admin.
chaos ΘΔ& (it/he)@whitequark feck it, grub on the network card
Fiona@whitequark That somehow seems like the least weird part of the whole setup… 
Arch 
@whitequark it's got systemd installed and running too right?
this is amazing
@arch naturally
Viss@whitequark uhhh wtf
Jonathan McDowell@whitequark From my experience with such things I assume it has a full UEFI firmware too.
@whitequark I mean, what should they do? Boot their linux via a tinier linux like IBM pesants? (looking at you, petitboot)
@krono the BMC does run Linux and is booting the main SoC
@whitequark 🤣 why grub then?
btw, what kind of bmc is it?
EDIT: nvm, I'm gonna look it up my self, no need to distract you
@whitequark O dear it *is* the BMC supermicro uses, too. I hope it is a newer, less...laggy variant.
@krono very dfiferent firmware
@whitequark I didn't know they made so different firmwares for the aspeed. interesting
elly@whitequark @krono BMC on a network card? 
This is giving me stronk "Bigfoot Killer 2100" vibes (NIC from ~2011 that had PowerPC SoC and used U-Boot and Linux, I have one of those)
This is giving me stronk "Bigfoot Killer 2100" vibes (NIC from ~2011 that had PowerPC SoC and used U-Boot and Linux, I have one of those)
twilight@whitequark it's more like a cluster of linux boxes which incidentally have a GPU network access and can communicate over PCIe with a host computer
@confusomu it does not have a GPU
@whitequark sorry, i don't know why i was imagining a graphics card. too tired and didn't pay attention…
@whitequark that's probably the most common config for an x86 based embedded device of any kind, tbh. Sometimes that gets hidden behind an older proprietary CLI, like Cisco ROMMON, when you can clearly see GRUB booting Linux actually doing the heavy lifting over the serial console.