✧✦Catherine✦✧

so i took apart an NVIDIA BF-3 card and would anybody who understands it please enlighten me why it has on board:

  • Puya P24CM02 (256 KB)
  • Winbond W25Q256JVBM (32 MB)
  • Winbond W25Q512JVBQ (64 MB)
  • MXIC MX66L2G45G (256 MB)
  • a Kingston EMMC128-TY29 (128 GB)
  • a Microchip CEC1736 root-of-trust controller (which embeds 2 or 4 MB of internal SPI flash and is connected to the other three SPI flashes... I think)
  • and an ASPEED chip with a p/n I did not extract from under thermal compound looking suspiciously similar to those awful BMC devices, connected to 1 GB of Samsung DDR4

I wasn't prepared for a 1 Tbit eMMC connected directly to the main ASIC. what could it possibly be storing there? the field engineer's porn stash?

Mar 28 at 6:00pmWeb
Show thread✧✦Catherine✦✧Mar 28
this is literally more flash than in most laptops, even if you include Thunderbolt controllers and the display firmware
Show thread✧✦Catherine✦✧Mar 28
apparently the network card runs ubuntu
Show threadmorganMar 28
@whitequark only three toots deep in the thread and the plot is already fantastic
Show threadRyan FinnieMar 28
@whitequark Saw this without thread context, assumed cursedness, was not disappointed
Show threadmccMar 28
@whitequark what would be necessary to "reverse the flow" so that this is the "computer" and the card can run with no other peripherals except power
Show thread✧✦Catherine✦✧Mar 28
@mcc I literally just assembled this on my desk and it doesn't do this out of the box but I am led to believe that you can get an SKU which has a PCIe Root Complex and can talk to NVMe all on its own
Show threadmccMar 28
@whitequark fantastic. Or terrible. Whichever
Show thread✧✦Catherine✦✧Mar 28
@mcc it is pretty cool I think to be able to do that
Show thread✧✦Catherine✦✧Mar 28
@mcc however the reason I wanted to do that is "to remove the Linux sitting in between networking and storage" but if the card literally runs Linux it's sort of lukewarm
Show threadmccMar 28
@whitequark so related to https://social.hackerspace.pl/@implr/116308523937232041 I guess the thing that would make me consider it "terrible" is the question of, if they can make this, it implies they could have just as easily made an embedded board I would find much more interesting
Show threadJohn RipleyMar 28
@mcc It’s such a horribly inefficient chip, it unfortunately wouldn’t make an interesting standalone board. It’s only compelling to use if you need the firehose of that much PCIe and network bandwidth, and even then it only makes sense in a niche of legacy x86 servers. All its functions can be achieved on the host with less power used, which if you have a rack power budget to stay within, means it’s actually overall a _decelerator_.
Show threadimplrMar 29
@jripley
Yes, a BF2/3 is basically a rpi4 or 5 glued to a cx4 NIC, with power use higher than both of those.
However, I don't think it's fair to compare an old bluefield with a modern x86 host. The newer incarnations have Neoverse V2 cores which should be just as (if not more) efficient than current x86.
There's a reason most hyperscalers use a variant of this tech for their clouds, and they're extremely sensitive to power OPEX.
Show threadJohn RipleyMar 29

@implr My biggest takeaway from spending a while redesigning data center hardware from the ground up, is that everyone says they care about OPEX, but when you look under the hood, it’s full of gross inefficiencies because it is fully Conway’s Law inside. Nobody is willing to do work which crosses organizations.

That’s where things like BF come in: they’re supposedly hardware (Linux in a trench coat) and a driver, so you can outsource all the new features to a vendor, and everyone’s happy.

Show threadJohn RipleyMar 28
@whitequark One of the selling points is you can use it as a remote attached storage. It’s a weird, egregiously inefficient and power hungry product, with so much thrown in that you don’t really need the server it’s plugged into.
Show threadInga stands with 🇺🇦 🇵🇸Mar 28
@whitequark @mcc "PCIe Root Complex" sounds like "God complex" in this context
Show threadimplrMar 28
@whitequark @mcc
The silicon should be the same. Some coworkers tried making it work on a BF2 (they're really cheap on ebay) with a collection of cursed pcie adapters, but that's not enough - iirc they determined you need the right firmware from the SKU sold as a root complex.
Probably hackable if you could figure out which magic registers to poke, probably incredibly tedious to figure this out.
Show threadIgnas Kiela6d ago
@whitequark @mcc If you want an amazon-flavored experience of this, Microtik has a "server" using a similar type of chip https://mikrotik.com/product/rds2216 (as well as a pcie card)
ROSE Data server (RDS) | MikroTik

RDS is a high-performance, all-in-one storage, 100G networking, and container platform designed for enterprise environments. Featuring 20 U.2 NVMe storage slots and a special RouterOS Edition for Storage & Compute (ROSE).

Show threadAndrewMar 28
@whitequark and I thought the hardware I worked on was cursed…
Show threaddram🎀Mar 28
@whitequark https://xkcd.com/801/ but it's linux, not java
Golden Hammer

xkcd
Show threadcrzwdjk ✅ Mar 28
@whitequark Ah, so basically a beefier Killer NIC, with ARM instead of PowerPC.
Show thread✧✦Catherine✦✧Mar 28
@crzwdjk oh is that how they do QoS?
Show threadcrzwdjk ✅ Mar 29
@whitequark I don't actually know anything about the Killer NIC other than that a guy I knew used it as a debugging tool since it sits right there on the PCI bus. Which is great until the device you're debugging locks up the bus.
Show threadVal Packett 🧉Mar 28
@whitequark oh yeah bluefield iirc you can even boot freebsd on those, it has acpi i think
Show threadAnnaMar 29
@whitequark not only that but some of them run kubernetes
Show thread✧✦Catherine✦✧Mar 29
@annathyst terrifying
Show threadrjMar 28
@whitequark "we CAN and WILL patch the hardware"
Show threadvalMar 28
@whitequark the firmware is an electron app
Show thread✧✦Catherine✦✧Mar 28
@val I genuinely don't know if it's a joke or not
Show threadvalMar 28
@whitequark it's a joke (I hope)
Show thread✧✦Catherine✦✧Mar 28
@val people did ship the configurator for some network card as a copy of firefox inside option ROM so there's that
Show threadvalMar 28
@whitequark oh no
Show threadRMar 28

@val @whitequark never heard of this, but this vibes as something that would've used XULRunner?

option rom is cursed, but XULRunner is arguably architecturally way less fucked than electron (and predates it by like a decade too)

webshit could've gone totally differently if the winds had been even slightly different

Show thread✧✦Catherine✦✧Mar 28
@r @val i'm pretty sure it was literally firefox 2, sarah would know
Show threadprom™️Mar 28
@whitequark sounds "lean". i'm bet it's very "good". lets hope they have great "success"... /s
Show threadbenjojoMar 28

@whitequark I mean you have basically bought a nice computer _with_ a nice on-board nic, rather than a NIC.

The BF's are mostly nice for "cloud" workloads where you need to make imaginary NICs/VirtIO storage on the thing it's attached to, though you can have the card itself by the root bridge with the right semi cursed adapters

Show thread✧✦Catherine✦✧Mar 28
@benjojo what does it need to start booting?.. does the damn thing has a power switch?
Show threadbenjojoMar 28
@whitequark is yours not enumerating? I don't know how you acquired it but it might have a different OS on the card that is trying to do something interesting
Show thread✧✦Catherine✦✧Mar 28
@benjojo oh no I tried to give it 12V from a power supply for funsies. (it just sits there consuming no power)
Show threadbenjojoMar 28
@whitequark did you put the power in the 12v connector on the side or on the gold finger? You almost definitely need to supply both so the BMC can start the show etc
Show thread✧✦Catherine✦✧Mar 28
@benjojo side, don't have a good way to supply the gold finger rn but looking into it
Show threadequinoxMar 28
@whitequark @benjojo possible you also need +3V3SB (and worst case +3V3 though that seems unlikely)
Show threadTrammell HudsonMar 28
@whitequark this chonky NIC from IBM has four sticks of DDR3 and a CF card slot.
Show threadlymenziesMar 28

@th @whitequark

Do you have the part number for this NIC? I'd like to read more about it.

Show threadTrammell HudsonMar 29
@lymenzies @whitequark it was a pre-production prototype of the PowerEN network accelerator. Some details on the architecture still survive: https://cseweb.ucsd.edu/classes/wi20/cse232B-a/papers/IBMPowerEN.pdf
Show thread✧✦Catherine✦✧Mar 29
@th @lymenzies line rate XML processing dates this to probably around 2006
Show thread✧✦Catherine✦✧Mar 29
@th @lymenzies okay, 2014 isn't very close but in fairness i didn't realize it was sponsored by IBM
Show threadlymenziesMar 28

@whitequark

I've had success with the #BlueField2 so I assume the instructions are very similar.

You'll need Nvidia's DOCA all package
https://developer.nvidia.com/networking/doca

I've only ever used it in a PCIe slot (no auxiliary power on the 2). You must have sufficient cooling.

But once you get the RSHIM tool working to talk with the Bluefield console, you can restart the card and interact with it. Best option is to load the BFB firmware and set a password.

Then you can use the RJ45 connector to SSH in.

DOCA Software Framework

Accelerate application development for the NVIDIA BlueField DPU.

NVIDIA Developer