✧✦Catherine✦✧Mar 28

so i took apart an NVIDIA BF-3 card and would anybody who understands it please enlighten me why it has on board:

  • Puya P24CM02 (256 KB)
  • Winbond W25Q256JVBM (32 MB)
  • Winbond W25Q512JVBQ (64 MB)
  • MXIC MX66L2G45G (256 MB)
  • a Kingston EMMC128-TY29 (128 GB)
  • a Microchip CEC1736 root-of-trust controller (which embeds 2 or 4 MB of internal SPI flash and is connected to the other three SPI flashes... I think)
  • and an ASPEED chip with a p/n I did not extract from under thermal compound looking suspiciously similar to those awful BMC devices, connected to 1 GB of Samsung DDR4

I wasn't prepared for a 1 Tbit eMMC connected directly to the main ASIC. what could it possibly be storing there? the field engineer's porn stash?

Show thread✧✦Catherine✦✧
this is literally more flash than in most laptops, even if you include Thunderbolt controllers and the display firmware
Mar 28 at 6:04pmWeb
Show thread✧✦Catherine✦✧Mar 28
apparently the network card runs ubuntu
Show threadmorganMar 28
@whitequark only three toots deep in the thread and the plot is already fantastic
Show threadRyan FinnieMar 28
@whitequark Saw this without thread context, assumed cursedness, was not disappointed
Show threadmccMar 28
@whitequark what would be necessary to "reverse the flow" so that this is the "computer" and the card can run with no other peripherals except power
Show thread✧✦Catherine✦✧Mar 28
@mcc I literally just assembled this on my desk and it doesn't do this out of the box but I am led to believe that you can get an SKU which has a PCIe Root Complex and can talk to NVMe all on its own
Show threadmccMar 28
@whitequark fantastic. Or terrible. Whichever
Show thread✧✦Catherine✦✧Mar 28
@mcc it is pretty cool I think to be able to do that
Show thread✧✦Catherine✦✧Mar 28
@mcc however the reason I wanted to do that is "to remove the Linux sitting in between networking and storage" but if the card literally runs Linux it's sort of lukewarm
Show threadmccMar 28
@whitequark so related to https://social.hackerspace.pl/@implr/116308523937232041 I guess the thing that would make me consider it "terrible" is the question of, if they can make this, it implies they could have just as easily made an embedded board I would find much more interesting
Show threadJohn RipleyMar 28
@mcc It’s such a horribly inefficient chip, it unfortunately wouldn’t make an interesting standalone board. It’s only compelling to use if you need the firehose of that much PCIe and network bandwidth, and even then it only makes sense in a niche of legacy x86 servers. All its functions can be achieved on the host with less power used, which if you have a rack power budget to stay within, means it’s actually overall a _decelerator_.
Show threadimplr6d ago
@jripley
Yes, a BF2/3 is basically a rpi4 or 5 glued to a cx4 NIC, with power use higher than both of those.
However, I don't think it's fair to compare an old bluefield with a modern x86 host. The newer incarnations have Neoverse V2 cores which should be just as (if not more) efficient than current x86.
There's a reason most hyperscalers use a variant of this tech for their clouds, and they're extremely sensitive to power OPEX.
Show threadJohn Ripley6d ago

@implr My biggest takeaway from spending a while redesigning data center hardware from the ground up, is that everyone says they care about OPEX, but when you look under the hood, it’s full of gross inefficiencies because it is fully Conway’s Law inside. Nobody is willing to do work which crosses organizations.

That’s where things like BF come in: they’re supposedly hardware (Linux in a trench coat) and a driver, so you can outsource all the new features to a vendor, and everyone’s happy.

Show threadJohn RipleyMar 28
@whitequark One of the selling points is you can use it as a remote attached storage. It’s a weird, egregiously inefficient and power hungry product, with so much thrown in that you don’t really need the server it’s plugged into.
Show threadInga stands with 🇺🇦 🇵🇸Mar 28
@whitequark @mcc "PCIe Root Complex" sounds like "God complex" in this context
Show threadimplrMar 28
@whitequark @mcc
The silicon should be the same. Some coworkers tried making it work on a BF2 (they're really cheap on ebay) with a collection of cursed pcie adapters, but that's not enough - iirc they determined you need the right firmware from the SKU sold as a root complex.
Probably hackable if you could figure out which magic registers to poke, probably incredibly tedious to figure this out.
Show threadIgnas Kiela6d ago
@whitequark @mcc If you want an amazon-flavored experience of this, Microtik has a "server" using a similar type of chip https://mikrotik.com/product/rds2216 (as well as a pcie card)
ROSE Data server (RDS) | MikroTik

RDS is a high-performance, all-in-one storage, 100G networking, and container platform designed for enterprise environments. Featuring 20 U.2 NVMe storage slots and a special RouterOS Edition for Storage & Compute (ROSE).

Show threadAndrewMar 28
@whitequark and I thought the hardware I worked on was cursed…
Show threaddram🎀Mar 28
@whitequark https://xkcd.com/801/ but it's linux, not java
Golden Hammer

xkcd
Show threadcrzwdjk ✅ Mar 28
@whitequark Ah, so basically a beefier Killer NIC, with ARM instead of PowerPC.
Show thread✧✦Catherine✦✧Mar 28
@crzwdjk oh is that how they do QoS?
Show threadcrzwdjk ✅ Mar 29
@whitequark I don't actually know anything about the Killer NIC other than that a guy I knew used it as a debugging tool since it sits right there on the PCI bus. Which is great until the device you're debugging locks up the bus.
Show threadVal Packett 🧉Mar 28
@whitequark oh yeah bluefield iirc you can even boot freebsd on those, it has acpi i think
Show threadAnna6d ago
@whitequark not only that but some of them run kubernetes
Show thread✧✦Catherine✦✧6d ago
@annathyst terrifying