Kenneth FinneganApr 23, 2024
I think #Fedora 40 shipped at 6:30 AM Pacific this morning. #Linux
Show threadKenneth FinneganApr 23, 2024

The ask4 #MicroMirror is making a significant difference for the #Fedora update experience in the United Kingdom.

A year ago, there was only a single 1Gbps Fedora mirror in the UK. We deployed the first 1G MicroMirror there before the Fedora 39 release, and that node was flatline 100% NIC for four days, so we made a point of finding a 10G host to be able to deploy more bandwidth.

So we now have 3x 1G MMs and this 10G mirror in the UK, so this should be making a significant difference for people trying to download the new release in the UK.

The wild part? This 10G mirror is still only an HP T620plus with a Mellanox 10G NIC and a 2TB SSD in it. $250 of hardware added an order of magnitude of mirror bandwidth capacity in a country.

Show threadKenneth FinneganApr 23, 2024
libdnf is using TLS for all the package downloads, which means the bottleneck on these HP T620plus nodes with 10G NICs is the 5.5Gbps we can get out of the AES-NI engine on the AMD GX-420CA CPU. So that's a bummer, but trying to talk distros out of using TLS for package management these days is a losing affair.
Show threadElectronic EelApr 23, 2024

@kwf maybe you can use a Mellanox card with TLS offloading enabled? I did some tests with one and posted about it here:
https://social.treehouse.systems/@electronic_eel/112079325395601770

The gist is that setting up the crypto offload is slow, but then it saves quite a lot of cpu. So when downloading an whole iso or many megabytes of rpms from a system with slow cpu it could pay off.

Electronic Eel (@[email protected])

Attached: 2 images I hoped that TLS-offloading would increase throughput or at least keep throughput but reduce cpu load. But when you look at the throughput graph, the TLS-offloading (nginx+hw) is completely useless for small transfers. I'd have needed log charts to better show this, 8.7 MByte/s total for 500 clients repeatedly requesting a file of 10 kBytes. The regular userspace-only nginx can do 323 MBytes/s for the same load. Even with 100 kBytes requests it is still useless (83 MBytes/s). It only becomes useful in the region somwhere between 1 and 10 MBytes file size. While offloading TLS to the kernel (kTLS) has some setup cost, it pays off from shortly after 100k, offloading the transmission to the network card seems to be much slower. Since the CPU is nearly idle during this time it seems like setting up the offload is somehow implemented inefficiently.

Treehouse Mastodon
Show threadKenneth Finnegan

@electronic_eel it might, but with so many of the requests being for <10MB files, maybe not.

My main motivation for sticking with the X3 gen NIC was just for cost. We try and keep the total node cost below $320 per POP

Apr 23, 2024 at 7:16pmWeb
Show threadElectronic EelApr 23, 2024
@kwf I can fully understand the cost argument. I payed about $200 for the nic - that would make your nodes quite a bit more expensive.