I'm continuing with my tests of the Mellanox/Nvidia ConnectX-6 Dx cards I got.

Over the last days I tested the integrated switching apabilities, called ASAP² by Mellanox. So what is it good for? Providing fast network access to virtual machines.

The conventional methods how this is done for KVM-based VMs on Linux is either a kernel bridge device, MacVTap or regular routing. But as you can see in mybenchmark graph, these methods have quite severe performance limits.

The alternative is Single Root I/O Virtualization (SR-IOV). A network adapter with this capability (nearly all server adapters offer this today) splits out several "virtual function" PCIe-devices. Then the virtual function devices
are made directly accessible to the VMs with the IOMMU of the CPU. It is faster because now the CPU doesn't have to do context switching between the VM and host.

While SR-IOV is available for quite some time now (it was first introduced around 2008), the implementations often had a few downsides:

• no communication between VMs or host and VM
• just basic briding supported, but newer designs often also offer VLAN capability
• no support for bonding two physical network ports with LACP for higher availability

Mellanox have implemented a complete switch ASIC that you can use to hook your physical
ports and SR-IOV virtual functions together. This switch can be controlled with the kernel
switchdev interface and it is able to apply complex switching rules.

As you can see in the light-blue line in my benchmark graph, it is able to do LACP bonding of two physical ports and applying a layer3+4 xmit_hash_policy to utilize both bonded ports. So a VM is hooked up with just one virtual function and doesn't have to care about bonding at all. If either port of the bond is disconnected, the other one is used (I tested this to be really sure it is supported).

This is quite a good feature and something I haven't seen from other vendors.

#networking #mellanox #homelab #virtualization

You may remember my post about the completely failed EMC test from about 3 weeks ago - I finally got proof of what the issue was: the EMC test lab bodged the test method and didn't ensure proper impedance on the measured cable.

The two attached graphs are from the same device, but just one measurement is done properly...
#electronics #emc #emi #emissions

thanks to @gsuberland for pointing me to his blog post about multichannel configuration on Samba - this triggered me researching if there is something similar for NFS - and it is: the nconnect= mount option.

And as you can see in the chart below it makes some difference...

In this scenario the hw crypto offload gets you quite near to the unencrypted performance. Also the bonded channels are properly utilized too. The only issue is that your application on the client has to send multiple requests in parallel to make use of this.

Since I was already benchmarking this topic, I thought about what the best way to improve encrypted fileserver performance would be.

So I replaced my slowish Epyc 7232P (8-core, Zen 2) with another Ryzen 7950X3D (16-core, Zen 4) like on my client.

As you can see investing in a faster CPU gives much better results than the crypto offloading and either NFS RPC-with-TLS or Samba become reasonable options.

After experimenting with the (underwhelming) performance of HTTPS offload of my Mellanox/Nvidia ConnectX-6 Dx smart NIC last week, I had one more idea how the crypto offload could be useful to me: encrypting NFS on a fileserver.

My test was one client doing random reads over NFS with fio. Since the test with the many HTTPS clients last week didn't perform well probably due to setting up the TLS session being slow, I now did a test with just one client and one TCP connection.

As you can see in the chart, offloading NFS encryption with the new RPC-with-TLS mode works and is even the fastest NFS option on this server hardware - but it won't even saturate a 10G link and is far slower than the unencrypted variant.

As the card is also able to do IPSec crypto offloading I also tried running the unencrypted NFS through an IPSec tunnel for auth and encryption, but the performance of it is totally useless.

So unfortunately I'm still underwhelmed with the performance of the crypto offload.

#networking #nfs #mellanox

I hoped that TLS-offloading would increase throughput or at least keep throughput but reduce cpu load. But when you look at the throughput graph, the TLS-offloading (nginx+hw) is completely useless for small transfers. I'd have needed log charts to better show this, 8.7 MByte/s total for 500 clients repeatedly requesting a file of 10 kBytes. The regular userspace-only nginx can do 323 MBytes/s for the same load. Even with 100 kBytes requests it is still useless (83 MBytes/s).

It only becomes useful in the region somwhere between 1 and 10 MBytes file size.

While offloading TLS to the kernel (kTLS) has some setup cost, it pays off from shortly after 100k, offloading the transmission to the network card seems to be much slower. Since the CPU is nearly idle during this time it seems like setting up the offload is somehow implemented inefficiently.