The HPC community long made the case that TCP/IP over Ethernet would not meet their needs, due to two factors. (1) The involvement of the OS in moving data from the application to the network was viewed as an unacceptable performance hit. (2) The best effort nature of IP and Ethernet meant the potential for congestion loss and delays. InfiniBand basically worked around issue (1) with DMA moving data directly from application memory—something that was also done in TCP/IP as early as 1994. Issue (2) was addressed with flow control and per-priority queues, and that idea was subsequently adopted as PFC (priority flow control) in Ethernet switches for the datacenter (aka Converged Ethernet). Modern RoCE implementations also use ECN (explicit congestion notification) or related techniques in an effort to reduce congestion at its outset, leaving PFC as a fallback. 3/n

There is an interesting progression from the early days of InfiniBand for high-performance computing, which was a parallel networking universe alongside the Internet, to the converged datacenter networks of today. The significant industry focus on networks to support the training of AI systems has given this convergence a big push. Notably, there is the need to support parallel computing APIs such as those that were provided by InfiniBand, but using standard Ethernet switches. This has been a multi-decade effort starting with the first versions of RoCE (RDMA over Converged Ethernet) and it continues to play out today. 2/n

More of our appreciation of Dave Farber can be found in this week's newsletter:
https://systemsapproach.org/2026/03/09/dave-farber-and-gigabit-networks/
/FIN