Mastodawn

As we continue revising our textbook for its 7th edition, we have turned our attention to "datacenter networking". As people with a lifelong focus on the Internet, it has been an interesting challenge to answer the question of "what's different about the datacenter". We think we are starting to get to the bottom of this, and we trace it back to the special-purpose networks that were originally built for high-performance computing applications, now getting a big boost from AI-related workloads. Our latest newsletter has the details:
https://systemsapproach.org/2026/03/23/convergence-takes-off-in-the-datacenter/

🧵 1/n

Convergence Takes Off in the Datacenter - Systems Approach

RDMA over converged Ethernet (ROCE) is an old idea that has moved into the mainstream thanks to AI workloads

Systems Approach

Show thread

Systems Approach 6d ago

There is an interesting progression from the early days of InfiniBand for high-performance computing, which was a parallel networking universe alongside the Internet, to the converged datacenter networks of today. The significant industry focus on networks to support the training of AI systems has given this convergence a big push. Notably, there is the need to support parallel computing APIs such as those that were provided by InfiniBand, but using standard Ethernet switches. This has been a multi-decade effort starting with the first versions of RoCE (RDMA over Converged Ethernet) and it continues to play out today. 2/n

Show thread

Systems Approach 6d ago

The HPC community long made the case that TCP/IP over Ethernet would not meet their needs, due to two factors. (1) The involvement of the OS in moving data from the application to the network was viewed as an unacceptable performance hit. (2) The best effort nature of IP and Ethernet meant the potential for congestion loss and delays. InfiniBand basically worked around issue (1) with DMA moving data directly from application memory—something that was also done in TCP/IP as early as 1994. Issue (2) was addressed with flow control and per-priority queues, and that idea was subsequently adopted as PFC (priority flow control) in Ethernet switches for the datacenter (aka Converged Ethernet). Modern RoCE implementations also use ECN (explicit congestion notification) or related techniques in an effort to reduce congestion at its outset, leaving PFC as a fallback. 3/n

Show thread

Systems Approach 6d ago

This topic continues to drive plenty of research (especially as the drawbacks of PFC are now pretty apparent). AI workloads continue to demand low latency and minimal packet loss, including at the extreme tail as that impacts training performance.

Two examples from last year's SIGCOMM conference:
Falcon: A Reliable, Low Latency Hardware Transport
https://dl.acm.org/doi/10.1145/3718958.3754353
Revisiting RDMA Reliability for Lossy Fabrics
https://dl.acm.org/doi/10.1145/3718958.3750480

4/n

Falcon: A Reliable, Low Latency Hardware Transport | Proceedings of the ACM SIGCOMM 2025 Conference

ACM Conferences

Show thread

Systems Approach 6d ago

From our perspective the good news is that we finally have a handle on the questions "what is special about data center networking" and "what does networking for AI look like" although we have also opened up a bigger set of questions that we need to keep exploring as we push on with the new edition. Further details here:

https://systemsapproach.org/2026/03/23/convergence-takes-off-in-the-datacenter/

/FIN

Convergence Takes Off in the Datacenter - Systems Approach

RDMA over converged Ethernet (ROCE) is an old idea that has moved into the mainstream thanks to AI workloads

Systems Approach