I just discovered something really subtle about WireGuard... TL;DR if you are adjusting interface MTUs precisely, and you have mismatched MTUs between peers in some cases, make sure your smallest MTU is always a multiple of 16!

WireGuard header overhead is said to be 32 bytes + UDP + IP, so 80 bytes for IPv6 and 60 bytes for IPv4. That's where you get the default MTU of 1420 (1500 - 80, so it works with IPv6).

But that's not precisely true! Actually, WireGuard will add up to 15 bytes of padding to the data, to make it a multiple of 16, as long as it doesn't exceed the MTU on that side of the connection.

So let's say you have a server with the MTU set at 1440, but you also have a client that is using IPv4 over PPPoE. So you set its MTU to 1432, subtracting the PPPoE overhead of 8 bytes. That should be fine, since the client will figure out the right path MTU for any connections, right?

Wrong!

The TCP client and server will negotiate an MSS that gives 1432 byte IP packets within the tunnel. But 1432 is not a multiple of 16! However, the client WireGuard instance knows that there is no headroom, so it will send 1432 + 60 = 1492 byte packets, which is the maximum PPPoE MTU. But on the way back, the server thinks it can go up to 1440! 1432 % 16 == 8, so it will try to round up to 1440. Then, it sends 1500 byte packets, which don't fit in PPPoE!

The fix is to either set both the client and server MTU to 1432, or to round down the client MTU to 1424.

Found a low res trans coloured IBM thinkpad wallpaper online, but couldn't find a better quality version, so I made one myself, and thought about sharing it here.

#thinkpad