poll: have you ever designed a protocol that uses TCP or UDP directly? (vs using protocols that are based on TCP/UDP like DNS, HTTP, SSH, etc)

(you can pick more than one)

yes, TCP
36.1%
yes, UDP
24.3%
no, but I’ve used protocols based on them
37%
other
2.6%
Poll ended at .

I'm so curious about what the 30% of you who are designing custom UDP-based protocols are doing, I would love to hear more!

(please only reply if you responded “yes, UDP” to the above poll)

@b0rk maybe multiplayer gamedev or smth that'd really not like the tcp overhead
@b0rk this was 30 years ago, not today. A networked video game, and I didn't know any better.
@b0rk Game dev, i'm guessing?
@b0rk High volume traffic generator for network testing.
@jef @b0rk And now I remember that I wrote one of those too, back in 2008. The reason we wrote it was that we were using NFS over UDP at the time, and I wanted to stress-test UDP on some new NFS servers without having to figure out how to make the disks and NFS itself go fast enough. So I wrote something that simulated 'small request, big reply' over UDP (one typical NFS packet pattern). It was an actual protocol because it had embedded sequence numbers to track losses.
@b0rk It will be nice if you'll share your experience in replies!

@b0rk I've written 2ping, a utility which can determine directional packet loss. Has its own UDP protocol (IANA-registered even! 15998/udp)

https://www.finnie.org/software/2ping/
https://github.com/rfinnie/2ping/blob/main/doc/2ping-protocol.md

2ping, a bi-directional ping utility

@b0rk I used this once on an ESP8266 that converted pulses from an electrical power meter to UDP packets containing the number of pulses received so far. Using UDP (over WiFi) meant that the ESP8266 would never block, and because of the counter, missing a packet wasn't that much of a problem.
@bert_hubert thank you that's a great example!

@b0rk @bert_hubert

I voted "yes udp" for a very similar microcontroller based project, though the opposite direction - LED 8-segment sign that you blasted UDP at to update. It was two-digit, so was just a single byte in a UDP packet. I didn't have a proper network stack at all, just an atmega with some ethernet adapter.

@mattm @b0rk @bert_hubert Another very similar case. But for 433MHz radios on an esp8266. Both sending and received packages were just raw UDP broadcasts with very small headers to keep it as simple as possible

@b0rk Years ago, having a lot of fun.

Also, fun exercise if you want to understand TCP, is to fun to implement it on top of UDP. :)

@b0rk multiplayer video games!
@b0rk Two part answer: I worked on Apple Remote Desktop and we had two protocols: VNC with TCP by default and the devices management protocol which was purely UDP. It’s essentially a mesh network so UDP (in late 2000s) was a great boost for reducing network load on the “task servers”.

@b0rk Reliable multicast delivery, for a distributed application that ran in company datacenter. Had to get the same data to lots of different computers at the same time.

Long time ago, though.

These days LAN bandwidth is much higher, so unicast is less of an issue. Plus the 32-bit RAM limit we were dealing with that required distributed system just to fit the data in RAM doesn't apply. And CPU parallelism is more of a thing, so you can scale with way fewer machines with many more cores.

@b0rk things I've built with UDP:

- a lot of bespoke lighting systems based on ESP8266 modules, where I couldn't be bothered to write a proper ArtNet interface

- a system for doing reactive backlighting on large projector screens, similar to Philips Ambilight but on a full event stage scale

- a silly network buffer storage system where you "store" stuff by repeatedly forwarding packets

- some basic P2P stuff

- a few toy VoIP systems

- a few attempts at multiplayer game code along the years

@b0rk also, if it counts, I've written interfaces to other people's custom UDP protocols, both for personal reasons and at work. Pioneer Rekordbox (DJ software) has some custom UDP stuff for timecode and OSC and I tinkered with that for a bit.

I see custom UDP stuff a lot in industrial control systems where it's easier to deal with protocol-level retries and acknowledgement in exchange for stateless multicast/broadcast capability than it is to deal with stateful unicast, for various reasons.

@gsuberland @b0rk yep, at my old employer we had designed a custom UDP protocol for talking to our DALI light control devices. It's easier to implement a simple UDP via multicast where the recipient just sends back an acknowledgement packet, especially if you want to connect to devices where physical access is difficult (stuff built in the ceiling for example) and have a "backdoor" when the customer messes up their config.

@b0rk We were building digital ink: hardware pen streaming points sampled at 10ms. When it was working as a pen, we streamed the data over TCP. Custom chips, hardware and software.

We then used the same basic hardware as a realtime pointing device. We were no longer interested in a perfect stream of samples, and the realtime ‘where’ of the device was much more useful. We switched the same protocol from TCP to UDP: “better never than late.”

RFC 5389: Session Traversal Utilities for NAT (STUN)

Session Traversal Utilities for NAT (STUN) is a protocol that serves as a tool for other protocols in dealing with Network Address Translator (NAT) traversal. It can be used by an endpoint to determine the IP address and port allocated to it by a NAT. It can also be used to check connectivity between two endpoints, and as a keep-alive protocol to maintain NAT bindings. STUN works with many existing NATs, and does not require any special behavior from them. STUN is not a NAT traversal solution by itself. Rather, it is a tool to be used in the context of a NAT traversal solution. This is an important change from the previous version of this specification (RFC 3489), which presented STUN as a complete solution. This document obsoletes RFC 3489. [STANDARDS-TRACK]

IETF Datatracker

@b0rk for a project where I needed to stream and re-embed virtual framebuffers and commands into a sort of encoding / streaming / rendering farm.

it started just piping things through socat over raw udp, and having a sidecar message queue, but we needed to eventually add some metadata on top of the stream for timestamps, sequences, and some encoding hints into "frames".

it was a wonky enough and deeply integrated setup that nothing normal quite fit the bill.

@b0rk Google home and and sonos stuff! (Not me, but prob the most common example of udp in a household nowadays)
https://support.google.com/chrome/a/answer/12256492?hl=en
Network requirements for cast moderator - Chrome Enterprise and Education Help

Applies to managed ChromeOS devices or Windows and macOS users who use managed Chrome browser on M108 or later. This page is for administrators. If you are a user and want to learn more, see Cast usi

@b0rk logs/stats. Lots of small events, delivery can be made more reliable by sending to localhost, but I *really* don’t want telemetry to become a bottleneck itself
@b0rk Loooong time ago, a network layer replacement when porting games from Windows to Mac.

@b0rk I have only used UDP in situations where I’m either sending very sparse packets of data on a timer say and I actually don’t care if any one particular one makes it, or when doing time sensitive data over a LAN as in teleoperation.

Anything else and you risk being a bad citizen by ignoring the TCP exponential back off algorithm. I strongly advise against using UDP unless you have a fixed data rate and you really care about not building a backlog of packets.

@b0rk if you have latency concerns, try TCP but turn on TCP_NODELAY first and see how that treats you

@unsaturated @b0rk

I didn't write this code, so I didn't count it in the poll, but the software I currently maintain uses UDP in local network broadcast to query for the hardware devices it could connect to.

This is another pretty good use case as it doesn't need to establish a connection.

As it's only when the GUI has been commanded to connect to something, and polls slowly even then, one doesn't need to worry about saturating the network, backoff, or similar issues.

@b0rk Many years ago for fun I used UDP multicast on a LAN to signal multiple computers. They all ran a little Java program that would beep the PC speaker to try to play a polyphonic melody together.

It was just a crazy project just to play with multicast, and would quickly get out of sync (no major attempt was made to correct for that).

IIRC the packets were mostly for timing, they didn’t contain the melody.

Also learned a lot thunking to 16 bit Windows PC speaker calls from Java too.

@b0rk IPX to UDP gateway, to interface with old (old) equipment. IPX is also a packet-oriented protocol, so bridging it to UDP kept the same semantics.

Also a health and heartbeat protocol, where dropped packets are useful as a metric. Neither are still in use any more.

Oh, and a high speed camera protocol using jumbo UDP packets over a 10G link, from a custom scientific grade system. That one is likely still in use.

@b0rk I once contributed to a university project where they wanted to use high-powered lasers for projecting a sort of digital firework over the city. They got the project approved but only under the condition that they take highest care for installing a high-quality safety sensor (which would detect if the laser was interrupted and shut it down ASAP). Part of that installation was a custom fiber link over which they needed a common network protocol with minimum overhead and latency, hence UDP
@b0rk Peer to peer protocols. :)
@b0rk saying I "designed" it is maybe a stretch but I set up UDP hole punching for a peer to peer messenger I was programming. I think the actual messaging part used TCP. this was years ago so the details are hazy
@b0rk Discovery, notifications and presence in cluster software
@b0rk e.g. serial port data bridge, to connect serial buses in multiple locations to behave as if there was a cable connecting them. UDP to avoid latency due to TCP resends; the serial protocols handle resends as needed.
@b0rk Real-time multiplayer computer games 😁

@b0rk For a device to let a one-way flow of information through an air gap, using commodity components with a simplex network connection.

In many ways it's similar to multicast file transfer, you can't use TCP or even ARP.

@b0rk Multi cast transmission of live radar data over 56K modem connections. As you can imagine, compression was essential, and there was no back channel.

I should write it up sometime, though it was niche, and not sure anyone would be interested.

@b0rk Basically two machines exchanging applicative states periodically
@b0rk recording normalised backtesting/model debugging data from a "distributed" (100s of independent copies of the same service) ML inference service.

@b0rk WLED is an open source controller for WS2812 and alike addressable LEDs (strips), running on ESP8266 and ESP32.
It can use UDP to sync multiple devices for bigger lightshows

https://kno.wled.ge/features/settings/ (scroll down to 'Sync settings')

Settings - WLED Project

@b0rk In all three of my professional lives: UDP, and especially UDP-multicast, is very heavily used in robotics; Raw UDP is ubiquitous in the low levels of telecom stacks; And UDP can P2P-traverse a lot of NAT equipment more pleasantly than other approaches.
More generally: TCP replaces packet loss with long delays; any time that a sufficiently delayed message is worthless, UDP is going to shine. That includes all forms of telemetry and most streaming.

@b0rk super simple exchange of a few bytes from application servers to a daemon which then uses the info gathered this way to talk over TCP to another service.

UDP in this case was chosen because low latency and zero wait time was preferred over guaranteed delivery

@b0rk Mostly CTF challenges actually. But also a small audio streaming thing.

@b0rk old game networking, where you don’t care about old data, but just need the latest stuff as fast and cheaply as possible. It’s easier to design your own reliability requirements.

For this, old data can also be incorporated into new data: For example, instead of resending “0:x+=4”, “1:x+=1”, and “2:x+=3” until you acknowledge you got them, you can just send “0-2:x+=8” on the next update, accumulating all updates since the last ack.

@b0rk 30 years ago, when I first learned (about) these protocols, I wrote a bunch of toy network programs - things like my own implementation of talk(1), stuff like that.
@b0rk
(Didn't design it, did an independent implementation and a lot of packet level debugging - and I didn't see any other multicast examples) LCM is a multicast messaging system used in high speed robotics - it's UDP primarily because it is multicast; it has different message types for longer multipacket messages.
@b0rk at $PREVIOUS_JOB, part of the code deployment system I built included a multicast system for notifying nodes of events, which (since it used IP multicast) was UDP-based.
@b0rk I worked on the core network for a walkie-talkie system used in emergency situations (forest fires monitoring, power grid intervention forces, that kind of thing) where any one of the base antennas could be cut from the network at anytime, be reconnected later, etc. All based on multicast and udp, but then, some network providers wouldn't do multicast, so we had a version with tcp and a centralized server relaying the data to all nodes as well.
@b0rk part of it was standard protocols (rtp for sending the audio, igmp for multicast), and part of it was custom (signalling protocol to initiate a conversation, call someone, hang up, etc). The very decentralized system based on multicast, and resilience constraints, left us with not many choices of existing protocols
@b0rk The ability of UDP to broadcast to all computers on a local network is handy. As part of a GUI for an industrial robot, I wrote a simple UDP protocol to discover all robots on the local network. The GUI would use this information to allow connecting quickly to the robot without typing its IP address.
@b0rk what does "used HTTP to do handshaking because it has convenient headers, then used the request and response bodies as a bidirectional channel for a different protocol" count for? :')
@iximeow @b0rk have you considered using WebSocket? Was it too onerous? (Serious question because a few of the WS features are a PITA if I recall)

@unsaturated @b0rk i've remarked that what i ended up with is basically "websockets but bad", it's true!

in my case the data in headers is very different from data in bodies, to the point that if it were over TCP or WS i'd have probably have different connections for the two to make the data processing context very clear. otherwise either would be totally fine instead, yeah

@b0rk notably the one and only time I used UDP directly was for a video game project I worked on with a friend in highschool and we didn't handle the unreliability of the underlying protocol at all. It wasn't until after we submitted the project and attempted to play on real network conditions did we realize that the game state between clients became desynchronized! (We still got a good grade on it though since it was nonetheless an impressive attempt for a couple of 11th graders)

@b0rk How strictly should one interpret "designed a protocol"?

E.g. When I was in grad school and not good at software engineering, I communicated between some programs I was writing with what was effectively a bunch of length-prefixed arrays.

I wouldn't put it on my resume, but it fits the minimum definition of a protocol.

@lynndotpy I think anything that seems like it could conceivably be useful in a real-life application counts!