poll: have you ever designed a protocol that uses TCP or UDP directly? (vs using protocols that are based on TCP/UDP like DNS, HTTP, SSH, etc)
(you can pick more than one)
poll: have you ever designed a protocol that uses TCP or UDP directly? (vs using protocols that are based on TCP/UDP like DNS, HTTP, SSH, etc)
(you can pick more than one)
I'm so curious about what the 30% of you who are designing custom UDP-based protocols are doing, I would love to hear more!
(please only reply if you responded “yes, UDP” to the above poll)
@b0rk I've written 2ping, a utility which can determine directional packet loss. Has its own UDP protocol (IANA-registered even! 15998/udp)
https://www.finnie.org/software/2ping/
https://github.com/rfinnie/2ping/blob/main/doc/2ping-protocol.md
I voted "yes udp" for a very similar microcontroller based project, though the opposite direction - LED 8-segment sign that you blasted UDP at to update. It was two-digit, so was just a single byte in a UDP packet. I didn't have a proper network stack at all, just an atmega with some ethernet adapter.
@b0rk Years ago, having a lot of fun.
Also, fun exercise if you want to understand TCP, is to fun to implement it on top of UDP. :)
@b0rk Reliable multicast delivery, for a distributed application that ran in company datacenter. Had to get the same data to lots of different computers at the same time.
Long time ago, though.
These days LAN bandwidth is much higher, so unicast is less of an issue. Plus the 32-bit RAM limit we were dealing with that required distributed system just to fit the data in RAM doesn't apply. And CPU parallelism is more of a thing, so you can scale with way fewer machines with many more cores.
@b0rk things I've built with UDP:
- a lot of bespoke lighting systems based on ESP8266 modules, where I couldn't be bothered to write a proper ArtNet interface
- a system for doing reactive backlighting on large projector screens, similar to Philips Ambilight but on a full event stage scale
- a silly network buffer storage system where you "store" stuff by repeatedly forwarding packets
- some basic P2P stuff
- a few toy VoIP systems
- a few attempts at multiplayer game code along the years
@b0rk also, if it counts, I've written interfaces to other people's custom UDP protocols, both for personal reasons and at work. Pioneer Rekordbox (DJ software) has some custom UDP stuff for timecode and OSC and I tinkered with that for a bit.
I see custom UDP stuff a lot in industrial control systems where it's easier to deal with protocol-level retries and acknowledgement in exchange for stateless multicast/broadcast capability than it is to deal with stateful unicast, for various reasons.
@b0rk We were building digital ink: hardware pen streaming points sampled at 10ms. When it was working as a pen, we streamed the data over TCP. Custom chips, hardware and software.
We then used the same basic hardware as a realtime pointing device. We were no longer interested in a perfect stream of samples, and the realtime ‘where’ of the device was much more useful. We switched the same protocol from TCP to UDP: “better never than late.”
Session Traversal Utilities for NAT (STUN) is a protocol that serves as a tool for other protocols in dealing with Network Address Translator (NAT) traversal. It can be used by an endpoint to determine the IP address and port allocated to it by a NAT. It can also be used to check connectivity between two endpoints, and as a keep-alive protocol to maintain NAT bindings. STUN works with many existing NATs, and does not require any special behavior from them. STUN is not a NAT traversal solution by itself. Rather, it is a tool to be used in the context of a NAT traversal solution. This is an important change from the previous version of this specification (RFC 3489), which presented STUN as a complete solution. This document obsoletes RFC 3489. [STANDARDS-TRACK]
@b0rk for a project where I needed to stream and re-embed virtual framebuffers and commands into a sort of encoding / streaming / rendering farm.
it started just piping things through socat over raw udp, and having a sidecar message queue, but we needed to eventually add some metadata on top of the stream for timestamps, sequences, and some encoding hints into "frames".
it was a wonky enough and deeply integrated setup that nothing normal quite fit the bill.
@b0rk I have only used UDP in situations where I’m either sending very sparse packets of data on a timer say and I actually don’t care if any one particular one makes it, or when doing time sensitive data over a LAN as in teleoperation.
Anything else and you risk being a bad citizen by ignoring the TCP exponential back off algorithm. I strongly advise against using UDP unless you have a fixed data rate and you really care about not building a backlog of packets.
I didn't write this code, so I didn't count it in the poll, but the software I currently maintain uses UDP in local network broadcast to query for the hardware devices it could connect to.
This is another pretty good use case as it doesn't need to establish a connection.
As it's only when the GUI has been commanded to connect to something, and polls slowly even then, one doesn't need to worry about saturating the network, backoff, or similar issues.
@b0rk Many years ago for fun I used UDP multicast on a LAN to signal multiple computers. They all ran a little Java program that would beep the PC speaker to try to play a polyphonic melody together.
It was just a crazy project just to play with multicast, and would quickly get out of sync (no major attempt was made to correct for that).
IIRC the packets were mostly for timing, they didn’t contain the melody.
Also learned a lot thunking to 16 bit Windows PC speaker calls from Java too.
@b0rk IPX to UDP gateway, to interface with old (old) equipment. IPX is also a packet-oriented protocol, so bridging it to UDP kept the same semantics.
Also a health and heartbeat protocol, where dropped packets are useful as a metric. Neither are still in use any more.
Oh, and a high speed camera protocol using jumbo UDP packets over a 10G link, from a custom scientific grade system. That one is likely still in use.
@b0rk For a device to let a one-way flow of information through an air gap, using commodity components with a simplex network connection.
In many ways it's similar to multicast file transfer, you can't use TCP or even ARP.
@b0rk Multi cast transmission of live radar data over 56K modem connections. As you can imagine, compression was essential, and there was no back channel.
I should write it up sometime, though it was niche, and not sure anyone would be interested.
@b0rk WLED is an open source controller for WS2812 and alike addressable LEDs (strips), running on ESP8266 and ESP32.
It can use UDP to sync multiple devices for bigger lightshows
https://kno.wled.ge/features/settings/ (scroll down to 'Sync settings')
@b0rk super simple exchange of a few bytes from application servers to a daemon which then uses the info gathered this way to talk over TCP to another service.
UDP in this case was chosen because low latency and zero wait time was preferred over guaranteed delivery
@b0rk old game networking, where you don’t care about old data, but just need the latest stuff as fast and cheaply as possible. It’s easier to design your own reliability requirements.
For this, old data can also be incorporated into new data: For example, instead of resending “0:x+=4”, “1:x+=1”, and “2:x+=3” until you acknowledge you got them, you can just send “0-2:x+=8” on the next update, accumulating all updates since the last ack.
@unsaturated @b0rk i've remarked that what i ended up with is basically "websockets but bad", it's true!
in my case the data in headers is very different from data in bodies, to the point that if it were over TCP or WS i'd have probably have different connections for the two to make the data processing context very clear. otherwise either would be totally fine instead, yeah
@b0rk How strictly should one interpret "designed a protocol"?
E.g. When I was in grad school and not good at software engineering, I communicated between some programs I was writing with what was effectively a bunch of length-prefixed arrays.
I wouldn't put it on my resume, but it fits the minimum definition of a protocol.