Lennart PoetteringMar 2

2️⃣ Here's the 2nd post highlighting key new features of the upcoming v260 release of systemd. #systemd260 #systemd

systemd is adopting the IPC system in more and more places. One nice thing about Varlink is that it is conceptually close to HTTP, in its request/response semantics. This is on purpose to make bridging the web world and the low-level system world easy.

@michaelvogt has beeing working on a Varlink/HTTP bridge in Rust, with various bells and whistles, including…

Show threadLennart PoetteringMar 2

…the ability to authenticate connections via classic SSH keys (mutual TLS is supported too of course).

The bridge can be used to expose local Varlink services to remote peers via HTTP, opening up those services to the HTTP world. But sometimes the other direction is useful too: i.e. being able to access remote Varlink services exposed via HTTP via sd-varlink + varlinkctl. With v260 that's easy to do: if you specify a Varlink service not via an AF_UNIX socket path, but via an HTTP URL…

Show threadLennart PoetteringMar 2

…then sd-varlink/varlinkctl will fork off /usr/lib/systemd/varlink-bridges/http (and analogue for https, or any other protocol scheme), and communicate through it.

In fact, Michael's bridge implements this client side too, and hence after installing it, you can do stuff such as:

varlinkctl call httqs://my.web-server.com:8080/ws/sockets/io.systemd.Hostname io.systemd.Hostname.Describe {}

and the right thing will happen. (I replaced a p with a q in the above, to trick Mastodon's formatting.)

Show threadLennart PoetteringMar 2
Michael's bridge is hosted here: https://github.com/systemd/varlink-http-bridge
Show threadArian
@pid_eins i think websocket is overkill here. Should use SSE instead as varlink streaming isn't bidirectional. Varlink is practically the same as Model Context Protocol which is funny as you can now market Systemd as AI native. They also switched to SSE for simplicity
Mar 2 at 8:02amWeb
Show threadLennart PoetteringMar 2
@arianvp you can recycle varlink connections, and it#s a good idea to do that.
Show threadArianMar 2
@pid_eins yes but http2 already takes care of recycling connections. You don't need websockets for that
Show threadLennart PoetteringMar 2
@arianvp nah, it's nice if the varlink connection after the http/websocket parts survives multiple calls, for Accept=yes varlink services.
Show threadArianMar 2

@pid_eins

the `more` support and keeping connections open can easily be implemented with just native HTTP without having to switch to websockets. Doesn't even have to be SSE. Can just be a response that streams its results and splits the stream by NUL byte.

The varlink semantics of queuing multiple requests and the need for responses to come back into the same order as requests makes it behave basically the same semantics as HTTP 1.1

Map each HTTP path to a varlink method call. And map the parameters to the request body

Then simply stream back the reponse of the varlink call back over the http connection. If the varlink response streams multiple answers, just keep the http connection open

Here is an example that reuses the same varlink connection over multiple HTTP calls; works fine!

I really don't see the need for websockets here

Here's a demo in a few lines of go:

https://gist.github.com/arianvp/314113d4309c7d7d3047535e167352ac

Varlink HTTP

Varlink HTTP. GitHub Gist: instantly share code, notes, and snippets.

Gist
Show threadLennart PoetteringMar 2

@arianvp so the bridge supports both the ws and the non-ws mode iirc. @michaelvogt please comment.

(the other reason why ws is sometimes nice, is that varlink knows a concept of upgrading protocol, i.e. just being something to set up a connection and then switching to something else)

Show threadMarco d'Itri Mar 4
@pid_eins @arianvp @michaelvogt websockets should be avoided: they are hard to proxy, impossible to inspect by a WAF and often not implemented by HTTP/3.
Show threadMichael VogtMar 4
@rfc1036 @pid_eins @arianvp Thanks for the feedback! We have a websocket and non-websocket mode currently in the bridge (and no-websock is much nicer with e.g. curl). Its just that varlink map extremely nicely to websockets, especially things like the protocol switching in varlink. I will look harder into this though, the "more" streaming looks pretty simple in plain http and that should cover most use-cases of today.
Show threadMichael VogtMar 11
@rfc1036 @pid_eins @arianvp Thanks again @arianvp for the very nice gist - I created https://github.com/systemd/varlink-http-bridge/pull/13 that should implement this "--more" streaming via json-seq and connection pipelining in the bridge now without websockets- really cool to get these kind of improvement suggestion so quickly/nicely.
Show threadMichael VogtMar 2
@arianvp @pid_eins Thanks Arian for the gist! The main reason for the websocket stuff is/was the protocol switching that Lennarts mentioned. But of course your right that just for the --more we won't need it and can use http streaming or SSE. I expect that we will add either http stream or sse (or both) pretty soon to make it easier for traditional curl usage to call e.g. io.systemd.JournalAccess.GetEntries. But I also want to make sure we DTRT so this kind of feedback is much appreciated!