We've been busy implementing p2panda's high-level Node API!
It's an opiniated - but also much easier to use, out-of-the box p2panda stack: the thing which orchestrates all of our individual modules without all the footguns you might encounter when trying to assemble them yourself.
Example:
let node = p2panda::spawn().await?;
let (tx, rx) = node.stream(topic).await?;
This does networking, discovery, bootstrap, sync, event streaming, ordering, storage and more! Let us tell you more about it:
p2panda is designed with a clear separation of the "event delivery" and "event processing" layers. p2panda applications should not worry about _where_ their messages come from but only how they are processed.
We make sure that other nodes who are interested in the same topic are confidentially discovered, to sync up on all missed messages, to deliver them to the processing layer.
This gives them flexibility to run on top of any infrastructure. We also call this a "Walkaway Stack".
So far we do this over the Internet using iroh to manage direct p2p connections, but we're prepared to offer the same level of abstraction for running over Radio-Protocols like LoRa or BLE, including mesh topologies.
Application developer will mostly use the API to monitor and inspect networking and sync activity, restrict access and select available transports (QUIC, Tor, Mesh Networks, BLE, ..) while we make sure data is delivered as securely and privately as possible.
Any message you send is delivered on top of our "Operation" data-type. That's an single-writer append-only log with some cool features: You can have as many logs as you want, you can remove any past entries in the log (pruning) and it's robust to forks. It's a simple CRDT actually if you look closer!
With append-only logs we are prepared to offer the same functionality over radio-based meshes as sync is cheap (state vector is exchanged in one message, constant-time) and broadcast-only friendly.
We can compose additional "system-level" features on top of Operations, by extending the header with information to help garbage collection / pruning, tombstones to delete multiple logs, vector clocks for causal ordering of messages for multi-writer documents, data to aid key agreement for group encryption and more!
It would also be possible to add existing solutions, like Willow's Meadowcap or UCAN, to check against permissions with certificate capabilities.
Developers generate a Topic (random bytes) to publish messages into it and subscribe to for receiving everyone else's messages. This is PubSub!
Topics are a way to group your data, a topic is for example a Reflection text-document, or a single chess game session, or a chat room.
Later we also want to divide the space up even more into human-readable strings you can then filter with wildcards (*).
This is partial replication: You only sync the data you need.
Now we have a system where we publish, sync and persist operations grouped by topics on top of any p2p network.
With this data being available to us now we want to make sure to correctly process and stream it towards the application layer. For this we're borrowing many conceps from general "event streaming" theory, you might know this from NATS JetStream, Kafka or something like that, with the difference that we're not aiming for "big data centres" but decentralised and autonomous p2p networks.
When being subscribed to a topic you have something like a stateful "stream consumer" now.
Every operation which runs through that stream gets processed, first on an internal system-layer where we check the log-integrity, prune the log on demand, order operations causally and later we'll also decrypt them here.
After this we're forwarding the message to your application-layer, with a bunch of meta data and debugging info attached.
The application-layer can continue processing the message delivered by this operation, now based on your own application logic. Often it'll be something like "move the pawn to field E4", "change the title of the document" or a chat message, or a Text CRDT etc.
Finally we want to acknowledge the operation. This will mark the operation internally as "acked" and next time we subscribe to the same stream we will not receive the message anymore. It's something like a "read receipt".
By default we're automatically acking messages from the stream for you, since this is a more advanced feature, but why is this important?
Imagine your application crashes mid-way or you use it on a mobile phone where it got closed while it was processing something. If we didn't have a chance to ack the message it'll be re-played next time when the app opens again, giving us a chance to make sure we've processed every message at least once.
Re-plays are also cool for rolling out updates or fixes btw! We can re-play everything or move the cursor back to a point where we need to re-materialise our application's state because something went wrong in the past.
Okay, enough about event streaming, you get the idea now that p2panda's high-level API is actually something like a feature-complete event streaming system, but with an eventual consistency guarantee and causal ordering.
You can follow the work here where mostly already is implemented:
https://github.com/p2panda/p2panda/tree/main/p2panda
There's not much left for a release, except of the final touches on the acknowledgments and more documentation.
Of course from this point on we'll continue still, there's more to do.
We're getting closer to a final overview of what p2panda is and how our APIs look like, where we draw boundaries with all our separate building blocks and so on.
All of this is going towards a first release candidate for p2panda v1.0.0!
So, what's left for the roadmap?
Mostly it's last touches to our data formats, sync protocol and API design, naming of methods, concepts, clarifying semantics around error handling, failed streams and so on.
All of this would mean that we can at least settle on a "core functionality" around event delivery and event processing.
This requires us to let the system "simmer" a bit, have some time other developers use it and give us more feedback, we're learning slowly about p2panda from using it mostly.
After defining this core concepts and APIs we really need to integrate our group encryption!
We've been doing a lot of work on our offline-first, decentralised continuous group encryption scheme (p2panda-encryption). Combined with group management (p2panda-auth, renamed soon to p2panda-groups) we've built p2panda-spaces last year.
There's a lot about it written here:
We want to integrate p2panda-spaces into the high-level API so you can conveniently add / remove members to groups which will allow us to encrypt everything towards the group's encryption secret, probably again scoped around a topic.
Having a key agreement system like this protects your messages when relaying them across intermediaries, like "support nodes" or in mesh networks, with Post-Compromise Security and Forward Secrecy!
Felicitas Pojtinger 🌅
p2panda
