Trolli Schmittlauch 🦥

I'm so excited to present my work on #Fediverse-wide hashtag federation, search, and subscription at #ActivityPubConf.

The recording will be available later, but if you're interested in enabling a global consistent view on hashtags, know stuff about DHTs or #ActivityPub relays, you can have a look at the paper: https://git.orlives.de/schmittlauch/paper_hashtag_federation/src/branch/master/paper_hashtag_federation.pdf

Please contact me about any questions, remarks or other feedback!

So here's a TL;DR:

paper_hashtag_federation

Großer Beleg: Hashtag Federation

Orlives Forge: Git with a cup of Mate tea
Sep 7, 2019 at 11:24amWeb
Show threadTrolli Schmittlauch 🦥Sep 7, 2019

The problem about the current state of hashtags in the fediverse is that users have a fragmented view on posts, depending on their instance. Some posts containing a hashtag never reach your own instance, so you won't see it. This is bad for decentralisation and coordination.

I plan to throw some additional P2P stuff at the fediverse: All instances are distributing the responsibility for relaying or storing posts (just their IDs) among themselves using a Distributed Hash Table. ->

Show threadTrolli Schmittlauch 🦥Sep 7, 2019

Now we a defined entities in the network where we can subscribe to posts of a hashtag, query their history and send posts to when publishing.

I considered load balancing, redundancy and security concerns. My solutions may not be perfect, but they're the best I could have come up with.

Show threadTrolli Schmittlauch 🦥Sep 7, 2019
Btw, he slides of my #ActivityPubConf talk are available here: https://git.orlives.de/schmittlauch/paper_hashtag_federation/src/branch/master/talk-slides.pdf
paper_hashtag_federation

Großer Beleg: Hashtag Federation

Show threadJason Robinson (inactive!)Sep 7, 2019

@schmittlauch Wow, much respect for thinking about this. It's the largest problem IMHO since the beginning.

I still think the solution is separate software so that every platform does not have to implement more complex stuff than extra search REST API calls. I have not read your (long!) paper yet.

My current preferred solution is a network of index servers that share data over a p2p or similar. Platforms communicate with them over REST API's.

Show threadJason Robinson (inactive!)Sep 7, 2019
@schmittlauch the index servers wouldn't need the whole content of the posts, only 1) ID and 2) hashtags. This would allow platforms to make search requests for ID's and then they can fetch the ID's for actual content. This would make running the specialized index servers lightweight and would not introduce additional complex p2p development requirements to platforms themselves. Implementing AP alone is hard enough.
Show threadTrolli Schmittlauch 🦥Sep 8, 2019

@jaywink I also propose the separation into a transparent application proxy component, lain even suggested to implement it as a relay.
Regarding "index servers" it depends on what you mean by that: If the index servers are supposed to crawl the Fediverse themselves then good luck with keeping up with its load: At the scale of  that'd be ~140,000 posts/second. Furthermore, the indexers might not even know each server.

Thus I propose that as an opt-in instances actually push ->

Show threadTrolli Schmittlauch 🦥Sep 8, 2019

@jaywink their published post to the responsible indexing server. Though it's not an indexing server but just a relay server, which will itself forward the post to a longer-term indexing/ storage server.
For distributing the load and avoiding a central point of authority or failure, I make each server just responsible for a subset of hashtags to handle.

I'll let you know once the recording of my talk is released, in case you need an easier start than a 24 pages paper ;)

Show threadJason Robinson (inactive!)Sep 9, 2019

@schmittlauch Great, sounds good 👍 Yeah no totally didn't mean crawling, I mean push from opt-in servers, just like the current relays work (well, the diaspora one at least, not sure how the AP ones work).

Just to understand, the network of hashtag servers would support anyone hosting one as part of the network? So you don't mean that some organization is in charge of hosting it?

Show threadJason Robinson (inactive!)Sep 9, 2019
@schmittlauch Science papers are a bit difficult for me - I tend to have a really hard time focusing after a page. So looking forward to a tl/dr ;)
Show threadJason Robinson (inactive!)Sep 9, 2019

@schmittlauch BTW just reading your talk slides.

One error: the #diaspora relay allows subscribers to choose either "all" or "a list of tags". Your talk indicated only the former.

Slides look good, though most of the science speak goes over my head ;) I think the important thing to consider is that whatever the complexity of the relay/indexers/distributed nodes, the API towards implementing social platforms MUST be simple and easy to understand.

Show threadTrolli Schmittlauch 🦥Sep 21, 2019

The recording of my talk at #ActivityPubConf on Hashtag Federation is now online.
It is a nice overview of my work.

https://conf.tube/videos/watch/340eb706-28c0-4a43-9364-700297ca96cb

“Decentralised Hashtag Search and Subscription in Federated Social Networks” – Schmittlauch

PeerTube
Show threadTrolli Schmittlauch 🦥Sep 21, 2019
/cc @Gargron @kaniini @denschub @deusfigendi @nightpool @heluecht @jaywink @cj @yaaps
Show threadDeus Figendi.jwdSep 21, 2019
@schmittlauch Thanks for the notification.
Show threadseeh #UnPlugTrump #SL5auraNov 4, 2019
@schmittlauch fleissig fleissig. Weil ich zu faul bin. ? Hashtag anstatt Links? Konnte mich nie richtig daran gewöhnen. Kommt mir komisch vor. Naja Suchworte,Schlüsselworte gabs ja schon immer
Show threadmondsternJan 17, 2021
@schmittlauch schön erklärt
Show threadTrolli Schmittlauch 🦥Feb 27, 2020

I finally started implementing my #HashtagFederation relay I presented at #ActivityPubConf last year.

Feel free to follow the progress of #Hash2Pub and give me some feedback: https://socialhub.activitypub.rocks/t/decentralised-hashtag-search-and-subscription-relay-implementation-progress-report/542

#ActivityPub #Haskell 

Decentralised Hashtag Search and Subscription Relay: Implementation Progress Report

I have finally started implementing my ActivityPub relay that is going to allow subscription to hashtags and querying a more or less consistent list of posts of that hashtag. The current working title for that software is Hash2Pub. The final goal for that project is to have a component that runs alongside to the main AP server and talks to it using the existing ActivityPub relay mechanism. But all these relay instances build a fully-decentralised DHT infrastructure to manage the assignment, re...

Show threadTrolli Schmittlauch 🦥Feb 28, 2020
@lain @lain You wanted to create a SocialHub account anyway, right?
Show threadbanned for burgerpostingFeb 28, 2020
@schmittlauch very interested in all Haskell activitypub projects, will definitely be following this 👀
Show threadTrolli Schmittlauch 🦥Feb 28, 2020
@pea The ActivityPub part is one of the later implementation stages though, I need to implement the DHT used for hashtag resolution, assignment and coordination first.
Show threadAus.Social AdminSep 7, 2019
@schmittlauch yessssssssssss.
Show threadwakest likes your bugs ⁂Feb 5, 2020
@schmittlauch have you seen this @adz
Show threadwakest likes your bugs ⁂Feb 20, 2020
@schmittlauch tagging @grishka: you might want to read their proposal.
Show threadKaren E. Lund 💙💛Nov 2, 2022
@schmittlauch The techie stuff is beyond my skill set, but the consistent view of hashtags sounds great. Thank you!
Show threadTrolli Schmittlauch 🦥Nov 3, 2022
@Karen5Lund Unfortunately I haven’t had time to continue that work in the last year 😿