I wrote a nano tool that tries to extract the #Fediverse accounts of your #Twitter followings: https://fedifinder.glitch.me/
It searches for the patterns @[email protected], [email protected] and host.tld/@user in the screen name, description, location and URL field. It displays them to you in the correct format for easy copying as well as a CSV download that can be imported to #Mastodon.
New version: https://fedifinder-backup.glitch.me/
You can now check if the handles in your own Twitter profile are found. They are not added to the export. If there are issues with your profile, let me know.
Thanks for the idea @nelson. #twittermigration #neuhier #nieuwhier
I was able to reduce the wait time by around 40% by moving the domain checks to after the page got rendered. I fear some rate limiting issues came from people reloading before receiving a response.
False positives are kept in the visible list, but striked through. They are still removed from the exported CSV to reduce server load on imports.
Next up: Lists. They are the most requested feature. #fedifinder
Support for lists arrived in #fedifinder.
It shows you your own lists as well as the lists you are subscribed to. You can scan each of them and the found handles are added to the visible list as well as the download. (False positives of the previous scan are removed from the visible list with each additional scan.)
I made the scan of the followings optional in #fedifinder to improve speed, reduce the amount of rate limiting errors (eg. users reloading the page) and make it easier for people who only want to scan lists or maybe something else that gets added in the future.
Rate limiting errors aren't displayed as json anymore, but tell you how to fix them (wait, wait and wait).
More improvements for #fedfinder.
The results now display basic stats for each instance and if the registration is open. Thanks for the hint @pfefferle.
Better check if something is part of the #fediverse.
Data handling is still a mess, but less messy now, which results in fewer requests to the server and less jumping text.
Tiny changes on the frontend, massive ones in the backend. #fedifinder
- Option to scan the profiles of your followers
- Twiter handle where the fediverse account was found
- Handles of failed servers can be displayed
- Switched to API v2
- Pinned Tweets are included
- You can now scan up to 15k followings and 15k followers
- Better handling of rate limits
- Caching of reason of unreachability of servers (and limited amount of retries for servers that timed out)
- better regex
I am currently looking through the #fediverse software options to find more URL patterns of profiles. Here is what I have at the moment. Most of them are easy to detect. But for pleroma I saw some servers with and some without the @ in the username. Makes it harder.
domain.tld/@name
mastodon, misskey, pleroma, funkwhale
domain.tld/profile/name
friendica, hubzilla
domain.tld/u/name
diaspora
domain.tld/c/name/videos
peertube
domain.tld/name
pleroma, gnusocial
Today wasn't fun. While I was outside, I got many messages that #FediFinder didn't work. I checked it and saw that it was overloaded and crashed every few minutes. I then tried to fix it from my mobile, but with each line of code it seemed like I added multiple new bugs. Not good. So I added a message that its broken and to check back later.
I spent the whole afternoon and evening trying to fix the mistakes of past me and clean up the code to make it at least slightly easier in the future.
The important thing: #FediFinder is back!
While https://fedifinder.glitch.me seems to work fine again, I added two backups that you can use as well:
https://fedifinder-backup.glitch.me and https://fedifinder-backup2.glitch.me
New feature: You can now click on the usernames for the fediverse (may not always work because of different formats of software) and Twitter.
Instead of building a better export, I made the sessions persistent, so that not everyone re-logins when I change something and the server restarts.
You can now watch as #FediFinder checks the domains. And even retry if it gets stuck (should add a 10 second timeout there as well).
And I added links to the backup servers and info about donations to the homepage.
I added an API (kinda) to #FediFinder. The only endpoint is /api/known_instances.json. It returns all instances it knows at the moment.
Currently, It is used to share data with @debirdify.
I don't know if it will stay public/available and/or if the structure will change in the future.
#FediFinder Day 9(?). I moved the handle identification from the backend to the frontend to improve performance. The server is only responsible for getting the data from Twitter and checking domains. No changes for the users.
With the functions, I moved some tests to the frontend. Only visible on the staging server or the code.
If handles are detected on the user profile, I show links to their import. Thanks @nart for the idea.
JSON export (including accounts without handles).
Fixed bugs.
There are people who had problems using #fedifinder. Not everyone knows what a CSV is or where to find the import. But I have a solution: Follow buttons!
If one or more #fediverse handles are identified on the profile of the person using fedifinder, they can now toggle on follow buttons to get directly to the follow request on their own instances. This should make it much easier.
(At the moment they only work for mastodon instances. I am open to pull requests for other software.)
But Luca, not everyone is in the fediverse. The export is useless for the other accounts. And in my Twitter archive are only the IDs of the people I followed. Not their names or websites. What about them?
If you scroll to the bottom of #Fedifinder, you can now view a table of the accounts that you have scanned. Even if they have no handle in their profile. Or one that wasn't correctly detected.
I updated the ReadMe to help people run their own #Fedifinder (no coding skills needed) and how to contribute.
https://github.com/lucahammer/fedifinder
And fedifinder got a new API endpoint. api/check. You can use it with a domain and/or handle as query parameters and get back the cached information fedifinder has for that: https://fedifinder.glitch.me/api/check?handle=@[email protected]
Full speed! Thanks to the latest update, #Fedifinder scans my 2k followings within 10 seconds. From login to exported file.
The solution to a slow databases wasn't a fast database, but no database at all. Finally, I can tell you to share fedifinder with all of your contacts without fearing that the
service will go down because of it.
Go and find your people! https://fedifinder.glitch.me/
@wurzelmann I am caching the results to not send thousands of requests.
Some day, I plan to add a button to the frontend for people to request new data.
Until then, adding "&force" to this endpoint tells #fedifinder to update the data. Without force, it returns the cached one, which is up to date again: https://fedifinder.glitch.me/api/check?handle=@[email protected]
@reichenstein Following indicators are on my list. https://github.com/lucahammer/fedifinder/issues/48
I'm unsure about profile pictures because of the additional load on servers.
@onatcer Same.
(It only were the user tokens, which are useless without the app tokens, but still. Trust, responsibility. Wah.)
Ich versuche die sowieso schon stark beanspruchten Server nicht noch weiter zu belasten, daher teste ich nur auf Domain-Ebene. Also ob .well-known/nodeinfo vorhanden ist und dann was in der nodeinfo steht. Das kommt dann in die Datenbank.
Wie hier (https://amok.recoil.org/@avsm/109302706429920745) diskutiert gibt es Fälle, wo das nicht reicht. Deswegen plane ich in Zukunft noch bei Domains, wo die Abfrage nicht klappt, nach webfinger endpoint zu schauen.
@rbairwell Now I get it. Like a crowd sourced database, but with consent. Because it only links profiles that people published themselves. And the can delete the tweet to not be found anymore.
I don't think my current setup can support that, but I will add it as an idea on GitHub.
@Luca thank you so much for this very useful tool. I used it today and it worked well.
I'm not sure if it handled all 150k followers though - but it seemed to handle all my followed OK.
@jh It's an Twitter API Limit.
Two options:
- Make the server wait on rate limits (they are refreshed after 15 minutes). Here https://github.com/lucahammer/fedifinder/blob/main/server.js#L571 (that's what I plan to implement)
- Save pagination_token from the response and add it to next request. Again after 15 minutes.
https://github.com/lucahammer/fedifinder/blob/main/server.js#L648
Luca Hammer EN
Wurzelmann
Curd Knüpfer
Alexander Svensson (wortfeld)
Oliver Reichenstein
yetzt
Jörg Seidel
InI4
kerstin altendorf
Onatcer
Sandra 
Fuzzy Leapfrog
Thom Slattery
Björn 🗯️
nicolas ⁂
Davis
Farhan
Torsten Kathke
Robert Szasz
Erwin Ernst eest9 Steinhammer
Cornelius Roemer
Richard Bairwell
Jeremy Howard