Mastodawn

@nineofseven
Hello! I also use #Yacy to crawl Mastodon. But the results are mixed. Sometimes none of the words are indexed because of Mastodon's Javascript.

My solution has been to crawl the RSS feeds instead of the posts. A scheduled crawl checks for new content every hour, so I can get fresh #search results.

klokanek Feb 23

@Earl @nineofseven

Maybe, this piece could be of some use:
https://codeberg.org/chrlns/linkfed/src/branch/main/linkfed.sh
(bash script to extract URLs from #Mastodon for feeding #YaCy crawler)

linkfed/linkfed.sh at main

linkfed - Script for retrieving links from Mastodon

Codeberg.org

@klokanek @Earl fantastic... just comes handy...