Jake Huennekens@robb Hey Robb! Could you explain how allowing RSS/read-later apps to continue accessing a site like MacStories will prevent bot scraping? Besides just saying they won’t, is there anything stopping Perplexshitty, or others, from scrapping that feed?
Robb Knight@jhuenne RSS would be governed by the same rules as accessing anywhere else on the site so if it has the user agent it would be blocked. Yes there are ways around that (like we're seeing with Perplexity) but we're just doing what we're able to right now.
@robb I got it! So, is this the best option besides putting everything behind a paywall of some kind? Is it possible to require an email or something to access an RSS feed to ensure that whoever is accessing the feed is real?
I know that still doesn’t guarantee a real person; just using it as an example. Kinda like Patreon podcast feeds, I think? I’m a nerd but not a web developer, so kinda out of my depth lol
@jhuenne Yes if someone wanted to make sure that AI scrapers couldn't get to content, a paywall like Patreon or similar would be the solution. I certainly wouldn't want to do that and have no intention to (and I don't speak for MacStories but I'm sure they don't want to do that either).
@robb Totally get that. In my example, I was thinking, hypothetically, that a site like yours could have a basic RSS reader tier of membership that’s free and only requires an email verification for people to access the full feed.
Again, this solution sucks; just interested to hear if it’s a possible solution. I’m guessing this cuts against the idea of the open web, but I just hate watching AI companies steal from my para-social friends