Rod HiltonJul 4, 2023

I have some insider knowledge here that I wanted to share.

This is not happening because Google scrapes Twitter and is now unable to. Google has been a paying customer (with a special negotiated rate) of the Firehose API for nearly a decade. Presumably, that deal was still in effect, barring API rate changes having an impact.

So this decision is solely because the results can no longer be viewed by non-logged-in users.

https://universeodon.com/@TomWellborn/110657677903620346

Tom Hates Fascism! :verified: (@[email protected])

Attached: 1 image LOL

Universeodon Social Media
Show threadKennethJul 4, 2023
@rodhilton while google has/had firehose access, they’ve always still scraped the site. There was fairly complex infra to allow that.
Show threadRod Hilton
@kpk hmm, I wasn't aware of that. They continued web scraping through the new web ui launch too? Why did they have Firehose access if they were just scraping anyway?
Jul 4, 2023 at 9:53pmTusky
Show threadKennethJul 4, 2023
@rodhilton yup. I asked that question a really long time ago and folks shrugged and said “different teams”. I think that’s fair, they’d have needed to do some transformation and stuff.
Show threadKennethJul 4, 2023
@rodhilton there was also a point where they stopped paying for the firehose for a bit. It must have been really hard to work out the pricing on it.