Mastodawn

Peter Adamson Jul 5, 2023

I have some insider knowledge here that I wanted to share.

This is not happening because Google scrapes Twitter and is now unable to. Google has been a paying customer (with a special negotiated rate) of the Firehose API for nearly a decade. Presumably, that deal was still in effect, barring API rate changes having an impact.

So this decision is solely because the results can no longer be viewed by non-logged-in users.

https://universeodon.com/@TomWellborn/110657677903620346

Tom Hates Fascism! :verified: (@[email protected])

Attached: 1 image LOL

Universeodon Social Media

MJBotte1 Jul 4, 2023

@rodhilton so everything is still Elon’s fault? Checks out

Rod Hilton Jul 4, 2023

@MJBotte1 it's possible when he raised the rates it affected Google and they switched to scraping but I really doubt it.

They may have threatened to scrape in the face of higher rates.

I theorized (unsubstantiated) that the API access was used as a cudgel as part of the GCP negotiations - contract was set to expire June 30 and he implemented these login & rate limiting the next day.

Raise rates on Google access, then offer a discount in exchange for GCP discount. Just a theory though.

Merovius Jul 5, 2023

@rodhilton @MJBotte1 I think the point was that the decision to no longer make Twitter viewable by non-logged-in users was still made by Musk. So even if it has nothing to do with scraping and Google still uses the API and only removed the results because it would be user-unfriendly - it would still be Musk who is to blame.

Kenneth Jul 4, 2023

@rodhilton while google has/had firehose access, they’ve always still scraped the site. There was fairly complex infra to allow that.

Rod Hilton Jul 4, 2023

@kpk hmm, I wasn't aware of that. They continued web scraping through the new web ui launch too? Why did they have Firehose access if they were just scraping anyway?

Kenneth Jul 4, 2023

@rodhilton yup. I asked that question a really long time ago and folks shrugged and said “different teams”. I think that’s fair, they’d have needed to do some transformation and stuff.

Kenneth Jul 4, 2023

@rodhilton there was also a point where they stopped paying for the firehose for a bit. It must have been really hard to work out the pricing on it.

matdevdug Jul 4, 2023

@rodhilton I mean it makes sense. The point of google search is to return information. If you cannot see that information without going through a login process that also limits how many posts you can see, it stops making sense.

Dreamwinder Jul 4, 2023

@rodhilton So why does google provide so many Pinterest links in image search? That’s also required login.

Rod Hilton Jul 4, 2023

@dreamwinder that's a great question. News sites too, most of them are paywalled now.

This is one of the reasons why I theorize that it's part of negotiations falling through, to reduce Musk's leverage.

Jérôme Jul 4, 2023

@dreamwinder @rodhilton just switched to private mode and looked for Pinterest results. They don’t require login. You can browse boards entirely to the bottom.

Marty Jul 5, 2023

@rodhilton I can't ever recall a time when I searched for something on Google and ended up linking to a tweet, but... being essentially invisible to Google search seems *really bad* for a site as prominent as Twitter.

Jeff Grigg Jul 5, 2023

Still sounds to me like Google did not do anything to twitter. Rather, twitter locked up all the pages behind a login requirement, and so Google dropped the data, basically, at twitter's request.

Google's search is only supposed to show you publicly accessible pages. Lock your pages; google stops sending people to you, as they won't be able to read your content without creating a login.

Pusher Of Pixels (old account)Jul 5, 2023

@rodhilton would Google simply delist pages it can't verify it can crawl?

Having so many non functional links in results looks bad for their product. Given the public policy change from the Twit, seems reasonable to cull the links just to ensure they aren't serving things ppl can't read.

Rob Landley Jul 5, 2023

@rodhilton The ratelimiting happened at 9am eastern time of the 1st of the month because Twitter owed $42 million for Google cloud services and refused to pay the bill.

https://mstdn.jp/@landley/110653445043472923

David Pollak Jul 5, 2023

@rodhilton and this is how GOOG drives down the price of Twitter so they can buy Twitter for maybe 10% of what musk paid… and maybe escape anti-trust scrutiny in the process…

@dpp @rodhilton yo then add it to the list of Things Google Killed? I’m sure they’d love a social property to put ads on but I doubt a dead Twitter would be it.