ProPublicaFeb 24, 2023

After a Decade of Tracking Politicians’ Deleted Tweets, Politwoops Is No More
--
Since Elon Musk took over Twitter, it has disabled the function we used to track the deleted Tweets of elected officials and political candidates — and the new method that Twitter says should identify deleted tweets appears to be broken.

Regardless of one's feelings about Twitter, this is a loss for transparency.

Thanks to @derekwillis for coming back to write this obit.

https://www.propublica.org/article/politwoops-deleted-tweets-twitter-politicians-musk

After a Decade of Tracking Politicians’ Deleted Tweets, Politwoops Is No More

Whether officials were deleting an embarrassing post or just correcting a typo, Politwoops tracked them all. But service changes made after Elon Musk’s acquisition of Twitter have rendered it impossible for us to continue tracking these tweets.

ProPublica
Show threadRickFeb 24, 2023
@ProPublica @derekwillis I feel like it should be possible to build a o tool to do this
Show threadDerek WillisFeb 24, 2023
@JetForMe @ProPublica it is possible; the problems are scale (thousands of accounts to track) and timeliness. And, not insignificantly, cost. Twitter could just run this themselves, tbh, but I understand why they don't.
Show threadRickFeb 24, 2023
@derekwillis @ProPublica 10k accounts, 10 tweets/day, 100k tweets/day, what, 1k of data per tweet, less than 500 GB over 10 years, I feel like it wouldn't more than $100/mo in cloud services, would it? I don't have any sense of the budget available for something like this, though.
Show threadDerek WillisFeb 24, 2023

@JetForMe @ProPublica it's complicated, Politwoops relies on the streaming API, which means it has to run 24/7, because in order to know which tweets have been deleted you have to have copies of them first (the notification is just the ID).

That latter bit is what's currently broken - Twitter isn't sending those notifications. So we would have to, I guess, go back and check if tweets had been deleted, which is possible but you lose time context.

Also, ppl delete stuff after a long time.

Show threadRick
@derekwillis @ProPublica Yeah, it's not pretty, but it is doable. The worst part is re-checking every tweet frequently, which scales poorly. I suppose you could check older tweets less frequently, but there's no getting around the poor scaling. Still, might be worth it for public accountability reasons. Not sure how Twitter API rate limiting would affect things.
Feb 24, 2023 at 10:13pmWeb
Show threadbasisbit 🦈🇪🇺🇺🇦 🔜 KnStFeb 25, 2023
@JetForMe @derekwillis @ProPublica You'd probably have to build your own Twitter website scraper and have that use various available open http proxy servers, or frequently spin up new virtual servers with new IP addresses to avoid Twitter's rate limits. But yes, definitely doable if you got plenty of spare time and a credit card (otherwise the cloud server providers don't want you as customer).