David ThielDec 19, 2022

I wrote a bit about an effect that I'd seen for a while but had difficulty explaining: we call it "Content Moderation Survivor Bias", and it's an effect that can muck up social media analyses and lead to dubious conclusions.

I define it thusly: in a retrospective sample of moderated social media platform, ToS-violating or inauthentic content tends to appear most prevalent in the immediate past. This appearance is misleading, however.

https://cyber.fsi.stanford.edu/io/news/content-moderation-survivor-bias

Content Moderation Survivor Bias

Show threadDavid Thiel
This is because content enforcement is not typically immediate — it can take days, weeks or longer. As you search back in time from the present, it becomes more likely content has been altered or removed. So you're drawing a conclusion on data whose completeness is not evenly distributed over time. This (among other things) led to some speculation regarding spam people found during recent COVID protests in major Chinese cities.
Dec 19, 2022 at 12:49pmWeb
Show threadDavid ThielDec 19, 2022
It appeared that adult spam using Chinese city names as hashtags was suddenly surging as the protests took off — making people suspect the CCP was trying to drown out information about the protests. Not initially implausible, but there are a number of problems with that assumption — for one, look at this plot showing volume of tweets mentioning these cities (which were overwhelmingly spam):
Show threadDavid ThielDec 19, 2022
The graph above was from a search conducted on Nov 29th. It sure looks like tweets really took off on the 27th, increasing toward the present, right? That's the quick conclusion you would draw if you looked at just the previous few days. Problem is, this isn't enough data, and it's biased. Here's another one week query conducted after the protests had wrapped up:
Show threadDavid ThielDec 19, 2022
This was conducted Dec 4th. Sure looks like...an increase in the past couple days, again. This is content moderation survivor bias in action, and if you did a similar search today, you'd probably find the same thing. Conversely, if you searched one of the original weeks today, you'd see the curve flatten partially. There are a few other problems that also led to these likely mistaken conclusions:
Show threadDavid ThielDec 19, 2022
One is just plain recency illusion. Most people weren't searching for these hashtags before, so the surge *seemed* sudden to them. Secondly, people didn't go far back enough. Twitter's API isn't blazing fast and has rate limits as well as query caps when using elevated API access. Gathering enough data to make strong conclusions isn't free or fast. If you expand that initial search to go back 2 weeks instead of one, a different picture emerges:
Show threadDavid ThielDec 19, 2022
Third, social media analysis and studies of CIB have often focused on state-backed actions, so we've kind of conditioned ourselves to attribute shady behavior to the governments of Russia or China. This is not entirely unreasonable given the inauthentic online operations those countries have engaged in, but it is a bias. And lastly, journalists and even academics want to publish fast about emerging events. Alas, sometimes we should be slow and as correct as possible, given messy data.
Show threadDavid ThielDec 19, 2022
So here I am, weeks later with the boring and un-splashy conclusion that sometimes a cigar is just a cigar. I'm of course not saying the Chinese government were sitting on their hands during this time, just that this particular spam is very unlikely theirs. Twitter would be better placed to make attribution here, but all lines of communication to collaborate with their Trust and Safety teams have been severed. Even stopping the spam is beyond their current capacity.
Show threadDavid ThielDec 19, 2022
Anyway, I don't want to sound like a scold to other researchers about this — I've been wrong about CIB before. I drew a totally wrong conclusion about malware in a report due to misunderstanding RiskIQ. I mistook a picture of Barry Goldwater as being a GAN. And I fell for this survivor bias multiple times before putting my finger on it. So, I hope this helps other people not trip up in the future.
Show threadDavid ThielDec 19, 2022

Though they don't really focus on the data bias issue, reporters at the NYT actually conducted a separate analysis and came to the same conclusion — and you can probably do so yourself if you have academic API access!

https://www.nytimes.com/interactive/2022/12/19/technology/twitter-bots-china-protests-elon-musk.html

How Twitter Bots Drowned Out Posts About Protests in China

Chinese protesters shared videos and photos to Twitter. But their posts were swarmed by escort and gambling ads.

The New York Times
Show threadDavid ThielDec 19, 2022
This is of course the straightforward approach, I just didn't want to pass "can I chat with an escort service" by the IRB
Show threadAlex StamosDec 19, 2022
@det I appreciate your discretion.