Sheila Dang
Yoel RothI’m deeply reluctant to wade into the TikTok ban madness, but: There are some weird edges to the Project Texas stuff that I worry will make TikTok more susceptible, not less, to malign foreign interference.
1/
Full disclosure: I don’t use TikTok, because my personal threat model actually does involve the Chinese government.
It can also be true that the moral panic about TikTok right now is overblown and driven by xenophobia.
Anyway...
2/
A key part of how companies, TikTok included, expose and disrupt coordinated manipulation is by aggregating an ENORMOUS amount of data about users and their behavior, and looking for anomalies. In infosec jargon land, we call this "centralized telemetry."
Centralized telemetry is incredibly important when you're dealing with adversarial issues, because the threat actors you're trying to find usually aren't stupid enough to leave a wide trail of evidence pointing back to them.
3/
To give a rudimentary example: If you look at posts containing a hashtag like ElectionNight2022, group those posts by the IP address they were sent from, and find that there’s a bunch of them being sent from an IP address in Russia, you might investigate the accounts responsible to see if there’s something fishy going on. (And yes, sometimes even government-sponsored trolling campaigns are this poorly-done.)
The posts, individually, look fine; the anomalies become clear in the aggregate.
4/
TikTok... do a lot of this work! They've hired a lot of very smart people to work on coordinated manipulation, fake engagement, and what they call "covert influence operations" — and they're doing a pretty good job!
There's a ton of data about their efforts in TikTok's (quite good!) transparency report: https://www.tiktok.com/transparency/en-us/community-guidelines-enforcement-2022-3/
5/
Now, you can ask very reasonable questions about whether TikTok's highly capable threat investigators would expose a PRC-backed covert influence operation if they found one.
And I personally find it a little... fishy... that the Q3 2022 transparency report discloses a Taiwanese operation, but not, say, the TikTok rendition of the unimaginably prolific and persistent Spamouflage Dragon.
6/
The basic problem with Project Texas and the whole "we're going to air-gap US user data from everything else" thing is that you're establishing geographic limits to the work threat investigators can do. Adversaries don't respect those limits.
TikTok have said they have about 1.5 billion global users, and about 150 million located in the US. That means the Project Texas / US Data Security folks will be working with a 10% view of the threat landscape.
7/
Side note about US Data Security (https://usds.tiktok.com): TikTok seems to be building... an entirely separate TikTok!... under the auspices of USDS. There are more than 200 job postings on the TikTok careers site for USDS roles alone: https://careers.tiktok.com/position?keywords=usds&location=
That is crazy! This is not how technology companies operate! It's a reflection of TikTok's essentially limitless resources, and existential dread about getting banned, that they're doing this.
8/
USDS may be a great solution if your goal is not to get banned in the US. But it's a terrible solution if your goal is to let threat investigators find the bad guys actually targeting your platform.
(Okay, actually, judging by today's hearing it seems like USDS is in fact a terrible solution when it comes to not getting banned, too. But that's a separate issue.)
9/
A siloed data model either means the USDS team will have to reinvent threat analysis using only their 10% slice of the platform, or TikTok's global analysts will have to find US-targeted threats without the benefit of any US-based user data.
(And yes, non-US threat actors regularly house their data and accounts in the US.)
10/
At the end of the day, this is another example of the privacy/security tradeoffs that come up over and over in the counter-IO space. This work just isn't possible to do without massive troves of incredibly privacy-sensitive user data and logs. It's a hard problem to solve!
But, whatever your anxieties about TikTok (and I have many!), banning it, and the haphazard Project Texas reaction to a possible ban, won't necessarily help national security, and arguably could just make things worse.
11/
Oh, and, spoiler alert: Mastodon and the fediverse have these same issues. (I'm working on a paper about this at the moment. Get in touch if you want to chat about it!)
12/fin
Dr. Samuel Wein@yoyoel
It strikes me that Mastodon's federated model is more vulnerable to these issues. Unless we figure out a way to federate telemetry.
It strikes me that Mastodon's federated model is more vulnerable to these issues. Unless we figure out a way to federate telemetry.
@samweingamgee Yes, exactly. 22,000+ instances each with separate data is quite the attack surface for adversaries looking to cover their tracks.
@yoyoel
Sharing that data in manner that maintains user privacy seems like it would be really hard. I follow the somewhat similar problem at work: how to share insights from medical data while maintaining health privacy, and as far as I can tell there is no easy solution.
Sharing that data in manner that maintains user privacy seems like it would be really hard. I follow the somewhat similar problem at work: how to share insights from medical data while maintaining health privacy, and as far as I can tell there is no easy solution.
Emily@yoyoel @samweingamgee So is single-platform analysis really the only source of info about these campaigns? Are there any other analytic models based on, say, campaigns hitting multiple platforms?
scott f@samweingamgee I came to post the same response. @yoyoel, I would be interested in your thoughts on how best to mitigate this, if that's something that ends up in the paper.
PJ "Vote Green" Coffey
Jiří Fiala Total Landscaping@yoyoel Please share when it's done
Xavier@yoyoel it always seems like there are only two solutions to this problem: a great western firewall or… education?
S★m V★rm★@yoyoel thanks for sharing your thoughts here. Really appreciate your insight. All the best!
Avi Rappoport (avirr)@yoyoel Maybe something like the CSAM protections with hashing avoiding sharing the actual suspect data?
Travis@yoyoel Super excited to hear more about this.
mlapalmYoel Roth (@[email protected])
I’m deeply reluctant to wade into the TikTok ban madness, but: There are some weird edges to the Project Texas stuff that I worry will make TikTok more susceptible, not less, to malign foreign interference. 1/
newsorpigalThe biggest threat to the free internet is the US government. Every major social media shares data with the US government and engages the US government for content moderation.
Tiktok is the only major social media that the US government can't directly influence, and Uncle Sam does not like it.
The TikTok debate can be simplified to one sentence: Entitled Americans are throwing a temper tantrum because there is a big player they don't control.
Looking for more context about TikTok and Project Texas? I've turned this thread into a Techdirt guest post (thanks, @mmasnick!) looking at the thorny and counterintuitive national security implications of siloing off US user data:
How Forcing TikTok To Completely Separate Its US Operations Could Actually Undermine National Security
Back in August 2020, the Trump White House issued an executive order purporting to ban TikTok, citing national security concerns. The ban ultimately went nowhere — but not before TikTok and Oracle …
WndlB
Zecharias Zelalem@yoyoel Hope you've been well, great to see you posting!
Doctor Biobrain@yoyoel @mmasnick It’s times like this when I think maybe we need more branches of the federal government.
I’ve already thought we needed a Culture Government led by Trump so all the Culture War MAGA lunatics can screech impotently as the real government gets work done. Maybe we need a Tech Government run by people who actually understand how computers and the internet works.
msgbi Markus Sagebiel
Andrew Couts@mmasnick @yoyoel Great points, thanks for sharing your insights on this! One nagging question I have is, how does storing US user data in the US inherently prevent foreign access? Is just… doesn’t, right? I may have missed how TikTok plans to handle this with Project Texas, but I haven’t heard anyone—besides you touching on it—addressing this specifically.
@infosec_jcp 🐈🃏 done differently
Graham Webster@yoyoel @mmasnick great piece! A question: you mention the Grindr case as an example of a Chinese government strategy to extricate data, yet the Reuters report only notes engineering access in Beijing with unknown transfers. Is there further info about a Chinese gov role or are you making an assumption? (FWIW I think the concern is legit even if it’s just a possibility, but obviously more acute if there’s evidence the possibility has been actualized.)
@gwbstr @mmasnick The short version is, CFIUS are not in the business of explaining the reasons they block transactions or demand divestments, but it's uncommon enough that you can fairly confidently infer that if they do something, there's a reason. More details: https://www.washingtonpost.com/politics/2019/04/03/why-is-us-is-forcing-chinese-company-sell-gay-dating-app-grindr/ and https://www.nytimes.com/2019/03/28/us/politics/grindr-china-national-security.html
That, coupled with Grindr's long-standing and notoriously lax data privacy standards, creates... a lot of risk.
But no, there's no confirmed/known transfers.
@yoyoel @mmasnick thanks for clarifying. I followed pretty closely at the time and thought indeed it was about the risk.
Examples of Chinese government intention to siphon off big piles of data on Americans need not touch on ownership or location of engineers. OPM, Experian, Marriott, etc.
Which is why a focus on nationality of ownership is both reasonable and hilariously blinkered if Chinese intelligence is the adversary of concern.
Mark Krueger@yoyoel Awesome thread Yoel 👍🏻 Really helped clarify the TikTok situation for me.
Derek Brauders@yoyoel I found this thread quite informative on the whole TikTok scenario. (This is where I prefer quotes - oh well 😊).
Point 7 is very telling though.
Point 7 is very telling though.
Nicole ParsonsSo much of GOP activity is geared towards promoting the agenda items of Republican billionaire donors.
The TikTok ban creates the legal precedents required to create captive consumers.
The donors really really like captive consumers because it shields home grown domestic monopolies.
They aren't objecting to the malign influence campaigns, the sale of user data, or surveillance capitalism.
They're objecting to these because they aren't part of their donors monopolies.
Locksmith
Southern Leftist@yoyoel
On the other hand, if the database is located in TEXAS (aka the Fascist Red Land), who knows what Abbott and his cronies can do with that data, let alone share it with other non-Texan Fash pals.
On the other hand, if the database is located in TEXAS (aka the Fascist Red Land), who knows what Abbott and his cronies can do with that data, let alone share it with other non-Texan Fash pals.
Dennis Conklin@yoyoel It’s not like an American company wouldn’t sell our data to China or anyone else with $$. It’s not like Facebook, Google, etc don’t have our data. It’s not like you can buy all our data from data brokers today and the brokers don’t need TikkyTokky to get our data. Tempest in a teapot!