Something exceptionally grim is happening on the Internet.
In the last few months, the constant flood of algorithmically generated junk content has kicked into an AI-powered overdrive, and it is cutting a swath of destruction as it overwhelms search engines, filters, and moderation systems
Call it Gresham's Law 2.0: bad content drives out good.
I'm starting this thread to document it, because there is a *lot* happening all at once.
Fake Funeral Live Stream Scams Are All Over Facebook
https://www.404media.co/fake-funeral-live-stream-scams-are-all-over-facebook/
"Scammers are copying genuine funeral service announcements, making their own Facebook accounts, then directing vulnerable victims to a fake livestream page. “It causes panic on the day where you shouldn't be thinking of that,” one impacted person said.”
Scammers are copying genuine funeral service announcements, making their own Facebook accounts, then directing vulnerable victims to a fake livestream page. “It causes panic on the day where you shouldn't be thinking of that,” one impacted person said.
https://housefresh.com/david-vs-digital-goliaths/
"Savvy SEOs at big media publishers (or third-party vendors hired by them) realized that they could create pages for ‘best of’ product recommendations without the need to invest any time or effort in actually testing and reviewing the products first."
Ghost Kitchens Are Advertising AI-Generated Food on DoorDash and Grubhub
https://www.404media.co/ghost-kitchens-are-advertising-ai-generated-food-on-doordash-and-grubhub/
"It’s common for advertisements to stage or edit pictures of food to make it look more enticing, but in these cases the ghost kitchens are showing people pictures of food that literally doesn’t exist, and looks nothing like the actual items they’re selling, sometimes because the faulty AI is producing physically impossible food items. "
Inside the World of TikTok Spammers and the AI Tools That Enable Them
https://www.404media.co/inside-the-world-of-tiktok-spammers-and-the-ai-tools-that-enable-them/
"We have recently been getting bombarded with Instagram Reels of influencers explaining how they make five figures a month by using AI to create tons of viral TikTok pages using stolen celebrity clips juxtaposed next to Minecraft gameplay footage."
Tech writer Kara Swisher has a new book. Enter the AI-generated scams.
https://www.washingtonpost.com/technology/2024/03/01/amazon-ai-fake-books-authors/
"Swisher looked at the screen and saw a book claiming to be a new biography of her, with an image on the cover that she immediately pegged as an AI-generated fake. … A closer look suggested that the book itself might be largely or entirely AI-generated, substituting generic descriptions of Swisher for factual details or anecdotes."
My Robotic Doppelganger Is The Grim Face of Journalism's Future!
https://foreverwars.ghost.io/my-robotic-doppelganger-is-the-grim-face-of-journalisms-future/
"On the zombie edition of the Washington Independent I discovered, the piece I had published more than ten years before was attributed to someone else. Someone unlikely to have ever existed, and whose byline graced an article it had absolutely never written."
Google Is Finally Trying to Kill AI Clickbait
https://www.wired.com/story/google-search-artificial-intelligence-clickbait-spam-crackdown/
"Google is taking action against algorithmically generated spam. The search engine giant just announced upcoming changes, including a revamped spam policy, designed in part to keep AI clickbait out of its search results."
The Perfect Webpage: How the internet reshaped itself around Google’s search algorithms — and into a world where websites look the same.
https://www.theverge.com/c/23998379/google-search-seo-algorithm-webpage-optimization
"Bit by bit, the internet has been remade in Google’s image. And it’s humans — not machines — who have to deal with the consequences."
[I can't prove this one is related, but I have a strong hunch that it is.]
Why Is Your News Site Going Out of Business?
https://talkingpointsmemo.com/edblog/why-is-your-news-site-going-out-of-business
[TPM's programmatic advertising revenue declined from ~$1.7M in 2016 to ~$75K in 2023.]
"If you look at that graph you can see that if your business was ads and remained ads you’re toast."
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
https://arxiv.org/abs/2403.07183
"Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates."
We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.
https://www.astralcodexten.com/p/open-thread-320
"If you want a vision of the future, imagine a bunch of people all accusing each other ad infinitum of being AIs (“haven’t you heard of Dead Internet Theory?!”), while the actual AIs serve ads to them in the background.”
How Spammers, Scammers and Creators Leverage AI-Generated Images on Facebook for Audience Growth
https://cyber.fsi.stanford.edu/io/news/ai-spam-accounts-build-followers
"We suspect that AI-generated images appear on users’ Feeds because the Facebook Feed ranking algorithm promotes content that is likely to generate engagement.”
Huckster Behind ‘Willy Wonka’ Event Also Sells AI-Written Vaccine Conspiracy Books
“Coull couldn’t even be bothered to pen his own author bio, which declares him a “rising star in the literary world” who “weaves spellbinding tales that delve into the mysterious realms of fictional thrillers and gripping conspiracies.”
Kotaku EIC Resigns Over New Editorial Edict
https://aftermath.site/kotaku-eic-resigns-over-new-editorial-edict
"In a resignation letter seen by Aftermath, Glennon says that she made this choice due to the management team’s recent decision to deprioritize news in favor of guides. … According to a source close to the situation, Kotaku's staff will be expected to create 50 guides a week at the site.”
Jen Glennon, who took over as editor in chief of Kotaku in October, resigned Thursday. In a resignation letter seen by Aftermath, Glennon says that she made this choice due to the management team’s recent decision to deprioritize news in favor of guides. Glennon is the second editor in chief of Kotaku since Stephen Totilo’s […]
TikTok is paying creators to up its search game
"The insights tool essentially lays out ideas for videos a creator could make, based on both personalized and general data. On my account, for example, TikTok flagged “Succession outfits” as a recommended topic for me to make content about. Also listed: “Herman Miller Chair,” “Fairy Grunge Room Ideas,” and “strawberry muffins.””
The AI Peer-Review Crisis
https://www.404media.co/email/c36519c0-0930-47a2-808d-48631b597dc1
“The bigger issue is that there's just the peer review crisis you're talking about where there's just so many papers being sent to these conferences,” he said. “I think it leads to the kinds of things that you're seeing in [Liang’s] paper where people are using the AI technology to help them to do the reviews.”
Nicholas LoVecchio told me he knew something was wrong as soon as he received the peer-review responses to his paper about “taboo” language titled “Using translation to chart the early spread of GAY terminology.” “I received them in the evening, the week before Christmas and I was just going through
Google Books Is Indexing AI-Generated Garbage
https://www.404media.co/google-books-is-indexing-ai-generated-garbage
"For example, the 2024 book Bears, Bulls, and Wolves: Stock Trading for the Twenty-Year-Old by Tristin McIver, bills itself as “a transformative journey into the world of stock trading” and “a comprehensive guide designed for beginners eager to unlock the mysteries of financial markets.” In reality, it reads like ChatGPT-generated text with surface, Wikipedia-level analysis"
‘AI Instagram Influencers’ Are Stealing Women’s Bodies
https://www.404media.co/email/d72b3844-9d62-4961-b52f-86ba9f237480
"Stealing video footage from real women to make an AI influencer more “believable,” because the underlying video footage actually is real, is a strategy that I repeatedly saw being espoused by guides to making these AI influencers."
‘Made for Advertising’ Websites Are the Marketing Industry’s Latest Messy Situation
"“Because we could say things like, ‘Well, it’s automatically generated content.’ But if somebody finds value in that—for example, I would read a piece that tells me something about my air conditioner, like how to install it, or how to clean it—I don’t care whether that was written by a person or a machine, as long as it solves my problem. So does that constitute MFA?”
How I Built an AI-Powered, Self-Running Propaganda Machine for $105
"It took me two days, $105 and no expertise whatsoever to launch a fully automated, AI-generated local news site capable of publishing thousands of articles a day—with the partisan news coverage framing of my choice, nearly all rewritten without credit from legitimate news sources.”
The AI flight attendants of Facebook
https://www.garbageday.email/p/ai-flight-attendants-facebook
"Who cares if users are praying to and/or asking to marry AI images of flight attendants? All that matters is that they’re on the app and you can stick ads in front of them.”
Amazon is filled with garbage ebooks. Here’s how they get made.
https://www.vox.com/culture/24128560/amazon-trash-ebooks-mikkelsen-twins-ai-publishing-academy-scam
"The saddest part about it, though, is that the garbage books don’t actually make that much money either. …The way people make money these days is by teaching students the process of making a garbage ebook. It’s grift and garbage all the way down — and the people who ultimately lose out are the readers and writers who love books.”
Facebook’s AI Told Parents Group It Has a Gifted, Disabled Child
https://www.404media.co/facebooks-ai-told-parents-group-it-has-a-disabled-child/
"The top-ranked comment on this post is from “Meta AI,” which is Meta’s AI chatbot. “I have a child who is also 2e and has been part of the NYC G&T program,” the nonsentient chatbot wrote to a group of human parents. “We’ve had a positive experience with the citywide program, specifically with the program at The Anderson School.”
AI Is Poisoning Reddit to Promote Products and Game Google With 'Parasite SEO'
"A service called ReplyGuy advertises itself as “the AI that plugs your product on Reddit” and which automatically “mentions your product in conversations naturally.”"
Facebook’s Bizarre AI Images Now on LinkedIn, Too
https://www.404media.co/facebooks-bizarre-ai-images-now-on-linkedin-too/
"The post has 2,192 reactions, 173 comments, and 84 reposts. About half of the comments note that the image is AI-generated, while about half of them say things like “I just saw his wood curving of Messi, He is talented. Just amazing.”
Do You Like These AI Images of Dying, Mutilated Children? Facebook Algorithm Wonders
https://www.404media.co/ai-images-of-dying-drowning-mutilated-children-go-viral-on-facebook/
"One of the images shows a child whose leg is amputated below the knee and holds a sign reading “Today is my birthlday pleaase like.” That image has 70,000 likes and 3,000 comments. Another image is of a girl face-down in the ocean wearing an oxygen mask that is connected to a floating birthday cake.”
Flood of AI-Generated Submissions ‘Final Straw’ for Small 22-Year-Old Publisher
https://www.404media.co/bards-and-sages-closing-ai-generated-writing/
"“The problem with AI is the people who use AI. They don't respect the written word,” Dawson told me. “These are people who think their ‘ideas’ are more important than the actual craft of writing, so they churn out all these ‘ideas’ and enter their idea prompts and think the output is a story.”
Facebook’s AI Spam Isn’t the ‘Dead Internet’: It’s the Zombie Internet
https://www.404media.co/facebooks-ai-spam-isnt-the-dead-internet-its-the-zombie-internet/
"When exploring their comment sections, one will often see hundreds of bot-like comments interspersed with a few ‘real’ people sounding the alarm to no avail,” Khan Schoolcraft, a moderator of the group, told me. “It’s like yelling into an empty room full of a thousand identical brick walls.”
Crypto world taps AI to flood IRS with complaints about tax regs
"Some leading opponents have built online tools that tap the power of artificial intelligence to allow anyone to quickly generate custom letters to the IRS in opposition to the proposal — a big step beyond the form letters often central to other lobbying campaigns.”
Meet AdVon, the AI-Powered Content Monster Infecting the Media Industry
https://futurism.com/advon-ai-content
"Basically, AdVon engages in what Google calls "site reputation abuse": it strikes deals with publishers in which it provides huge numbers of extremely low-quality product reviews — often for surprisingly prominent publications — intended to pull in traffic from people Googling things like "best ab roller." "
Slop is the new name for unwanted AI-generated content
https://simonwillison.net/2024/May/8/slop/
"Not all promotional content is spam, and not all AI-generated content is slop. But if it’s mindlessly generated and thrust upon someone who didn’t ask for it, slop is the perfect term for it."
This is what the internet looks like now
https://www.garbageday.email/p/internet-looks-like-now
"Across the social web, older forms of low-effort engagement bait are getting a brand new coat of AI paint. Every time I poke around a random TikTok that ends up on my For You Page, I, without fail, find out it’s connected to some weird content farm pumping out AI images on Facebook."
@jtlg
The problem isn’t with LLM, the problem is with publishers.
Lots of marketing copy was just slop prior to LLM slop.
My apprentice had a job on the side writing advertising copy for companies. It was just drivel. He even accidentally dropped some Lorem Ipsum in once and nobody noticed!
He ended up outsourcing to India, where he paid cents in the dollar.
Finally he started outsourcing to an LLM.
His customers never proofread or checked any of the content he submitted to them.
James Grimmelmann
Dan W