TriniFietser šŸ‡¹šŸ‡¹Jan 12
Prof. Emily M. Bender(she/her)

Health experts: Your synthetic text "AI" overviews are misleading, for example see this about liver function tests.

Google: Okay, we'll block "AI" overviews on that query.

The product is fundamentally flawed and cannot be "fixed" by patching query by query.

A short šŸ§µ>>

https://www.theguardian.com/technology/2026/jan/11/google-ai-overviews-health-guardian-investigation

ā€˜Dangerous and alarmingā€™: Google removes some of its AI summaries after usersā€™ health put at risk

Guardian investigation finds AI Overviews provided inaccurate and false information when queried over blood tests

The Guardian
Jan 11 at 11:23pmWeb
Show threadProf. Emily M. Bender(she/her)Jan 11

"Block that search in particular" is also what Google did when the inherent racism in their image search algorithm was pointed out, in 2018.

https://www.theguardian.com/technology/2018/jan/12/google-racism-ban-gorilla-black-people

>>

Google's solution to accidental algorithmic racism: ban gorillas

Googleā€™s ā€˜immediate actionā€™ over AI labelling of black people as gorillas was simply to block the word, along with chimpanzee and monkey, reports suggest

The Guardian
Show threadProf. Emily M. Bender(she/her)Jan 11

And when Dr.
@safiyanoble pointed out the racist results returned for queries like "black girls" in ~2016. See her amazing book Algorithms of Oppression:

https://nyupress.org/9781479837243/algorithms-of-oppression/

>>

Algorithms of Oppression

A revealing look at how negative biases against women of color are embedded in search engine results and algorithms Run a Google search for ā€œBlack girlsā€...

NYU Press
Show threadProf. Emily M. Bender(she/her)Jan 11

Back to the current one, the quotes from Google in the Guardian piece are so disingenuous:

>>

Show threadProf. Emily M. Bender(she/her)Jan 11

I've been shouting about this since Google first floated the idea of LLMs as a replacement for search in 2021. Synthetic text is not a suitable tool for information access!

https://buttondown.com/maiht3k/archive/information-literacy-and-chatbots-as-search/

/fin (for now)

Information literacy and chatbots as search

By Emily This post started off as a thread I wrote and posted across social media on Sunday evening. I'm reproducing the thread (lightly edited) first and...

Mystery AI Hype Theater 3000: The Newsletter
Show threadJPJan 11
@emilymbender we're living through a mass psychological engineering campaign and the results have been, and will continue to be, horrifying https://azhdarchid.com/are-llms-useful
Are LLMs 'useful'?

Here's a thought experiment...

Azhdarchid
Show threadNicole ParsonsJan 12

@jplebreton @emilymbender

The question is also "LLM'S are useful to whom?"

The wealthiest seem overjoyed with it so far.

So much so, they are funding one of the largest coercive & forced user adoption campaign in history.

https://www.forbes.com/sites/mattdurot/2025/07/17/bill-gates-charles-koch-and-three-other-billionaires-are-giving-1-billion-to-enhance-economic-mobility-in-the-us/

It's the best at:
1. Election interference
2. Malign influence campaigns
3. Automated cyberwarfare
4. Manipulation of public sentiment
5. Automated hate campaigns
6. Plausible deniability for funding a fascist movement
7. Frying the planet

Five Billionaires Pledged $1 Billion To Boost Economic Mobility Using AI

Five of Americaā€™s top philanthropists are teaming up for a new venture aimed at helping low-income Americans rise from poverty. An AI giant has signed on to help.

Forbes
Show threadJPJan 12
@Npars01 @emilymbender yeah, it's clearly one of those technologies that accelerates just about everything patriarchal white supremacist capitalism does in various ways, and provides a greater means of plausible deniability to the people behind that than previous systems. it enables the monsters running the world to Capitalism Harder, at the exact moment when we need to be doing the opposite and take better care of one another and our planet. so in that sense it's definitely working as designed.
Show threadLRJan 12
@jplebreton @Npars01 @emilymbender or not designed, as this shit was ripped from labs before it was ready, because moneydudes got the fomo. the devil finds work in underengineered solutions.
Show threadJPJan 12
@lritter @Npars01 @emilymbender of course yeah, a lot of modern corporate "design" processes are like that, a thing is shown to have certain strategy-affine effects and it's immediately productized with no long term thinking needed.
Show threadKyle Memoir šŸ‰šŸ§Jan 12

@Npars01 @jplebreton @emilymbender

Bang on.

Show threadDr. Juande Santander-VelaJan 12
@Npars01 @jplebreton @emilymbender so they pledge to give 1/320th of their net worth so that they can pretend even a 1% tax on their total net worth would not work better. Got it.
Show threadTomƔs HradckyJan 12
@Npars01 Boy are they not even subtly advertising their running from the coming guillotines. Just cracks me up how little they think they need to spend to avoid their own collapse.
Show threadnimu ĪµŃ—Š·šŸ––šŸ»šŸ¦Jan 12
@Npars01 ah yes, bill gates, the "good billionaire"
Show threadNicole ParsonsJan 12

@coracinho

Lol. Are there good billionaires?

Show threadRokosunJan 12

@jplebreton @emilymbender

That blog post was an interesting read, thanks for sharing.

Show threadKaren CampeJan 11
@emilymbender say it louder for those in the back!!
Show threadGrassroots JoeJan 12

@emilymbender

People should use search options that either have no AI summary or at least allow you to easily turn it off permanently as DuckDuckGo does.

Show threadEvannakitaJan 12
@joeinwynnewood @emilymbender Do you have recommendations for ones that aren't DuckDuckGo?
Show threadBrianJan 12
@evannakita @joeinwynnewood @emilymbender I use Startpage at times! @StartpageSearch Also, cool profile amd background picture!
Show threadEvannakitaJan 12
@Brian @joeinwynnewood @emilymbender Thank you so much, both for the recommendation and for the compliment! I drew both of those and it means a lot to hear :)
Show threadBrianJan 12
@evannakita You're welcome!^_^ā¤ļøā¤ļøā¤ļø
Show threadGrassroots JoeJan 12

@evannakita

Sorry, no. I think DuckDuckGo is a decent option. They don't keep your search history and produce good search results.

@emilymbender

Show threadEvannakitaJan 12
@joeinwynnewood @emilymbender No worries, and agreed on it being a decent option! It's what I've been using for a couple years now. I've just been troubled by some recent stuff the company's been doing, so I wanna keep my options open.
Show threadJason_DoddJan 12
@emilymbender In Google's defense, they have been so good at making their search engine pure shit, "AI" search just might be better.
Show threadleeJan 12
@Jason_Dodd google doesn't need you defending it
Show threadJohn TinkerJan 12
@emilymbender
The con is to allow the user to imagine that the chatbot is AGI and not LLM. Once it is clear what the LLM is, then it is useful for finding normative language related to any number of topics, has been my experience. The AI market capitalization is criticized as a bubble. I think it is, too. I think that the misunderstanding regarding the chatbot will bite. I discussed this matter with DeepSeek: https://johntinker.substack.com/p/misunderstanding-as-a-commutator
Misunderstanding the Chatbot, seen as a Commutation Operator

There appears to be a general misunderstanding that the large language model is a form of artificial general intelligence.

Johnā€™s Substack
Show threadProf. Emily M. Bender(she/her)Jan 12

@johntinker See pinned toot:

https://dair-community.social/@emilymbender/109339391065534153

Also, no, LLMs are not what you think they are, if you are "discussing" anything with them.

Show threadAdrian CockcroftJan 12
@emilymbender Right now Iā€™m finding that when I have a conversation with chatGPT about a topic itā€™s a much better and more accurate and useful experience than using Google or DuckDuckGo searches. It includes links to sources but it also has a useful wider context that informs where it looks (like whether Iā€™m looking for UK or USA based info), and is persistent, so I can come back to the topic months later and continue.
Show threadBob ThomsonJan 12
@adrianco @emilymbender thatā€™s because the an LLM model is now more effective than the broken Google Search algorithm is at surfacing the better results out of all the Gen AI slop the web is now drowning in.
I donā€™t have conversations because there isnā€™t anything to converse with, but i just type what I would have done into Google back when it was useful. I find it supremely ironic that this is my main use case for LLM.
Show threadKim Crawley šŸ˜· (she/her)Jan 12

@bobthomson70 @adrianco @emilymbender

Why don't use actually use a good search engine rather than burn down a forest and drain a lake by "prompting"?!

Show threadBob ThomsonJan 12
@kimcrawley @adrianco @emilymbender if I could find one that was any use I would.
Show threadKim Crawley šŸ˜· (she/her)Jan 12

@bobthomson70 @adrianco @emilymbender

And how much effort did you put into that? I'm willing to bet, no more than a couple of minutes.

My late father was born in 1937, television didn't enter his life until he was 18 or so, and he wasn't a computer geek.

And yet, in 1993, being completely unable to ask the internet or anyone for help, he switched from his Smith Corona electronic typewriter to a Windows 3.1 PC and he taught himself how to use it.

He was no spring chicken, even then.

Age is no excuse.

But your utter lack of curiosity and effort is furthering environmental destruction and enabling evil.

"All it takes for evil to prevail is for good men to do nothing."

DuckDuckGo with "Duck.ai" turned off.

There you go.

But as for me, I taught myself everything too. I wasn't fucking helpless. And I guarantee you, I'm older than you think I am.

Show threadSarah GentileJan 13
@bobthomson70 @kimcrawley @adrianco @emilymbender hi allā€”a vote for duckduckgo searchs, but with -ai added every time. Yes, I wish it would default to ai off, but like many things this yearā€¦ *sigh*
Show threadAdrian CockcroftJan 13
@forever_archives @bobthomson70 @kimcrawley @emilymbender Iā€™ve been using DDG for several years. However itā€™s not the search results themselves, itā€™s the conversation and iteration where ChatGPT will do several searches and figure out what is relevant and ask questions to clarify.
Show threadKim Crawley šŸ˜· (she/her)Jan 13

@adrianco @forever_archives @bobthomson70 @emilymbender

You're a clanker! Blocked!

Show threadAdrian CockcroftJan 12
@emilymbender For example, hereā€™s a conversation about what species of bat Iā€™m looking at. Itā€™s a much better experience than web search. Regardless of how accurate it is, the experience is going to drive usage. However it asked good clarifying questions and the answers are correct as far as I can tell. https://chatgpt.com/share/6964daa3-4a64-8009-86e9-4a1b804998a7
ChatGPT - Bat Identification Help

Shared via ChatGPT

ChatGPT
Show threadProf. Emily M. Bender(she/her)Jan 12

@adrianco I do my very best to ever avoid reading synthetic text, so no thank you.

Meanwhile: Just because you have a system that is producing text that looks like what you want, does not make it useful.

Show threadQybatJan 12
@emilymbender The world cries out for a better search. One that can work even on an Internet full of malicious SEO engineered to generate false positives for fake reviews and other scam sites. Such a technology is desperately needed. Unfortunately we got LLMs instead, which exchange one set of problems for another: They can only return information that is true most of the time.
Show threadCogito ergo mecagoendiosJan 12

@emilymbender Do you know when you watch a movie and the caricature evil investor says something like "ĀæLifeboats? Ā”Nonsense! This ship was advertised to be unsinkable, and therefore that would be bad for our brand and a useless expense."?

For some reason, this feels like the same problem. They have already decided that the final destination must be marketing a computer that talks like a human servant, and any evidence that natural conversation is not a good machine interface will be ignored.

Show threadNazaniJan 11
@emilymbender Uh-huh. I've seen overview attempt to combine mutually exclusisve theses.
Show threadgeolawJan 12
@emilymbender Good News! We told our AI to audit the results of our AI and it told us it passed!
Show threadbjb  Jan 12
@emilymbender If the approach to mking AI return good results is to put in a bunch of special cases, can we just go back to writing proper expert systems?
Show threadWesDymJan 12
@emilymbender I just wanna say, I'm thrilled that rugbies are back. I always loved rugbies.
Show threadBlake C. StaceyJan 12
@emilymbender Incredibly, I still hear scientists defending this damn technology. It's like, "I'm always on time for my morning classes thanks to my radium watch! Golly gee, the bus to the math conference runs so much smoother on leaded gasoline."
Show threadMichael McWilliamsJan 12
@emilymbender The developers do not know what answer an AI will give to a specific prompt in advance. It's a black box of answers. Therefore there is no QA of the product. It is unpredictable and therefore dangerous. Need I continue?
Show threadMegatronicThronBanksJan 12
@emilymbender The theme to I DREAM OF JEANIE is now playing in your head!
It's outa that bottle!
Show threadChris FoxJan 12
@emilymbender That Guardian headline is itself misleading; it should say something like, "Google's false and inaccurate AI overviews continue to put users' health at risk."
Show threadSolomon SchulerJan 13
@emilymbender the comment "OK we'll block AI overviews for that query" demonstrates how even google doesn't want to deal with their own shit. the fact that they're trying to BLOCK it instead of fixing it shows how little fucks they give to it. They know it's shit and how temporary it is, and they're just prolonging the inevitable collapse of it to make even more money while it still lives.
Show threadEdwardJan 13
@emilymbender it's probably more important to educate people about how AI works so they can be reasonable about fact checking, but then that will probably lead to people not being dependent on AI and big tech won't make any money from it