We're so fucked.

"Our analysis of a selection of questionable GPT-fabricated scientific papers found in Google Scholar shows that many are about applied, often controversial topics susceptible to disinformation: the environment, health, and computing. The resulting enhanced potential for malicious manipulation of society’s evidence base, particularly in politically divisive domains, is a growing concern."

https://misinforeview.hks.harvard.edu/article/gpt-fabricated-scientific-papers-on-google-scholar-key-features-spread-and-implications-for-preempting-evidence-manipulation/

GPT-fabricated scientific papers on Google Scholar: Key features, spread, and implications for preempting evidence manipulation | HKS Misinformation Review

Academic journals, archives, and repositories are seeing an increasing number of questionable research papers clearly produced using generative AI. They are often created with widely available, general-purpose AI applications, most likely ChatGPT, and mimic scientific writing. Google Scholar easily locates and lists these questionable papers alongside reputable, quality-controlled research. Our analysis of a selection of

Misinformation Review
@ct_bergstrom Oh wow, the scholarly papers already a majority of those cited: "Roughly two-thirds of the retrieved papers were found to have been produced, at least in part, through undisclosed, potentially deceptive use of GPT. The majority (57%) of these questionable papers dealt with policy-relevant subjects (i.e., environment, health, computing), susceptible to influence operations. Most were available in several copies on different domains (e.g., social media, archives, and repositories)"

@cobalt @ct_bergstrom I note "A sample of scientific papers with signs of GPT-use found on Google Scholar was retrieved".

So a majority of the papers that were flagged as potentially fraudulent were in fact made using the text generator.

That is already quite bad enough, since the number of fraudulent papers indexed by Google Scholar with just a couple of obvious tells of ChatGPT is apparently >100.

@ct_bergstrom

Is it time to find a proper Butler and begin a Jihad?

I always thought Herbert was a little over the top. But now I'm not sure. He may have been on to something.

@johntimaeus @ct_bergstrom

> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.

> The target of the Jihad was a machine-attitude as much as the machines ... Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments.

It’s funny, both Tolkien and Herbert had strong Ludditism in their stories, and many AI nerds are likely fans.

@johntimaeus @ct_bergstrom @maegul see also the unveiling of the Torment Nexus, inspired by classic sci fi novel “don’t invent the torment nexus”
@ct_bergstrom Yet academia has embraced it, especially in certain disciplines (like business).
@dalfen @ct_bergstrom calling business as discipline 😅

@peteriskrisjanis I figured someone would respond like that here.

@ct_bergstrom

@ct_bergstrom I am starting to become quite radicalized by this; I am starting to feel that in order to save society, we must classify AI development as terrorism.
@WhyNotZoidberg @ct_bergstrom I'm a teacher at a university. I strongly believe that the emergence of AI/ChatGPT etc is going to set back humanity's development by at least a decade or two. It seems that a lot of people are just becoming reliant on it and are losing the ability to think for themselves.
@artdragon86 @WhyNotZoidberg @ct_bergstrom to be fair the previous tech fads (especially crypto/NFTs) and the way social media has been weaponised by the far right have already set us back about 20 years.
@ct_bergstrom
This reads like the three Body Problem - Sophons and the end of Science o_O
But compared to that, AI shit would easier to Mitigate...
Just stop Publishing AI-Crap
@ct_bergstrom while it is very disheartening, it doesn't change much in reality, does it? Like global warming - if you claim to be skeptic, no amount of valid papers gonna convince you. Same with other issues.
While cooking up "evidence" for political agenda is nothing new, and I don't see how that's gonna change people minds, it is saddening.
@peteriskrisjanis Systemic reviews. Search engines. Training for future LLMs. Daubert decisions. Automated discovery. Need I continue?

@ct_bergstrom

The issue here is human bias being proliferated at an exponential rate. These problems were here before. How many scientists in the 50-70’s said lead in gasoline was fine?

People point at AI because it’s new.

@ct_bergstrom honestly, add this to the many reasons. we stopped having functional societies in most places on earth a long time ago, IMO
@ct_bergstrom just a wild guess: these same papers produced bij AI are used bij LLM to give answers to search-questions? "Garbage is even more garbage out"?

@ct_bergstrom

All scientific publications and journals should immediately require a declaration from authors that no LLM/AI systems were used in the generation of results or the paper being published.

@simonzerafa
> All scientific publications and journals should immediately require a declaration from authors that no LLM/AI systems were used in the generation of results or the paper being published

... and convincing evidence of a false declaration ought to be grounds for being banned from publishing, by all credible academic journals. For at least 5 years, if not permanently.

@ct_bergstrom

@simonzerafa @ct_bergstrom many do, but only some authors comply. That casts doubt on all submissions, unfortunately. And, at least for now, we can only really identify LLM use when authors submit with obvious editing failures. That's what this paper caught - also I document here as best I can:
https://peeraireview.com/search.html

The future is pretty simple... We have to view all papers starting in 2023 as potentially containing GenAI content, unless we know and trust the authors.

PAIR

GPT-fabricated scientific papers on Google Scholar: Key features, spread, and implications for preempting evidence manipulation | HKS Misinformation Review

Academic journals, archives, and repositories are seeing an increasing number of questionable research papers clearly produced using generative AI. They are often created with widely available, general-purpose AI applications, most likely ChatGPT, and mimic scientific writing. Google Scholar easily locates and lists these questionable papers alongside reputable, quality-controlled research. Our analysis of a selection of

Misinformation Review
@ct_bergstrom
Here's an example of an extremely brash example of such a paper (link below).
The problem is not only the spill of this 'content' in general, but also for-profit paper mills like #Frontiers which let that shit happen.
https://arstechnica.com/science/2024/02/scientists-aghast-at-bizarre-ai-rat-with-huge-genitals-in-peer-reviewed-article/
Scientists aghast at bizarre AI rat with huge genitals in peer-reviewed article

It’s unclear how such egregiously bad images made it through peer-review.

Ars Technica
@ct_bergstrom
I dunno - I think google and any other corporate entity trying for content inperpertuum have fucked themselves.

As it stands authors will email copies of their papers if asked - as they want people reading their stuff more than insisting people go through the hoops of the sites "publishing" the papers.

Academia will simply move away from publishing entities and companies that pump ai slop into everything.

@ct_bergstrom

Already now this is a big problem. The next big thing after the IT revolution… or a next phase of it…

@ct_bergstrom Of course, no one could have seen this coming... cripes.
@ct_bergstrom you know what's bad then? The fact that they are not reviewed properly. If they are, then we wouldnt be here in the first place.

@tengkuizdihar Yeah, so read the article.

By and large this is not peer reviewed literature.

(1/2)

@ct_bergstrom
> Our analysis of a selection of questionable GPT-fabricated scientific papers found in Google Scholar

BTW I'm guessing this problem is even worse in China. Where there have long been questionable attitudes towards academic publishing, influenced by decades of the CCP's MiniTru approach to information. Also where training MOLE (Machine Operated Learning Engines) is even more hyped and funded, and less regulated, unless for MiniTru purposes.

(2/2)

"China continues to have problems with research integrity and has the highest number of retractions of any country due to plagiarism, invented data and fake peer review, but is seeking to improve this by removing cash incentives and use of plagiarism software."

#KenHyland, 2023

https://doi.org/10.1002/leap.1545

@ct_bergstrom It's poisoning the well, misinformation and ignorance are great for creating an underclass to exploit and who aren't smart enough to argue back.

@nini @ct_bergstrom And that bit about reaching a point where there's no longer an aim of twisting or replacing the truth but destroying the notion of truth altogether.

Vanishing all signal in a total wave of noise means inviting all who fear, or are overwhelmed by, the responsibility of decisionmaking to amputate all higher mental function, reason, values, offering carte blanche to run with whatever is most gratifying, easiest, most comfortable. Become livestock.

@ct_bergstrom and that is just the beginning. Have you seen this? https://sakana.ai/ai-scientist/ (link to arXiv is also in the post)
Sakana AI

The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

Carl T. Bergstrom (@[email protected])

Attached: 1 image Taylorism is a management philosophy based on using scientific optimization to maximize labor productivity and economic efficiency. Here's the result of making the false Taylorist assumption that the output of scientific research is scientific papers—the more, faster, and cheaper, the better.

FediScience.org
@ct_bergstrom funny how the people who have the most money in AI are also the ones who have funded or made money from disinformation 🤔
@ct_bergstrom ok so not only do LLMs suck more energy than we can decently produce, and destroy the internet by flooding it with more BS than we can debunk, but its next target is now science.
Of course, LLMs are not a scient thing trying to fuck with humanity. We are doing this to ourselves in enlightened consent.
@ct_bergstrom Do they say what % is in fake journals (pay to publish, etc)?
@ct_bergstrom Hi Carl, I wrote this 10-minute thought experiment about the epistemological brokenness of LLMs, and compare them to the epistemology of biological organisms. Since you're an LLM critic and biologist, I'd be interested in your thoughts https://medium.com/@timkindberg/the-baby-room-an-ai-thought-experiment-6e1fd9ddb922
The baby room - Tim Kindberg - Medium

Note to readers: this is a purely intellectual exercise to help us compare humans to large language models like ChatGPT. It involves things we would never do to an actual child. Think of it as a dark…

Medium