Mastodawn

Andrew Mar 26, 2024

I have a preprint out estimating how many scholarly papers are written using chatGPT etc? I estimate upwards of 60k articles (>1% of global output) published in 2023. https://arxiv.org/abs/2403.16887

How can we identify this? Simple: there are certain words that LLMs love, and they suddenly start showing up *a lot* last year. Twice as many papers call something "intricate", big rises for "commendable" and "meticulous".

#bibliometrics #scholcomm #chatgpt

ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature

The use of ChatGPT and similar Large Language Model (LLM) tools in scholarly communication and academic publishing has been widely discussed since they became easily accessible to a general audience in late 2022. This study uses keywords known to be disproportionately present in LLM-generated text to provide an overall estimate for the prevalence of LLM-assisted writing in the scholarly literature. For the publishing year 2023, it is found that several of those keywords show a distinctive and disproportionate increase in their prevalence, individually and in combination. It is estimated that at least 60,000 papers (slightly over 1% of all articles) were LLM-assisted, though this number could be extended and refined by analysis of other characteristics of the papers or by identification of further indicative keywords.

arXiv.org

Show thread

Monika Barget Mar 26, 2024

@generalising We also have the discussion if our students should still be allowed to use Grammarly etc. as the AI-abilities of these tools increase. What I think should happen: raise overall expectations for writing quality and simply fail papers on grounds of imprecise writing etc. when they are full of flowery AI adjectives. If someone uses AI tools so smartly that a decent argumentation comes out, they probably did put in a lot of work and grey matter. At least that's still the case now.

Show thread

Andrew

@mob I wonder a lot about the grammarly thing, but also stuff like "AI assisted search tools". At what point does it cross the line between something we're happy for students to use, and something that's too much? Really hard to say, and very blurry sometimes.