Annalee Newitz 🍜Apr 29, 2025
I downloaded a scientific paper of great interest to me. It is of typical length, about 15 pages in the PDF. I opened it in Acrobat, and got this message highlighted at the top of the page next to an AI sparkle icon: "This appears to be a long document. Save time by reading a summary." I cannot put into words how much this enrages me. First, this is not a long document. Have you met documents? Second, and more important, telling someone to not read a thing they want to read??! Fuck you, Adobe.
Show threadCarolannieApr 29, 2025
@annaleen not only that, but the summary has probably a 50% likelihood of being wrong
Show threadRuediger Gad
@carolannie @annaleen was about to ask if you tried it. Am curious about real-world experience.
Apr 30, 2025 at 5:16amWeb
Show threadfanf42Apr 30, 2025

@ruedigergad @carolannie @annaleen I expects it will be abysmal. I tried on several very well known, very documented short texts (article, short essay). Really, things like "the tyranny of structurlessness". The AI (all of the common ones I tested: Claude, ChatGPT, Gemini, local ollama/deepseek...) gave extremely deep sounding and assured summaries, which were also horrible results, missing extremely important, often foundational nuances to the point of misinterpretation.
For things they must have been trained on.

So, AI (llm) summary is the same silver bullet than llm for defending your case in court. Worse than useless

Show threadNicola RomanòMay 3, 2025

@ruedigergad @carolannie @annaleen I haven't tried the one in Acrobat, but generally ChatGPT or Gemini do a good work at summarising. I don't think hallucinations are a big problem honestly (for summarisation , they are a big problem for other things) I maybe haven't used them extensively but I don't recall seeing any obvious errors when summarising a document. It's the critical evaluation that is probably a bit more limited, but again if you want a quick summary, probably you're not looking for that either, and the tool does the job.

Honestly, aside fron the initial "oh that's cool" novelty-item moment I would say that mostly it's faster and more efficient to quickly scan papers manually yourself but there are other uses.

Show threadCarolannieMay 3, 2025
@nicolaromano Hmmm, your experience is different from mine. Discussing generically, summaries seem to be either useless because too vague, or inaccurate. But I usually look at articles in botany or ecology and can spot errors easily.