Pedro MendesNov 10, 2023
Ethan White

Yesterday @biorxivpreprint announced it was piloting adding Large Language Model summaries to its preprints

https://connect.biorxiv.org/news/2023/11/08/summaries

I looked at the summaries for our most recent #preprint & the results were meh. One was pretty good, one focused on a side note for the 1st half, & one included a very minor point in the summary. LLMs may have a role in this space, but if so it should be with author consent, supervision, & sign-off.

Further discussion of the announcement: https://hachyderm.io/@ethanwhite/111380087325572771

Broadening audience, increasing understanding

bioRxiv - the preprint server for biology, operated by Cold Spring Harbor Laboratory, a research and educational institution

Nov 10, 2023 at 7:42pmWeb
Show threadpowersoffourNov 10, 2023
@ethanwhite hey i sent an email that should clear this all up for you
Show threadEthan WhiteNov 10, 2023
@powersoffour that made my day Matt. You’re awesome
Show threadEthan WhiteNov 10, 2023
@powersoffour I particularly love that even with that prompt the LLM recommends only using the LLM output as a starting point and making sure to edit it
Show threadpowersoffourNov 10, 2023
@ethanwhite I was pre-disposed to being annoyed because Bard kept giving me nonsense (inventing references, GLSL code that flatout did not work, etc) when I was kicking the tires this morning.
Show threadporitzjNov 11, 2023
@ethanwhite @biorxivpreprint
I'm so confused about this. Isn't a thoughtful human with understanding *writing the abstract* better than fancy autocomplete doing something like this?
OK, maybe the author's abstract doesn't point out subtle inconsistencies and issues in a paper, and tensions between the paper and other things in the literature... but fancy autocomplete does? Really?
Show threadEthan WhiteNov 11, 2023
@poritzj I think the idea is that abstracts are written for domain specialists and these summaries are intended for a more general audience, but I’d definitely prefer to see these written by people with expertise (or at the least edited by them).
Show threadporitzjNov 11, 2023
@ethanwhite
OK: scientific outreach is a good thing, but
(a) I don't think that's the way it will be used (I know a microbiologist who has been talking for months about AInt articles summaries to make his life easier) and
(b) won't the lack of understanding, context, what is important, etc., - plus the inevitable errors! - make this pretty bad outreach?
If biorxiv want to do good scientific outreach, that would be great, but I suspect this approach will do more harm than good.
Show threadEthan WhiteNov 11, 2023
@poritzj I agree. I think it's a nice big picture idea but I would have approached it very differently. Feels like some folks got a little too excited about the current capability of LLMs, but that's not uncommon these days. It's also interesting given the larger dialog around the target audience for preprints (i.e., do we even want to be doing outreach based on work that hasn't yet been formally peer reviewed yet). Not stating a position on this last bit, but it's an ongoing topic of discussion