Papers are not the output of scientific research in the way that cars are the output of automobile manufacturing.

Papers are merely a vehicle through which a portion of the output of research is shared.

We confuse the two at our peril.
‪
The entire idea of outsourcing the scientific ecosystem to LLMs — as described below — is a concept error that I can scarcely begin to get my head around.

sakana.ai/ai-scientist/

"While there are still occasional flaws in the papers produced by this first version..."

Meanwhile the authors note that the output itself fails to meet standards of scientific rigor, but treat this as a minor wrinkle, not a fundamental barrier imposed by using the wrong tool for the wrong job.

Scientists: We need to slow down the publication race and produce higher quality papers at a slower rate to make the literature manageable again.

Engineers: We hear you. Now every lab in the world will be able to produce hundreds of medium-quality papers (with a few mistakes in each) every week.

I do appreciate the authors' candor in detailing failure modes.

A system that makes difficult-to-catch mistakes in implementation, fails to compare quantitative data appropriately, and fabricates entire results—maybe I have high standards but I don't see this as writing "medium-quality" papers.

Here's the weird Taylorism again. The system produces work at the level of an early trainee requiring substantive supervision. This is not good ROI for producing papers.

The primary output of time invested in trainee research is the development of independent scientists—not the research papers.

In the end, how one judges this paper probably comes down to how one assesses the claim that is always used to justify this kind of work.

The authors "believe" that future versions will be
greatly improved.

Given what I know of fundamental limits to what LLMs can do, I see no reason to agree.

When I fail to do something, I either don't publish or very occasionally I publish describing that failure. When I do so, I don't pretend it was a success and promise that it'll magically get better.

@ct_bergstrom
> 'The authors "believe" that future versions will be greatly improved.'

I'm wondering if there's a name for this effect - where a machine does something resembling a human activity, and people ascribe it further human qualities - then assume it will advance in ability in a similar manner to a human.

See also: self-driving cars. Shuffling mindlessly around a car park? Great! It'll soon be driving perfectly.

Or not.