Taylorism is a management philosophy based on using scientific optimization to maximize labor productivity and economic efficiency.
Here's the result of making the false Taylorist assumption that the output of scientific research is scientific papers—the more, faster, and cheaper, the better.
Papers are not the output of scientific research in the way that cars are the output of automobile manufacturing.
Papers are merely a vehicle through which a portion of the output of research is shared.
We confuse the two at our peril.
The entire idea of outsourcing the scientific ecosystem to LLMs — as described below — is a concept error that I can scarcely begin to get my head around.
sakana.ai/ai-scientist/
"While there are still occasional flaws in the papers produced by this first version..."
Meanwhile the authors note that the output itself fails to meet standards of scientific rigor, but treat this as a minor wrinkle, not a fundamental barrier imposed by using the wrong tool for the wrong job.
Scientists: We need to slow down the publication race and produce higher quality papers at a slower rate to make the literature manageable again.
Engineers: We hear you. Now every lab in the world will be able to produce hundreds of medium-quality papers (with a few mistakes in each) every week.
I do appreciate the authors' candor in detailing failure modes.
A system that makes difficult-to-catch mistakes in implementation, fails to compare quantitative data appropriately, and fabricates entire results—maybe I have high standards but I don't see this as writing "medium-quality" papers.
Here's the weird Taylorism again. The system produces work at the level of an early trainee requiring substantive supervision. This is not good ROI for producing papers.
The primary output of time invested in trainee research is the development of independent scientists—not the research papers.
In the end, how one judges this paper probably comes down to how one assesses the claim that is always used to justify this kind of work.
The authors "believe" that future versions will be
greatly improved.
Given what I know of fundamental limits to what LLMs can do, I see no reason to agree.
When I fail to do something, I either don't publish or very occasionally I publish describing that failure. When I do so, I don't pretend it was a success and promise that it'll magically get better.
@ct_bergstrom
> 'The authors "believe" that future versions will be greatly improved.'
I'm wondering if there's a name for this effect - where a machine does something resembling a human activity, and people ascribe it further human qualities - then assume it will advance in ability in a similar manner to a human.
See also: self-driving cars. Shuffling mindlessly around a car park? Great! It'll soon be driving perfectly.
Or not.
@FeralRobots @Spring @coprolite9000 @ct_bergstrom
And then there is the problem of an AI that has defeated the top-level human players is somehow still beatable using a strategy that only an intermediate level player would consider deploying
@coprolite9000 @ct_bergstrom
In #Robotics, it's (reasonably) well-known that people will ascribe human cognitive abilities and intent to absolutely anything.
Roboticists are particularly bad about this (see linked paper below).
I think the optimism that it will get better is not just universal but required for publication and grant approval.
Link: https://www.sciencedirect.com/science/article/abs/pii/S0921889013001863
In the 1960s, without realizing it, AI researchers were hard at work finding the features, rules, and representations needed for turning rationalist philosophy into a research program, and by so doing AI researchers condemned their enterprise to failure. About the same time, a logician, Yehoshua Bar-Hillel, pointed out that AI optimism was based on what he called the “first step fallacy”. First step thinking has the idea of a successful last step built in. Limited early success, however, is not a valid basis for predicting the ultimate success of one’s project. Climbing a hill should not give one any assurance that if he keeps going he will reach the sky. Perhaps one may have overlooked some serious problem lying ahead. There is, in fact, no reason to think that we are making progress towards AI or, indeed, that AI is even possible, in which case claiming incremental progress towards it would make no sense. In current excited waiting for the singularity, religion and technology converge. Hard headed materialists desperately yearn for a world where our bodies no longer have to grow old and die. They will be transformed into information, like Google digitizes old books, and we will achieve the promise of eternal life. As an existential philosopher, however, I suggest that we may have to overcome the desperate desire to digitalize our bodies so as to achieve immortality, and, instead, face up to and maybe even enjoy our embodied finitude.
Carl T. Bergstrom
Coprolite9000
Bornach
Ed W8EMV 
Fabian Transchel
Jon Spring
FeralRobots
Robotistry
Dianne Patterson
Gemma ⭐️🔰🇺🇸 🇵🇭 🎐
Jonas