nate berkopecLLMs do not meaningfully "refactor" at anything other than a junior engineering level. They can basically do some window dressing and move code around between files. True refactoring means creating new abstractions, which LLMs can't do because they can't form world-models.
José Albornoz@nateberkopec @jardo wasn’t it in 2023 when AI started producing better sorting algorithms than we could as humans? https://deepmind.google/blog/alphadev-discovers-faster-sorting-algorithms/
AlphaDev discovers faster sorting algorithms
In our paper published today in Nature, we introduce AlphaDev, an artificial intelligence (AI) system that uses reinforcement learning to discover enhanced computer science algorithms – surpassing those honed by scientists and engineers over decades.
Jared Norman@nateberkopec @eljojo yeah, that's neat but AlphaDev isn't an LLM and what it was doing wasn't refactoring.
DaveyYeah if you hear that AI was used to do something groundbreaking, it's always* something made specifically to do that groundbreaking thing, very often reinforcement learning, and never an LLM.
*so far
@jardo @nateberkopec AlphaEvolve, using LLMs, has advanced the state of the art in matrix multiplication and other fields like hardware design. https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
how do you define refactoring? what I see here is that we have new algorithms that work on the same input and produce the same output as previous algorithms, but do it better. that’s a refactoring to me.
the software rewrote parts of verilog to make it faster.
@eljojo so? optimization has completely different goals than refactoring despite both sharing the requirement of not changing the output. this all has nothing to do with what Nate was saying
@jardo it really feels like we’re splitting hairs on what “true refactoring” means. Nate’s thesis is that the machine can’t go beyond what a junior engineer can do, because they lack world models, while I see the machine making better code than we humans can. I guess my thesis is that you don’t need a world model in order to write superior abstractions?
@eljojo optimization isn't about abstractions... you've completely missed the point of what Nate was saying... I'm out.