Gabriele SveltoI've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools. Well, sed was developed in 1974 and - according to Wikipedia - first shipped in UNIX version 7 in 1979. On modern machines it can process files at speeds of several GB/s and will not randomly introduce errors while processing them. It doesn't cost billions, a subscription or internet access. It's there on your machine, fully documented. What are we even talking about?
Daniele@gabrielesvelto I don't think the comparison is entirely fair tho. Both sed and syntax tree based editing are really powerful (and I use both when it makes sense), but if you need to do a one off migration you might be spending hours trying to figure out how to make it work right, while an llm will do a good enough job where you need to review the changes and fix a few mistakes, usually at the first try, without you having to actively spend time on it.
@fourlastor what about the time spent setting up the LLM, sandboxing it and then reviewing all the changes? What about the risk of the code containing prompt-injections that might be designed to introduce vulnerabilities or simply take over your machine or credentials for a state-sponsored attacker to use? What about the reliance on a single closed-source paid-for commercial tool? Those are a lot of disadvantages to make up for.
@gabrielesvelto answering in order:
>what about the time spent setting up the LLM, sandboxing it and then reviewing all the changes?
This for what I'm working on is usually between 30 and 40 minutes, start to end (minus the time that the LLM takes to do its own work in its own git subtree, while I do other stuff). For context, claude doesn't commit, I review the changes locally (git is blacklisted). In my case this is been pretty stable on 100-150 tasks where I did the same kind of migration
@gabrielesvelto prompt-injections
The project is closed source, and we don't have places where we randomly include text files, if someone IN THE COMPANY manages to introduce malicious code, imho they'd just infect gradle instead of hoping on someone running an LLM to trigger something (other than devs having access to only what they need). State sponsored hackers specifically are really not in my list of things I can defend from, be it from LLMs or whatever introduced attacks
@fourlastor you don't need to do anything special to be a target of state-sponsored actors if your rely on an LLM for your coding tasks. State-sponsored actors have almost certainly poisoned the training data of major commercial LLMs, you don't need to add anything yourself. Remember, these things are trained on anything that's dredged from the internet. *Anything*. Do you really trust what happens within the model? Remember the xz compromise? It can now be done automatically *at scale*.
@gabrielesvelto and ok, but what is the *actual* scenario you're imagining? because my coding tasks go as such when I use LLMs:
1. I have 10-15 classes that need to change the way we do X from Y to Z
2. I prompt the LLM, telling it "change A,B,C so that they use Z instead of Y"
3. I review the code, fixing mistakes as I see them
1/x because post length limits
1. I have 10-15 classes that need to change the way we do X from Y to Z
2. I prompt the LLM, telling it "change A,B,C so that they use Z instead of Y"
3. I review the code, fixing mistakes as I see them
1/x because post length limits
@gabrielesvelto
The code change is frankly pretty simple, we're talking of stuff on the level of "migrate Book so instead of using function calls, uses annotations for ABC, update the call sites", we're not talking about "change this complex piece of code so that it does complex ABC in another complex XYZ way". The realm of errors is "I know that Foo doesn't work well by itself and needs extra care"
The code change is frankly pretty simple, we're talking of stuff on the level of "migrate Book so instead of using function calls, uses annotations for ABC, update the call sites", we're not talking about "change this complex piece of code so that it does complex ABC in another complex XYZ way". The realm of errors is "I know that Foo doesn't work well by itself and needs extra care"
@gabrielesvelto anything that goes over the bar of "this is stupid but boring" goes into the "I'll do it by hand because if anything I need to learn how it works before touching it"