Jack RusherāThe LLM generated what was described, not what was needed.ā
https://blog.katanaquant.com/p/your-llm-doesnt-write-correct-code
GLCThere's one line there that would be valuable in a broader discussion (beyond coding):
"This is not a syntax error. It is a semantic bug:"
Given that LLMs don't even have a semantic component, this seems foreseeable. But outside the specific context of coding I see people struggling to formulate the point cogently.
This would be relevant to contexts like technical writing, etc., but I don't think the vocabulary exists for expressing it neatly in such cases.
tuban_muzuru
Jš
įįįµįįįį¢į¾į į¾įį¾įį
@jack Unfortunately, I've had the honor working with senior developers who operate in similar way. I mean, they produce code much faster than they think (or that anyone else can read/review). Creating huge "working" unmaintainable code bases that reinvent every possible wheel. Sales and management are super happy. Then someone else have to spend years cleaning up their mess.
They are senior and super fast so everyone has to adapt. After all, these devs produce lot of code and must be pro...
@jack I recently spoke to one of these developers. He claimed that his greatest use of coding LLMs is to reduce the size and complexity of the code he writes. He simply can't stop himself writing insane amounts of code. I'm quite the opposite. An interesting anecdote I believe. People are different.
@jack My current favorite is āGenerate a list of this kind of organizations (there are n of them) and their CEOsā
āI created a list [well done], but I couldnāt find all CEOs, so I guessed the rest based on the names I found!ā
The result looks plausible, of course, so the LLM did its job :-)
āI created a list [well done], but I couldnāt find all CEOs, so I guessed the rest based on the names I found!ā
The result looks plausible, of course, so the LLM did its job :-)
@chris They are not grounded in world models, so poor performance on tasks involving facts is not very surprising š¤·āāļø