@ikari @aral you... know that the LLM when asked to summarize code is just giving you the most likely summary based on the closest matches it could find in the training database, right?

I mean, you clearly don't understand the implications of that, so maybe you didn't know that? But your comment about the importance of understanding the tools capabilities implied that?

LLMs are a *terrible* tool for explaining or summarizing code, precisely because they will get it right some high percentage of the time and imperceptibly but potentially disastrously wrong some low percentage of the time.

To give a very concrete example: if I find a function with 17 if/else cases and hard-to-follow juggling of multiple state variables, and I just want to know: will it always return a string? If I prompt an LLM with that question, it doesn't answer the question by doing static analysis of the code paths to try to prove or disprove the answer (a type checker would do that). Instead it effectively is searching through its training data for instances where other people asked the same question about different code, with some bias towards more-similar code to the code you're looking at, and then predicting what an answer would look like based on that search. The answer will sound plausible and might even include a plausible-sounding explanation, but the relationship between the answer you get and the code you asked about is extremely tenuous. In this example if it's gives you a false positive and you use the function assuming it does always return a string, you're in for some fun later when a non-string gets returned. The worst part is that you won't remember later what the likely culprit is because when the LLM-generated answer assured you the function always returned a string, you mentally dropped your suspicions about whether that code would always return a string, so you don't even have a lingering "if it crashes due to a non-string value in a place a string is needed I should double-check function X" mental note.

@me @aral

Your example reminds me of those flimsy plastic shopping bags.

Made for single use, really handy if you forgot to bring a bag, firm chance of falling apart before you reach the kitchen, but(!) at that moment the cheap and quick way to get stuff home...

... and causing immeasurable damage for centuries on a global scale.

The right acronym for AI should be EC. Externalized Cost.

PS: I claim no moral high ground to stand on; not about plastic bags, nor AI usage. I too live in this world.

@slotos

I love this!

@aral

@daj @aral to agree pedantically here: it is *already* causing failures. Amazon admitted it outright, instituting new rules for code review (hah, not a good solution) after LLM-"assisted" changes led to several outages. Looking at Github's abysmal uptime recently it's hard not to imagine they're suffering too.

Plus there's the spectacular *lack* of any new significant proudly-LLM-coded apps entering the market. If there claims of the boosters were true, we'd be drowning in new software right now. Instead, besides apps for AI itself, I haven't heard of a single new app with a significant userbase recently. The reality seems to be: the existing apps that have pushed LLMs onto their programmers are now struggling, and new companies that attempt to use it don't succeed enough for anyone to hear about them.

@indigoparadox Perhaps. But even games have bugs and updates, etc. My reticence is mostly from a security viewpoint: even tightly-scoped libraries, etc., might have to be updated due to newly-discovered threats (either within their own code bases or the greater environment in which they operate).

And, of course, the alternative to evolution is replacement or abandonment where library A is retired and replaced with library B. Or the game gets a separate sequel, or the author halts further fixes and makes the next game…