Mastodawn

Cap Ybarra

@davidgerard @doriantaylor even these nuanced takes get a lot wrong and assign too much usefulness where none is deserved

@cap_ybarra @davidgerard cool, what do they get wrong

@doriantaylor @davidgerard

"it's good for writing unit tests"

is it really though? how are you sure the tests fail in the circumstances you need them to? are all unit tests equally valuable?

"i didn't have to read the documentation for x"

so you understand all the ways it can fail, and the circumstances under which you expect it to succeed? you're sure it didn't miss some critical detail?

the answer to all of these questions, btw, comes from grokking the code, which will mean you need to read the code and toy with it enough to sufficiently inhabit it, in which case how much time did you save, really?

@cap_ybarra @davidgerard

1) at no point did i say it was *good* for writing unit tests, i just said it was *possible* to generate them; whether they're any good is a separate consideration

like these things are known to make bad tests and even alter tests to pass; my point was you can't get away with skipping test coverage whether you write the tests by hand or it generates them, because of the way it works ("works")

but as we both pointed out, no guarantee generating tests will save any time

@doriantaylor @davidgerard you're right, my read was insufficiently close. i'm a jerk

@cap_ybarra @davidgerard

2) web api client boilerplate is a consummate pain in the ass, because every vendor does theirs ever so slightly differently. it's also likely one of the things that's super well-represented in the training data, plus it'll either work when you run it or it won't; it's pretty low-risk. the failure mode is i have to correct it by hand (and it isn't like i didn't have the reference docs open contemporaneously anyway).

@doriantaylor @davidgerard but, and bear with me here, what if we decided to not accept boilerplate as an inevitable thing that must be prompted through and instead wrote some code to ease the burden

@doriantaylor @davidgerard the boilerplate could be made unnecessary deterministically instead of accepted as a price of doing business

@cap_ybarra that i agree with completely; my own work is completely deterministic and i'm a big proponent of open standards, and resent the fact that every web API is ever so slightly different from every other.

i guess my observation is that people *are* using LLMs to generate code, and will likely continue to (despite being a blunt tool in my opinion), but i suspect when measured from the outside, the net gains are going to vary dramatically.

(@davidgerard has written similar-ish things)

@doriantaylor @davidgerard i'm perhaps over sensitive to the "llms are ok sometimes" takes. they're overrepresented and you do not, under any circumstances, gotta hand it to em

@cap_ybarra @davidgerard the empirical reality i'm seeing is a) people seem to believe they're useful and b) they're already expensive, and those people are not going to want to pay 10-20x what they're currently paying, and so the situation will equilibrate eventually around locally-runnable models that are "good enough" for rote coding tasks over existing mundane procedural languages.

like i would bet money that the halo of LLM as panacea is eventually going to dissipate, but that will remain.

@doriantaylor @davidgerard i will remain steadfast in my belief that rote coding tasks are a skill issue

@cap_ybarra @davidgerard i mean maybe but i find most people just flatly do not understand higher-order programming and cannot be made to

@doriantaylor @davidgerard there are other lines of work those people might be better suited for instead of forcing us to eat slop to cover for them

@cap_ybarra @davidgerard there is nothing more than i would like to see than purely-functional, content-addressable code, but instead we get cash furnaces generating javascript and python