Veviser
Sue SmithPeople keep talking about llms killing saas but we were quoted a ludicrous amount for a previously free platform during the week and I'm rustling up an alternative that's completely free aside from a few days of my own labour using the magical automation that is ✨bash scripting✨
The fun part is that I gave claude a few shots at building it, it's a low stakes project, I don't really care about the implementation and just need something that works, but it failed miserably because it's a relatively novel app using niche libraries that aren't well documented and will be almost non existent in training data, but it's a super simple system I could teach a beginner coder to build
Thinking about this again and I think when these tools encounter ambiguity, from the perspective of their logic when there's no way to verify correctness from the training data, instead of coming back to the human and asking them to make a judgment they keep cycling or hallucinate an "answer"
The traditional way of resolving this would be for the human to make a choice they take responsibility for, but since these tools are partially accountability sinks they're not designed for that
The real "human in the loop" would be for the assignment of accountability lol, I find it absolutely bananas how many traditionally risk averse organisations have embraced this stuff without meaningful guardrails, but I do believe the legal shit, compliance, risk management etc will be the thing that enforces some regulation of these practices rather than the quality or reliability of the output itself
I wonder at what point the affected skills and understanding become business critical in different settings, because in practice that's when things might change, I joked about this on linkedin but I could envisage a situation similar to what happened with observability and monitoring tools as everything moved to the cloud but for codebase comprehension, when does it become urgent that your teams know how the fuck your software works
When I look at how effective incident management works and how the tooling is designed to enable human understanding and action, I feel like we're headed towards codebase understanding becoming a more central part of that world
And hopefully your incident was critical enough that your retro results in introducing interventions to build that understanding proactively in future, I can dream lol
I think llm code generation is essentially similar to using dependencies, you can outsource well worn shit to it and not need to know about the implementation, but the parts of a system that are specific to a context (domain, company, etc) remain dependent on human understanding.. The folk funding this would love to make software a production line where workers don't need to understand how what they build works but any remotely complex system can't be built that way imo
Robee? Na! 🌈@sue It’s not the first time in history people without a clue on what makes good engineering (ie. board members) are dictating what engineering should be using to solve problems. Blockchain, Web 2.0…
@sue And when it goes wrong then the accountability won’t be at the board level.
Matt Palmer@sue it's clear that the "software development as a production line" people have never, ever seen a production line, let alone know how they are created and maintained. Engineers, leading hands, and the workers on the line are constantly changing and innovating to keep it all going and adapting to ever-changing requirements.
Doug Belshaw@sue I think you hit the nail on the head r.e. accountability: whether it's LLMs or an external contractor, the aim is to claim victory for successes and point to someone/something else when things go wrong.
bogosity
Lord Kelvin@sue
I believe you’re spot-on, this whole thread.
A VC firm provided a (larger than us) company the funds to buy us, & now has a controlling interest in the new whole. We’re being forced to use AI to code. I suspect the VC also has AI investment, the way they ignore arguments about protecting our IP.
It’s gut-wrenching at times, the short-sightedness.
But your story jibes 100%.
@sue "we don't need to know how it works, we can just ask the stochastic parrot how it works"
HashRaydamon@sue that or a colosal mistake that cost either millions of dollars or outright kills people
@hashraydamon yeah unfortunately the first one is more likely to force an intervention
Juha Autero@hashraydamon There are hundreds of billions riding on this. Millions of dollars or few dead people are not big enough to have an effect.
Nick@sue Yeah, my own experiences of testing a different LLM have been similar.
Couldn't find docs for how to do something with an obscure library, tried asking LLM for an example of how to do the thing.
It obligingly produced code but completely hallucinated the only part I needed info on. When I challenged it on this, it cheerfully admitted making it up.
A human developer who repeatedly behaved like this wouldn't stay in their job for long...
@pythoneer Yeah if they can't find examples they just seem to get into an endless spiral of generating more and more insanity
Claude@pythoneer it's horrible how humans will get fired over even mundane mistakes but LLMs generating false data over and over just get the "just one more trillion dollars and a few datacenters bro" treatment. Society's willingness to pour billions into this while refusing to secure healthcare and adequate training for human employees is appalling. @sue
RootWyrm 🇺🇦
@pythoneer @sue my favorite fucking part is that what is now also happening as a direct result is that people who were barely documenting things to begin with, now just don't document them AT ALL. "Oh, just ask the chatbot!" Nevermind that the chatbot is always fucking wrong. It also doesn't know anything that happened after 2023.
lornajane@sue what scares me is that there are people paid to work on software who think that bash scripting is some kind of old magic.
@lornajane honestly I am currently feeling like doing a bash scripting beginner coding exercise lol, it feels like the kind of activity that can give people the thrill of having made computers do stuff that prompting an llm doesn't
@sue the real irony is that I mostly write shell type scripts in python now. I do still use a Makefile in any project where I can't be overruled about that sort of choice though
@lornajane I weirdly find I write more of this type of thing now than ever, it feels especially odd as everyone is evangelising about LLMs
Mateusz 🏳️🌈@sue I’ve noticed that Anthropic’s models do excessively use these old-fashioned tools you use for bash scripting (grep, sed, awk…). It’s partially a reason for their success in coding tasks, I guess 🤷🏻♂️
John Maxwell@sue Heretic! Sorcerer!