Sue SmithPeople keep talking about llms killing saas but we were quoted a ludicrous amount for a previously free platform during the week and I'm rustling up an alternative that's completely free aside from a few days of my own labour using the magical automation that is ✨bash scripting✨
The fun part is that I gave claude a few shots at building it, it's a low stakes project, I don't really care about the implementation and just need something that works, but it failed miserably because it's a relatively novel app using niche libraries that aren't well documented and will be almost non existent in training data, but it's a super simple system I could teach a beginner coder to build
Thinking about this again and I think when these tools encounter ambiguity, from the perspective of their logic when there's no way to verify correctness from the training data, instead of coming back to the human and asking them to make a judgment they keep cycling or hallucinate an "answer"
The traditional way of resolving this would be for the human to make a choice they take responsibility for, but since these tools are partially accountability sinks they're not designed for that
The real "human in the loop" would be for the assignment of accountability lol, I find it absolutely bananas how many traditionally risk averse organisations have embraced this stuff without meaningful guardrails, but I do believe the legal shit, compliance, risk management etc will be the thing that enforces some regulation of these practices rather than the quality or reliability of the output itself
I wonder at what point the affected skills and understanding become business critical in different settings, because in practice that's when things might change, I joked about this on linkedin but I could envisage a situation similar to what happened with observability and monitoring tools as everything moved to the cloud but for codebase comprehension, when does it become urgent that your teams know how the fuck your software works
When I look at how effective incident management works and how the tooling is designed to enable human understanding and action, I feel like we're headed towards codebase understanding becoming a more central part of that world
And hopefully your incident was critical enough that your retro results in introducing interventions to build that understanding proactively in future, I can dream lol
I think llm code generation is essentially similar to using dependencies, you can outsource well worn shit to it and not need to know about the implementation, but the parts of a system that are specific to a context (domain, company, etc) remain dependent on human understanding.. The folk funding this would love to make software a production line where workers don't need to understand how what they build works but any remotely complex system can't be built that way imo
Robee? Na! 🌈@sue It’s not the first time in history people without a clue on what makes good engineering (ie. board members) are dictating what engineering should be using to solve problems. Blockchain, Web 2.0…
@sue And when it goes wrong then the accountability won’t be at the board level.
Matt Palmer@sue it's clear that the "software development as a production line" people have never, ever seen a production line, let alone know how they are created and maintained. Engineers, leading hands, and the workers on the line are constantly changing and innovating to keep it all going and adapting to ever-changing requirements.
Doug Belshaw@sue I think you hit the nail on the head r.e. accountability: whether it's LLMs or an external contractor, the aim is to claim victory for successes and point to someone/something else when things go wrong.
bogosity
Lord Kelvin@sue
I believe you’re spot-on, this whole thread.
A VC firm provided a (larger than us) company the funds to buy us, & now has a controlling interest in the new whole. We’re being forced to use AI to code. I suspect the VC also has AI investment, the way they ignore arguments about protecting our IP.
It’s gut-wrenching at times, the short-sightedness.
But your story jibes 100%.
@sue "we don't need to know how it works, we can just ask the stochastic parrot how it works"
HashRaydamon@sue that or a colosal mistake that cost either millions of dollars or outright kills people
@hashraydamon yeah unfortunately the first one is more likely to force an intervention
Juha Autero@hashraydamon There are hundreds of billions riding on this. Millions of dollars or few dead people are not big enough to have an effect.