I’ve been thinking a lot lately about the hollowing-out of software. Specifically, how do I protect myself from a formerly useful program auto-updating with vibe-coded (and potentially compromised) AI-slop?
More and more development teams are getting axed. Their corpses are getting stuffed with cheap, rotten code that is at best unfit for purpose, and at worst actively dangerous.
I can’t possibly track of it all. It seems a matter of time before something bites me.
Obviously, turning auto-updates off is also extremely unsafe.
But when you fire your engineering department, demand ten times the output from the survivors, and then outsource that output to the Lying Machine, how can I trust that in six months anyone at your company will know what your code does anymore?
This isn’t a problem that can be solved with using open-source tools either. As we saw with the whole chardet incident, volunteer communities are equally vulnerable to this thinking.
Everyone is rightly upset about the plagiarism, and the class war angle of it. But I don’t really see anyone talking about the (in my opinion) very real possibility of software declining in reliability across the board.
It’s one thing if your check flow has errors in it. I might be annoyed if my shopping takes 15% longer, or if there are 15% more bugs, or your shitty chatbot hallucinates whenever I’m trying to get support.
Those things are definitely annoying, but I’ll survive.
But at this point EVERYTHING touches a computer at some point. When things start going wrong in a hospital or prison, or within the government, someone is going to die.
The risk introduced by each Silicon Valley idiot-factory compounds the more that those systems interact with one another.
So I genuinely wish I could just deep-freeze my devices until we can sort this nonsense out.
If you were hoping for an end of thread insight, some kind of solution or proposal, I’ve got bad news for you. I’ve got nothing.
All I know is that my threat model is a whole lot broader recently.
Trying to think ahead just a little bit, it seems safer to assume that every program which can update without my input, will in short order become a liability.
And that I better start planning how I can best navigate the world without my phone or my laptop, even when those things are mandatory.
@Haste My hope, and I know you already know this, is that regulated industries like medical device software will be some of the first to hit the termination shock of the reality of vibe coding.
And I know that won't stop everyone else from doing it ... but it'll (I hope) at least show that someone knows it's not viable.
I absolutely hate the idea that it's going to take adverse patient effects to get people to start to back off. I wish we could learn the lesson without having to experience the harm.
@alex oh god I hadn’t even thought about outages just becoming more common.
In some ways that almost sounds preferable to a program being confidently wrong in a way that you have to be paying attention to notice.
I think I’d rather have a traffic light turn off than give me the wrong answer, you know?
EVHaste
The Witch of Crow Briar
C Sawula
Rick Osborne
Alex Hyett