My experience with generative-AI has been that, at its very best, it is subtly wrong in ways that only an expert in the relevant subject would recognise. So I don't worry about us creating super-intelligent AI, I worry about us allowing that expertise to atrophy through laziness and greed. I refuse to use LLMs not because I'm scared of how clever they are, but because I do not wish to become stupider.
I will say one thing for generative AI: since these tools function by remixing/translating existing information, that vibe programming is so popular demonstrates a colossal failure on the part of our industry in not making this stuff easier. If a giant ball of statistics can mostly knock up a working app in minutes, this shows not that gen-AI is insanely clever, but that most of the work in making an app has always been stupid. We have gatekeeped programming behind vast walls of nonsense.
We seem to have largely stopped innovating on trying to lower barriers to programming in favour of creating endless new frameworks and libraries for a vanishingly small number of near-identical languages. It is the mid-2020s and people are wringing their hands over Rust as if it was some inexplicable new thing rather than a C-derivative that incorporates decades old type theory. You know what I consider to be genuinely ground-breaking programming tools? VisiCalc, HyperCard and Scratch.
You know what? HyperCard was a glorious moment in time that I dearly miss: an army of non-experts were bashing together and sharing weird and wonderful stacks that were part 'zine, part adventure game and part database. Instead of laughing at vibe-coders, maybe we should ask ourselves why the current state-of-the-art in beginner-friendly programming tools is a planet-boiling roulette wheel.

@jonathanhogg this is my central response to the "AI makes software development accessible" argument.

Once upon a time anyone could program their personal computer using a book that came with it. We taught it to all the kids in my tiny town's elementary school. My shopkeep neighbor and our local mechanic wrote their own custom software with no CS background.

BASIC, Hypercard, personal computers, printed manuals > LLM's.

@requiem @jonathanhogg I am old enough to remember the days before gcc when I wanted to learn how code everyone laughed and said first I'd need $500 for a copy of the Borland C compiler. It wasn't until linux became popular that I actually had a way to compile my C programs.
@skotchygut I'm pretty sure I still have a "backup copy" of Turbo C++ somewhere... 😇 @jonathanhogg
@requiem @skotchygut believe it or not, I actually learned C on the BBC Micro in the 80s with the Beebug C compiler, which was a strange contraption that compiled to a 16-bit virtual machine code and then interpreted that on the 8-bit 6502