When I was writing The Intelligence Illusion in 2022, I put together a list potentially positive applications of LLMs that were doable without having to magic away the tech's flaws or invent something completely new. It mostly consisted of variations on "might improve debugging"
That's clearly not enough to warrant the massive investment and ongoing costs, the ethics, or buying into the technological vision of a bunch of political extremists and their allies, so I thought nothing of it
But, four years later there are indictions that this might end up partially panning out as tools for discovering and reporting security vulnerabilities
I'd be very careful in trusting these anecdotal reports
We're still in a bubble. With the money involved people have a strong incentive to believe. Other people have a strong incentive to mislead. That is a dangerous combo. We don't know what went into these "AI" security reports or how many false positives or negatives they generated
For an "AI" security software startup, on the hook to generate returns for VC investors, the temptation to spend countless hours filtering through slop reports, polish them up, and submit them to a few high-profile open source projects would be hard to resist
With billions at stake, a large constituency of believers, and lax regulation, the incentives in the market favour marketing, research, and financial fraud—you could call it a "poor information environment"
Many in tech are predisposed to believe but they should be predisposed to disbelieve
I still think that the ethical and political issues with LLMs are insurmountable. Even if we did get past that, the prompt UI is a fatal flaw, even with productive uses for LLMs as a system
But I wouldn't complain if, at some point in the distant future, we end up using improved ethically made descendents of LLMs to clean up after the code slop-acopalypse
Baldur Bjarnason
Tyr