AI coding agents : Claude Code, Codex, Copilot and their kin are changing how software gets built. Faster. Smarter. More autonomous. And that's exactly what keeps me up at night.
I’ve been playing with these agents for a bit, you can see some of the code that me and AI have authored together (Github link in the comments) - PRs are welcome!
Our adoption is outpacing our ability to secure things. Three open research problems explain why:
1. Detection is broken.
When an LLM agent runs on your developer's endpoint, what is it actually doing? Bypassing a control because it's buggy? Being "enterprising"? Or is that just a bad actor on your network? We lack the telemetry, the baselines, and the tooling to answer that question at scale. There is more noise than signal, your detection and response team isn't sleeping.
2. More code ≠ good code.
Multi-thousand-line PRs are trivial to generate. But are they correct? We abandoned KLOCs as a productivity metric in the 90s for good reason. As agent generated code volumes grow, human PR review won't scale. We need to find better ways of maintaining our invariants.
3. The supply chain problem just got recursive.
We spent years hardening software supply chains. Now we're injecting code from models trained on data we don't fully understand, via pipelines we don't fully control, into systems we absolutely must protect. And if the bad guys compromise your AI provider? Are they surveilling training data, or intentionally shaping outputs?
So I'll ask what most vendors aren't:
How are you monitoring what your AI agents do on developer endpoints and separating signal from noise?
How would you know if an agent made an "enterprising" decision that introduced a vulnerability?
How do you secure your LLM supply chain, and what will you do when they're compromised?
I don't have all the answers. And to any vendor about to drop a "turnkey solution" in my comments, I assure you: neither do you. Fair warning ahead of RSAC, sorry I won't be there.
So, keep your eyes peeled, because:
"Sleep is the cousin of death."
— Nas, Chief AI Vigilance Officer, Queensbridge (Illmatic, 1994)
S&K Scholarship
Seth Hanford 🐡