- ok, LLM find me bugs in this codebase
- aye aye! I found a lot of bugs! :heart: :thumbup: :rocket:
- categorize them, keep the ones you're the most confident with
- done! in /tmp/vuln.md file, one is CVSS10, 100% confident!
- ok, LLM analyze all the bugs in /tmp/vuln.md and tell me
if you're confident with the bug?
- mmmh, not sure, but bugs are here
- write a poc.py
- done!
$ ./poc.py
(nothing happens)
$
- ok, LLM, can you analyze vuln.md and poc.py to figure whats wrong?
- I don't think there is abug, the code is unreachable, structure of data is bad.
- Try again
- wrote /tmp/poc2.py
- still doesn't work
- Maybe no bug.

Tons of manual review later in obscure code format.
Ok, no bug, code is good, all paths are checked correctly.

Are LLM really helping?

At least, poc.py is consistent. It does nothing. There's no bug either.

Maybe that's the real truth.

Hesitate between crying or closing the laptop lid with force.