Do LLMs actually help hackers reverse engineer and understand the software they want to exploit?
We ran the first fine-grained human study of LLMs + reverse engineering.
To appear at NDSS 2026.
Interested? Some quick findings in 🧵👇
Paper: https://www.zionbasque.com/files/papers/dec-synergy-study.pdf
We implemented an LLM plugin and instrumented a decompiler to track everything:
function visits, renames, types, comments, and every LLM interaction.
48 participants (experts + novices) solved 2 CTF binaries
-> 109 hours of recorded reversing, all in-browser via @pwncollege.
First, a lot of reverse engineering is front-loaded, and LLMs do well at that task.
The very first glance (a few seconds) at a function often played a large role in determining success in understanding it.
LLMs are very good at surfacing more information on that first visit.
Novices using LLMs reached expert-level reversing rates.
Not necessarily because they reasoned better, but because LLMs give fast semantic orientation “for free," which often only comes to experts.
But if people got better at that first glance, did experts really gain anything?
Experts got negligible gains.
They offloaded known algorithms, then spent more time on custom logic.
LLMs summarize. Humans still do the hard reasoning.
We speculate that to shift this dynamic and actually help experts, new LLM collaboration methods are needed (like better MCP)
Even rare hallucinations were dangerous.
A few false vulnerability reports completely derailed participants, sending people chasing bugs that didn’t exist (for a while).
We emphasize that in RE, where you speculate often, an untrustworthy speculator can waste a lot of time
Auto-generated names and comments didn’t improve understanding.
Only artifacts created by humans correlated with comprehension.
Sometimes, the act of naming matters more than the name itself. That is often mutually exclusive from MCP-based solutions in this space.
If you care about RE, hacking, or human-AI teaming, go check out the full paper. Additionally, if you just want to mess with the same LLM interface participants had (DAILA), check the code below:
Paper: https://www.zionbasque.com/files/papers/dec-synergy-study.pdf
Code: https://github.com/mahaloz/DAILA
zh4ck
mahaloz
TinheadNed