Stefan Eissing

The security reporting situation that I see at the ASF and in #curl is

- huge increase in reports
- increase of valid reports
- appearance of duplicate/triplicate reports of the same issue by different people

A high profile project needs to deal with 2-4 new reports each day. This is nuts.

One *may* hope this to go down again later this year bc
- unhallucinated issues are finite (see the fuzzing wave)
- eventually it will cost real money to generate these reports

Apr 4 at 7:28amWeb
Show threadMartin ███████2d ago
@icing I almost, but not quite yet, see a market for CAPTCHA programming challenges that are trivial for a human but will cause LLMs to choke. 🤢
Show threadStefan Eissing2d ago
@unixtippse It's actual humans submitting those, using LLM tools. And they find valid bugs.
Show threadÉtienne Parmentier2d ago
@icing based on your description, it sounds like report submiters don’t check if the issue was already reported ? Or are they spamming reports intentionally and Hope one is getting through ?
Show threadck2d ago
@tinylittleenormous @icing You cannot see undisclosed reports, so I'd think it's not unlikely to get duplicates since everyone is using all the same tools.
Show threadÉtienne Parmentier2d ago
@icing @ck that’s why ! Thx
Show threadRoger Sen2d ago
@icing a reporting tool able to identify duplicates (reports around he same bugs) would be useful and not to difficult to code.
Show threadSteve Loughran2d ago
@rogersm @icing issue is that security reports are not processed through the same issue tracker, and if there is a real CVE they'll be fixed "discreetly" and not fully announced until a patched release ships.
If the same shipped code is scanned multiple times by the same agent, it's probably going to find the same issues, real or not
Show threadSteve Loughran2d ago

@icing everyone reporting a real vulnerability to a project is at least someone not exploiting it.

I wonder how well the same LLMs do vulnerability scans of decompiled code? If they can do that then closed source binaries may be equally exposed

Show threadElias Probst2d ago
@icing if those valid reports are based on tool (LLM?) usage, there's a case to be made to improve languages/frameworks/pre-commit analysis to catch those earlier in the future and to reduce the post-release noise & efforts.
Show threadNils Goroll 🕊️2d ago

@icing i am not sure if my mental model of what is happening in the "useful llms" corner is adequate, but i lean towards "if you look you will find": if llms stir up potential candidates for security issues, attention shifts towards improvements.

in this way i think your reference to the fuzzing wave is helpful, as presumably it was driven by the same overall principle. #llm

Show threadwiredfool2d ago
@icing Seeing something similar with pillow in the last month or so, a significant uptick in valid reports, and one duplicate.