Mastodawn

finding myself repeating "fuzzers are stochastic, with enough cpu time you will always find a bug, results are a usually demonstration of resources not algorithms"

like, hacking, along with spam and fraud are the sorts of activities where things only need to work 0.1% of the time to be successful

it isn't a demonstration of clever code or tooling but the uncompromising effectiveness of sheer brute force heh

Show thread

✧✦Catherine✦✧1d ago

@tef i don't think this is correct, actually. afl (when it got introduced) has massively reduced the amount of resources because it is coverage-guided

Show thread

tef 1d ago

@whitequark * unless comparing to other fuzzers

like if you only have a list of bugs there's no study on the techniques

@tef yea sure

@whitequark was genuinely mulling over adding it but i shrugged as my post was already too wordy heh

Show thread

✧✦Catherine✦✧1d ago

@tef i do think it's relevant. LLM-based fuzzers do seem like a significant and consequential achievement, for better or worse; not as world-changing as some fearmonger, but still something we have to take into account

Show thread

Graham Sutherland / Polynomial 1d ago

@whitequark @tef the main thing they seem to be novel at is guiding based on semantics on top of the traditional deterministic analysis methods. which is very neat, but yeah, as you say, not world changing.

Show thread

Graham Sutherland / Polynomial 1d ago

@whitequark @tef also be extremely suspicious of anyone who references any article that's like "here are hundreds of REAL VULNERABILITIES identified by an LLM!" because not a single one of those I've looked at has had anything close to 10% of them be real actual security bugs, and most of the rest are low/info stuff you could've found with a linter.

Show thread

✧✦Catherine✦✧1d ago

@gsuberland @tef yeah i don't care about that, my assessment is based on personal examination of bug reports and talking to some people who i expect are not full of shit

Show thread

✧✦Catherine✦✧1d ago

@gsuberland @tef realistically we'll have a 1-2 year spike in discovered bugs and then that'll smooth out to normal, like it happens every time when fuzzers improve

we are definitely not facing a "Skynet" scenario, as some tech people would lead you to believe

Show thread

Graham Sutherland / Polynomial 1d ago

@whitequark @tef and I am fairly confident that a large part of the 1-2 year spike will primarily be driven by simple renewed interest in fuzzers, rather than the actual capabilities of an LLM integration.

Show thread

✧✦Catherine✦✧1d ago

@gsuberland @tef I'm reserving judgement on that until I get a lot more experience

Show thread

Graham Sutherland / Polynomial 1d ago

@whitequark @tef fair. my experience is that "oh neat, I wonder what would happen if I ran it on X" is a powerful force. every time there's some new fuzzing or SAST toy people wanna throw it at everything to see what drops out, and that often results in more bugs being found simply because nobody had bothered on that target before. they could've been found with the prior versions of the new toys, it's just that the new shiny got someone to run it.

Show thread

✧✦Catherine✦✧1d ago

@gsuberland @tef people definitely do run a lot of fuzzers on e.g. curl

Show thread

Graham Sutherland / Polynomial

@whitequark @tef yeah for sure. I mean more in a "total bugs being discovered across software" sense. on specific heavily fuzzed targets then obviously new things are indicative of new capabilities. some of that will come directly from the new shiny, some of it will come from basic grunt work that was done because someone finally got around to implementing it, either by coincidence or because the new shiny made them interested enough in the fuzzer project to make a PR.