tef

finding myself repeating "fuzzers are stochastic, with enough cpu time you will always find a bug, results are a usually demonstration of resources not algorithms"

like, hacking, along with spam and fraud are the sorts of activities where things only need to work 0.1% of the time to be successful

it isn't a demonstration of clever code or tooling but the uncompromising effectiveness of sheer brute force heh

Apr 8 at 1:44pmMastodon for iOS
Show threadPozorvlak23h ago
@tef dude, this is cope. What you say is all true, but they have (apparently) dramatically reduced the resources required to find and exploit vulnerabilities which resisted many rounds of previous investigation. This is a big deal.
Show threadCassandrich22h ago
@pozorvlak @tef Sounds more like they've drastically subsidized the resources to push the narrative they want investors and corporate C suites to buy.
Show threadLars Marowsky-Brée 😷19h ago

@pozorvlak @tef I'd question they've "dramatically reduced" the resources, given that they've thrown a metric fuck ton of resources at the problem space.

But yes, they also have found very severe real issues and dismissing this as "any linter/fuzzer could have" is a bit of a cope - because they clearly _didn't_ in the past.

So the real world impact of this is, well, real.

Show thread✧✦Catherine✦✧23h ago
@tef i don't think this is correct, actually. afl (when it got introduced) has massively reduced the amount of resources because it is coverage-guided
Show threadtef22h ago

@whitequark * unless comparing to other fuzzers

like if you only have a list of bugs there's no study on the techniques

Show thread✧✦Catherine✦✧22h ago
@tef yea sure
Show threadtef22h ago
@whitequark was genuinely mulling over adding it but i shrugged as my post was already too wordy heh
Show thread✧✦Catherine✦✧22h ago
@tef i do think it's relevant. LLM-based fuzzers do seem like a significant and consequential achievement, for better or worse; not as world-changing as some fearmonger, but still something we have to take into account
Show threadGraham Sutherland / Polynomial22h ago
@whitequark @tef the main thing they seem to be novel at is guiding based on semantics on top of the traditional deterministic analysis methods. which is very neat, but yeah, as you say, not world changing.
Show threadGraham Sutherland / Polynomial22h ago
@whitequark @tef also be extremely suspicious of anyone who references any article that's like "here are hundreds of REAL VULNERABILITIES identified by an LLM!" because not a single one of those I've looked at has had anything close to 10% of them be real actual security bugs, and most of the rest are low/info stuff you could've found with a linter.
Show thread✧✦Catherine✦✧22h ago
@gsuberland @tef yeah i don't care about that, my assessment is based on personal examination of bug reports and talking to some people who i expect are not full of shit
Show thread✧✦Catherine✦✧22h ago

@gsuberland @tef realistically we'll have a 1-2 year spike in discovered bugs and then that'll smooth out to normal, like it happens every time when fuzzers improve

we are definitely not facing a "Skynet" scenario, as some tech people would lead you to believe

Show threadGraham Sutherland / Polynomial22h ago
@whitequark @tef and I am fairly confident that a large part of the 1-2 year spike will primarily be driven by simple renewed interest in fuzzers, rather than the actual capabilities of an LLM integration.
Show thread✧✦Catherine✦✧22h ago
@gsuberland @tef I'm reserving judgement on that until I get a lot more experience
Show threadGraham Sutherland / Polynomial22h ago
@whitequark @tef fair. my experience is that "oh neat, I wonder what would happen if I ran it on X" is a powerful force. every time there's some new fuzzing or SAST toy people wanna throw it at everything to see what drops out, and that often results in more bugs being found simply because nobody had bothered on that target before. they could've been found with the prior versions of the new toys, it's just that the new shiny got someone to run it.
Show thread✧✦Catherine✦✧22h ago
@gsuberland @tef people definitely do run a lot of fuzzers on e.g. curl
Show threadGraham Sutherland / Polynomial22h ago
@whitequark @tef yeah for sure. I mean more in a "total bugs being discovered across software" sense. on specific heavily fuzzed targets then obviously new things are indicative of new capabilities. some of that will come directly from the new shiny, some of it will come from basic grunt work that was done because someone finally got around to implementing it, either by coincidence or because the new shiny made them interested enough in the fuzzer project to make a PR.
Show threadGraham Sutherland / Polynomial22h ago
@whitequark @tef when it's something like the linux kernel or android or chromium then of course this doesn't apply.
Show threadDominykas Blyžė19h ago
@gsuberland @whitequark @tef it might be that 90% of those reports are invalid, and slop sucks, but as someone on the side where I have tools alerting me about CVEs in my code I have my workload increased significantly over the past three months, so there's definitely signal somewhere in there too.
Show threadGraham Sutherland / Polynomial19h ago
@dominykas @whitequark @tef I talked about this more in this thread, but generally yes.
Show threadtef22h ago
@whitequark oh, for sure, being a script kiddie has gotten a little bit easier while token prices are low heh
Show thread✧✦Catherine✦✧22h ago
@tef it seems to be a pretty big help for defenders too, in a way that wasn't the case a year ago. i've been meaning to evaluate the efficiency, though haven't gotten around to it
Show threadsilverwizard21h ago
@tef Seriously! I've been reading the bugs they found - and the reason these are old bugs is that they're bugs that any linter would have found but no one bothered to run against ancient working code.
Show threadmx alex tax1a - 2020 (6)21h ago
@tef @rootwyrm and the slop companies are running a protection racket in open source: "nice project you got there. accept our slop vulnerability fixes and our slop triage for the slop reports, or else" and people are knuckling under (:
Show threadtef20h ago
@atax1a @rootwyrm it's that or vlc drops support for 16bit xbm files
Show threadGabriele Svelto17h ago
@tef as they say, in all things computing if brute force doesn't work it's because you're not using enough of it
Show threadAddison16h ago
@tef of course, the cleverness of that brute force is a factor

This post brought to you by someone who's entire PhD rests on fuzzing
Show threadLisPi15h ago

@tef > with enough cpu time you will always find a bug, results are a usually demonstration of resources not algorithms"

I'm inclined to think that's largely a reflection of bad design (either of the program, tools or language).

If the program can enter invalid states, it supports logical operations to put it in such a state (unless you're fuzzing for hardware flaws, at which point the program is incidental and barely if at all related). That's bad design (or implementation thereof, it's quite possible to have a good design and screw up the coding part) unless it specifically must support arbitrary operations for its intended purpose.

Some languages make proper implementation of a design harder though, such that running fuzzers can be useful when for whatever reason proper semantics for the intended operation cannot be directly encoded. (The real solution is discontinuing use of those languages or use without tooling that provides such properties.)