@andreasdotorg @LukaszOlejnik Right, buffer overflows are vulnerabilities in the application that overflows its buffer, not in the compiler. And a non-transitive argument is a vulnerability in the calling program, not qsort().
One can and should take a step back and say "actually C itself is a vulnerability", but then there's no point picking on just qsort().
@jyasskin @andreasdotorg @LukaszOlejnik
I fully agree and go a step further: The signed-integer overflow is already undefined behaviour on its own, meaning that it is a bug to use this function in any context that can contain INT_MIN as second argument, even if you just call it and do nothing with the result.
This is literally the same as writing to *NULL inside the comparison-function and complaining that your program crashes: You are not allowed to do that in C!
Now, should this be prevented in glibc? I’m genuinely not sure: This is bound to be a widespread issue, but if such a fix comes at the cost of performance to people who do things correctly, this comes down to rewarding the guilty and punishing the innocents, which goes against my axiomatic ethical convictions.
@jyasskin @andreasdotorg @LukaszOlejnik
This honestly reminds me of the time someone use C++’s std::sort with a random-function to shuffle data (instead of using std::shuffle like competent people) and was pissed when that ended up crashing, because efficient sorting-algorithms rely on the specified invariants holding.
I think Sean Parent(?) shared that story and that it resulted in lots of people at the company in question (Google? Adobe?) using std::stable_sort everywhere, which didn’t crash but was just as much clear-cut undefined behavior.
(There is nothing wrong with stable_sort, besides it being underused in places where stable sorting would be desirable, but there is a reason why the algorithms are called differently and that is unstable sort tending to be faster.)
Lukasz Olejnik
Jeffrey Yasskin
Ela
Fiona