Mastodawn

@lisyarus
> In reality it is a pretty deterministic (modulo compiler options, CPU flags, etc)

and then det(v, v) suddenly doesn't return 0.0 anymore because the compiler now targets a newer CPU with FMA and uses regular multiplication for `a.y * b.x` and FMA for `a.x * b.y - prev_result`, and FMA uses infinite precision so `a.x * b.y` can be slightly different from the result of the regular multiplication -_-

It annoys me that IEEE754 allows this and compilers (ab)use it

Arseny Kapoulkine 1d ago

@Doomed_Daniel @lisyarus This is not an IEEE754 problem; IEEE754 defines FMA and unfused mul/add. It's up to each language to figure out how to compile compound expressions and what is permissible; C/C++ allow contraction by default, so you do get different results based on the platform or the underlying model - but you can also turn contraction off to fix this, or use languages that don't allow this implicit conversion.

@zeux @lisyarus IEEE754 explicitly allows contractions

IMHO compilers should put them behind -ffast-math or similar, not enabled by default

Arseny Kapoulkine 1d ago

@Doomed_Daniel @lisyarus I agree that contraction should not be enabled unconditionally; Rust is an example of a language that doesn't do that - it follows IEEE 754 strictly. IEEE 754 does not explicitly allow contractions. In fact it has language that prohibits these by default, which C doesn't follow (clang will synthesize fma unconditionally without optimizations when supported)

@zeux @lisyarus
IEEE 754 at least seems to allow this as long as it can be switched off, doesn't require it to be disabled by default - or at least that's what the C/C++ standards and/or compilerwriters think

(GCC is extra fun because it only allows using -ffp-contract but ignores the FP_CONTRACT pragma)

@Doomed_Daniel @zeux @lisyarus again, that's not IEEE 754.

But it's not like the FP standard gets a vote on this. The language standards do whatever they want.

See also Kahan's rants on Java's choices wrt FP implementation in the 90s and early 2000s. Mind, in a lot of that he's railing against the idea of trying to aim for exact reproducibility since it was at the time impractical. By now it wouldn't be, though.

@rygorous @zeux @lisyarus
IIRC in a discussion about this years ago (and/or GCC bugtracker?) compiler-adjacent people argued that it does not violate IEEE 754, because the standard allows it as long as it can be controlled with a flag, or sth like that.

The language standard can do whatever it wants, but not claim that it conforms to IEEE 754 then?

@Doomed_Daniel @zeux @lisyarus Again, I don't understand what power you think the standard has here, though.

They can write all kinds of things into IEEE-754 that are then summarily ignored.

This is pretty much exactly how IEEE 754s traps have been handled by most langs for the past 40 years.

@Doomed_Daniel @zeux @lisyarus See also langs and environments that always treat subnormals as 0 and so forth.

It's just words on a page. What do you think the enforcement mechanism is?

The reality of the matter is that anything written in IEEE-754 is exactly as important as its users think it is. Historically speaking, very good track record on the basic stuff (correctly rounded fundamental operations, RN default, certain math lib fns being available), beyond that, very touch and go.

@Doomed_Daniel @zeux @lisyarus For most of the 80s and 90s we had a bunch of very different evaluation strategies, at least three:
1. basic RISCs with just +,-,*,/,sqrt and either only float or float+double but no extended format (and often limited support for traps etc.)
2. RISCs with FMAs (usually of the "almost everything reduces to FMA variety", e.g. the RS/6000, PPC and descendants)
3. M68k and x87 with internal extended precision and extra exponent range.

Fabian Giesen

@Doomed_Daniel @zeux @lisyarus all three of these provided reasonable error bounds in the same ballpark but disagreed on the exact results, and simulating each of those with any of the others was impractically expensive. Hence compilers doing a best-effort mapping and language standards not saying much on the topic. What IEEE thinks about it, nobody really asked in the first place.

@Doomed_Daniel @zeux @lisyarus at this point, #3 is functionally dead, and most things that used to be in category #1 by now have moved to #2, i.e. FMAs used to be a "depends on the platform" thing and are now almost everywhere.

The side effect being that a thing that used to show up on ports to a different platform (i.e. going non-FMA <-> FMA targets) is now something that is showing up within targets, and the numerical env in general is much more homogeneous.

@Doomed_Daniel @zeux @lisyarus My point being, legislating any particular evaluation strategy was wholly impractical until at least the 2010s or so.

At this point it would be a lot more reasonable. It's just that we have a path-dependency where both the compiler options and the standards come from a world where insisting that things be nailed down in a particular way was simply impractical, and what compiler support there is for turning this off is also what used to be a total niche feature.

@Doomed_Daniel @zeux @lisyarus Because in the 90s, if you're porting numerical SW to say a PPC, you wouldn't want to not use FMAs, for the most part.

That's making a lot of things 2x more expensive because even just basic FP mul and FP add on that HW would execute as FMAs, and their was only one FP arithmetic unit which was the FMA unit!

Standards have more recently started to try to tighten down on things but it's just a hard retro-fit due to many historical accidents.

@Doomed_Daniel @zeux @lisyarus FWIW I also agree that FP contractions being enabled by default is very unfortunate these days, but it makes perfect sense for that to be the default given how we got the option in the first place, namely, it was introduced on HW where FMA was the only way to do FP adds or muls and turning contractions off could literally halve your FP throughput.