Took me a day to figure out what was happening.
After checking my code a few times I isolated it out on being related to the DFG LUT.
Inverting its value (one minus) was somehow fixing the shading and brightness issue. This was very confusing.
Then I extracted the LUT from Filament and compared it from Learn OpenGL and mine.
Here is what they look like in Designer:




























