mmitchell_ai
Great reporting from @benjedwards on where we're at now with understanding GPT-4 perfomance "How are we meant to build dependable software on top of a platform that changes in completely undocumented and mysterious ways every few months?- simon willison (1/2)
https://arstechnica.com/information-technology/2023/07/is-chatgpt-getting-worse-over-time-study-claims-yes-but-others-arent-sure/
Study claims ChatGPT is losing capability, but some experts aren’t convinced

Either way, experts think OpenAI should be less opaque about its AI model architecture.

Ars Technica
Jul 19, 2023 at 11:49pmWeb
Show threadmmitchell_aiJul 19, 2023
@benjedwards But I also want to note Jiahao Chen @jiahao and Anima's insightful thoughts here about RLHF destroying the calibration of base models and the potential degradation of performance being a function of the bias/variance tradeoff RLHF plays into.