Fabian GiesenOct 4, 2023

"OMG it's a 25GB game patch, must be tons of amazing new content that they're keeping quiet about."

*Could* be, sure, and stealth launches are a thing sometimes, but generally speaking, if it's a big patch for no discernible reason not attached to anything upcoming that's already been announced, the actual reason is probably something very boring and usually non-game-related

Show threadFabian GiesenOct 4, 2023

(Very) incomplete list of reasons for big patches that are completely unrelated to new content, from personal experience:

- Somebody added an extra field to some header somewhere, so everything that comes after the header moves and has slightly different context, plus content chunking puts chunk boundaries in different places now, so all the compressed data after changes
- We updated compilers and our assets are built using fast math, YOLO

Show threadFabian GiesenOct 4, 2023
- Sound assets use Opus in float (not fixed-point) mode which uses RCPSS/RSQRTSS during encode, and we switched our build machines between Intel and AMD, so presto, new audio for everyone
- we changed audio codecs
- we changed lossless data codecs for some or all data
- we changed _something_ and don't know for sure what or why it's a 15GB patch but we only noticed 4 days before cert and it was too scary to figure out and fix before then, so here's hoping it's fine
Show threadFabian GiesenOct 4, 2023
- this one simple bugfix in our vector math library had bigger ripple effects than we thought
- this one simple bugfix in the platform libm had bigger ripple effects than we thought
- this one simple bugfix in our compiled asset caching had bigger ripple effects than we thought
- who are we kidding, past a certain scale there's really no such thing as a simple bugfix
Show threadFabian GiesenOct 4, 2023
- given all of the above, reproducible asset builds are Actually Hard, therefore we use content-addressed storage and just cache build artifacts... unfortunately something went wrong and corrupted that store so we had to rebuild everything and this is what happened
Show threadstilescrisisOct 4, 2023
@rygorous This brings up repressed World of Warcraft memories... I built the asset server. 😰
Show threadFabian Giesen
@stilescrisis I built one 15 years ago for the (long defunct) game company I was at at the time, and felt simultaneously validated and disheartened after coming to Epic and learning UE had a conceptually very similar system (DDC) with mostly the same strengths and headaches...
Oct 4, 2023 at 10:05pmWeb
Show threadstilescrisisOct 4, 2023
@rygorous WoW's system is probably a lot more database-centric than most, but a lot of these asset problems are very relatable. (Although we didn't ever YOLO things... if there was a large patch, we'd know why and it'd have a darn good reason.)
Show threadFabian GiesenOct 4, 2023

@stilescrisis I've perpetrated one of these that should've been a database but wasn't, interacted with a few more, and at Epic UE has its own variant (DDC) with multiple backends that offer different sets of guarantees.

Our monitoring for this for Fortnite has some gaps and systemic problems (that are being worked on). I wish I could say we never rolled out a big patch primarily because we noticed too late to realistically fix it, but I'd be lying.