Was thinking again a while ago what waste PBR textures can be under most lighting.
Kind of weird to do a 4x texture memory increase - assuming BC1-5 and no alpha/metalness, e.g. BC1 base color + BC5 normal map & BC4 roughness - that will only show up under specific lighting conditions and elsewhere appears flat.
Though doubling texture resolution in both dimensions is also a 4x increase that might never show up (esp. with upscaling) so all things considered, maybe 4x isn't that bad.
@archo Yeah but at sensible resolutions, the higher-rez textures will never be loaded. So all they're wasting is disk space and their own production time. Whereas PBR is burning my precious DRAM for minor LSB differences. Boooooo.
(I say this with honest love to all my PBR shader writers)
(it's a joke. This is a bit)
(or is it)
@TomF Texture loading time from disk and download time/bandwidth would be wasted as well, which seems relevant with modern 150 GB games. (Also not a lot of such games fit on 1 TB consoles.)
But I suspect for high res textures there could be a way to stream the biggest mips on-demand from the CDN to the GPU based on what the GPU needs which AFAIK nobody's doing yet. PBR seems uniquely disadvantaged in this regard (but at least the number of parameters doesn't seem to be growing infinitely).
@rygorous @TomF IIRC those methods mainly dealt with correlated color/normal/roughness. Though I've seen a post from 2020 that deals with tile repetition as well.
Many emissive/metalness maps I've seen could also benefit from tile deduplication in VRAM (lots of solid black/white space).
Geometry data (normal, AO, curvature) seems extremely difficult to compress though, despite looking very regular visually.
@mjp @archo @rygorous Absolutely. But you can do that flatten/baking step based on predictive heuristics as well, and they work just fine.
Now if you're arguing that you could use sampler feedback for the SOURCES of that baking - since you don't really care about the latency there - absolutely! Although I suspect the heuristics work Just Fine Enough even there.
@dotstdy @TomF IIRC there was also some software that used a virtual file system to download Steam game files on-demand (I forgot the name), which together with local mip streaming from separate files would work similarly.
The part I was specifically thinking nobody's done yet was detecting which mips are actually being requested by the pixel shader. It seems this wasn't mentioned in the video but I'm curious if they've been experimenting with something like that as well.
@archo @dotstdy It's called "virtual texturing" and it's been tried many ways over the years. My favourite talk is this one by Sean Barrett: https://www.youtube.com/watch?v=MejJL87yNgI
Also called "megatextures" and implemented in the "id Tech 5" engine and used in a bunch of games, most notably Rage.
I don't remember if it's persisted in newer engines like UE5. There's a certain overhead to using it, and I think most of the time analytic solutions are just as effective and cheaper e.g. https://tomforsyth1000.github.io/blog.wiki.html#%5B%5BKnowing%20which%20mipmap%20levels%20are%20needed%5D%5D
@archo @dotstdy Hardware implementations go as far back as 1998 when we showed a Permedia3 streaming a giant dataset, all demand-paged in only when the pixel shader requested the data.
The problem with all these demand-paged approaches is you get this gigantic stall in the middle of rendering your scene.
@archo @dotstdy So you need to solve this with two things:
1. have a fallback, i.e. lot the pixel shader use the mipmap level it has, not the one it wants, and fetch the desired data async.
2. predict what the shader will want and prefetch aggressively. OK, but how do you do that? You use the analytic methods. But this reduces the advantage of the pixel shader method, but the costs are still there.
@TomF @dotstdy Oh yeah I'm aware of VT, was specifically talking about detecting shader demand in the context of streaming content from the internet (not even as granularly as VT impls do it, whole mips would be fine), sorry for the confusion.
It seems as if the solutions have been implemented 90% from one end (megatextures going from disk to GPU) and 90% from the other end (mips downloaded from a CDN depending on upcoming content/possibly draw calls) but haven't seen an end-to-end solution.
@dotstdy @archo Also if you rely only on sampler feedback, as the camera turns, the edge of the screen (which is where your eyes are naturally looking) are always wrong. I like the term "Just Too Late" rendering instead of Just In Time. It looks kinda crappy.
So as Josh says, you always need good PREDICTIVE heuristics anyway, and if you do then why even use sampler feedback?
@TomF @dotstdy @archo Turnkey sampler feedback failed the "no implicit behavior" test of inserting bits into your shaders, as well. But the primary piece is as you note, that your predictive heuristics are needed and Good Enough(tm). The most interesting uses for sampler feedback are actually for offline analysis in tooling to build offline CPU-side predictive guidance hierarchies/etc. for your streamer.
@wadeb is on here if you want to ask specifics about COD on-demand texture streaming.
@TomF @dotstdy I would agree with "just too late" looking crappy but it's also ubiquitous and happens in ways that are far more noticeable (e.g. culling).
I also specifically recall GR:Wildlands (2017) doing "just too late" mip streaming, most noticeably with road textures (but probably at draw call granularity and going all the way between low and high LOD).
I agree with SF not adding much though. It at most removes the need for reducing detail in graphics options manually.
@archo @dotstdy Yeah I really hate just-too-late pixel-feedback culling especially. It's really obvious. Pop pop pop.
In general the problem with occlusion culling is it doesn't help the worst case, and although it helps framerate in the normal case, it also adds artifacts. Makes me very hesitant to add it to any engine - very poor bang-for-the-buck.
@mjp I meant that it would be possible to replicate huge chunks of scenes even with the PBR textures removed (may need baking/keeping lighting into the color map aka old-school diffuse maps).
Depends on the game and the scene of course but most materials have high roughness and a good amount of games have lots of heavily shadowed areas or cloudy/foggy skies.
Attached some examples where the numerous flat-appearing areas should be easily visible:
@mjp I'm not saying that the entire screenshot/scene doesn't benefit from PBR textures, just that there are huge parts that don't - for example, the walls in the first screenshot and grass/most trees in the second one.
My note on diffuse baking was about being able to assume that lighting will generally come from known average directions, which allows the normal map to be baked into base color with hard-to-notice loss assuming (near-)max roughness (e.g. for the characters in the screenshots).
@mjp I specifically mentioned lighting as a precondition in my initial post.
As for grazing angles, they shouldn't make a difference for (near-)max roughness materials.
Grass may have tons of specular elsewhere, but not in that screenshot.
Also not concerned with solutions that require >1 baked color maps.
All I'm saying is that with frequently appearing low-contrast lighting, PBR maps have little value.
Certainly not claiming there's a better way to support different lighting conditions. 🤷♂️
Arvīds Kokins
Tom Forsyth
Fabian Giesen
Matt Pettineo
Josh Simmons
James Milne
Michael Vance
Wade Brainerd
Tommy Schmid
Andrew Haining 