Bart Wronski 🇺🇦🇵🇸I am excited to finally share our recent paper "Filtering After Shading With Stochastic Texture Filtering" (with @mattpharr @marcosalvi and Marcos Fajardo), published at ACM I3D'24 / PACM CGIT, where we won the best paper award! 1/N
Aras Pranckevičius@BartWronski ah, I remember a year or two ago you were asking around about Unreal jittered sampling and other “strange” texture filtering approaches. Now we know where all that went! Really nice!
@aras yes, we had a tech report with our initial findings and a ton of folks reported some great precedents in old games. We knew of all the academic literature, but game developers just use them and often not even report. :)
The coolest example was this old Star Trek game and the first Unreal, we had no idea! This helped us a lot to contextualize our research. :)
Game developers, please report your findings and even "hacks"! :)
@aras Even if writing a full paper might seem intimidating and a ton of work (plus sometimes dealing with gatekeeping reviewers), GDC or Siggraph "Advances" presentations, blog posts, JCGT articles or arXiv tech reports are good enough to find and reference and much easier to write. :)
@BartWronski there’s at least 10x effort (and prestige?) difference between a blog post and a GDC/Siggraph talk, but yes even a blog post is 1000x better than nothing. From personal experience though, “hey I found a gross hack!” the first instinct is to *not* write about it :) But of course you have no idea if your “gross hack” is actually a sensible application of a theory that has not been formulated yet.
Nikita Lisitsa@aras @BartWronski "a sensible application of a theory that has not been formulated yet" is pretty much all particle physics since 1950s 😄
@aras reminds me of one talks at Stephen's and Steve's course where someone from the film industry explained how in the first Toy Story, artists requested a "hack" for diffuse power remapping to make lighting look more natural. Later it turned out that artists intuitively compensated for the lack of gamma correction. :) @self_shadow do you remember which talk it was?
@aras also, I don't think most gamedevs find a "real" paper in some journal or a more "academic" conference any more prestigious than a GDC talk, from my anecdotal experience - it's the other way around. :)
@BartWronski true from gamedev side, but feels like complete opposite from “science” side. Like half a year ago, I toyed with Gaussian Splats wrt data compression, but these were not “papers”. How many of the 100+ new 3DGS papers mentioned it? About one, out of like 20 that were about data compression ;)
@aras yeah and some of them had even worse compression ratios than yours :( visibility of blog posts is close to zero for non-academics. And I even had one academic explicitly refusing to cite my blog post about a similar method as their paper "because it's not peer reviewed". :(
FWIW, I think if you wrote a LaTeX version of your post, just put it on arXiv with references - you'd get a ton of citations. And at many conferences, chance of a "best paper" award. :)
@aras arXiv (even not peer reviewed, just a "preprint") is better than a blog post, as it will show in Google Scholar, including back-references from citations. Academics often look at some paper, look who cited it, and read and cite those.
@BartWronski oh yeah I’m very aware of how and why that happens. My wife’s a professor, and “citation indexes” are a real thing there that can determine your employment status etc. Citing blog posts is not incentivized in any way, and because you get what you incentivize… well that’s what you get. It’s just funny from the outside :)
demofox@aras @BartWronski it sure is. I've tried to cite posts in both STBN and now FAST noise papers, and got a lot of push back on both papers. It's interesting cause the "real competitors" we ought to compare against live in blog posts IMO. (not as much push back at i3d btw Bart! that was nice.)
@aras @BartWronski like i have it on my personal blog todo list that i need to compare FAST vs these 2 types of noise.
https://tellusim.com/improved-blue-noise/
https://acko.net/blog/stable-fiddusion/
https://tellusim.com/improved-blue-noise/
https://acko.net/blog/stable-fiddusion/
@aras sonetimes it's incentives (your reviewers will be academics, so you better cite them all to not offend any potential one 😅), but discoverability is a real issue as well...
Tom Finnigan@BartWronski @aras Yeah, a lot of people’s literature search is just crawling up and down the citations tree, which blog posts typically aren’t a part of
@tomfinnigan @aras yeah, and putting something on arXiv makes you part of this citation tree. 90% of new CS/CV/ML papers submit there before final publication and already get cited.
Putting something together in LaTeX and submitting there is not much work (if you need "vouching" before submission to the arXiv CS.GS group, I am happy to recommend you! :) ), especially since Overleaf got really good recently (including some WYSIWYG editing!) and it's free for a single user and small projects.
Putting something together in LaTeX and submitting there is not much work (if you need "vouching" before submission to the arXiv CS.GS group, I am happy to recommend you! :) ), especially since Overleaf got really good recently (including some WYSIWYG editing!) and it's free for a single user and small projects.
Stephen Hill@BartWronski @aras I’m fairly certain that was “Art Direction within Pixar’s Physically Based Lighting System” (Ian Megibben & Farhez Rayani). Sadly we never got permission to post final slides at the time. I should really ping Ian again.
@BartWronski your paper is a perfect blend of “proper literature” and “documenting gamedev practices” by the way. The latter is very often not well documented or even understood (I’m sure you are aware of a million reasons why :)). But it is curious that production environment sometimes stumbles upon actually sound theory by accident, without realizing it.
@aras I had this observation in an older blog post of mine that artists manually sharpening mipmaps (which seemed like a gross hack) is actually an intuitive compensation for ugly bilinear filter and correct, best-fit optimization-based solution gives similar results :) https://bartwronski.com/2021/07/20/processing-aware-image-filtering-compensating-for-the-upsampling/
@BartWronski @aras artists are smarter than we give them credit for. I'm pretty sure incorrect lighting falloff was making up for the renders not being sRGB correct :P
@demofox @BartWronski “life finds a way”
@demofox @BartWronski definitely. But my point is, in gamedev (or generally outside of "research"), many of these things are not because someone wanted to find a theory; they are because someone wanted to save half a millisecond. I'm 99% sure stochastic mip sampling happened in gamedev only because of in a virtual texturing system manually doing full trilinear is very costly. Someone had an idea of random mip choice, and went "hey that does not look too bad!" and so it shipped.
@demofox @BartWronski which again is why I'm very happy for paper like this (and a handful of others) that "bring industries together". I think Bart mentioned it too, but many graphics people are blissfully unaware of most of signal processing things done by audio people, for example. It might be useful! (or it might not, lol)
@demofox @BartWronski I'm totally fanboying Bart here though -- since you have experience in gamedev *and* research *and* music -- 🤯 -- excellent! ❤️
@aras @BartWronski Yeah, Bart is a mensch haha. & I love that the best research comes from people that are well versed in both worlds.
@demofox @aras switching fields for almost 5y helped me a lot :) I would kind of recommend it to everyone (assuming they are ok with re-starting almost from scratch...). The terminology difference was funny ("what the hell is optical flow? oh, you mean motion vectors?"), but I contributed knowhow from games and graphics to some CV/ML research and camera products. :)
"What do you mean games already do robust temporal multi-frame super-resolution???"
"What do you mean games already do robust temporal multi-frame super-resolution???"
@BartWronski @demofox "What do you mean games already do robust temporal multi-frame super-resolution???" is funny. It goes the other way too though, like half of all the ECS stuffs within gamedev are along the lines of "ok so you've invented an extremely limited relational database, right" and they go "what's a relational database?" :)
@BartWronski @demofox complete topic jump, but thanks Bart and Alan for this "discussion" right here. Since 2020 I've been pretty much sitting at home and working in isolation, and right now I'm on the brink of spiralling into some depression episode and/or realizing how much I miss in-person discussions. This one is not in-person, but still. Thanks.
Jon Valdés@BartWronski @aras @demofox yep, I've done two work from home stints (one in 2010, another in 2020), and I burned out both times.
Aras, take care of yourself, it's easy to end up in a bad spot
About all the folks saying "come visit!" around here, every rendering team on the planet would love to have you show up at their office for a week and have random discussions over lunch.
You could really start the nerdiest world tour ever. And we'd all love it 😄
Leszek Godlewski@jon_valdes @BartWronski @aras @demofox indeed, that's a pretty awesome idea. And I'm sure it could even be crowd-funded!
For now, maybe saving the date for https://www.graphicsprogrammingconference.nl/ is a good step? 🙂
Maciej Sinilo@TheIneQuation @jon_valdes @BartWronski @aras @demofox Digital Dragons, too, not far from Aras, but I think in 2 days so maybe not enough heads up :) It's a cool little conference though
@msinilo @jon_valdes @BartWronski @aras @demofox it's grown too big for its own good, though. Last year I had trouble seeking out friends in the crowd, and the party venue has been too small to accommodate everyone who wants to get in for a number of years now.
@TheIneQuation @jon_valdes @BartWronski @aras @demofox Ah interesting. Last one I've attended was, uh, 2017 I think. Sounds like that's just how it goes though, GDC was actually a nice little conf back in 2000s 😆
@msinilo @jon_valdes @BartWronski @aras @demofox http://www.gic.gd/ on the other hand still has some room to grow. And I'm no longer on the program board, so there's no conflict of interest in bragging! 😜
forrestthewoods@aras come to Seattle for breadsticks! :)
Ben Throop@forrestthewoods @aras Yes do it
Anders Lindqvist@aras (sorry for jumping in) I can somewhat relate. I work 100% from since a year before covid. I have smallish kids and it is hard to deny the convenience of working from home, but I feel that it will not be sustainable as they become a little older and can take care of themselves. Hopefully by then there are still offices left that I can start going to!
Michael Vance@aras @BartWronski @demofox I'm also serious, come hang out in Portland with Wade & I and work for a bit on whatever out of our office. Alternatively, I'm still trying to get out to Poland at some point and take a motorcycle trip with Drobot up to your neck of the woods.
@mtothevizzah @aras @demofox let me know when you decide when to go to Poland! maybe by some magic coincidence, I would be there as well? (right now aiming for September, but this might change; and generally, Poland is super awesome May-October)
I haven't seen Drobot for years now, it would be super fun. (though no motorcycle for me - I cannot drive one and they scare me :) )
I also plan to do "Fall Colors" trip this year again and Maine is definitely high on the priority list. :)
Inês Almeida@BartWronski @mtothevizzah @aras @demofox I'm jumping in clueless about motorcycles and who Wade is, but completely up to get on a train to Poland and join in 😊
I very much relate to missing inspiring discussions 💜
@iralmeida @BartWronski @aras @demofox @wadeb naturally. Amsterdam is still on my hit list for Europe as well!
Wade Brainerd@mtothevizzah @iralmeida @BartWronski @aras @demofox Maine is a nice place to visit, especially in Summer+Fall, and I too miss in-person collaborations! Bart, do you think anyone is using inverse rendering now to automate these artist mipmap tricks? Like, directly updating the vertices + mipmaps of a low-LOD model as parameters, to try to match downsampled images of a high-LOD model.
@wadeb @mtothevizzah @iralmeida @aras @demofox 1. there was a paper from my colleagues that does that for both meshes and materials: https://research.nvidia.com/publication/2021-04_appearance-driven-automatic-3d-model-simplification but back then was arguably way too slow. 2. We wrote an article to the new GPU Zen about exactly that for materials :) and it's IMO practical and fast. Uses Slang.D, so no need to rewrite BRDFs in a different language. Should be out in a few weeks, but if you're curious, I'll ask around if I can share a preprint
Appearance-Driven Automatic 3D Model Simplification | Research
We present a suite of techniques for jointly optimizing triangle meshes and shading models to match the appearance of reference scenes. This capability has a number of uses, including appearance-preserving simplification of extremely complex assets, conversion between rendering systems, and even conversion between geometric scene representations.
@wadeb @mtothevizzah @iralmeida @aras @demofox doing it with vertices is also probably doable today with all the advances in differentiable rasterization, it got fast! This latest branch in particular should be of interest: https://github.com/NVlabs/nvdiffrec/tree/slang not just rewritten in Slang, super fast and clean, but also includes many advances like: https://research.nvidia.com/publication/2023-08_flexible-isosurface-extraction-gradient-based-mesh-optimization
And for Maine, definitely want to come back. Just need to decide when, and train vs road trip vs flying. :)
@BartWronski @mtothevizzah @iralmeida @aras @demofox Thanks! It looks like they did what I was thinking of re: optimizing mipmaps directly. Should get things like normal-to-gloss automatically. Props to them for using Simplygon output as a reference and seed since that tracks production practice. Re: speed and renderers, I've been playing with PyTorch and Kaolin but wondering about just trying https://ggx-research.github.io/publication/2024/04/17/publication-intel-diffrast.html and starting with our existing LODs + mipmaps, to just try and improve quality.
@BartWronski @aras @demofox It may be Sept or Oct, will let you know. Michal and I have also talked about doing a bike-packing trip across Scotland, we'll see what happens, and go bug @neil if we do…
@aras @BartWronski @demofox IIRC @wolfpld has an anecdote about a colleague independently discovering transformation matrices. 😉
Bartosz Taudul@TheIneQuation @aras @BartWronski @demofox Yeah. He was a smart guy, did some impressive things. But lack of formal education made for some funny moments at work.
BrianKaris
Julien Barnoin@aras @BartWronski If all my gross hacks were removed there wouldn't be much left of my work 😅
@aras @BartWronski ironically a blog post will have 100x to 1000x the reach of a GDC/Siggraph talk!
I kinda think blog posts are the ideal format for knowledge sharing. Everyone should write more!