@b0rk It's reductive, but in general, don't ever set it to *.
The one place where it's reasonable (but still not great) is multi-tenant hosting. That is, a single site/app which could be homed under any number of domains. In such a scenario, it may be more trouble than it's worth to try to have preflight responses check the Origin.
If you know you aren't serving JS assets or anything which could reasonably be exploited, then, on paper at least, you can justify just using *.
The trick with that, though, is if you are serving any user-uploaded content, then you can't 100% trust even that. Things like JS-in-SVG exploits are very real, as are things like the big WebP problem a few years back. You can try to rely on your media upload and handling pipeline to detect such problems, but you're then accepting some risk.
Another place it's commonly used is embeddable content. But even that, I'd argue, should be handled on a dedicated domain. It's work, but it's worth it.