drmorrOk, now I'm at "Unlock High-Frequency Deployments without Blowing Up Prometheus" by Ganesh Vernekar from Reddit. He's talking about one of my personal enormous pet peeves, Stale Metrics in Prometheus ™️
In this case we're trying to handle stale prometheus series by turning on stale series compaction. Which can help with your memory utilization, but it makes your CPU utilization worse, particularly if you have lots of recording rules.
OR -- and hear me out now -- what if we just DIDN'T MAKE UP DATA THAT DOESN'T EXIST.
Man aging with madness@drmorr it's a shame it's only in 3.10.0, and has that bug already. Fix eta 3.12.0 or 3.13.0