So this was a whole shit show that as of now has cost me 2 days of investigation, clean-up, mitigation, will likely cost me more time & also feels extremely crappy all in all.
I'm just glad I didn't base any *really* important decisions on those manipulated numbers, that would have been REALLY bad.
Now I have to live with knowing that most of the growth I saw the past 2 years was likely faked, which feels quite terrible tbh. Feels like not doing a good job after all.
Well, the analysis and mitigation is ongoing, and in the process of that I found another ... thing. The verdict is still out on this being a case of a rampant running CI, or some weird VPN endpoint, or something evil. The traffic from the cloud IP I found was definitely organic, but still had some issues and also was too short lived per instance identifier.
Nuked everything from that source. Goodbye 100k instances 😢
Shitty week, really. And still not done with analysis and mitigation.
Gina Häußge
__Miguel_