So this was a whole shit show that as of now has cost me 2 days of investigation, clean-up, mitigation, will likely cost me more time & also feels extremely crappy all in all.
I'm just glad I didn't base any *really* important decisions on those manipulated numbers, that would have been REALLY bad.
Now I have to live with knowing that most of the growth I saw the past 2 years was likely faked, which feels quite terrible tbh. Feels like not doing a good job after all.
Well, the analysis and mitigation is ongoing, and in the process of that I found another ... thing. The verdict is still out on this being a case of a rampant running CI, or some weird VPN endpoint, or something evil. The traffic from the cloud IP I found was definitely organic, but still had some issues and also was too short lived per instance identifier.
Nuked everything from that source. Goodbye 100k instances 😢
Shitty week, really. And still not done with analysis and mitigation.
Well. FFS 😡 Even more manipulation. Spent the whole day analysing again, compiled a report, confronted Obico about it and they admitted it. Will post about that on the OctoPrint blog tomorrow. I have no energy left today.
https://obico.io/blog/2024/07/03/my-apologies-for-the-mistake/
I'm writing this post to apologize to Gina Häußge, one of the most important and respectable person in 3D-printing community, as well as to the OctoPrint community for the stupid and selfish mistake I made and the harm I have caused:
Just did a round on the punching bag, with Limp Bizkit's "Break Stuff" blasting from the sound system.
Doesn't solve this of course, but at least the adrenaline is out of my system again. I can seriously not recommend it enough to have a punching bag in your office... And that's not a joke. I literally have that.
Gina Häußge
🪬🍄🌈🎮💻🚲🥓🎃💀🏴🛻♓🧿