So this was a whole shit show that as of now has cost me 2 days of investigation, clean-up, mitigation, will likely cost me more time & also feels extremely crappy all in all.
I'm just glad I didn't base any *really* important decisions on those manipulated numbers, that would have been REALLY bad.
Now I have to live with knowing that most of the growth I saw the past 2 years was likely faked, which feels quite terrible tbh. Feels like not doing a good job after all.
Well, the analysis and mitigation is ongoing, and in the process of that I found another ... thing. The verdict is still out on this being a case of a rampant running CI, or some weird VPN endpoint, or something evil. The traffic from the cloud IP I found was definitely organic, but still had some issues and also was too short lived per instance identifier.
Nuked everything from that source. Goodbye 100k instances 😢
Shitty week, really. And still not done with analysis and mitigation.
Well. FFS 😡 Even more manipulation. Spent the whole day analysing again, compiled a report, confronted Obico about it and they admitted it. Will post about that on the OctoPrint blog tomorrow. I have no energy left today.
https://obico.io/blog/2024/07/03/my-apologies-for-the-mistake/
I'm writing this post to apologize to Gina Häußge, one of the most important and respectable person in 3D-printing community, as well as to the OctoPrint community for the stupid and selfish mistake I made and the harm I have caused:
Gina Häußge
balloob