So this was a whole shit show that as of now has cost me 2 days of investigation, clean-up, mitigation, will likely cost me more time & also feels extremely crappy all in all.
I'm just glad I didn't base any *really* important decisions on those manipulated numbers, that would have been REALLY bad.
Now I have to live with knowing that most of the growth I saw the past 2 years was likely faked, which feels quite terrible tbh. Feels like not doing a good job after all.
Well, the analysis and mitigation is ongoing, and in the process of that I found another ... thing. The verdict is still out on this being a case of a rampant running CI, or some weird VPN endpoint, or something evil. The traffic from the cloud IP I found was definitely organic, but still had some issues and also was too short lived per instance identifier.
Nuked everything from that source. Goodbye 100k instances 😢
Shitty week, really. And still not done with analysis and mitigation.
Well. FFS 😡 Even more manipulation. Spent the whole day analysing again, compiled a report, confronted Obico about it and they admitted it. Will post about that on the OctoPrint blog tomorrow. I have no energy left today.
https://obico.io/blog/2024/07/03/my-apologies-for-the-mistake/
I'm writing this post to apologize to Gina Häußge, one of the most important and respectable person in 3D-printing community, as well as to the OctoPrint community for the stupid and selfish mistake I made and the harm I have caused:
Just did a round on the punching bag, with Limp Bizkit's "Break Stuff" blasting from the sound system.
Doesn't solve this of course, but at least the adrenaline is out of my system again. I can seriously not recommend it enough to have a punching bag in your office... And that's not a joke. I literally have that.
And here's my post on round two of the shitshow that's been keeping me busy for the past week.
I just hope that's it for now, I'm completely and utterly drained by this, both emotionally and physically.
As a consequence of all of this, we've decided that commercial plugins will no longer get their stats tracked publicly, among some other consequences, effective as soon as I can implement the necessary filtering on the stats data exports.
https://octoprint.org/blog/2024/07/04/more-stats-manipulation/
@foosel ugh, what a disappointing turn of events!
glad you've invoiced them for your time.
@foosel wow that was a shit show, and I am really sorry for how this ended up. I wished I could send consensual hugs per TCP. For now all I can say I understand your actions and feelings. For me both seem very valid, and I am glad you spoke out about your feelings as well.
Yet I hope this whole matter doesn't disturb you financially so much with blocking the culprits and removing them from the sponsor list.
Be well ❤️🩹
@foosel
at least the Obico guy owned up instead of attributing it to some anonymous "member of the community" or sth
but this is definitely a shitty situation and not publicly tracking commercial plugins anymore sounds like a reasonable consequence
if you want. What a shit show@foosel JFC. Even though I'm using Octo less (thanks for the closed ecosystem and GFY Bambu) this makes me queue up for a donation. If they suspected someone was jacking up stats, why TF didn't they just reach out? Fools!
Just know that your work is loved and appreciated.
@foosel you are showing an _enormous_ amount of civility and restraint. Kudos to that.
If I were to judge, these people owe you not only compensation for the cleanup time, but they also owe the community for the deception, something they will never be able to make up for in my opinion.
I like this idea, though why as a deferred invoice? Just send them a normal invoice. Even if they did not pay, they would have to wittingly ignore the invoice and see how much in money quantified effort the have taken from @foosel and the community - leaving out the emotional harm that was caused.
Gina Häußge
DJGummikuh
Zlasha
Graham Sutherland / Polynomial
Harald now @ treehouse.systems
Pxl Phile
Olivier Saraja ☕️📚🦖🛸👻
gadgetoid
Greg Saunier
piofthings
Cam Davidson-Pilon
Daniel 🏳️🌈 Jilg 
Daniel Gibson
Sandzwerg
Alan Langford 🇨🇦🧤🧊摏
David
🪬🍄🌈🎮💻🚲🥓🎃💀🏴🛻♓🧿
normen
concretedog