Mastodawn

There is currently an issue with OVH (the data center provider for Masto.host) that is causing service degradation and servers to be slow or unresponsive.

Currently on the phone with them and still don't have much information besides that there is an issue with Object Storage https://public-cloud.status-ovhcloud.com/incidents/ck365yfh1f1x that for sure will cause issue with media uploads but something else must be going on.

I apologise for the trouble this might cause and will update here when I have more information.

[GRA/UK/DE/WAW/SBG][Public Cloud] - Object Storage

Public Cloud's Status Page - [GRA/UK/DE/WAW/SBG][Public Cloud] - Object Storage.

Masto.host Feb 28, 2023

It looks like the issue was in fact just related to the Object Storage. There was an authentication problem and requests were getting stuck and causing everything to stall.

They implemented a fix and it looks like it solved the problem. Now most servers have a lot of media files to process and it will take a while for the timelines to catch up but I estimate that in the next hour everything should be back to normal.

Masto.host Feb 28, 2023

Please let me know if in an hour you are still experiencing any issues.

Again, really sorry for any trouble this caused.

Niléane Feb 28, 2023

@mastohost thank you for the updates :)

Masto.host Feb 28, 2023

@nileane In this case it was all I could do, just try and keep you updated :) Thanks and hope it wasn't too bad on your end.

Niléane Feb 28, 2023

@mastohost I didn’t even have time to notice anything 😅. So all good!

Niléane Mar 1, 2023

@mastohost not sure if related, but on my end I’ve got many profile pictures that aren’t loading in my timeline this morning…?

Masto.host Mar 1, 2023

@nileane don't think it's related, is it just profile pictures?

Niléane Mar 1, 2023

@mastohost I think so.
Although it mostly seems to be profiles on remote servers that I’ve probably never interacted with before. Perhaps it takes time to federate?

Masto.host Mar 1, 2023

@nileane So, it's probably related with the new remote cache feature https://mastodon.social/@mastohost/109864531841763112 that it's running in order to reduce media storage space usage for Mastodon servers.

This is a new feature and I am not sure but I don't think the impact should be noticeable. At least I have got no reports regarding that or have notice any issues myself. Strange. Are the accounts missing avatars using Mastodon or other software like Pleroma or something?

Niléane Mar 1, 2023

@mastohost They’re running Mastodon it seems

Masto.host Mar 1, 2023

@nileane Sorry for the questions but want to make sure the new feature is not causing any problems.

Is this in the federated timeline or in your home feed or both?

Niléane Mar 1, 2023

@mastohost No worries, happy to help :)

Both. But it’s exclusively accounts that I don’t follow

Masto.host Mar 1, 2023

@nileane OK, thanks. I will look into it and please do let me know if you notice anything else or the issue gets worse.

Niléane Mar 1, 2023

@mastohost will do! thanks

Niléane Mar 9, 2023

@mastohost hi!
Little update: the issue is still there, although it hasn’t gotten any worse

Masto.host Mar 9, 2023

@nileane Hello, thanks for the update.

To be clear are avatars just the Mastodon default empty profile or is the media file really missing, like broken interface?

Niléane Mar 9, 2023

@mastohost yes it’s the default Mastodon empty profile avatar (grey elephant)

Masto.host Mar 9, 2023

@nileane That's good.

I don't see any logged errors, so I am assuming that like you mentioned it will take time to update: "Although it mostly seems to be profiles on remote servers that I’ve probably never interacted with before. Perhaps it takes time to federate?" https://nileane.fr/@nileane/109947843255777216

I would say that this was not so obvious in the past because your server was caching a lot of remote avatars/headers and now that those are regularly removed, that delay starts to be more obvious.

Niléane (@[email protected])

@[email protected] I think so. Although it mostly seems to be profiles on remote servers that I’ve probably never interacted with before. Perhaps it takes time to federate?

Niléane

Niléane Mar 9, 2023

@mastohost I see, that makes sense

Masto.host Mar 9, 2023

@nileane if I am correct, it's a trade-off. Either we have Mastodon use less media storage or we occasionally get those for remote accounts that are not actively federating with your server.

Obviously Mastodon can/could (will?!) improve that but I don't think there is a problem on your installation.

Niléane Mar 9, 2023

@mastohost I understand yea.
Thank you for your insight :)

@mastohost As always, your transparency is deeply appreciated — thanks for telling us!

Masto.host Feb 28, 2023

@josh Thank you so much :) and sorry for the disturbance.

Dave KT Feb 28, 2023

@mastohost thank you for the transparency and keeping everyone in the loop!

Masto.host Mar 1, 2023

@dave Thanks :)

esmevane, sorry Mar 1, 2023

@mastohost you’re good! I didn’t even notice. You rock

Masto.host Mar 1, 2023

@ironchamber Glad it was not much of a problem :) Thanks

Timo Hetzel Feb 28, 2023

@mastohost Actually, I got a heads-up about this a few days ago from them.

Masto.host Feb 28, 2023

@timo I received a notice about a 6am update to the Object Storage infrastructure. This one was not planned and it's marked as an incident https://public-cloud.status-ovhcloud.com/incidents/ck365yfh1f1x

It could be related to the earlier planned maintenance but if it is, it was a side effect, I think.

[GRA/UK/DE/WAW/SBG][Public Cloud] - Object Storage

Public Cloud's Status Page - [GRA/UK/DE/WAW/SBG][Public Cloud] - Object Storage.

Timo Hetzel Feb 28, 2023

@mastohost right. Used it this morning at 8 without any issues without any auth config change.

Masto.host Feb 28, 2023

@timo it did fail on my end for a while at around 9am but nothing compared to what happened now, all auth stopped working.

hybrid havoc Feb 28, 2023

#DarkfriendSocial heads up that the instance has been slower than usual for about the past hour or so I think, and service providers are aware.

From @mastohost

https://mastodon.social/users/mastohost/statuses/109944793520601781

Masto.host Feb 28, 2023

@hybridhavoc It should all be back to normal now. Sorry about the disturbance.

TILvids Feb 28, 2023

This is part of why I migrated #tilvids away from OVH, downtime plus one of their data-centers burned down.

Been happier after the migration.

Nemo_bis 🌈Mar 1, 2023

@tilvids You're on DO, right? (Couldn't find your privacy policy from the frontpage.)

OVH is a good option for those who want to respect GDPR. I know if I managed Ceph on my own it would probably be worse than what they do.

Status page says "The incident was due to a load increase on the GRA clusters" and it was fixed in 1 hour from the time of reporting, not too bad.

Timo Hetzel Feb 28, 2023

@mastohost lost my blog to the data center fire at ovh. Good luck!

hybrid havoc Feb 28, 2023

@mastohost Thanks for the heads up. Didn’t see this until about 30 minutes after you posted it. 🤣

Seeing my queues getting back down to zero now, so could be getting back to normal.

Masto.host Feb 28, 2023

@hybridhavoc Glad it's back to normal :) Sorry about the disturbance.

@infosec_jcp 🐈🃏 done differently Mar 1, 2023

Wait, wasn't this that container last year or so ago that was on 🔥🥵🧯😳?

Daniel G.Mar 1, 2023

@mastohost thank you!

FeedMe Doughnuts Mar 1, 2023

@mastohost considering their wooden built no sprinkler host farm burned to the ground last year, I would not trust them. Hope for the best.