Mastodawn

Masto.host Feb 28, 2023

There is currently an issue with OVH (the data center provider for Masto.host) that is causing service degradation and servers to be slow or unresponsive.

Currently on the phone with them and still don't have much information besides that there is an issue with Object Storage https://public-cloud.status-ovhcloud.com/incidents/ck365yfh1f1x that for sure will cause issue with media uploads but something else must be going on.

I apologise for the trouble this might cause and will update here when I have more information.

[GRA/UK/DE/WAW/SBG][Public Cloud] - Object Storage

Public Cloud's Status Page - [GRA/UK/DE/WAW/SBG][Public Cloud] - Object Storage.

Masto.host Feb 28, 2023

It looks like the issue was in fact just related to the Object Storage. There was an authentication problem and requests were getting stuck and causing everything to stall.

They implemented a fix and it looks like it solved the problem. Now most servers have a lot of media files to process and it will take a while for the timelines to catch up but I estimate that in the next hour everything should be back to normal.

Masto.host Feb 28, 2023

Please let me know if in an hour you are still experiencing any issues.

Again, really sorry for any trouble this caused.

Niléane Feb 28, 2023

@mastohost thank you for the updates :)

Masto.host Feb 28, 2023

@nileane In this case it was all I could do, just try and keep you updated :) Thanks and hope it wasn't too bad on your end.

Niléane Feb 28, 2023

@mastohost I didn’t even have time to notice anything 😅. So all good!

Niléane Mar 1, 2023

@mastohost not sure if related, but on my end I’ve got many profile pictures that aren’t loading in my timeline this morning…?

Masto.host Mar 1, 2023

@nileane don't think it's related, is it just profile pictures?

Niléane Mar 1, 2023

@mastohost I think so.
Although it mostly seems to be profiles on remote servers that I’ve probably never interacted with before. Perhaps it takes time to federate?

Masto.host Mar 1, 2023

@nileane So, it's probably related with the new remote cache feature https://mastodon.social/@mastohost/109864531841763112 that it's running in order to reduce media storage space usage for Mastodon servers.

This is a new feature and I am not sure but I don't think the impact should be noticeable. At least I have got no reports regarding that or have notice any issues myself. Strange. Are the accounts missing avatars using Mastodon or other software like Pleroma or something?

Niléane Mar 1, 2023

@mastohost They’re running Mastodon it seems

Masto.host Mar 1, 2023

@nileane Sorry for the questions but want to make sure the new feature is not causing any problems.

Is this in the federated timeline or in your home feed or both?

Niléane Mar 1, 2023

@mastohost No worries, happy to help :)

Both. But it’s exclusively accounts that I don’t follow

Masto.host Mar 1, 2023

@nileane OK, thanks. I will look into it and please do let me know if you notice anything else or the issue gets worse.

Niléane Mar 1, 2023

@mastohost will do! thanks

Niléane Mar 9, 2023

@mastohost hi!
Little update: the issue is still there, although it hasn’t gotten any worse

Masto.host Mar 9, 2023

@nileane Hello, thanks for the update.

To be clear are avatars just the Mastodon default empty profile or is the media file really missing, like broken interface?

Niléane Mar 9, 2023

@mastohost yes it’s the default Mastodon empty profile avatar (grey elephant)

Masto.host Mar 9, 2023

@nileane That's good.

I don't see any logged errors, so I am assuming that like you mentioned it will take time to update: "Although it mostly seems to be profiles on remote servers that I’ve probably never interacted with before. Perhaps it takes time to federate?" https://nileane.fr/@nileane/109947843255777216

I would say that this was not so obvious in the past because your server was caching a lot of remote avatars/headers and now that those are regularly removed, that delay starts to be more obvious.

Niléane (@[email protected])

@[email protected] I think so. Although it mostly seems to be profiles on remote servers that I’ve probably never interacted with before. Perhaps it takes time to federate?

Niléane

Masto.host Mar 9, 2023

@nileane if I am correct, it's a trade-off. Either we have Mastodon use less media storage or we occasionally get those for remote accounts that are not actively federating with your server.

Obviously Mastodon can/could (will?!) improve that but I don't think there is a problem on your installation.

@mastohost I understand yea.
Thank you for your insight :)

Masto.host Mar 9, 2023

@nileane I will continue to keep my eye out for similar reports and information about it and let you know if I find any problem that is specific to your server but I don't think that is the case.

But please let me know if this somehow changes for the worst or some other issue that you notice.

Niléane Mar 18, 2023

@mastohost I think it's become a bit worse?
Pretty much 80% of the time when I'm not following an account (even from a known remote server), its avatar is missing. :/

Masto.host Mar 18, 2023

@nileane Sorry about that but it's a combination of the way Mastodon deals with federation from non followed users and only they can fix it. I don't think there is much I can do on my end besides disabling or increasing the number of days for remote avatars pruning.

Still, v4.1.1 introduced a bug https://github.com/mastodon/mastodon/issues/24149 and remote avatar pruning will be disabled for the next days. See if that makes a different, so we are sure this is the source.

v4.1.1 - Index update strategy is undefined for current context · Issue #24149 · mastodon/mastodon

Steps to reproduce the problem Running tootctl media remove --prune-profiles in an instance with ElasticSearch Expected behaviour Process the command Actual behaviour Returns an error for each file...

GitHub

Niléane Mar 18, 2023

@mastohost Alright :)

Masto.host Mar 18, 2023

@nileane Obliviously this will not completely change overnight but if that is the source of the problem in the next couple of days you should see an improvement.

Can you please keep an eye out and let me know?

Niléane Mar 18, 2023

@mastohost I will! Thanks!

Masto.host Mar 18, 2023

@nileane Perfect, thanks.

Again, sorry for all the trouble but these new features require some tuning and learning.