DavidWe're back!
Firstly, let me apologise for the down time on genealysis.social. On Wednesday 25th March at 3:20am I received an automated notification from a service which monitors the server. It noted that it was down. I was asleep at the time. At 9am in the morning I could see that there was a problem so I contacted my hardware provider. Their entire focus was on "what have you changed", trying to establish if it was something I had done. I assured them nothing had changed in several weeks. It took the entire day of me bouncing back and forth with them to finally realise it was an apparent hardware issue in the data centre.
Since then I have been constantly chasing for updates. On Thursday 26th March at 6:11pm I was told they have escalated it. That has been the last response from them, despite my constant chasing.
On Saturday morning (today) 28th March, I took the decision to purchase a new cloud server and rebuild the server, using a backup. This takes several hours and would mean I had to restore to a backup on Tuesday morning at 6am. The Wednesday 6am backup had not started as the server failed. It meant we had lost a days data, but at this point we had nothing so the restore was the better option.
And now.... 13:30 on Saturday 28th March the server is up and the data restored. We are now receiving data and I have posted some tests. Thousands of queued messages are now pouring in from other servers.
The server cache needs to be rebuilt for all the account avatars and header images. This takes time as it has to re-fetch these across the entire fediverse for every account which has had contact with genealysis.social. That represents 552,596 images. It is running now and it claims it will take 12 hours and 54 minutes!
In the meantime some accounts might have blank avatars.
Changes...
For the last three years I have used the services of IONOS to host the server. They are a multi-national company. I am appalled at the support I have received for this incident and I will be complaining and seeking compensation for the costs I have paid up front.
We are now hosted by Hetzer, which I use for several other servers. The only change is that data is now stored in a Finland data centre, rather than the UK. I will update the Privacy Notice to reflect this.
Once again I am truly sorry for the outage. Thank you for your patience.
David.
#Genealysis Server Admin
DianneR
Sidaba
glassnet
Mary Rohrer Dexter