Later today I need to reboot the server. Should take less than 20 minutes.
libranet.de • venera.social
LIBRANET.de Announcements
@libranet
- 320 Followers
- 8 Following
- 712 Posts
LIBRANET.de is a Friendica node. This is its public channel for maintenance announcements and general informations about the service. Also applies to venera.social🍀 misskey.de and hub.libranet.de.
Need to reload the database service. Downtime approx. 15 minutes. Will do it at 1500 CET / 1400 UTC.
Everything is done. Database is restored and checked. I will bring back libranet.de, wait some time and then re-enable venera.socisl.
Grrrrr. The only way to fix that is a full restore of the latest backup. Good news: backup is very new. Was created 1 hour before the crash. Bad news: This takes a lot of time.
venera.social and libranet.de are in maintenance mode. There was a database crash which resulted in some InnoDB tables being corrupted. :-(
Currently, database is in recovery mode. I have to drop both databases and re-create them from dump/backuo.
Currently, database is in recovery mode. I have to drop both databases and re-create them from dump/backuo.
Drive has been replaced. Server is back online. Currently, the RAID1 recovery is running in the background. Good night! 😴
I have ordered the replacement of the dead nvme drive. Hetzner service will shut down the server as soon as someone has walked to the rack. 🙂
Ok. Managed to find out what the problem is. One of the NVMe disks is dead. I have removed it from the RAID array. Server is running again. But there is no safety net anymore. I have to replace the dead disk asap.
Server is up again. But I cannot connect via SSH or webbrowser. I have to check that later. Maybe I'm blocked by fail2ban. If this isn't the problem I have to access the server via rescue console.
Server is not reachable since 0:20 CET. :-( Cannot login via SSH because of a timeout error. Started a hardware shutdown now. When this is finished I can try to restart the server.