Harry Sintonen

I updated my Debian server to 13.2 and in the process managed to render the server unbootable. The server is at Hetzner DC, so I had to reboot it in rescue mode, mdadm assemble the raid, cryptsetup luksOpen the crypted rootfs, mount the rootfs, mount /boot, /proc, /sys and /dev inside the rootfs and then chroot the rootfs. I could now inspect the issue I had caused: I messed up while fixing the earlier kernel package install failure, and the initrd was missing entirely from /boot. A quick reinstall of the kernel packages fixed the issue, and I could successfully boot into the dropbear-initramfs SSH environment, unlock the crypted root and fully bring up the server.

I know not everyone enjoys stuff like this, but I still do. It helps that this is a hobby server, of course.

EDIT: To clarify: This situation was entirely my own fault. I somehow managed to get the system in a state where a package wasn't properly installed. I should have noticed that before rebooting the system - but I did not.

Nov 15 at 5:19pmWeb
Show threadTimo LindforsNov 15
@harrysintonen I usually do a simulated boot with something like "kvm -hda /dev/sda -snapshot" before attempting real reboot. Useful with remote baremetal systems
Show threadSpaceLifeFormNov 16

@harrysintonen

How did the earlier install fail?

Show threadHarry SintonenNov 16

@SpaceLifeForm Very old install with far too small /boot and package install failing - I thought I sorted that out but somehow initrd was still missing - I really ought to resize those partitions some day.

The complication is that the remaining storage is LUKS encrypted (and several TBs of size) - so resizing isn't as easy as with regular installs.

Show threadSpaceLifeFormNov 16

@harrysintonen

So, you were not paying attention, and missed the ENOSPC error.

Why did your initrd grow in size so much?

Or, did you leave old kernels lying around wasting space?

Show threadHarry SintonenNov 16
@SpaceLifeForm I forgot to remove the old kernel package before upgrade. Upgrade failed. I removed the old package and completed the upgrade. In the end I had no errors, but somehow I managed to get the system to the state where the initrd was missing (which I could later see when I booted into the recovery system). I must have missed some error, even though I don't recall doing so. It definitely was my mistake not detecting this issue before rebooting.
Show threadSpaceLifeFormNov 16

@harrysintonen

I would not worry about making /boot larger.

I would just never keep more than 2 old kernel builds.

I am old, so I may be confused. /s

Show threadAndrew WipplerNov 16
@harrysintonen If only recovery was easier to perform.