Kevin5d ago

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

  • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
  • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
  • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
  • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
  • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

  • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
  • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
GitHub - kevinveenbirkenbach/hetzner-arch-luks: Guide to install Arch Linux with LUKS encryption on an hetzner server

Guide to install Arch Linux with LUKS encryption on an hetzner server - kevinveenbirkenbach/hetzner-arch-luks

GitHub
GeeklandMay 7
¿Cómo arranca Linux? Del BIOS/UEFI a systemd #teoría_de_tux #arranque_linux #bios_uefi_linux #boot_linux #grub_linux #initramfs #kernel_linux_arranque #proceso_de_arranque_linux #systemd_inicio
https://notilinux.com/como-arranca-linux-bios-uefi-systemd/
¿Cómo arranca Linux? Del BIOS/UEFI a systemd - NotiLinux.com

Descubrí cómo arranca Linux paso a paso desde BIOS o UEFI, pasando por GRUB, el kernel, initramfs y systemd hasta llegar al sistema listo.

NotiLinux.com
AskUbuntuMay 3

Ubuntu 24.04.4 LTS linux kernel 6.17.0-23-generic boots to initramfs alert UUID does not exist #boot #kernel #initramfs #busybox

https://askubuntu.com/q/1566364/612

Ubuntu 24.04.4 LTS linux kernel 6.17.0-23-generic boots to initramfs alert UUID does not exist

initramfs_alert_uuid available kernets I can't do much on busybox / initramfs because listing the partions / blocks returns nothing (I tested the ls /dev/sd* /dev/nvme* /dev/vd* 2>/dev/null I sa...

Ask Ubuntu
AskUbuntuApr 28

Sudden Jump in initrd.img SIze Filled /boot Partition #kernel #initramfs

https://askubuntu.com/q/1566167/612

Sudden Jump in initrd.img SIze Filled /boot Partition

I went to update my system today and encountered a strange problem that has left my system in a perilous state. I've done this hundreds of times so didn't expect anything to go wrong. I ran sudo apt

Ask Ubuntu
FlorianMar 29

After having it just lying around, I finally took the time to set up my Raspi again. I've now got a booting Pi and my first alpine contribution (https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/215)  

(the `usbdelay` command line parameter of alpine's initramfs `init`semms to get ignored in some cases - which is kind of an issue, if your root filesystem resides on an external USB HDD that takes some 20seconds to become available)

#alpine #alpinelinux #initramfs #mkinitfs

init: pass usbdelay if root= is set (!215) · Merge requests · alpine / mkinitfs · GitLab

This is required if the (encrypted?) root filesystem is on an external USB hdd which takes quite some time to spin up. Otherwise booting fails with a timeout...

GitLab
Hackaday [Unofficial]Mar 7

Remotely Unlocking an Encrypted Hard Disk

https://fed.brid.gy/r/https://hackaday.com/2026/03/06/remotely-unlocking-an-encrypted-hard-disk/

13Mar 7
Remotely Unlocking an Encrypted Hard Disk https://hackaday.com/2026/03/06/remotely-unlocking-an-encrypted-hard-disk/
#LinuxHacks #Diskencryption #Encryption #Initramfs #Linux #Systemd #Tailscale
Remotely Unlocking An Encrypted Hard Disk

Can you remotely unlock an encrypted hard disk? [Jyn] needed to unlock their home server after it rebooted even if they weren’t home. Normally, they used Tailscale to remote in, but you can&#…

Hackaday
Adhidarma Hadiwinoto Mar 3

A little tip to anyone use #bluetooth (not custom wireless) keyboard and using #LUKS disk encryption.

If you cannot enter the LUKS passphrase at boot, it is because the Bluetooth module and connection are not activated yet.

To fix this, create /etc/dracut.conf.d/bluetooth.conf file and put this line on that file:

add_dracutmodules+=" bluetooth "

Don't forget to regenate your #initramfs by:

sudo dracut --regenerate-all --force

Tested on #fedora 43

Thorsten Leemhuis (acct. 1/4)Feb 10

Half of the support for the #initrd (not to be confused with #initramfs!) was removed from #Linux 7.0 through a #vfs merge from @brauner:

https://git.kernel.org/torvalds/c/996812c453cafa042f2e674738dbf8fa495661f3 and https://git.kernel.org/torvalds/c/ef12d0573a7f5e7a495e81d773ae5f3e98230cd4

""Remove the deprecated linuxrc-based initrd code path and related dead code. The linuxrc initrd path was deprecated in 2020 and this series completes its removal. If we see real-life regressions we'll revert. […]

The no-op load_ramdisk= and prompt_ramdisk= parameters are dropped, and noinitrd and ramdisk_start= gain deprecation warnings.

Initramfs is entirely unaffected. The non-linuxrc initrd path (root=/dev/ram0) is preserved but now carries a deprecation warning targeting January 2027 removal""

#Kernel #LinuxKernel

Merge tag 'vfs-7.0-rc1.initrd' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs - kernel/git/torvalds/linux.git - Linux kernel source tree

аккаунт более неактивенJan 17

Во времена #EFI & #UEFI этот самый #GRUB уже и не нужен.
Во-первых, полно и других разнообразных UEFI boot manager'ов.

Во-вторых, не особо то они и нужны вообще как класс. Компьютер можно загружать и напрямую через EFI-stub ядра ОС, образы которых давно собираются как EFI-приложения (раз и два).
Т.е. надо идти в настройки UEFI материнской платы и смотреть загрузочные записи, хранящиеся в #NVRAM — дописывать туда все те параметры\аргументы, которые UEFI при старте компа будет подставлять в качестве аргументов командной строки. Так же как и GRUB или другой boot manager грузящий vmlinuz-файл.

Цивилизованный старт
Использовать то, что было известно как #gummiboot, а теперь стало называться #systemd-boot. Там вполне удобные текстовые файлы в /boot/loader/entries/... через которые можно передать нужные значения и аргументы в переменные. Например такие как используются для #LUKS разделов:

options rd.luks.options=password-echo=no options rd.luks.data=UUID=/dev/disk/by-id/nvme-VENDOR-partN options rd.luks.name=UUID=my-enc-swap options resume=/dev/mapper/my-enc-swap options rd.luks.data=UUID2=/dev/disk/by-id/nvme-VENDOR-partY options rd.luks.name=UUID2=my-enc-root options root=/dev/mapper/my-enc-rootНе обязательно задавать таким образом по одной опции на строке, можно устраивать свалку пихая всё в одну большую строку.

Комментарий №1
В данном примере, это когда swap идёт не файлом, а отдельным разделом на диске, такое размещение упрощает использование swap для hibernate.

Комментарий №2
Все эти LUKS-опции не обязательно указывать при каждом запуске системы в /boot/loader/entries/... файлах или через загрузочные записи UEFI внутри #NVRAM
Можно прописать всё это сопоставление и через файл /etc/crypttab.initramfs перед генерацией\созданием #initramfs образа.

#linux

The EFI Boot Stub — The Linux Kernel documentation