Life in Australia.
Do you live in an area with a high Drop Bear attack rate? The attacks are rare, but can be devastating. Fortunately there's been no fatalities in my suburb this year (so far).

#aus #DropBear #cycling

Arch Linux: LUKS-voll-verschlüsseltes System beim Booten per SSH entsperren (mittels systemd und Dropbear)

https://www.codingblatt.de/arch-linux-luks-per-ssh-entsperren/

#archlinux #luks #ssh #systemd #dropbear #opensource

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

• The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
• PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
• Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
• Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
• Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

• Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
• New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

My favourite Australian character trait is the willingness to jump in on a lie in order to trick someone (in good fun).

This is best shown by dropbears*, where any Aussie will happily join in on sharing how terrifying they are, how weird it is that no one outside Aus knows about them, the importance of eating vegemite as a deterrent, etc.

I love that my MPIC friends have gotten fully on board with this, happily rabbiting on about dropbears and their risks. It warms my heart and makes me feel like home!

* this is not to imply that dropbears are not real. Of course they are.

#DropBear #migrantLife #Australia

Today's desktop computer challenge: For the case I need access to my home desktop from remote, I trigger boot through wake on LAN and then remotely unlock the encrypted hard disks through SSH.

The challenge: Two encrypted hard disks. Solution: For the second hard drive add a key file located on the first hard disk as additional slot through cryptsetup, update the configuration in `/etc/crypttab`. Done.

1/n

#ubuntu #dropbear #cryptsetup #wol

TIL if you Google for “drop bear” there’s a little #EasterEgg

#dropbear #australia