David Chisnall (*Now with 50% more sarcasm!*)

This has saved me a bunch of times, so might be useful to other people. When I change firewall rules on a remote machine, I always run the following command:

# service pf restart && sleep 30 && service pf stop

And then I hit control-C after a few seconds. If my ssh session is still working, this cancels the service pf stop command. If I've managed to break my ssh connection with the rule change, the firewall is disabled after 30 seconds and I can revert the changes and reenable it.

May 10, 2025 at 11:19amWeb
Show threadWanjaMay 10, 2025

@david_chisnall I do something quite similar for network-related changes that might break SSH access:

```nixos-rebuild test; sleep 60; nixos-rebuild test --rollback```

It applies the entire system configuration I've defined and then rolls it back after 60s. I use ; instead of && because I want it to also roll back if there's an error returned.

It's similar to but not quite the same concept as a "dead man's switch".

Show threadEdwin TörökMay 10, 2025
@david_chisnall I assume you're running that in screen/tmux?
Show threadDavid Chisnall (*Now with 50% more sarcasm!*)May 10, 2025
@edwintorok I use a combination of autossh (to reconnect if the connection drops) and dtach (which gives me a buffered persistent pty, but no fake terminal-in-a-terminal nonsense). So if the connection drops and is reestablished, that's fine. If the connection drops and isn't reestablished, the pty is still live and so the third command runs.
Show threadmoeMay 10, 2025
@david_chisnall neat, similar to commit confirmed behaviour on Juniper routers
Show threadNetuxMay 10, 2025
@david_chisnall ok, that's pretty smart.
Show threadLudwig VielfrassMay 10, 2025
@david_chisnall genius is sometimes described as “that which is obvious only in retrospect” and this certainly qualifies. I’m totally stealing this!
Show threadtheothertomMay 10, 2025
@david_chisnall This is like my favourite feature on Firebrick kit - "apply config for 5 minutes". Why the hell can't more things do that?
Show threadDavid Chisnall (*Now with 50% more sarcasm!*)May 10, 2025

@tom I guess it’s hard in the traditional UNIX model because there are two steps:

  • Edit a config file.
  • Tell one or more things to reread their configuration.

    • Ideally, these should be an atomic transaction where you specify a change and then it is applied and rolled back. I think a lot of cloud configuration things use a model that enables this. But when the second step is the only one that the service manager sees, it doesn’t have enough information to be able to revert.

    I suspect you could do something like this if you made /etc a separate ZFS dataset and snapshotted it whenever you restarted a service. Then you could automatically roll back to the previous state. Keeping configuration in a revision control system or database would also make it possible. Flat files that are read as-hoc are the worst possible tool for this kind of thing.

    Show threadtheothertomMay 11, 2025
    @david_chisnall Yeah, it's far harder on a general-purpose system than an appliance.
    Part of why it works so well is that (as far as I can tell as a user) the entire config is applied atomically - so that thought about snapshots would achieve something similar.
    More what I was getting at though, is that systems that have an easy path to making mistakes less annoying to deal with seem under-rated, and I wish more of the technology I work with had a similarly easy recovery path.
    Show threadDavid Chisnall (*Now with 50% more sarcasm!*)May 11, 2025
    @tom Oh, absolutely. Raskin dedicated an entire chapter of his book to this point and the world would be a lot better if everyone had to read it before being allowed to write software.
    Show threadJeffMay 10, 2025
    @david_chisnall This is excellent advice, in the same category as keeping a separate root shell open when testing changes to sshd_config or access config (sudo/doas/pam/&m.).
    Show threadGarrett WollmanMay 10, 2025
    @david_chisnall Compare Junos `commit confirmed`.