Mastodawn

A habit I'm thinking of adopting for my shell scripting:

Write scripts in the form of a sequence of function definitions, with no commands at the top level that actually do anything. Then, at the very end, write a compound statement that calls one of the functions and ends with an exit. Like this:

#!/bin/bash
subroutine() { ... }
main() { ... }
{ main "$@"; exit $?; }

The idea is that the shell doesn't actually _do_ anything while reading this script until it reads the braced compound statement at the end. And then it's committed to exiting during the execution of that statement. So it won't ever read from the script file again.

With a normal shell script, it's dangerous to edit the script file while an instance of the shell is still running it, because the shell will read from the modified version of the file starting at the file position it had got to in the old one, perhaps reading a partial command or the wrong command and doing something you didn't want. But with a script in this style, the shell finishes reading the script before it does anything, so it's safe to edit.

(Of course, if your editor saves the new script file to a different file name and renames it over the top, you're safe anyway. But not all editors do: emacs, in particular, reopens the existing file and overwrites its contents.)

DJ Adams May 15

@simontatham I adopted this approach a while ago and am happy I did. Not least because it helps me organise my thoughts and also is a structural decision already made so one thing less to think about. Example https://github.com/qmacro/dotfiles/blob/main/scripts/strava-api/oauthflow-local

@qmacro @simontatham Same here - been using that layout for a long time now, think I first learned it from studying the Google Shell Scripting guide, which has lots of other interesting techniques.

https://google.github.io/styleguide/shellguide.html

Never considered it from the perspective of preventing mishaps from editing a running script tho, that's neat.

styleguide

Style guides for Google-originated open-source projects

styleguide

DJ Adams May 15

@mercutio @simontatham yes! In fact I think I might have picked it up from that (great) guide too - see https://qmacro.org/blog/posts/2020/10/05/improving-my-shell-scripting/

Improving my shell scripting

Improving my shell scripting 05 Oct 2020 | 3 min read I'm using a style guide along with the shellcheck and shfmt tools to help me improve...

DJ Adams

Simon Tatham May 15

@qmacro I think I'd be more impressed with it as a means of organising yourself if shell functions didn't default to having global variable scope! The same structure in Python protects you against accidentally having a disorganised mess of global variables everywhere, but in shell you don't get that benefit unless you're _also_ careful to use 'local' all over the place.

DJ Adams May 15

@simontatham I need a LOT of organising :-)

Read along with Matt May 15

@simontatham I've been doing this for years, entirely because I prefer the aesthetic of

main() {
step1
step2
}

step1() {
...
}

step2() {
...
}

main "$@"

over the usual practice of defining functions (if using any at all) before showing the order they get called. I like my scripts to start with the broad outline, and I can always scroll down to see more details.

The question of editing while it runs hadn't even occurred to me!

Simon Tatham May 15

@matt in that example, you won't get the editing protection. That comes from having the call to main be in a braced block together with an exit. (But of course that's only a one-line change to your existing habit.)

Read along with Matt May 15

@simontatham Oh word. Something to read up on, thanks.

(Specifically not asking for a more detailed explanation, reading bash docs in order to better understand the foundation I usually take for granted is like, sudoku, for me. But thanks for the heads up.)

Red Rozenglass May 15

@jonathanmatthews@fosstodon.org @simontatham@hachyderm.io I've been writing shell scripts on prod like that for more than half a decade, it's pretty good, can recommend. The biggest thing I like about it is that it frees me to order the code in an accessible way for fellow programmers, take them on a guided journey through the code, which is not always in execution order. Think "function hoisting" in some other programming languages.

The first thing the programmer would see when they open the shell script is main() with a high-level overview of what's going to happen in this script in general, then below that, key functions of the scripts are implemented, and then at the end, all the silly small utilities or irrelevant implementation details. The developer reading the code is free to go as shallow or deep as they want, the deeper they scroll, the more details they get, but they have the general map of the territory from the first page of code.

The alternative, of writing code at the top level, may take the reader through spiraling mazes of validation functions and help messages and setup procedures before they even know what the code is about, because everything is ordered for execution, not understanding.

@simontatham I would never use this... It requires that you avoid ctrl-c. Why? And if you are testing a script why is it in the background anyway?

If you are so scared of your program crashing and running in a type of zombie state, why use nested functions at all, that almost guarantees bad behaviour. Furthermore, don't normal people use 'set -e'?

Idk maybe I'm just overthinking it or just don't understand..

Simon Tatham May 15

@ponies I'm completely confused by everything you just said!

What's the problem you've spotted with Ctrl-C?

Who said the script was running in the background? More usually I'm running the script in one terminal window, and editing it in another, or in a GUI editor.

What nested functions are you talking about? I didn't show any.

How does 'set -e' help? That's solving a completely different problem.

@simontatham well my point is that stopping and starting the program again makes your issue irrelevant, right?

Simon Tatham May 15

@ponies yes, certainly. If you always stop the script running, then edit it, then start it again, you don't need this protection.

But it's _useful_ to be able to edit a program so that the next run will work better, while the previous run is still going. It means you can fix each problem as you see it, instead of having to remember what they all were once the program finishes running, maybe hours later.

In almost any other language – interpreted _or_ compiled – you can do this safely. It's only in shell where editing a running script causes a disaster. Why _wouldn't_ someone want to eliminate that exception, and make shell scripts behave the same as everything else?

@simontatham This is dumb, even with no burden of implementing it.

So yes bash may have this problem, find me an editor that actually behaves this way. Basically, you have to contrive some crackpot scenario to have this eventuate.

Not even with sed -i "s/code/replacement/" can I trigger this.

Yes I note your emphasis on "useful". Indeed, it's not useful at all and furthermore I have to add functions which simply do not need to be there in the first place

Simon Tatham May 15

@ponies I already mentioned an editor that behaves this way – emacs.

There's no weird edge case involved. You have a script wrapping some subcommand that takes a long time and you don't want to ^C it, like a backup. You run the script. You notice it could do something better. You edit the script, in emacs, now, while you can still remember what it was you wanted to change.

Then the backup (or whatever) finishes, bash reads more data from the script file, and gets confused by the fact that you edited it.

@simontatham no, I cannot do this in Emacs either.

Can you actually reproduce this at all? Please tell me how you encounter this

And if you can't.. why change your code style for a non issue 🧐

Simon Tatham May 15

@ponies certainly I can reproduce it – I checked it this morning before making the original post. And I've reproduced it again just now. Here's a demo video.

If you can't reproduce this in emacs, that's very interesting. Perhaps if you stopped trying to insult me we might investigate *why* your emacs is behaving differently from mine? Does your emacs write a new file and rename it into place?

(You could check by examining the inode number of the file before and after saving over it, or by watching in strace what it's actually doing during the save action. In mine, strace shows that it opens the existing file name with O_CREAT|O_TRUNC|O_WRONLY, and before and after saving, 'ls -li' reports the same inode number.)

@simontatham so to make the above example work you have to specifically tell your editor to keep the same inode. I can see the Emacs setting which by default is set to nil (gnu manual). On my distro it is also nil.

If you are changing settings and then crying wolf.. it's kind of a lie isn't it? I can break stuff too you know

Simon Tatham May 15

@ponies I didn't deliberately configure emacs to keep the same inode. I don't know why you think I did.

I did configure 'make-backup-files' to nil, to avoid the clutter of all those extra foo~ files. It looks as if that's what causes it: emacs in its default configuration renames the old file to the backup name and then makes a fresh
inode for the new file, but if you turn off make-backup-files, it just saves over the original file with no renaming.

But it's not unreasonable to turn off backup files. And emacs _could_ still make a new inode and atomically replace the old one, even if it's not making a backup file. It just doesn't.

What setting are you talking about, which defaults to nil? I haven't been able to find it. You apparently know its name but preferred to insult me again rather than actually saying what the name of the setting is?

@simontatham look I'm following and bookmarking you so 👍

I have also figured out how to make bash do this without Emacs. Ok it's real.

I'm just saying the idea of somebody running code and editing it at the same time (despite your use case) is for the most part a fiction, it requires

1. Editing configuration away from defaults, or
2. Overwriting the file from within the script whilst it's running

Edit: some possible setting, maybe
http://www.gnu.org/s/emacs/manual/html_node/emacs/Backup-Copying.html

Backup Copying (GNU Emacs Manual)

Backup Copying (GNU Emacs Manual)

Alfred M. Szmidt May 15

@ponies >I'm just saying the idea of somebody running code and editing it at the same time (despite your use case) is for the most part a fiction, it requires [...]

It is not fiction, and a very common (and known case) case. You have a semi-long running shell script, you run it, and then see something to improve -- edit, and boom. This can lead to real crap too, depending on how far bash has executed the file (not parsed!).

I'm not sure that @simontatham way is solves the problem.

@amszmidt @simontatham The whole discussion was about the real possibility that this will NEVER HAPPEN if somebody did what you just said, @amszmidt.

Editors don't behave this way unless you make them behave that way.

Alfred M. Szmidt May 15

@ponies Sorry, bullshit. This has happened to me during this single year multiple times. And this is with default settings on editors, and in this case #GNU Emacs.

@amszmidt @simontatham gnu documentation says otherwise.

All I ask is - what distro and which Emacs

Alfred M. Szmidt May 15

@ponies Nonsense. It is unrelated to GNU/Linux system, or even Unix. It is also unrelated to #GNU #Emacs. It is all about how the standard input/output/error streams buffer data.

Alfred M. Szmidt May 15

@ponies The solution @simontatham is/was presenting, could possibly (untested) be simplified by putting { and } around all the code. It forces #GNU #bash to read the whole block (storing it in memory, so buffered reading of the file won't be an issue -- in theory) before executing it.

@amszmidt @simontatham the only nonsense here is you ignoring what I'm actually saying -- 👌😆

@amszmidt @ponies @simontatham
Speaking as someone who's been dealing with bash for a *very long time* in occasionally weird situations, I have to weigh in here: your particular use case is not common.
It's *your* use case, though, and it looks to me like you could save yourself a lot of trouble by simply getting in the habit of not editing bash scripts while you run them.

That's an interesting workaround you have come up with, though I don't know how you're planning to exit main with an error code to pass to $? without actually exiting the script.

I'm not meaning to pry much, but what on earth are you doing that does not benefit from a restart with correct code?

Simon Tatham May 15

@http "simply get in the habit of not editing scripts while you run them" sounds to me a lot like "simply get in the habit of never having an accident". If that were easy, then a great many safety precautions in this world wouldn't be needed. And yet, here we are.

How to exit main with an error code in $?: if you've been doing bash for that long then I'd expect that would be obvious. First, the result of the last subprocess or other command is left in $?, so whatever you ran last _inside_ main will be the value of $? when main returns to the calling block. Second, you can also use 'return' in a shell function to set the $? seen by its caller. (But of course if you do that in main, you might as well run 'exit' directly; _that_ isn't the case where the fallback exit outside main is needed.)

What doesn't benefit from being restarted: something you don't want to interrupt! For example, a shell script wrapping a long-running thing which you'd have to start all over again if you interrupt it. If the thing you want to correct isn't critical, like "oops it would have been better to run the subcommand in verbose mode, but at least it'll do the right thing this time round even if it does it quietly", then it makes perfect sense to want to change the script for _next_ time, without interrupting _this_ run.

Yes, of course you _could_ wait until it finishes, and then remember to come back and make your change. But doing things now is a good way to avoid forgetting to do them later. In any other language you can just do it. And with this precaution, you can do it in shell too.

@simontatham @http

The result of $? Is never bloody obvious and I agree that you would have no reliable way of knowing what result is being returned there as your code becomes more complex

@simontatham @http the important question is: when do you watch YouTube if you can't stop coding while a long test suite is running?
XKCD 303!

@http @amszmidt @simontatham
🤩🙏

Alfred M. Szmidt May 16

@http > Speaking as someone who's been dealing with bash for a *very long time* in occasionally weird situations, I have to weigh in here: your particular use case is not common.

And speaking as soneone who has used bash since it existed, since before bash, yeah it is bloody common. Maybe stop telling people what their experience is. It is also unrelated to bash and applies to all shells that do buffered reading, go troll soneone else.

@ponies @simontatham

@ponies @simontatham It doesn't require changing editor configuration. vim.tiny, for example, writes into the original file by default (and I'd guess the full version of vim does the same, but I don't have that installed). I've been bitten by this problem enough times that I came up with a similar solution to Simon's independently. If the way you use your machines doesn't cause the problem, maybe you didn't need this solution - but that doesn't mean it's never a problem for others.

Adam Sjøgren May 15

It's not fiction for me, it happens regularly and I have to think about it when working on long running shell scripts.

It is as simple as automating something that processes a lot of files in a loop, and then, while running it, thinking "ah, I botched that output, it would be nicer if it also echo'ed this other thing the next time I run the script", adding a line, saving, and then boom the script goes haywire after the loop.

Plenty of editors overwrite the file when saving:

$ ls -li test.sh 
636 -rw-rw-r-- 1 asjo asjo 34 2025-05-15 21:28:51 test.sh
$ vi test.sh
$ ls -li test.sh 
636 -rw-rw-r-- 1 asjo asjo 24 2025-05-15 21:29:30 test.sh
$ jove test.sh
$ ls -li test.sh 
636 -rw-rw-r-- 1 asjo asjo 25 2025-05-15 21:30:02 test.sh
$ nano test.sh
$ ls -li test.sh 
636 -rw-rw-r-- 1 asjo asjo 28 2025-05-15 21:30:42 test.sh

Even vanilla, unconfigured Emacs:

$ emacs -Q -nw test.sh
$ ls -li test.sh 
636 -rwxrwxr-x 1 asjo asjo 30 2025-05-15 21:31:57 test.sh

Marius Gedminas May 15

@ponies @simontatham I had this happen to me with a git-versioned shell script that runs git pull && make && restarts a running daemon process, to automate updates somewhat.

If the git pull updates the script itself, well, interesting things happen.

@mgedmin @simontatham finally a reasonable case scenario 🎈

Adam Sjøgren May 15

Why do shells actually do this differently than other interpreters? Does anything depend on it?

I guess generating commands and piping them into as shell is the usecase?

@asjo @simontatham now that I am invested in this, I have read the claim that zsh does not do this, although I haven't tested. You would assume the fish guys would have fixed something like this too, right? But again, I am not sure

Adam Sjøgren May 15

I just tested with zsh - same as bash.

My test: I open a file called hep.sh with this content:

    for i in $(seq 1 10); do
        date
        sleep 2s
    done

    echo "HEP"

and then I run zsh hep.sh and I add a line before the "date" line saying echo "YAY" and save the file, while the script is running:

    $ zsh hep.sh 
    Thu 15 May 21:47:22 CEST 2025
    Thu 15 May 21:47:24 CEST 2025
    Thu 15 May 21:47:26 CEST 2025
    Thu 15 May 21:47:28 CEST 2025
    Thu 15 May 21:47:30 CEST 2025
    Thu 15 May 21:47:32 CEST 2025
    Thu 15 May 21:47:34 CEST 2025
    Thu 15 May 21:47:36 CEST 2025
    Thu 15 May 21:47:38 CEST 2025
    Thu 15 May 21:47:40 CEST 2025
    HEP
    hep.sh:7: command not found: ne
    HEP

I'm not sure if the same syntax works for fish; it does for bash:

    $ bash hep.sh 
    Thu 15 May 21:50:09 CEST 2025
    Thu 15 May 21:50:11 CEST 2025
    Thu 15 May 21:50:13 CEST 2025
    Thu 15 May 21:50:15 CEST 2025
    Thu 15 May 21:50:17 CEST 2025
    Thu 15 May 21:50:19 CEST 2025
    Thu 15 May 21:50:21 CEST 2025
    Thu 15 May 21:50:23 CEST 2025
    Thu 15 May 21:50:25 CEST 2025
    Thu 15 May 21:50:27 CEST 2025
    hep.sh: line 6: syntax error near unexpected token `done'
    hep.sh: line 6: `done'

Adam Sjøgren May 15

fish behaves differently (and the syntax is a little different), I tested with:

    for i in $(seq 1 10);
        date
        sleep 2s
    end

    echo "YAY"

and then fish yay.sh, adding a line inside the loop and saving while it was running; result:

$ fish yay.sh
Thu 15 May 21:54:18 CEST 2025
Thu 15 May 21:54:20 CEST 2025
Thu 15 May 21:54:22 CEST 2025
Thu 15 May 21:54:24 CEST 2025
Thu 15 May 21:54:26 CEST 2025
Thu 15 May 21:54:28 CEST 2025
Thu 15 May 21:54:30 CEST 2025
Thu 15 May 21:54:32 CEST 2025
Thu 15 May 21:54:34 CEST 2025
Thu 15 May 21:54:36 CEST 2025
YAY

Phairupegiont May 15

@simontatham Wow, I thought only cmd.exe was dumb enough to not make a copy of the script in memory before starting to execute it.

@phairupegiont @simontatham I suspect the reason the shell "runs in place" is because running code from standard input is a "feature":

echo 'typeset -p PWD' | sh

https://pubs.opengroup.org/onlinepubs/009604499/utilities/sh.html

sh

Phairupegiont May 16

@march38 @simontatham
Yeah ok, when running from stdin, not buffering everything first is fine, because what is read cannot be modified afterwards.

But when running from a file, I just don't see a reason not to buffer it first.

Simon Tatham May 16

@phairupegiont @march38 ah, here's a thought – some shell scripts are _incredibly big_ and composed mostly of a here-document. 'shar' archives spring to mind. When shar was popular, computers were also smaller, so quite likely there would have been archive scripts so big that the shell would run out of memory buffering the whole thing in RAM before running it!

Phairupegiont May 16

@simontatham @march38
Ok, shell archives are a case where buffering is unwanted.

A case that looks like it would be solved by having a flag in the shebang to disable input buffering...

Toni Aittoniemi May 15

@simontatham nice tip!

@gimulnautti @simontatham this is what I do; but I’ve found that shell check rule SC2317 is violated for functions so that has to be disabled explicitly which makes me sad.

@gimulnautti @simontatham Of course, I believe that my failure as a scripto-monkey is because I do a variation on this a lot

main() {
action="$1"
shift
action_"$action" "$@"
}

which happens to one of the conditions that shellcheck doesn't enjoy.

Simon Tatham May 15

@QuotidianEnnui @gimulnautti I can imagine that for some scripts that might be a security hole – if something doesn't trust its argv, then you might be handing it the ability to tell it to do more things than you intentionally permitted.

Is Shellcheck clever enough to spot the difference between what you wrote, and the same thing but with some kind of a passlist to reject any value of $1 except for a set of deliberately permitted ones?

@simontatham Nope it isn't

In fact I have this right now (and it always fails shellcheck with unreachable code)

main() {
local action=${1:-check}
if [[ ! "${action}" =~ ^$ACTION_LIST$ ]]; then
echo "Invalid action: $action"
action_help
exit 2
fi
action_"$action"
}

ACTION_LIST is defined elsewhere as "permitted|list|of|actions"

Simon Tatham May 15

@QuotidianEnnui I suppose _this_ kind of passlist might be OK because it avoids constructing a function name at all?

case "$action" in
foo) action_foo "$@";;
bar) action_bar "$@";;
# ...
esac

It's ugly, because you have to write each action name out twice, and repeat the "$@" every time too (which would be more of a pain if it had to be more complicated than that). But perhaps it would at least make Shellcheck not be annoyed?

@simontatham Oh yes, I agree with that (shellcheck will hate me less, and it's uglier 😉 )

but for some reason I have developed the habit that sees me using maps of functions where I could be using a "switch" in a compiled language; so my scripts have often reflected some of that thinking.

It's not a big deal and if people hated it I would switch but they don't seem to mind round here (or they just trust that the scripts will work).

@QuotidianEnnui @simontatham

I found that the "unreachable" SC2317 warning is also triggered when this trick is combined with "trap exit_handler EXIT" This was filed 3 years ago in https://github.com/koalaman/shellcheck/issues/2542 and 2660.

I posted a workaround there.

SC2317 false positive warning · Issue #2542 · koalaman/shellcheck

For bugs Rule Id (if any, e.g. SC1000): SC2317, SC2317 My shellcheck version (shellcheck --version or 'online'): online I tried on shellcheck.net and verified that this is still a problem on the la...

GitHub

Alfred M. Szmidt May 15

@simontatham Sightly code golfier .. but should do the same, since how #GNU #Bash processes blocks.

#!/bin/bash
{
echo "$@"
#subroutine stuff ...
exit $?
}

Ah yes, definitely. For long-running scripts I tend to just enclose the whole script (sometimes after any startup stuff that I know will go quickly) in brackets. (But braces, as you have it, would be more efficient, not spawning a subshell.) The shell (at least bash) seems to read whole lines at a time, so as long as the final "exit" is on the same line as the closing bracket it insulates the shell from edits.

@simontatham slightly longer version:
https://github.com/thesofproject/sof-test/issues/740

clean-up: move all shell script code to a function and use a "main" · Issue #740 · thesofproject/sof-test

This a very generic, clean-up task. Its main purpose is to document this good practice: almost all the code in a shell script should be in a function. In other words, the smallest script should loo...

GitHub

Tim Ward ⭐🇪🇺🔶 #FBPE May 16

@simontatham Why would someone want to edit a shell script whist it's running?

And even if they did, that'd be in a sandboxed developer test environment so quite harmless if it went wrong.

Simon Tatham May 16

@TimWardCam you're kidding, right? Shell scripts are useful for the wrapper layer _around_ projects that live in nice organised repositories.

The example I've used elsewhere in this thread is backups. You have some backup tool which takes a command line saying 'back up _this_ machine to _that_ disk', or whatever. That tool itself, of course, is Properly Organised. It has a source control repo and numbered releases, and its developers take care to test changes on non-live data before releasing them to users.

But as a user of it, you personally always want to back up the same set of three machines to a disk mounted at the same pathname. So you write a personal shell script that contains your usual three backup commands and maybe also a mount/umount around them.

Seriously, would you have a dev version of _that script_ and a live version, and mock up an entire test framework you can run before deploying the one to the other? There's "well organised" and then there's just ludicrous. If you set up all that infrastructure for every 5-line script, you'd never get _anything_ done.

But then one day you run it, and you realise you don't know how far through the backup it is, and you think "oh yeah, there's a --progress-report option to the backup tool, it'd be nice if I'd put that on the command line". So you want to edit your script.

If it were in Python, you'd be able to do that right now, while the backup is still running. Too late to affect this run (unless you want to abort the backup and restart from scratch), but it will be better next time.

But in shell, you've got to write a note to yourself for later, and wait until the backup finishes to make your change. Or organise the script the way I said, in which case you can do it now, avoiding forgetting to do it later.

Tim Ward ⭐🇪🇺🔶 #FBPE May 16

@simontatham My backup system exposes all that in the GUI that sits on top of the underlying tool, no need for me to mess with shell scripts.

But yeah, I appreciate that some people prefer doing this sort of thing in other ways.