Mastodawn

Stuart Presnell 4d ago

When to automate a repetitive task:

NO-BRAINER: "This is obviously going to be faster to automate than to do it by hand _even once_. Let's automate it right now, and not do it by hand at all."

FORESIGHTED: "Doing it once by hand is faster than automating it, but I'm going to have to do it a lot of times, so it still saves time to automate it first."

NEED A RUN-UP: "I don't yet understand this task well enough to automate it, so I'll do it a few times by hand first to get the idea."

RAN OFF THE RUNWAY: "Great, now I've done this by hand a few times, I think I can automate it reliably! Oh, oops, turned out I only had one more case of it left to do."

TERRIFIED OF RUNNING OFF THE RUNWAY: "This is a one-off, so it would be a waste of time to automate it, I'll just do it manually."
[next day] "Oh, oops, I made a mistake and have to do it again. But it should be fine this time."
[a month later] "Even though I've had to redo it 25 times already, surely _this_ is the last time? So it would still be a waste of time to automate it."

@simontatham and it can be really hard to tell the last three cases apart when you're in them

Simon Tatham 5d ago

@Taneb indeed, that's just what I've spent most of this morning failing to do!

geraldew 5d ago

@Taneb @simontatham quite true but I can say I've had multiple "ran off the runway" cases where I coded a good solution and then had another instance crop up ten or fifteen years later.

So now I still default to making something once I feel I understand the problem-solution pair - even though I definitely won't be doing the work in ten years time.

@simontatham
Step 0: recognizing that it is, in fact, a repetitive task.

@simontatham SEP field: This is a regular but infrequent task which we share in the team. It won't be my turn to do it again until December so I'll do it manually now and hope someone has automated it before then.

DasMammut 5d ago

@cunobaros @simontatham Automating something also provides complete documentation - that benefit shouldn't be underestimated 🧐

Sometimes I write a bash script which does (semi-)automagically a job which needs to be done quite rarely. Thus I don't forget something important 😇

Jyrgen N 1d ago

@rkbw @cunobaros @simontatham In some cases it goes for me like this:
1. I have to do something and have to find out how to do it.
2. Some time later, I have to do it again and have to find out how to do it again.
3. The third time I am annoyed I haven't written down how to do it before, so finally I write down a recipe how to do it. (1/2)

Jyrgen N 1d ago

4. The nth time, I write shell command snippets into my documentation to copy and paste, maybe even with variables for the parameters.
5. The (n + m)th time I finally convert this to a shell script.

Now this may sound a bit stupid like "why haven't I written documentation / a script the first tine?", but there may be months or years between the times I have to do that. (2/2)
@rkbw @cunobaros @simontatham

geraldew 5d ago

@cunobaros @simontatham I've certainly come into teams where it is apparent that has been the operating mode for some years.

Hugo Mills 5d ago

@simontatham I'm a fan of the variant of "need a run-up", where you don't know what you're doing, but want to make sure that when you have to redo it, you redo the working steps the same way.

The best way of doing this is to write down the steps. In a script. So when you do get it working, you have most of your automation already written.

Wolf480pl 5d ago

@darkling
I've had tasks that:
- I know I'll need to do exactly 5 times
- are complicated
- have many unknown unknowns - many thngs could go wrong, but I don't know which ones
- any of those unknown unknowns could do a lot of damage

I don't automate those, since I can't do error handling for an error I can't anticipate.

I write down what I'm doing, with commands, including a lot of sanity checks, and what to expect from an output.

Then I repeat these steps by hand.
@simontatham

Hugo Mills 5d ago

@wolf480pl @simontatham If any damage isn't reversible, then absolutely.

I've mostly used my pattern in things like new data-ingest pipelines, where you can keep the original upstream data, and just nuke the interim stuff and try again if it all goes wrong.

Raven667 4d ago

@darkling @simontatham that may be some of the popularity of Ansible, i see it working at the same level as this kind of script and i got into it when i had to set up pairs of servers, that in previous iterations had always been singletons set up one-time by hand, and wanted to make sure i set them both up the same, and i wamted documentation of what i did, since i didnt have any setup docs beaide the state of the old server i was migrating from. Now i can build the entire stack in vagrant or on bare metal in a day, maybe an hour, rather than months.

Agnieszka R. Turczyńska 5d ago

@simontatham If not automated that already, around the 4th-5th time max is when I am "I need to automate it this time, it will take some effort, but I am not going to do this manually again". However, the threshold might be lowered if the given environment is already tedious.

Krahabors 5d ago

@simontatham one more case - when task is too complicated so doing it manually is asking for a trouble -> automating makes it less prone to mistakes

@simontatham me when procrastinating on setting up ansible

Paul Moore 5d ago

REALITY: "Automating this 5 minute task is way more fun than doing it, so I'll spend all day automating it"

Sharif Naas 4d ago

@pfmoore @simontatham https://xkcd.com/1319/

Automation

xkcd

@pfmoore @simontatham There's the old saying, "Automating this task is taking many hours, but it will save valuable seconds later"

Simon Tatham 4d ago

@bit101 @pfmoore that _sounds_ like satire, but just occasionally the hours of automation happen when you had nothing better to do, and the valuable seconds are saved in an emergency!

@simontatham @pfmoore totally satire :)

evilstevie 4d ago

@pfmoore @simontatham I am in this comment and feel seen :)

cheetah_spottycat 1d ago

@pfmoore @simontatham It's worth it because this approach gets you out of ADHD paralysis and is just efficient as staring at this 5 minute job all day because it's so fucking dull.

Jyrgen N 1d ago

@cheetah_spottycat @pfmoore @simontatham Absolutely. Large dull manual tasks are a horror for me (and $deity knows I have enough of those), but I am always happy when I can transform them into a technical, scriptable problem. I am even happier when I can implement a clever generic solution for a whole class of dull tasks.

Andrew Jeffery 5d ago

@simontatham I've ended up putting together a shell script that maintains markdown checklists and records executions and artefacts in a git repo. It allows me to hone messy processes over time to the point where they can largely be automated, while still documenting the necessary steps where reliable automation hasn't yet been achieved. It gives me a lot of comfort

https://github.com/amboar/checklists

GitHub - amboar/checklists

Contribute to amboar/checklists development by creating an account on GitHub.

GitHub

[puts together an automation after having to do it 30 times]

[never have to use it again]

geraldew 5d ago

@simontatham from the mix of these things is why I have my own tool for handling hundreds (if not thousands) of SQL scripts that I've been tinkering with for over 20 years.

Which means I can add an extra item to the list:
- having built a tool for doing some amount of the automation occurring in my work, gives me a head start each time a new need for automation becomes apparent.

I occasionally shock myself at how early that became true, seeing innovative components dated 2005 or so.

Ratsnake Games 🔞5d ago

@simontatham don't forget "if i automate this, i'll have to do something else that's worse"

Pete / Syllopsium 4d ago

@ratsnakegames @simontatham This. So much this. When the reward for being efficient is more work that you don't like, there's not much motivation.

potion_genderqueer

@simontatham Sometimes I'll do a task by hand the first time (and f*** it up a bit) and then go back and cherry-pick my shell history to write the automation, subbing in variables as I go... but yeah. Generally legit.

@simontatham @dtl when I’d prefer to write software to do the thing for me rather than doing the thing myself, that’s when I automate it

10 I know I should automate this, but I just need to get through this one task, then I'll automate it later.
20 GOTO 10

@simontatham I had one - I did a bunch of checks by hand every morning on some processes. I knew it needed automating but it took me 5 mins & it was a _reasonably_ big job to automate but it was going to need doing.

THEN I handed it over to someone else WITH the instruction that it was worth automating

I found out several years later it was STILL being doing by hand and now took over 2 hours

I SO wish I'd automated it when I had the chance!!!

Henryk Plötz 5d ago

@simontatham Some time ago, Randall prepared a handy table: https://xkcd.com/1205/
(Somewhat after https://xkcd.com/974/)

Is It Worth the Time?

xkcd

@simontatham Ah, the eternal Automate the Boring Stuff ROI calculation.

I once had a colleague leave the company and so inherited an ad-hoc process of his. I took one look at the instructions doc and decided this wasn't for me, at all. I *think*, over the years, this eventually fell into the FORESIGHTED category.

There is the old maxim “why bother spending 1 hour on a chore when I can spend 8 hours failing to automate it.“

christian mock 4d ago

@simontatham also, probably, "foresighted": I'll only have to do it once every year, which is why I'll automate it so it is documented instead of re-discovering how it's done every time.

Adam Sjøgren 4d ago

WANKING INFRASTRUCTURE: when somebody decides that to automate this process, a whole new bunch of systems and tools need to be installed, configured, and put into production, and so much time is spent on this, that no work is done solving the actual business problems.

Jeff Grigg 4d ago

"ONE-OFF, BUT BOSS"

"This is a one-off, so I do it manually."

…

But somehow The Boss ends up with it.

[Boss] "Somebody said they changed something. Run that report again, to see if anything changed." (And then again, and again, and again, … Always In A Bug Rush, of course.)

😱

😢

Ray McCarthy 4d ago

@simontatham
I've been called in to automate this, but while they produce the stuff, there is no documentation. Each person only knows their own bit.

I'll decline to bid as I might never discover how they do it. Their totally manual system at least works.

Richard Rathe 4d ago

This from Larry Wall (of Perl fame) seems relevant...

The Three Virtues [Paraphrased]

1) #Laziness - The quality that makes you go to great effort to reduce overall energy expenditure.

2) #Impatience - Because you are impatient, you will build systems and shortcuts for your daily work.

3) #Hubris - You have the confidence that you can do better than the status quo.

#LarryWall #Perl #TheThreeVirtues

Blackoverflow 4d ago

@simontatham
FAILURE WAITING TO HAPPEN: "l write down what (I think) needs to be done beforehand and delegate the liability for the manual execution to some support team."

ACCOUNTABLE: "I do this by hand and write down what steps I had to do.
This way, I won't have to reinvent this if it needs to be done again or if it will be automated.
Furthermore, others can learn from it or point out errors.
I am aware that these steps are not guaranteed to work in two months time, because things change."

TheConstructor (he/him)4d ago

@simontatham I remember from my SPSS days, that most dialogues would offer to emit the action as commands. While they were not yet generalized, they offered a great start for later automatization at almost no additional time needed (you set up everything in the dialogue and just had to hit execute on the output command)

Simon Tatham 4d ago

@TheConstructor yes, I love GUI programs that show you the underlying CLI commands the GUI actions translate into! I've used a couple as well. When I joined my current employer they used a bug tracker based on a database, whose Windows client was able to show its log of SQL commands in an extra pane. And their current flagship GUI debugger has a command-line mode too and a log window showing the CLI equivalent of each GUI action you take.

They are twice as much work to write, of course. But it's worth it.

@simontatham CLIENT IS A FASHION BRAND RUN BY SELF-IMPORTANT IGNORAMI: "This task could be automated in about 10 minutes, but the client doesn't trust computers to produce reports so every month my colleague has to follow instructions in a Word document and spend hours colouring, then copying and pasting cells in a spreadsheet based on their colours and no one else seems to think this is a monstrous way to treat a person."

Lise Andreasen 4d ago

@simontatham
Deadly boring, high probability of doing it wrong: Automate sooner.

Steveg58 4d ago

@simontatham
CONFIGURATION MANAGER (Traditional):
I've done this new thing once by hand and recorded what I have done. If I need to do it again I won't have to reinvent the wheel and, oh look, there are some step I can streamline there and maybe if I re-order these operations it will be even faster. And should I get run down by a bus, my replacement will be able to hit the ground running. Also if it seems like we will have to do this regularly I can swiftly automate away all the boring bits.