Mastodawn

bilbo_b 14h ago

Two different approaches to debugging a software problem:

The Sudoku approach: stare at the limited set of clues you have, and think harder and harder about them until you find a way to deduce something useful.

The Minesweeper approach: don't even try to figure out the solution from only the clues you have right now. Instead, focus on finding a way to acquire another clue, and then using that to get another, and so on. Eventually you've collected so many clues that the answer is obvious.

Sometimes the Sudoku approach is necessary, because you've got all the clues you're ever going to get. But I think my new motto is "Never Sudoku a problem when you can Minesweeper it."

@simontatham This sounds interesting. I'd like to see examples of both.

Simon Tatham 18h ago

@a32 my type example is where you first become aware of a bug because some kind of automated test run fails, and presents you with a log file containing a cryptic error message.

The Sudoku approach to this involves trying to figure out what the error message might mean, e.g. by grepping the source code for the message string and backtracking from there to see where the failing function might have been called from. Sometimes there's more than one possibility: a characteristic of the Sudoku approach is that you try to _guess_ which one seems more likely in this case.

The Minesweeper approach starts by reproducing the same failure on your own machine; stripping off all the layers of test-runner script and makefile until you have a single shell command that runs the actual process that crashes; then running that command again and again in ways that give you more information, such as
• in a debugger
• under strace
• with extra verbose or diagnostic options
• with modified versions of the input file that provoked the failure

A Minesweeper-oriented developer wouldn't bother trying to _guess_ which part of the program had called the failing function. They'd put a breakpoint on it, and get the debugger to _tell_ them.

@simontatham I will keep an eye on my own debugging style.

aspragg 11h ago

@simontatham @a32 Now I'm trying to figure out which of the two approaches the tactic of "make educated guesses about the best place to put a bunch of `printf()`s and try again" falls into 😆

@aspragg minesweeper

You're gathering more clues

Alexander The 1st 6h ago

@aspragg @simontatham @a32 I mean, that's still Minesweeper-y to an extent; even if the printfs don't actually trip when you reproduce the bug, that tells you more information - namely, that the bug is *not* where you guessed where it was.

Glen, waiting for the pre-poll 18m ago

@simontatham @a32 My question would be why this test suite doesn't core dump immediately upon the failed test? Then we can minesweeper back from the moment the suckage was detected.

Agnieszka R. Turczyńska 16h ago

@a32 @simontatham
Industrial solution, which gives some set of logs and reported behaviour, but you can hardly get more, as it would disrupt the production process even more, and causes a money loss for the customer using the appliance. Whatever data you got, you got. Try to make something from it, at least to the point you have a reproducible scenario to fail in your own lab. That's sudoku.

Sometimes you are able to reproduce it in the lab. Sometimes analysis from existing data leads you to a strong candidate of the root cause. Sometimes you may prepare a patch for the customer to get more data, to confirm or reject your hypothesis. And that is the minesweeper approach, as you are actively looking for more clues. Sometimes guessing wild, sometimes educated.

@agturcz @a32 @simontatham I once managed to fix a problem while having only a screenshot of a debugger showing a partial stack dump and the CPU register values when it crashed. That probably qualifies as the sudoku approach.

Agnieszka R. Turczyńska 15h ago

@mansr I would say this qualifies as a fucking wizardry approach 😅
@a32 @simontatham

Passwordsarehard4 15h ago

@mansr @agturcz @a32 @simontatham did you even notice a change at the end of The Matrix or could you just see the text the whole movie?

scmbradley 18h ago

@simontatham I think this is true if you're debugging something on your machine. But if you're trying to help a client with an issue, where each round of minesweeper can take days of back and forth emails to get your new clue, sometimes you should sudoku a problem you could, in theory, minesweeper.

Simon Tatham 18h ago

@Scmbradley true, you have to interpret "can" with a certain amount of pragmatism. I agree that there are situations in which Minesweepering is technically possible but prohibitively expensive.

In this situation I generally try extra hard to get the client to provide a reproducible example. Sometimes they still can't, or won't, but sometimes they just haven't thought of trying, so in my experience it's worth a shot.

James Wells 14h ago

@simontatham
And there is the problem... It is almost impossible to get the "client" to give you the answers you need and when they do, it is often like pulling teeth to get the second question answered.

Often in that situation, I will start with sudoku until the client gives me enough that I can re-create their failure myself and then switch to minesweeper to do the actual troubkeshooting.

John Philip Bell 18h ago

@simontatham
I love the theory, but the optics are terrible "miss/misinterpret a clue and 'BANG'" 🤪

Simon Tatham 10h ago

@john_philip_bell for the full Minesweeper "one misclick and kaboom" experience, you need to be in a situation where you have no choice but to attempt your diagnostic re-runs on the writable live data set.

Justin Derrick 18h ago

@simontatham For particularly difficult problems, I find that breaking it EVEN MORE is a way to glean additional insight.

@JustinDerrick @simontatham Simplify the code over and over in ways that preserve the bug.

@simontatham I wonder if this is related to the "overlapping bug" problem? Sometimes when you come across a hard to understand bug in a (probably legacy) codebase, you spot another bug which is much easier to explain mixed in.

I always advocate fixing that small bug first unless it would turn into a major side-quest. Anything which contributes to the general confusion, even in a small way, is bad for solving the big one.

The goal in finding and fixing most "eh?" bugs is to make it increasingly explicable. (And often the fix is then trivial). And in bad codebases (more common than folk think) there could be maybe two or three "annoyances" which interact in strange ways to hide the underlying problem.

Bai Shen 17h ago

@chiffchaff @simontatham 1000 percent this.

I think a lot of the problems come up because devs are scared to touch more than the bare minimum because they might break something else. I once saw someone change a return type but not the method name.

It's also the kinds of things management complains about spending time on, because "it works, why fix it?"

@baishen @simontatham It can sometimes be hard to know whether you're hitting Chesterton's Fence or not, especially if code is weak on tests. I used to work on code that was 15+ years old and had 10k+ bugs fixed against it, and it could be really hard to tell if something was a bug or deliberate.

One reason for that was it was modelling science, the world is messy, and much of the code written by non-programmers, so it was very hard to distinguish "coding good-practice they didn't understand" from "science I don't understand".

Bai Shen 17h ago

@chiffchaff @simontatham TIL Chesterton's Fence.

I agree that it can be difficult to discern sometimes, but my example and others that spring to mind are far removed from the fence. They're obvious things that should be removed/fixed but no one wants to do it. Reasons vary, but are generally all signs of bad culture/environment.

Iván Sánchez Ortega 18h ago

@simontatham But... when I'm playing sudoku (or, rather, "solo") I don't even try to figure out the solution - I use the clues to derive new clues (usually in the form of filling a square) until all unknowns are cleared.
Does that mean... that I minesweep sudokus?

Simon Tatham 17h ago

@IvanSanchez (and also @artemis who made a similar comment in parallel): true, of course, but I'm thinking in particular of the part where even deducing _one_ extra number to fill in can require arbitrarily complicated thinking about the clues you already have.

Also, in Sudoku, the "extra information" was already contained within the existing information – the clues you fill in are logical consequences of what you already had. In Minesweeper you have to get the extra information from outside yourself. That makes them philosophically different.

(Indeed, in the Sudoku style of debugging, the same phenomenon can happen!)

Iván Sánchez Ortega 17h ago

@simontatham @artemis I feel, then, that the key is deriving abstractions from the immediate clues, then using those abstractions as clues.
e.g. in sudoku a clue is a cell with a "1"; an abstraction is "which rows already have a "1". In 'sweeper, an abstraction is "in this pair of adjacent cells there must be exactly one mine".
If one doesn't have the right abstractions, then the problem must be thought-out and brute-forced. But with the right abstractions, problems can be mecanically solved.

Tim Ward ⭐🇪🇺🔶 #FBPE 18h ago

@simontatham I often do both at once. Whilst my brain is thinking through the existing clues my fingers are typing away trying to collect new clues.

Artemis 17h ago

@simontatham
Hmm. I would have said this describes solving a Sudoku: "finding a way to acquire another clue, and then using that to get another, and so on." You identify how to acquire one piece of information, one clue. As you add pieces of information, you can eventually solve the puzzle. You don't stare at it until you solve it...you figure out how to use the information you have to get another piece of information.

I guess I think Minesweeper & Sudoku are solved in a very similar manner?

David JONES 17h ago

@artemis @simontatham no, because in minesweeper you must click to find out new information. Where you click is up to you. In Sudoku there is no clicking, right from the beginning you have all the clues you will need. The search is inside of oneself.

Artemis 16h ago

@drj
But the thought processes used to do the solving itself are extremely similar.

Cookie & Oreo 14h ago

@artemis @drj right! I play both and they do feel similar, in minesweeper you click to get an extra clue but the clue you had originally tells you where to click. In sudoku you use the original clues to give yourself another clue, further along until you complete it.
The one difference I can think of is early vs late game farina. In minesweeper you gain the most clues the fastest at the early game while in sudoku is in the late game where you start moving fast and filling the gaps faster.

Tiota Sram 8h ago

@biscuitcats @artemis @drj I think the relevant difference here is that in Sudoku, you could theoretically solve the whole thing without writing down anything you discover: the entire solution is deductible from the initial clues given (modern fog of war digital Sudoku being an exception to this rule). In contrast, you must interact with the Minesweeper board to get more clues, the nature of additional clues is not deductible from the original clues.

Aris Adamantiadis

💲Paid 17h ago

@simontatham This is very relevant to security research. My mind is a sudoku solver when most of the time we can minesweeper it.

@simontatham Your description of the minesweeper approach reminds me of an attacker identifying the first vulnerability, then the next one from there, and so on until they get what they came for.

Curt Olson 17h ago

@simontatham When things don't sodoku or minesweep, I'll start at the top and start tracing values through until I see something completely wrong. But if you are debugging AI written concurrent code or event driven code or some unholy combination of distributed technologies across several computers, then 🤷🤷‍♂️🤷‍♀️ ?

The other big trick I learned that can help a lot when nothing is making sense is to ask myself what stupid thing I've done. Seriously, hunting for the stupid thing I did is usually far more productive than hunting for some obscure compiler or library or OS bug. ... just please don't be the one to remind me of this, it's no fun when people imply I do stupid things! :-)

Chris L 11h ago

@clolsonus @simontatham I find Claude makes more stupid mistakes than I do (of the sort that the compiler/linter misses which requires debugging of tests). But logging every value used in a if statement that affects program flow usually makes things tractable.

Diane Bruce 16h ago

@simontatham FWIW the Sudoku approach was pretty much how all debugging had to start in the days of punch card and core dumps. It's still of use IFF you have the source code as you can trace through the source and figure out what it should be doing. Too much minesweeper and you end up with code peppered with printfs. 😀

@DianeBruce @simontatham Debugging ASICs and FPGAs is very Sudoku-heavy too - and when the Minesweeper approach does come in to it, you start by playing a game of Sudoku to work out how you're going to look for that next clue!

Diane Bruce 12h ago

@FenTiger @simontatham Oh we furiously agree both tactics are useful! I do worry that the "kids" overuse the spray and pray method of debug.

@DianeBruce @simontatham Well, in both approaches you have to start by thinking about what your next move will be.

Experts might be able to do the thinking bit very quickly - but they still do it.

"Spray and pray" suggests to me that they're skipping over the thinking part and going straight for the printfs ...

Diane Bruce 12h ago

@FenTiger @simontatham Ideally yes. One should know what the code should be doing and hopefully be able to unit test each piece before putting it together and testing after it's put together. I find testjigs very handy for that.

Chris L 11h ago

@DianeBruce @FenTiger @simontatham printfs, or debug logging, can set the stage for faster Sudoku debugging, as you have more concrete constraints on what must be true at each log line. Important to include the value of the variable affecting program flow.

Diane Bruce 11h ago

@jayalane @FenTiger @simontatham We are furiously agreeing!

Chris L 10h ago

@DianeBruce @FenTiger @simontatham maybe I feel defensive since my last commit was removing a bunch of prints from this “memoize python functions into SQLite” module I use (I had to bump it from one column to ten due to max width in SQLite, and so I had to debug it for the first time in ten years. Usually I always keep all prints I add for debugging at least in some lower log level, but it wasn’t worth it to me since this is really supposed to:be invisible infra that just works.

Diane Bruce 10h ago

@jayalane @FenTiger @simontatham Hey guilty as charged as well. One problem with debug prints is the debug lines can obfuscate what you are trying to grok! Even if they are hidden from the output with a log level.

samiamsam 16h ago

minesweeper works for me when problem solving

What computers are best at is doing slightly different things over and over very, very fast.

@simontatham Both games reveal more given digits as you work on the problem. I get it: There's a qualitative difference between the puzzle revealing additional facts as a reward for correct deductions, and the solver filling in additional clues for themself. But that doesn't really impact the approach to solving all that much, so I don't find this metaphor particularly meaningful or revealing.

Btw., look up “fog of war Sudoku” for a fun combination of both puzzle types 😺

@simontatham And then there's the all too common kaleidoscope approach: keep stirring randomly until it looks pretty.

Shubham Arora 15h ago

@simontatham I have worked my whole life with Minesweeper approach. Just never knew it.

Mark Dominus 15h ago

https://blog.plover.com/prog/katara-advice.html

“Novice programmers often imagine that they can figure out what is wrong from looking at the final output and intuiting the solution Sherlock Holmes style. This is mistaken. Nobody can do this. Debugging is an engineering discipline: You come up with a hypothesis, then test the hypothesis. Then you do it again.”

I shouldn't have said "nobody", maybe you can do it. But as you said, it's not something to count on.

And novice programmers watching experienced programmers debug often *think* that is what is happening because they don't yet understand the process and because it goes by too fast to understand.

Advice to a novice programmer

From the highly eclectic blog of Mark Dominus

The Universe of Discourse : Advice to a novice programmer

Mark Dominus 15h ago

@simontatham Although your description doesn't seem to match my "finger of blame" model. Perhaps there are real differences.

I'm a blamey person. I don't pretend it's a good quality, but we all have to work with what we have.

Simon Tatham 15h ago

@mjd oh, I think both of the approaches I describe still end up pointing the finger of blame at a specific piece of code, it's only a question of how you get there.

I think the most common way that my debugging problems are more multidimensional than the kind you describe there (binary search through the data flow of the failing example until you find the point where correct data goes in and incorrect data comes out) is that it's not always obvious what _is_ correct data, because the data being manipulated is huge, or its correctness depends on a complicated spec that I don't keep all of in my head. (The intermediate representation inside a compiler is a good example.) So the question you have to answer at each step of the search is also nontrivial.

HolgerH 霍尔格 15h ago

@simontatham
Damn right, I agree!

Staden_スタデン 14h ago

I don't think I have ever done the sudoku approach. I don't even know how it would be done.

Chris L 11h ago

@Staden I used to work with programmers that had grown up in the Soviet Union-they rarely or never used a debugger but would read the code and simulate what must be going and and fix bugs that way, a skill developed by having access but limited access to computers so they had to use their minds more effectively.

Staden_スタデン 10h ago

@jayalane I would love to see them working and understand their thought process

Chris L 10h ago

@Staden They were impressive. They 8 spired me to take a moment to think before cranking up gdb (this was 1990s). I do think that the grow5 of software ecosystems over small APIs that we coded to reduce the effectiveness of this, as there is so much software in the dependency tree and you just can’t predict what people will do managing connections or certs or retries or numeric data or anything.

@simontatham This made me wonder if someone's made a mashup of those two games. Turns out they have, it's a cool idea - https://gamesforcrows.itch.io/sudo-sweep

Sudo Sweep by Games For Crows

Sudoku x minesweeper hybrid

itch.io

@simontatham *Looks at my "Doom" approach* 😅

Simon Tatham 13h ago

@rich ha! That reminds me of a totally different 'two approaches to debugging' incident I remember from university.

I was sitting in a friend's room while he wrote a program. He'd written a large amount of code before testing any of it – more than I would have been inclined to do myself. So when he started trying to run it, I observed:

"You know, when I debug, it's like hunting. Lay tempting bait, sit very still, try to entice the elusive bug to show itself, then bring it down with a single shot. But you're spurring your horse into a whole army of bugs, laying about you left and right with a broadsword, bringing several down in every stroke, but there are always more."

(He did get the thing to work, though, as I remember! Just a difference of preference in what order we liked to do things.)