Austin Clements from the Go team just landed a series of 17 patches. If everything goes well, you'll never notice that anything changed.

That being said, watching this come together has been mind blowing. This is serious engineering (TM).

🧵 Thread ...

https://go-review.googlesource.com/c/go/+/472956/5

From a high level point of view, these patches are "just" a refactoring of the stack trace implementation in Go.

The gnarly old gentraceback() function has been replace with a new iterator API that is more flexible, yet easier to understand.

To get why this is amazing engineering, first consider the old code.

gentraceback was a 500+ LoC monster function that will make you weep if you try to read it. And that's even before you realize how incredibly load-bearing it is.

https://github.com/golang/go/blob/go1.20.1/src/runtime/traceback.go#L24-L563

go/traceback.go at go1.20.1 · golang/go

The Go programming language. Contribute to golang/go development by creating an account on GitHub.

GitHub

When I say load-bearing, I mean that without it, the GC won't be able to scan the stacks of your goroutines, and your goroutines won't be able to grow their stacks (which needs pointer adjustments).

It's an absolutely vital organ of the Go runtime.

Any bug in this function has a chance to break all Go applications.

If you're doubtful, look no further than some recent issue that caused Go programs on arm64 to become completely unresponsive due to gentraceback getting stuck in an infinite loop.

https://github.com/golang/go/issues/52116

runtime: gentraceback() dead loop on arm64 casued the process hang · Issue #52116 · golang/go

What version of Go are you using (go version)? $ go version go version go1.18 linux/arm64 Does this issue reproduce with the latest release? Yes, it reproduce in the latest Go1.18 What operating sy...

GitHub

Speaking of arm64, it's part of why this function is complicated. ARM is a so called link architecture which works very different from intel when it comes to function calls.

A register, rather than the stack, is updated when CALLing a function. More here

https://blog.felixge.de/go-arm64-function-call-assembly/

Go arm64 Function Call Assembly

An in-depth analysis of the assembly code emitted by the Go compiler for function calls on arm64.

Felix Geisendörfer

Now let's explore why it's so difficult to refactor this function. The first reason is that it's insanely difficult to debug this code when you break it.

It'll produce the most cryptic panics, crashes, corruptions and issues you can imagine.

Secondly you're not really writing Go. You're writing "runtime code" that has to be signal safe. The rules for this are incredibly subtle, and you'll quickly learn about write barriers and how the compiler will refuse to build things when you introduce them into this code.
Next up are the build bots: Assuming you've got something work on your local machine, there is a decent chance you've broken things on another OS or architecture. You'll spend lots of time debugging these kind of failures.

Last but not least: Breaking up such a complex refactoring is necessary in order to give somebody a chance to review it.

And this makes everything harder. What if you're 10 patches in, and you notice a mistake in patch 3? It's possible, but very hard to make these kind of edits.

I've still skipped over many other aspects, such as designing a nice API, dealing with inlining, etc. - but hopefully you can see why this is really gnarly stuff to be hacking on.

So now look at these 17 patches again. It's a real work of art 🙇🏻‍♂️.

https://go-review.googlesource.com/c/go/+/472956

If you need a comparison (or laugh), feel free to look at my pathetic attempt from last year to do the same refactoring 🤣.

I spent a week on it and produced 54 chaotic patches that didn't even get me half way to a full iterator API.

https://github.com/felixge/go/pull/4

refactor gentraceback into iterator style by felixge · Pull Request #4 · felixge/go

The goal is to refactor gentraceback to use a more high-level iterator API internally. E.g. something like this: itr := newTracebackIterator(...) for n := 0; n < max; { if !itr.Next() { break }...

GitHub

So again, 👏🏻 to Austin Clements for this incredible work, but also
@prattmic for some fantastic review comments.

The people working on the Go runtime team are really incredible. ❤️

💐RIP gentraceback
2010-03-23 - 2023-03-10

Thank you for all the unwinding.

@felixge 🫡 there's no better attitude to replacing code than being grateful for what it's done. It's probably been executed billions (trillions?) of times.

Thanks for the thread btw, wasn't aware of this awesome work!

@javierhc yeah, gentraceback definitely accomplished a lot in its lifetime. More than most code, that's sure :).