New blog post: The Return of the Frame Pointers (Fedora, Ubuntu) https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
The Return of the Frame Pointers

The Return of the Frame Pointers

@brendangregg As someone who has worked in smaller organizations which didn't at the time consider it practical to recompile everything (and be responsible for the consequences of such), this makes me so happy.

(But angry that it took us so long in the x86_64 era to get there)

@heftig seems glibc in Arch still not rebuilt against frame pointers. Is it planning to land?
@ngz0 There's no concrete time plan I'm aware of. The next build of the package should have the frame pointers.
@brendangregg I’d say they are no longer valid for amd64 but they are still valid-ish for i386.
@brendangregg it’s all fine and good it being enabled by distros.. but I’d love to see it enabled by default in the compilers too. GCC if I’m not mistaken will omit frame pointers from β€œ-O1” up, meaning that so much non-distro software will still lack them unless the maintainer explicitly turned them back on.

@brendangregg As another data point: In the Wasmtime JIT for WebAssembly we've had frame pointers turned on unconditionally on x86-64 for several years, and started doing that for all architectures a couple years ago because we need to quickly capture a stack trace if the guest traps.

That said I'm confused by your suggestion that JITs can't produce DWARF, since Wasmtime does. That code is a regular source of bugs, and I'm not certain whether profilers can see the generated DWARF, but when it works, I believe at least gdb supports it.

That doesn't change your point that DWARF isn't a great choice for profiling, and for the same reasons we don't want to use DWARF for stack-walking.

But we've discussed that we may have to stop relying on frame pointers in the future to support WebAssembly exceptions proposals. We also don't currently do any inlining, so frame pointers can potentially be a significant hit if we have a ton of tiny leaf functions. So we still have open questions about what's the best thing for us to do.

@jamey @brendangregg Interesting - why are you outputting unwinding dwarf for JIT if you also have framepointers in the jit code? Is it needed for GDB unwinding to work?
@mstange @brendangregg We produce general-purpose DWARF (which register is this variable in, etc), which I believe happens to include enough for unwinding. I'm not super familiar with the details though, I could have some of this wrong.
@jamey Thanks for the info, I've updated the post as others have said JIT->DWARF is possible too. I'm not sure it's going to scale to the busy JVM workloads I've seen in production though, where just I tried realtime JIT symbol logging (timestamp, func name, address, size) and found the overhead for that was too high. Imagine a 64-CPU system with a significant amount of time in c2 compilation constantly.
@brendangregg Yeah, as I said, you're right that DWARF isn't a good choice for profiling, for several reasons. It just isn't literally impossible, that's all. 😁
@brendangregg we've spent considerable resources in terms of engineering at Mozilla to deal with the issue of stack traces, both for stability (crash reporting & diagnostics) and performance (profiling). Frame pointers are hugely helpful when present, but we also have to live without them since extensive inlining in C++/Rust code means they're just not enough.

@brendangregg you might want to check the framehop and wholesym Rust crates developed by my colleague @mstange

They've been written to provide very high quality stack traces while also offering very high performance, so that they can be used during online profiling:

https://crates.io/crates/framehop
https://crates.io/crates/wholesym

crates.io: Rust Package Registry

@gabrielesvelto @brendangregg Framepointers and inlining are two totally orthogonal issues though. Framepointers give you reliable and simple unwinding, and having them is super valuable regardless of whether you resolve inlining during symbolication.

@brendangregg The 2nd China eBPF Developers Conference Invitation!

Dear Mr. Brendan Gregg,

I hope this message finds you well. My name is Zhao Chenyu, representing Thoughtworks China and Xi'an University of Posts and Telecommunications in China. It is with great sincerity that I extend this invitation to you to participate in the upcoming 2nd eBPF Developer Conference in China as a keynote speaker.

Please contact me for more detailed information.