A tale of @iainireland 's performance win, and the question of: How do you find problems like this in your system if you don't know _what_ you are looking for?

https://www.mgaudet.ca/technical/2025/5/9/a-performance-investigation-challenge

A Performance Investigation Challenge — Matthew Gaudet

Matthew Gaudet

In this vein: Any `bpftrace` wizards out there (please feel free to boost for visibility):

Suppose I have the a trace file like this:

```
uprobe:/home/matthew/unified-git/obj-debug-shell-x86_64-pc-linux-gnu/dist/bin/js:*registerWithRootLists*
{
@call[ustack(perf,3)] = count()
}

uprobe:/home/matthew/unified-git/obj-debug-shell-x86_64-pc-linux-gnu/dist/bin/js:cpp:JS_ShutDown
{
printf("calling exit from ebpf");
exit()
}

```

right now I get unsymbolicated output like this:

```
@call[
55912bd662e0 0x55912bd662e0 ([unknown])
55912bde2918 0x55912bde2918 ([unknown])
55912c053dbd 0x55912c053dbd ([unknown])
]: 1538
```

Which is cool! This is almost certainly the data I'm interested in. But without symbols it's uninterpretable. How on earth can I get this symbolicated?

@mgaudet I have had this issue before, it’s most likely a timing problem of symbol resolution vs process exit. My workaround is to just put an infinite loop in the VM shutdown routine so bpftrace has a chance to read /proc and resolve symbols. See this issue for more context

https://github.com/iovisor/bpftrace/issues/246

ustack not symbolicated if traced process exits first · Issue #246 · bpftrace/bpftrace

I have process that allocates once and then sleeps: #include <unistd.h> #include <stdlib.h> int main() { void* a = malloc(2); sleep(1000); return 0; } And I trace it with this command: sudo bpftrac...

GitHub

@caizixian Oh wow, for me, it's even dumber.

If I explicitly print the map, it works, but if I wait till the at-exit map print it doesn't. Doesn't seem to have anything to do sleeping for me.

Thank you for your hint though!

@caizixian Oh but maybe I still need the sleep? Silly!
@mgaudet I wonder how you installed bpftrace. I think I tried compiling my own and installing from distro, but I find downloading the latest static binary from their GitHub releases to be the most reliable. A lot of bugs and corner cases are fixed over time.

@caizixian hah. I just used whatever my distro installed; never crossed my mind to check for a GitHub release

I am -so pleased- that what I have is already working.

Thank you again for your help. Invaluable.

Exploring a language runtime with bpftrace — Matthew Gaudet

Matthew Gaudet

@mgaudet I'm glad I could help.

@iainireland @smarr

Following up on the start of this thread on microarchitectural performance debugging, you might find Claire Huang's undergraduate thesis interesting (co-supervised by @steveblackburn). She took a very detailed look at GC in terms of cache behaviors, and measured optimization headroom. https://www.steveblackburn.org/pubs/theses/huang-2025.pdf

She also previously led the eBPF tracing in GC work (MPLR'23).