Ritesh Oedayrajsingh Varma5d ago

New article! A user *cough* @aras *cough* is reporting full system freezes while using Superluminal on Linux. What do you do? Cry? Well, we did a little bit.

But we also dove into the kernel...again, this time finding & fixing several issues in eBPF's spinlock implementation. Read all about it:

https://rovarma.com/articles/a-tale-about-fixing-ebpf-spinlock-issues-in-the-linux-kernel/

A tale about fixing eBPF spinlock issues in the Linux kernel | Ritesh Oedayrajsingh Varma

A system freeze led us deep into Linux spinlock internals, where we helped find not one but three bugs in the kernel's resilient locking code used by eBPF.

Show threadMichael Coupland1d ago
@rovarma Engaging read!
One lingering question: after the fixes, ringbuf_reserve in the NMI sampling event fails because the lock detects and rejects reentrancy, right? (Just quickly, instead of after 250ms.) That seems… scary?
I guess you just don’t get a sample, and that’s not the end of the world for a profiler? Are there other NMI you instrument where a failure like this might drop important information?
Show threadRitesh Oedayrajsingh Varma
@michaelc Yep, that’s exactly what happens. It’s fine in this case because with the high sampling rate and the sporadic nature of the race, you’ll lose a few samples at most. And, the race is between the context switch and the sample, and we also collect stacks for cswitches, so the sample wouldn’t even give us any extra information since it would be the same stack (remember this is a same-CPU issue, so it effectively means you’d be sampling the cswitch).
Mar 22 at 11:55amWeb
Show threadRitesh Oedayrajsingh Varma17h ago
@michaelc The sample interrupt is the only NMI we care about. There might be other NMIs on the system, but we don’t capture them.