Ritesh Oedayrajsingh Varma

New article! A user *cough* @aras *cough* is reporting full system freezes while using Superluminal on Linux. What do you do? Cry? Well, we did a little bit.

But we also dove into the kernel...again, this time finding & fixing several issues in eBPF's spinlock implementation. Read all about it:

https://rovarma.com/articles/a-tale-about-fixing-ebpf-spinlock-issues-in-the-linux-kernel/

A tale about fixing eBPF spinlock issues in the Linux kernel | Ritesh Oedayrajsingh Varma

A system freeze led us deep into Linux spinlock internals, where we helped find not one but three bugs in the kernel's resilient locking code used by eBPF.

Mar 17 at 2:36pmWeb
Show threadagersant3d ago
@rovarma @aras excellent read, thank you for explaining all this in an approachable and easy to read format! 👏👏
Show threadMorten Hilker-Skaaning3d ago
@rovarma @aras what a journey 😊 best wishes for pushing Superluminal to be excellent for Linux🥇
Show threadJake Shadle3d ago
@rovarma @aras Great article!....and now my anticipation for Superluminal on Linux is even higher. Take my money! :D
Show threadAras Pranckevičius3d ago

@rovarma great post! (kinda proud of myself that my very eyeball-y guess “200ms stalls” turned out to be almost exactly correct :))

Would be good to clarify why only certain kernel versions were affected, since now not clear why only I could see the problem initially

Show threadRitesh Oedayrajsingh Varma3d ago

@aras Good point! Added a bit about that in the conclusion.

TL;DR: usage of rqspinlock in the ebpf ring buffer was introduced with kernel 6.15, and our development machines (and most of our alpha testers) are on Ubuntu, which runs an older kernel than that (6.14).

It would eventually have shown up for us too, once Ubuntu caught up.

Show threadRitesh Oedayrajsingh Varma3d ago
@aras I’m planning on switching my dev machine to Arch, which is on the bleeding edge kernel usually, so that we’ll spot issues like this faster locally in the future
Show threadAras Pranckevičius3d ago
@rovarma nice! And again, oh man what a debugging story. Massive props both for writing it down, and for working with kernel folks to actually fix it. ❤️
Show threadDavid Gonzalez Martin3d ago
@rovarma Great post. I can't wait to use Superluminal on Linux :)
Show threadChris Green2d ago
@rovarma @aras Will a cheap rs232 to ethernet adapter + rs232 card work for kernel debuggjng? The advantage would be that you only need this on the target machine to gain access to it from any computer that can see its ip addr.
Show threadRitesh Oedayrajsingh Varma2d ago
@aachrisg @aras it might, but honestly a regular rs232 card is not expensive either, so it doesn’t matter much :)
Show threadChris Green1d ago
@rovarma @aras Its not about $, it's about getting a 1:N connection for kernel-level logs for about the same cost as 1:1. And w/o running any (severely limited length) thick cables. My home office is kind of sprawling :-). I tend to use tablets as log displayers too.
Show threadRitesh Oedayrajsingh Varma1d ago

@aachrisg @aras True! In my case, my machines are right next to each other, so the cable is fine. There is a KGDBOE (“KGDB Over Ethernet”) project that attempts to generically support kernel debugging over ethernet transports, but due to the complexity of the ethernet driver, it’s severely limited, and I never got it to work. If the ethernet driver is “external” to the machine being debugged, it should work in principle I think. One way to find out! ;-)

https://github.com/sysprogs/kgdboe

GitHub - sysprogs/kgdboe: A network interface for GDB for Linux Kernel

A network interface for GDB for Linux Kernel. Contribute to sysprogs/kgdboe development by creating an account on GitHub.

GitHub
Show threadChris Green1d ago
@rovarma @aras Thanks, I'll have to take a look. I had to figure out what was going on with a non-booting mini-pc recently and it sure would have been nice to watch kernel log output on my main desktop (and 200K faster throughput as a bonus).
Show threadMichael Knudsen2d ago

@rovarma @aras great story and writeup!

Also, it's good to hear Superluminal is coming to Linux. It might be useful should I one day decide to add bugs to my code. (:

Show threadLogicalErzor1d ago
@rovarma @aras what a great and approachable read! looking to see how i can use this same digestible prose for my blog