Sneak peek: LBR based stacks in tracing spans for failed syscalls through ebpf_exporter.

Think "retsnoop, but more structured". It be attached to existing traces too.

Also pictured: SRSO sadness.

#ebpf #kernel #ebpf_exporter

It's been too long since the last ebpf_exporter release, so I cut v2.5.0 today.

https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.5.0

#ebpf_exporter #ebpf #bpf #linux #kernel

Have you ever though "boy I love distributed tracing, but I wish I could add kernel produced spans to my traces"?

Well, now you can! I just published ebpf_exporter v2.4.0 and it can now create kernel spans that are integrated in your normal traces.

See: https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.4.0

The most interesting example is producing network related spans, like TCP retransmits or SKB drops. Here's a screenshot of a whole bunch of them attached to a userspace trace.

#ebpf #ebpf_exporter #kernel

It's been a while since the last ebpf_exporter release, so I cut a new one just now. Lots of changes in v2.3.0.

You can now start testing with just docker:

```
docker run --rm -it --privileged -p 9435:9435 ghcr.io/cloudflare/ebpf_exporter:v2.3.0 --config.dir=examples --config.names=timers
```

* https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.3.0

#ebpf #ebpf_exporter #kernel #linux

Empty ebpf getpid() tracepoint overhead: 15ns 🏎️, fentry is 24ns 🚗

Empty ebpf uprobe overhead: 1600ns 🐌

Benchmark: https://github.com/cloudflare/ebpf_exporter/pull/331

Is there a better way if I want arbitrary userspace programs to pass something to my ebpf code? Giving a ringbuf fd via some library seems convoluted.

#ebpf #ebpf_exporter

New cool example for ebpf_exporter: CFS delay histogram. In addition to knowing overall CFS throttlig delay from cgroups in cpu.state, now you can have a histogram of individual throttling durations in prometheus.

* https://github.com/cloudflare/ebpf_exporter/pull/311

As a bonus, you get a bpftrace command to observe these.

#ebpf #ebpf_exporter #cgroups #cfs #bpftrace #prometheus

I hacked together opentelemetry distributed tracing support for ebpf_exporter: https://github.com/cloudflare/ebpf_exporter/pull/297

So far I managed to add some block i/o tracing via tracepoints, but it's unclear how to tie this together to userspace traces, since there's no way for userspace to pass the trace id.

Are there any other kernel areas that people are interested in having integrated with distributed tracing? Sockets? Scheduling? Something else?

#ebpf_exporter #ebpf #kernel #opentelemetry #otel #otlp

ebpf_exporter now comes with a docker image with examples if you couldn't be bothered to build it yourself:

* https://github.com/cloudflare/ebpf_exporter/pkgs/container/ebpf_exporter

docker run --rm -it --privileged -p 9435:9435 ghcr.io/cloudflare/ebpf_exporter:master --config.dir=examples --config.names=timers

#ebpf #ebpf_exporter