BPF kprobes for latency tracing

eBPF started as "run tiny programs in the kernel safely." For observability, kprobes let you attach at function entry/return and export histograms without a custom kernel module — until you overdo it and measure the observer more than the system.

What a kprobe gives you

Attach to tcp_sendmsg, record timestamp delta to userspace, aggregate p99 latency. No recompile, no printk spam — verifier-checked bytecode instead.

latency.bpf.c

1SEC("kprobe/tcp_sendmsg")2int BPF_KPROBE(tcp_sendmsg_entry) {3    // store start ts keyed by pid/tid4    return 0;5}

Gotcha

Probes run in kernel context on hot paths. Heavy maps, string formatting, or unbounded loops fail verification or cost more than the syscall you're measuring.

When BPF beats strace

Production-safe sampling at high QPS.
Aggregated metrics in-kernel (histograms, per-cgroup counts).
No stop-the-world attach to every process.

When you need full argument capture or userspace stacks only, perf / eBPF stack walkers / tokio-console may be simpler starting points.

🦀

Ferris' hot tip

Start with off-the-shelf tools (bpftrace, bcc) before writing C. Custom probes shine when you have a specific kernel boundary and a SLO to defend — not for hello world.

Pair latency tracing with reading EXPLAIN on the database side — most "mystery slowness" is two hops, not one.