What Is eBPF?

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/30-ebpf.

Small programs that run inside the kernel

eBPF is a mechanism that loads a small user-written program into the Linux kernel and runs it when a specific event occurs. An event here means one of these: a program calls a system call, a packet arrives, or execution reaches a particular instruction address. You do not rebuild the kernel, and you do not write a kernel module.

You can write “when a program calls the openat system call, record the file name.” You can also write “when execution reaches this instruction address, note the time.” This book deals with the second kind, which places hooks at specific addresses in user-space programs.

As Chapter 3 showed, kernel space is the side without restrictions. Running code of your own there is dangerous, so eBPF needs a safeguard.

The constraints the verifier imposes

The verifier provides that safeguard. It analyzes the whole program at load time, and the load fails if the program does not pass. The verifier guarantees three things: that the program always terminates, that it reads no uninitialized memory and no uninitialized pointer, and that it touches no memory outside what the kernel permits. The check behaves like the type checking in a compiler. The Go compiler turns a bad assignment into a compile error, and the verifier turns a program whose safety it cannot prove into a load error. A rejected program never runs in the kernel at all.

To make that verification possible, the kernel puts several constraints on an eBPF program. The number of instructions has an upper limit, and a loop has to take a form whose end the verifier can confirm. Fixing the count as a constant is the basic approach, and dedicated helpers such as bpf_loop let you pass the count at run time, but no form gives you an unbounded loop. Code that walks a list to its end is plain enough to write, but the verifier cannot prove that it terminates, so it does not pass. Hurdle 4 runs into this constraint in a concrete form.

Figure 1: The arrows represent load attempts and their outcomes. The verifier runs once, before execution, and a program that fails it never reaches the kernel.

Maps: state that survives across events

The kernel calls an eBPF program once per event, and the program finishes right away. Anything equivalent to a function’s local variables does not survive to the next event. To hold state across events, you use a kernel-managed key-value store called a map. Its role is similar to Go’s map. The difference is that the data itself lives on the kernel side, and both eBPF programs and user-space processes can read and write it.

The basic instrumentation pattern records a start time at a function’s entry, retrieves it at the exit, and computes the elapsed time. A map is what connects those two events. Hurdle 4 uses another map, ongoing_goroutines, to hold parent-child relationships between goroutines.

Figure 2: The arrows represent the passage of time. The entry hook and the exit hook are separate events; what connects them is the map key.

uprobe and uretprobe

eBPF has two main mechanisms for placing a hook in a user-space program.

• uprobe: a probe that you place at a specific instruction address in a user-space binary. People often call it an “entry hook” because tools commonly place one at a function’s first address, but the mechanism accepts any instruction address in the binary.
• uretprobe: a special mechanism for catching the moment a function returns.

To “place” a uprobe is to rewrite the contents of the binary. The kernel saves the instruction at the given instruction address, then swaps its first byte for a breakpoint instruction. On amd64 that instruction is int3, a single byte cc in machine code. Take the double function that we disassembled in Chapter 4 and place one at its top. The 48 of 48 01 c0 (ADDQ AX, AX) becomes cc, and the remaining bytes stay as they are. Chapter 4 also showed that this machine code sits on a read-only shared page. The kernel does not write to the shared page directly. It uses copy-on-write to make a copy private to that process, swaps the one byte in the copy, and points the mapping table at the copy. Neither the executable on disk nor the original shared page changes.

When the CPU reaches that address, it executes the swapped-in byte, traps, and enters the kernel. One question remains. cc is the same single byte for every uprobe, and the byte itself carries no record of which uprobe it belongs to. The kernel looks at the value of the PC at the moment of the trap. It matches that value against its table of registered uprobes, which records the file each uprobe went into and the byte position within it. If the address is not in the table, the kernel treats it as a breakpoint that someone else placed, such as a debugger.

The eBPF program tied to the uprobe identified this way then runs. The kernel saved the full set of registers at the moment of the trap, and it hands the eBPF program a copy. Another process normally cannot see the contents of those registers, but it can read them through this copy. Hurdle 2 depends on that copy. When the program finishes, the kernel executes the saved original instruction in a separately prepared area, then hands control back to the instruction that follows. From the target program’s point of view, a single instruction ran. The registers and the stack come back untouched.

A uretprobe adds one step to this. The kernel first places a uprobe at the function’s entry. It fires at the moment execution reaches the first instruction. Chapter 5 showed that SP then points at the slot holding the return address, the number that CALL pushed. The kernel saves the value in that slot and writes the address of a trampoline of its own into the same place. It never has to disassemble the target function or work out the size of the frame.

When the function executes its final RET, it jumps to the trampoline instead of the caller, and traps there again. The kernel runs the return-side eBPF program, then jumps to the real return address that it saved¹.

The two mechanisms differ in what they rewrite. A uprobe rewrites one byte of machine code, and that byte lives in the code area. A uretprobe rewrites a value on the stack as well. That difference is what produces Hurdle 1².

A different target also means a different way of writing. The code area is a read-only shared page, so it needs the copy-on-write detour that you saw above. The stack is an area that the process itself reads and writes as it runs. No protection has to come off, and no copy or swap happens.

Chapter 3 said that you cannot read another process’s memory from outside by naming an address. That holds between user-space processes. The kernel is on the side that the limit does not reach. It writes the target process’s page in place, so the program uses the new value from the next time it reads that slot.

Figure 3: The arrows represent the order of operations. A uprobe swaps the first byte of an instruction to cause a trap, and the kernel runs the original instruction in a separately prepared area. A uretprobe is an entry uprobe that rewrites the return address on the stack to the trampoline’s address.

Typical instrumentation places a uprobe at the function’s entry and a uretprobe at its exit. The pattern is “record the start time at the entry, compute the elapsed time at the exit.” Most languages take this without trouble. In Go, the straightforward form fails at the first step.

A loader written in Go

You write the eBPF program itself in a restricted dialect of C and compile it to bytecode with a dedicated compiler. The side that loads the bytecode into the kernel, binds probes to real addresses, and reads maps is an ordinary user-space program. Most people write that loader in Go. The de facto library is github.com/cilium/ebpf, and OBI uses it too.

The code below attaches a single uprobe.

// assume the compiled eBPF program objs has already been loaded
exe, err := link.OpenExecutable("/proc/12345/exe")
if err != nil {
	return err
}

// place the probe at the function's entry by symbol name
up, err := exe.Uprobe("main.handleRequest", objs.OnEntry, nil)
if err != nil {
	return err
}
defer up.Close()

Pass a symbol name as the first argument, and the probe goes at that symbol’s first address. Another form takes an address directly instead of a symbol name.

up, err := exe.Uprobe("", objs.OnReturn, &link.UprobeOptions{
	Address: 0x9e290, // the instruction's offset within the file
})

That form, an empty symbol name plus an Address, is the workaround in Hurdle 1. The value you pass needs care. A uprobe registers by how many bytes from the start of the file, so what goes here is an offset within the file and not a virtual address. Pass a symbol name and the library converts it for you; pass Address directly and you compute it yourself from p_vaddr and p_offset in readelf -lW, as offset = the instruction's virtual address - p_vaddr + p_offset. Pass the 0x49e290 that the disassembly shows and the probe does not land on the instruction you aimed at. Chapter 12 shows where OBI does this conversion. Running the two snippets above needs Linux and root privileges, so read them rather than run them. The Go samples later in the book are the ones to run and check on your own machine.

Figure 4: Solid arrows represent the flow of processing; dashed ones represent auxiliary flows (placing uprobes and reading maps). The loader puts eBPF programs into the kernel, and only those that pass the verifier stay resident. A probe fires when execution reaches the instruction address that holds it, and user space reads state that spans events through maps.