What the Go Runtime Manages

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/25-go_runtime.

A Go binary carries a runtime alongside the code that you wrote. The runtime does its own scheduling and its own memory management instead of leaving either to the OS. Doing the work itself is what makes Go hard to observe from outside.

One statically linked binary

Chapter 4 showed that linking comes in two forms, static and dynamic. Go puts the standard library and the runtime into one executable. It needs no shared library unless you use cgo, and a binary built with CGO_ENABLED=0 gives back not a dynamic executable from ldd. Build a program that uses net/http with the default settings and a dependency on libc remains, but the Go code still sits in one file.

This suits the side that instruments the program, and one misreading is worth heading off first. A uprobe names a file and a byte offset within it, so nothing stops you from placing one in a shared library. OBI does exactly that in the OpenSSL shared library to read TLS traffic. The advantage of static linking lies elsewhere. Your application code, a library such as net/http, and the runtime all sit in one file, at positions that the build fixes. Analyze that one file through /proc/<PID>/exe and you find every function that you want to instrument.

This is also why Chapter 1 said that Go is the only language whose instrumentation reaches down to library functions. A JIT compiler or an interpreter generates the machine code for user code at run time. Analyzing the executable of such a language tells you nothing about where the functions are. The shared library of the runtime itself, libjvm.so or CPython, does have fixed addresses, and OBI places hooks there. What it cannot reach are the functions of the libraries that run on top. In Go, compilation fixes the position of every function, and the symbol table or .gopclntab gives it to you. Instrumentation that places a hook at an address works for that reason.

OS threads and goroutines

A goroutine is not an OS thread. It is a unit of execution that the runtime manages, and every go f() creates one with a small working area of a few KB. Creating tens of thousands of them is fine.

Three structs divide the roles inside the runtime. A g is one goroutine and an m is one OS thread. A p holds the right to run Go code, together with a queue of waiting work. Many g structs take turns running on a few m structs through the p structs. The Go runtime decides when to switch them, and the kernel does not. Only g returns later in this book, so the names m and p are all you need to keep.

Two things follow. The kernel cannot see a goroutine, because its knowledge stops at the thread. Goroutines from different requests can also run on the same OS thread. A thread ID therefore cannot tell you whether two moments of work belong to one request. Hurdle 4 starts from this mismatch.

Figure 1: Arrows show the “runs on” relationship, where each item above runs on the item below it. The Go runtime decides which goroutine runs when, and the kernel does not.

Goroutine stack growth

One point is worth settling before the mechanism: the stack of a goroutine is a separate region from the stack of the OS thread. The OS thread gets a stack of a substantial fixed size when it starts, and that stack never moves. The runtimes of many languages use the stack of the OS thread directly as the stack of each thread, and a JVM platform thread is one of them. (The virtual threads that arrived in JDK 21 are closer to Go, and keep their stacks in a region that the GC manages.) The Go runtime instead allocates a stack region of its own for every goroutine that it creates. The stack of the OS thread does not disappear, and the runtime uses it for internal work of its own.

The stack of a goroutine also differs in behavior from the stack of an OS thread. It starts small and grows when it runs out.

The procedure runs like this. The entry of each function carries an instruction that checks whether enough stack remains. If not enough remains, execution jumps to runtime.morestack. There the runtime allocates a larger region, copies the current contents whole, and rewrites every pointer into the stack with its new address.

The address of a local variable in a goroutine therefore changes while the program runs. The runtime corrects the pointers that it placed, so Go code never sees the relocation. An observer looking in from outside does see it. This book calls a stack that can move to another place during execution a movable stack. It is the cause of Hurdle 1.

Figure 2: Arrows show the order of events in time, from top to bottom. During the copy, the runtime walks the frames one at a time, using the return addresses as its guide. The move depends on the frames being walkable.

Go’s own calling convention

Go also decides for itself where a call puts its arguments. Since Go 1.17, a call passes its arguments in registers rather than on the stack.

One register gets special treatment as well. On amd64, R14 keeps pointing at the g struct of the goroutine that runs now. R14 therefore turns up everywhere in machine code compiled from Go source.

Hurdle 2 covers the details. For now, one fact is enough to carry forward: a Go function call does not follow the standard convention of the platform.