How Function Calls Work

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/20-function_call.

The chapters so far covered the contents of an executable and the way it lands in memory as a process. This chapter turns to the thing that happens most often while a program runs: the function call. The return address in Hurdle 1 and the argument passing in Hurdle 2 both come straight out of this chapter.

Stack frames

A call pushes a working area for the called function onto the stack. That area is a stack frame. It holds local variables and values saved for later use, and it goes away when the function returns. The stack grows from high addresses toward low ones, as Chapter 3 showed. The deeper the calls go, the lower the new frames land.

The frame that goes away is always the last one pushed. A function returns in the reverse of the order you called it, so no frame in the middle can disappear first. That order, last pushed and first popped, is last in, first out (LIFO).

A function has two places worth telling apart. Its instructions live in .text from Chapter 4 and stay put however often you call it. What the stack holds is the execution state of one call. In the experiment below, one function runs four times, and the four calls share one set of instructions while pushing four frames.

You can observe this with the pointers from Chapter 2. The program below recurses to deepen the call stack and prints the address of a local variable at each depth. The end of this chapter explains what //go:noinline means.

• Run in the Go Playground

package main

import (
	"fmt"
	"unsafe"
)

//go:noinline
func descend(depth int) {
	var local [64]byte
	addr := uintptr(unsafe.Pointer(&local[0]))
	fmt.Printf("depth %d: address of local = %#x\n", depth, addr)
	if depth < 3 {
		descend(depth + 1)
	}
}

func main() {
	descend(0)
}

depth 0: address of local = 0x17ef0a08eeb8
depth 1: address of local = 0x17ef0a08ee08
depth 2: address of local = 0x17ef0a08ed58
depth 3: address of local = 0x17ef0a08eca8

Each step down in depth drops the address by exactly 176 bytes. One call to descend therefore pushes a frame of 176 bytes. When the function returns, the whole of that frame goes away.

Figure 1: This figure has no arrows. Up and down correspond to high and low addresses. Each deeper call to descend pushes a frame 176 bytes lower than the last.

CALL and RET

The frame holds one more thing, the return address. On amd64, the CALL instruction pushes the address of the next instruction onto the stack and then jumps. The RET instruction at the end of the function pops that value and jumps to it. Where to return is therefore a number in one slot on the stack, and nothing more.

Disassembling the calling side of the double example from Chapter 4 shows you the real thing.

$ go tool objdump -s 'main\.main$' demo
  ...
  main.go:11   0x49e1b3   e8c8ffffff     CALL main.double(SB)   ← the call happens here
  main.go:11   0x49e1b8   440f117c2428   MOVUPS X15, 0x28(SP)   ← after returning, this runs next
  ...

The CALL instruction sits at address 0x49e1b3, and the instruction after it sits at 0x49e1b8. The difference of 5 bytes is the length of the CALL instruction itself. You can read that off the output, where the machine-code sequence e8c8ffffff is exactly 5 bytes. The moment this CALL runs, the number 0x49e1b8 goes onto the stack. The RET at the end of double pops that number off the stack and jumps to it.

Look at the called side as well. The body of double that Chapter 4 disassembled held only two instructions.

$ go tool objdump -s 'main\.double$' demo
  main.go:7   0x49e180   4801c0   ADDQ AX, AX
  main.go:7   0x49e183   c3       RET            ← pops 0x49e1b8 off the stack and jumps there

Nothing on the RET line says where to return. It is a one-byte instruction, c3, with no operands. The destination sits on the stack, so the instruction says only “return to the value that is pushed there.”

SP is the register that points at that value. Chapter 2 listed SP by name only. It holds the address of the top of the stack, which is the last location pushed. When CALL pushes the return address, SP moves down by that much. When RET pops the value, SP goes back to where it was. RET needs no operand because its destination is fixed as whatever SP points at.

Now look at the moment execution reaches the first instruction of the function. Nothing has gone onto the stack after the return address that CALL pushed. Because the stack is last in, first out, SP points at exactly that slot. The function lowers SP further when it reserves its own frame, but that happens after the first instruction runs. However deep the calls go, and whatever else already sits on the stack, one read of SP finds the return address at that point.

Figure 2: Arrows show the order of events in time, and the vertical position inside each table shows the height of the address. SP always points at the last location pushed. CALL pushes the address of the next instruction and SP moves down to it. RET pops the value that SP points at and jumps there. Values come off in the reverse of the order they went on, so what RET pops is always what the matching CALL pushed.

Hurdle 1 covers uretprobe, the mechanism where the kernel rewrites this number. Keep two things in mind. The thing it rewrites sits on the stack, and at the entry to a function SP says where.

Calling conventions

The arguments do not have to sit inside the frame. A calling convention is the agreement about where the caller puts the values and where the callee reads them. Being an agreement, it works with any location, as long as both sides follow the same one.

Two methods matter here: pushing the arguments onto the stack, and putting them in registers. With stack-based passing, an outside observer can read the arguments from fixed positions in the frame. Register-based passing skips memory and is faster, but you cannot read the arguments without the table that says which argument goes in which register.

The output above holds a live example. Look at the line just before CALL main.double(SB).

  main.go:11   0x49e1ae   b815000000   MOVL $0x15, AX   ← puts the argument 21 (=0x15) in AX
  main.go:11   0x49e1b3   e8c8ffffff   CALL main.double(SB)

The argument 21 in double(21) goes into the register AX rather than onto the stack, and then the call happens. The body of double from Chapter 4 was ADDQ AX, AX. The caller puts the value in AX and the callee reads it from AX, and that correspondence is the convention.

Every platform has a standard calling convention, and every C program on Linux amd64 follows it. The tools that observe from outside assume that standard. Go does not follow it, and that mismatch is the subject of Hurdle 2.

Figure 3: The same call to f(a, b, c) puts its arguments in different places under different conventions. With stack-based passing, the arguments start at SP+8, past the return address that CALL pushed. With register-based passing there are no arguments on the stack, and a reader that assumes the C convention looks at RDI for the first argument, which hands back whatever Go put in its fourth register. Nothing reports an error; you simply read another argument’s value.

Inlining

The chapter so far rests on one assumption: that a function call survives in machine code as a CALL. That assumption does not always hold.

The compiler sometimes replaces a call to a small function with the body of the function expanded in place. This is inlining. The round trip of CALL and RET disappears along with the argument passing, so execution gets faster. -gcflags=-m shows you what the compiler decided.

$ go build -gcflags=-m main.go
./main.go:6:6: can inline double
./main.go:11:20: inlining call to double

//go:noinline is the directive that forbids the expansion and keeps the call as a CALL. The samples in this book carry it again and again, because inlining would remove the very call that I want you to observe. You do not need it in an ordinary application.

Inlining reaches further than observation. To observe the entry and the exit of a function from outside, you name the instruction address of that function. An inlined function leaves no address to name, so small accessors and wrappers are the first to vanish from instrumentation. This is also why a function that you wrote yourself was missing from the symbol table in the Chapter 4 exercise. Placing an observation point where an address does exist continues the story from Chapter 4. The target is machine code on a read-only shared page, so the tool swaps the page through copy-on-write, and Chapter 7 shows the injection itself.

Figure 4: Arrows show the direction of the transformation. Inlining erases the call instruction itself, which leaves no address to place a hook on.

Key points for the hurdles

• The return address is a number that CALL writes into one slot on the stack. The uretprobe in Hurdle 1 rewrites it.
• The stack only ever works last in, first out. At the first instruction of a function, SP points at the return address that the matching CALL pushed. The uretprobe in Hurdle 1 uses that one point to decide where to write.
• The calling convention decides where the arguments live. Register-based passing is fast, but reading it from outside needs the table. Hurdle 2 is the story of that table.
• An inlined function has no address to place a hook on at all.

Exercises

1. In the descend experiment, change the array local from [64]byte to [128]byte. How does the spacing of the printed addresses change? Make a prediction, then check it.
2. Remove //go:noinline from double, rebuild, and look for CALL main.double in the output of go tool objdump -s 'main\.main$'. Why can you not find it?