Hurdle 3: Version-Dependent Field Offsets

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/50-hurdle3_offsets.

OBI identifies a g without reading its contents. Other values are not so easy. To get the method name of an HTTP request, or the target of a gRPC call, OBI reads inside a struct.

The protocol instrumentation from Chapter 8 looks like it could get the same values from the byte sequence that passes through the socket. For plaintext HTTP/1.1 it does. OBI reads the method and the path from a leading byte sequence such as GET /items HTTP/1.1. Go instrumentation still reads structs, because some values never reach the byte sequence.

One reason is TLS. Go carries its own crypto/tls and links it statically, so the Chapter 6 technique of placing uprobes on OpenSSL functions does not apply. Only the encrypted byte sequence reaches the socket. For a Go process, OBI places uprobes on crypto/tls.(*Conn).Read and Write, and reads the plaintext there: before encryption, and after decryption. The other reason is HTTP/2 and gRPC. HTTP/2 compresses header names and values with HPACK, which replaces them with indexes into a table built for each connection. From the second request onward, :path can be an index alone. An observer that did not watch the connection from its first byte cannot recover the original string. OBI then falls back to * for the span name. What the socket path reads by default is also just the first 256 bytes of a request; enable body capture in the configuration and HTTP/1.1 can go further. Some values, such as a routing template, never appear in the byte sequence at all.

So Go instrumentation reads the struct before serialization. It reads Method and URL in http.Request, and method in gRPC’s transport.Stream, by byte position. An external tool has to know where each field sits. This is the hardest of the four hurdles to deal with. A mistake here behaves like a mistake in Hurdle 2: no crash, and a trace whose contents are wrong.

What is a field offset?

As Chapter 2 showed, an external observer sees byte positions only. The number of bytes from the start of a struct to a field is the field offset. unsafe.Offsetof returns that same number.

Go source accesses a field by name, as in req.Method, but eBPF looks at the process from outside, where the name does not exist. Running memory holds byte sequences and addresses. So the external reader needs a rule stated in bytes: Header sits 56 bytes after the start of http.Request.

Checking your Go against OBI’s table

You can extract this offset from Go itself with unsafe.Offsetof.

• Run in the Go Playground

package main

import (
	"fmt"
	"net/http"
	"runtime"
	"unsafe"
)

func main() {
	var r http.Request
	fmt.Println(runtime.Version())
	fmt.Printf("Method        = %d\n", unsafe.Offsetof(r.Method))
	fmt.Printf("URL           = %d\n", unsafe.Offsetof(r.URL))
	fmt.Printf("Header        = %d\n", unsafe.Offsetof(r.Header))
	fmt.Printf("ContentLength = %d\n", unsafe.Offsetof(r.ContentLength))
}

go1.26.5
Method        = 0
URL           = 16
Header        = 56
ContentLength = 88

These four numbers match the offset table that OBI carries, pkg/internal/goexec/offsets.json.

"net/http.Request": {
  "Method":        { "versions": {"oldest": "1.17.0", "newest": "1.27.1"},
                     "offsets": [{"offset": 0,  "since": "1.17.0"}] },
  "URL":           { "versions": {"oldest": "1.17.0", "newest": "1.27.1"},
                     "offsets": [{"offset": 16, "since": "1.17.0"}] },
  "Header":        { "versions": {"oldest": "1.17.0", "newest": "1.27.1"},
                     "offsets": [{"offset": 56, "since": "1.17.0"}] },
  "ContentLength": { "versions": {"oldest": "1.17.0", "newest": "1.27.1"},
                     "offsets": [{"offset": 88, "since": "1.17.0"}] }
}

versions gives the range of versions across which OBI verified this entry. offsets gives the version from which the field sits at that position.

The value that the local Go printed matches the value in the table that OBI prepared in advance. To read a struct from outside is to trust this table and read byte +56.

net/http.Request is a lucky case, and each offsets array holds one element. These fields did not move once between Go 1.17 and 1.27.

The shift when one field is added

Add one field in the middle of a struct, then compare the two layouts.

• Run in the Go Playground

package main

import (
	"fmt"
	"unsafe"
)

type streamV1 struct {
	ctx    any
	id     uint32
	method string
}

type streamV2 struct { // one internal field added
	ctx    any
	cancel func()
	id     uint32
	method string
}

func main() {
	var v1 streamV1
	var v2 streamV2
	fmt.Printf("v1: id=%2d  method=%2d\n", unsafe.Offsetof(v1.id), unsafe.Offsetof(v1.method))
	fmt.Printf("v2: id=%2d  method=%2d\n", unsafe.Offsetof(v2.id), unsafe.Offsetof(v2.method))
}

v1: id=16  method=24
v2: id=24  method=32

Everything behind the added field moved together. The byte layout explains the numbers. An any takes 16 bytes: one pointer to the type, one to the data. A func() takes a single pointer, so 8 bytes, and a string takes 16 for a pointer and a length. A uint32 takes 4 bytes. The string behind it starts with a pointer, which must begin on an 8-byte boundary, so padding fills the gap.

Figure 1: The arrow represents the change of adding one field. The padding changes position and size too. The fields behind shift by the same 8 bytes that the new field added.

Code that hardcoded 24 as the position of method reads a meaningless position in v2, one that straddles id and the padding.

Offsets that change without notice

The streamV1 and streamV2 pair is not an invented example. OBI reads gRPC’s internal/transport.Stream to get the method name, and that struct shows the same shift. The table records four positions for method, so the field moved three times.

The type lives in the internal package internal/transport, so no compatibility promise covers it. In Go, nothing outside can import a package whose path contains internal, and the library never shows this type to its users. Its developers can therefore rearrange the fields in any update, and OBI pays for that freedom, because it reads memory positions from outside. In the same table, golang.org/x/net/http2.ClientConn.fr moved more: eight recorded ranges, seven changes of position, one of them back to a place the field held before. Across offsets.json, OBI tracks 89 structs and 154 fields, and 43 of them record at least one move. These counts are as of v0.13.0, and they exclude the entries for struct sizes and constants.

Figure 2: Arrows show which position the instrumentation code reads. A hardcoded +88 points at method in grpc 1.66, and at a different field in 1.69. The worst outcome is a read that produces a plausible value without crashing.

Hardcode an offset, and the code works perfectly against one version. Point it at an application that the user rebuilt on a newer dependency, and it reads the wrong place. The worst case is that it does not crash, and keeps working plausibly. The traces still arrive, with the wrong contents inside. For an observability tool, nothing damages trust in the observed system more.

Read DWARF first, fill the gaps from the table

OBI solves this by combining two sources. The entry point in pkg/internal/goexec/structmembers.go is where the two paths divide.

func structMemberOffsets(elfFile *elf.File) (FieldOffsets, error) {
	// first, try to read offsets from DWARF debug info
	var offs FieldOffsets
	var expected map[GoOffset]struct{}
	dwarfData, err := elfFile.DWARF()
	if err == nil {
		offs, expected = structMemberOffsetsFromDwarf(dwarfData)
		if len(expected) > 0 {
			log().Debug("Fields not found in the DWARF file", "fields", expected)
		} else {
			libVersions, err := findLibraryVersions(elfFile)
			if err != nil {
				return nil, fmt.Errorf("searching for library versions: %w", err)
			}
			offs = offsetsForLibVersions(offs, libVersions.versions, log())
			setGoAutoSDKActivationSupport(offs, libVersions, elfFile)
			return offs, nil
		}
	} else {
		// initialize empty offsets
		offs = FieldOffsets{}
	}

	log().Debug("Can't read all offsets from DWARF info. Checking in prefetched database")

	// if it is not possible, query from prefetched offsets
	return structMemberPreFetchedOffsets(elfFile, offs)
}

OBI tries DWARF first. DWARF is the debug information that holds types and field positions, and the compiler embeds it in the target binary itself. Chapter 4 listed it as the .debug_* sections.

DWARF holds the right answer because the compiler writes it down for the debugger: this field of this type sits this many bytes from the start. The same information lets a debugger show req.Method by name. So while OBI can read DWARF, it gets values that match the binary in front of it. Nothing here depends on the version.

A tool generates the offset table below automatically, so you may ask why OBI does not use the table alone. DWARF comes first because it holds the answer written into the binary in front of you. The table can carry only the versions that were known and built ahead of time. A freshly released version, a fork, or a private module is absent from it. So the order is to read the real answer first, then fill only the gaps from the table.

expected in the code is the set of fields that DWARF did not supply. If it is empty, DWARF was enough, and the function returns. If even one field remains, the function calls structMemberPreFetchedOffsets, and it passes the offsets that it already has. It never discards a value that it read from DWARF. The receiving function checks if _, found := fieldOffsets[constantName]; found { continue }, skips a field that already has a value, and fills only the missing ones from the table. The test TestPrefetchedOffsetsPreserveResolvedOffsets pins this behavior.

The table that fills the gaps is offsets.json, which //go:embed puts into the OBI binary.

//go:embed offsets.json
var prefetchedOffsets string

Nobody writes this table by hand. A tool called go-offsets-tracker extracts the offsets again at every release of Go and of the major libraries, and updates the table. Part of the work of developing OBI therefore goes into tracking the internal structures of Go and those libraries. This is the hidden cost of zero-code instrumentation.

The mapping between the C side and the Go side stays manual. This comment sits above the constant sequence in structmembers.go.

// this const table must match what's in go_offsets.h
type GoOffset uint32

const (
	// go common
	ConnFdPos GoOffset = iota + 1
	FdLaddrPos
	FdRaddrPos
	TCPAddrPortPtrPos
	TCPAddrIPPtrPos
	// http
	URLPtrPos
	PathPtrPos
	RawQueryPtrPos
	HostPtrPos
	SchemePtrPos
	MethodPtrPos
	StatusCodePtrPos
	ResponseLengthPtrPos
	ContentLengthPtrPos
	ReqHeaderPtrPos
	IoWriterBufPtrPos
	IoWriterNPos
	IoWriterWrPos
	// ...
)

If the iota order on the Go side and the order in the eBPF C header disagree, OBI reads an entirely different field. Nothing checks the correspondence. The C header carries its own note, // start at 1, must match what's in structmembers.go, and the maintainers hold the two sequences together through those comments alone. The compiler says nothing, and no test covers it.

The three constants at the tail of this excerpt, IoWriterBufPtrPos, IoWriterNPos, and IoWriterWrPos, correspond to buf, n, and wr of bufio.Writer. Offset tracking therefore covers unexported fields of the standard library. Hurdle 4 shows what they are for.

Working out the version from the binary

To use the table, OBI first has to know which version built this binary. It parses the build info blob that the Go linker leaves in the binary.

// The build info blob left by the linker is identified by
// a 16-byte header, consisting of buildInfoMagic (14 bytes),
// the binary's pointer size (1 byte),
// and whether the binary is big endian (1 byte).
var buildInfoMagic = []byte("\xff Go buildinf:")

From there OBI extracts runtime.buildVersion and runtime.modinfo. go version -m reads the same data: the Go version, and the path, version, and hash of every dependency module.

$ go version -m ./myapp
./myapp: go1.26.5
	path	example.com/myapp
	mod	example.com/myapp	(devel)
	dep	golang.org/x/arch	v0.30.0	h1:sB9h+1gRGa2+LauFSV0tm8bK1J2yo1bx6/Uyi/P6DTU=
	build	-buildmode=exe
	build	-compiler=gc
	...

Once OBI knows the versions of the dependency modules, it can pick the right row from the Stream.method table above. In some places OBI goes further and holds the module hashes themselves.

var goAutoSDKActivationModules = [...]activationModule{
	{
		path: "go.opentelemetry.io/auto/sdk",
		sums: map[string]string{
			"v1.1.0": "h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=",
			"v1.2.0": "h1:YpRtUFjvhSymycLS2T81lT6IGhcUP+LUPtv0iv1N8bM=",
			"v1.2.1": "h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=",
		},
	},
	// ...
}

These are the same hashes that go.sum records. OBI confirms the version string and the identity of the contents, and only then turns on the behavior specific to that version.

Figure 3: Arrows show the processing flow, top to bottom. OBI keeps every field that it read from DWARF, and fills only the unreadable ones from offsets.json. Looking the table up first requires working out the version from the build info, and the table returns the newest record at or below that version. The two join into a single offset table. Only the part that comes from the table depends on the version of the binary.

The answer to Hurdle 3 does not remove the fragility. It absorbs the fragility through a mechanism, release after release. OBI reads a field position from DWARF when the binary carries it, and fills the rest from a table that updates itself. That division of labor is how OBI keeps up with changes in Go and in its libraries.