Upstream Go and Closing Thoughts

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/65-conclusion.

The four hurdles in this book were all difficulties of looking into Go’s internals from outside.

Direct support for eBPF instrumentation

Upstream Go carries almost nothing that helps eBPF instrumentation directly. The uretprobe problem from Hurdle 1 (#22008) has sat at “Unplanned” since 2017. A proposal to allow a hook on goroutine creation (#73798) was closed as “not planned” in 2025.

The Go team holds a consistent position: the runtime’s internal structures are not a public API, and outside code should not depend on them. eBPF instrumentation depends on exactly those private internals, so support is hard to come by. The cost of tracking offsets, covered in Hurdle 3, is the other side of that same position. The tracking reaches the runtime itself as well. offsets.json lists runtime structs such as runtime.hchan and runtime.moduledata, and they account for close to half of the recorded moves. The net/http and gRPC fields in this book came from the standard library and from third parties. The policy of keeping internal structures out of a stable API holds for the runtime and for the libraries alike.

The decision not to expose goid follows the same reasoning. In Hurdle 2, OBI chose the address of g as its identifier because avoiding an unexposed value breaks less often than forcing a read of it. The external tool designs around the value instead of depending on it, and the two sides meet there.

How observability “from the inside” has grown

Go is not indifferent to observability. Observation from the inside has grown steadily.

• Flight recording (#63185, implemented in Go 1.25 as runtime/trace.FlightRecorder): keeps the most recent execution trace in a ring buffer, so you can pull out what led up to a problem at the moment it happens. The idea matches the flight recorder in an aircraft.

Figure 1: The arrows represent the flow of recording and retrieval. The runtime records continuously while keeping only the most recent window in a ring buffer, and it overwrites older records. On a signal at the moment of a problem, it writes out only the records from just before that point.

• goroutine leak profile: a profile of goroutines that have become unreachable and stay blocked, added to runtime/pprof in Go 1.26 under the name goroutineleak. It is an experimental feature that you enable by building with GOEXPERIMENT=goroutineleakprofile. Looking in from outside with eBPF cannot tell a leaked goroutine from one that is merely waiting. Deciding reachability needs the whole heap and every stack, which only the inside of the runtime holds. This is an answer that exists because the observer sits inside.
• Compile-Time Instrumentation SIG (formed in January 2025): an approach separate from eBPF that embeds instrumentation code at compile time. The tool is called otelc, and it needs no change to the source code. The next section covers how it works and what actually reaches the compiler.

The Compile-Time Instrumentation SIG

The Compile-Time Instrumentation SIG formed in January 2025 and reached its first stable release, v1.0.0, on July 14, 2026. That v1.0.0 was retracted soon after publication. A bug in the otelc pin command wrote incorrect module paths into the user’s go.mod, and a fixed v1.0.1 went out the same day. The line is still in the repository’s go.mod.

retract v1.0.0 // otelc pin generates incorrect module paths in user go.mod files; use v1.0.1

The latest release is v1.1.0, published on August 24, 2026 (release list).

Using it is simple.

$ otelc go build -o myapp .

You put otelc in front of go build and change no line of source code. The phrase “it uses -toolexec” is not a metaphor. Internally, otelc go build assembles and runs a real go build like this, in the buildWithToolexec function of tool/internal/setup/setup.go (source on GitHub).

$ go build -work -toolexec="<path to the otelc executable> toolexec" -o myapp .

-toolexec is the mechanism that the Go toolchain provides for placing your own program in front of the compiler. otelc uses it this way: when the package under compilation is a supported library such as net/http, gRPC, or database/sql, otelc inserts instrumentation code into the bodies of the target functions inside it, and only then hands the result to the compiler. What gets rewritten is the library side, not the call site.

Figure 2: The arrows represent the order of the build steps. An ordinary build turns source code straight into a binary. A build through otelc puts otelc itself in front of go build with -toolexec, which inserts instrumentation code into the bodies of the supported libraries’ functions before compilation.

What actually gets rewritten

The source code does not change, but what reaches the compiler does. Run otelc go build and you can see it.

The application code comes first. Not one character of it changes across the build.

resp, err := http.Get("http://localhost:8080/greet?name=world")

http.Get eventually calls RoundTrip inside net/http. Before the build, that function is the Go standard library source as it ships.

// net/http/roundtrip.go (before the otelc build; the Go standard library as it ships)
func (t *Transport) RoundTrip(req *Request) (*Response, error) {
	if t == nil {
		panic("transport is nil")
	}
	return t.roundTrip(req)
}

otelc go build passes -work internally, as shown above, so the rewritten source stays in Go’s build cache after the build finishes. Looking in there gives the following.

// net/http/roundtrip.go (what otelc actually rewrote it into; confirmed on a real machine)
func (t *Transport) RoundTrip(req *Request) (_r0 *Response, _r1 error) {
	if hookContext3038199408, _ := OtelBeforeTrampoline_RoundTrip3038199408(&t, &req); false {
	} else {
		defer OtelAfterTrampoline_RoundTrip3038199408(hookContext3038199408, &_r0, &_r1)
	}
	if t == nil {
		panic("transport is nil")
	}
	return t.roundTrip(req)
}

// Trampoline Template
func OtelBeforeTrampoline_RoundTrip3038199408(recv0 **Transport, param0 **Request) (hookContext *HookContextImpl3038199408, skipCall bool) {
	defer func() {
		if err := recover(); err != nil {
			println("failed to exec Before hook", "BeforeRoundTrip")
		}
	}()
	hookContext = &HookContextImpl3038199408{}
	hookContext.params = []interface{}{recv0, param0}
	hookContext.funcName = "RoundTrip"
	hookContext.packageName = "http"
	if BeforeRoundTrip != nil {
		BeforeRoundTrip(hookContext, *recv0, *param0)
	}
	return hookContext, hookContext.skipCall
}

func OtelAfterTrampoline_RoundTrip3038199408(hookContext HookContext, arg0 **Response, arg1 *error) {
	defer func() {
		if err := recover(); err != nil {
			println("failed to exec After hook", "AfterRoundTrip")
		}
	}()
	hookContext.(*HookContextImpl3038199408).returnVals = []interface{}{arg0, arg1}
	if AfterRoundTrip != nil {
		AfterRoundTrip(hookContext, *arg0, *arg1)
	}
}

//go:linkname BeforeRoundTrip go.opentelemetry.io/otelc/instrumentation/net/http/client.BeforeRoundTrip
func BeforeRoundTrip(hookContext HookContext, recv0 *Transport, param0 *Request)

//go:linkname AfterRoundTrip go.opentelemetry.io/otelc/instrumentation/net/http/client.AfterRoundTrip
func AfterRoundTrip(hookContext HookContext, arg0 *Response, arg1 error)

(I left out the mechanical getters and setters on HookContextImpl3038199408, such as GetParam and SetParam, which only move values in and out. The number 3038199408 at the end of each function name is a unique hash that code generation assigns so that several hooks do not collide.)

The generated code does three things. It calls a before hook at the top of the function, and it arranges an after hook with defer. It also swallows any failure of the hooks themselves with recover, so that the original work continues. The body of RoundTrip, meaning the t == nil check and the return statement, stays exactly as it was.

//go:linkname declares only the names of BeforeRoundTrip and AfterRoundTrip. Their bodies live in another package, and that package is where the OpenTelemetry spans get created.

// instrumentation/net/http/client/client_hook.go (the real hook implementation, excerpted)
func BeforeRoundTrip(ictx hook.HookContext, transport *http.Transport, req *http.Request) {
	ctx := req.Context()
	attrs := semconv.HTTPClientRequestTraceAttrs(req)

	ctx, span := tracer.Start(ctx,
		req.Method,
		trace.WithSpanKind(trace.SpanKindClient),
		trace.WithAttributes(attrs...),
	)

	// Inject the trace context into the request headers
	propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))

	newReq := req.WithContext(ctx)
	ictx.SetParam(requestParamIndex, newReq)

	ictx.SetData(map[string]interface{}{
		"ctx": ctx, "span": span, "req": req, "start": time.Now(),
	})
}

func AfterRoundTrip(ictx hook.HookContext, res *http.Response, err error) {
	span, ok := ictx.GetKeyData("span").(trace.Span)
	if !ok || span == nil {
		return
	}
	defer span.End()

	if res != nil {
		attrs := semconv.HTTPClientResponseTraceAttrs(res)
		span.SetAttributes(attrs...)
		if code, desc := semconv.HTTPClientStatus(res.StatusCode); code != codes.Unset {
			span.SetStatus(code, desc)
		}
	}
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
	}
}

(I left out the defensive code that sits away from the main line. One example is the User-Agent filter that stops the OTel exporter’s own requests from looping forever.)

That closes the circle. BeforeRoundTrip starts a span, injects traceparent into the headers, and swaps the request out with ictx.SetParam. The swap works because GetParam and SetParam on hookContext point at the arguments of RoundTrip themselves, as **Request. AfterRoundTrip then fills the span from the response and closes it. Hurdle 2 turned on arguments living in registers rather than on the stack. Here the compiler itself writes the instrumentation code, so the question of register or stack never arises. Hurdles 1 through 4 all grew out of the constraint of watching from outside with eBPF. Compile-time instrumentation avoids those hurdles because of where it sits.

Confirming it on your own machine

You can reproduce everything above with the following steps. I confirmed them on go1.26.5 linux/amd64.

$ git clone https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation.git
$ cd opentelemetry-go-compile-instrumentation
$ make build
$ cd demo/app/http/client   # any code that uses net/http will do
$ ../../../../otelc go build -o client .
# Note the WORK=/tmp/go-buildXXXXXXXXX line in the output,
# then look under it for the build directory of the net/http package.
# The rewritten roundtrip.go is still there.

eBPF observes from outside, and Go keeps adding ways to observe from inside. Observation from outside leaves the application untouched, and in exchange it must keep tracking the runtime’s internal structures. Observation from inside is accurate and hard to break, and in exchange it needs a change to the code or to the build.

Figure 3: The arrows point from the observer to the observed. The labels show the benefit and the cost of each approach.

Closing thoughts

The four hurdles line up against the Go traits behind them and the answers OBI chose.

These four problems did not arise separately. Each is the other side of what makes Go what it is: a stack that moves, a custom register ABI, private internal structures, and goroutines. The design that makes Go fast and easy to write is the same design that blocks anyone observing it from outside.

This knowledge belongs to more than the people who write eBPF tools. If you write Go applications, these hurdles let you explain two things for yourself. Why zero-code instrumentation traces your application on one day and not on another, and which Go version and build shape keep instrumentation stable. Whether you strip DWARF, which changes the path that offset resolution takes, was one example of that.

Return to the “works regardless of language” claim from Chapter 1. For the generic path that interprets protocols, the claim holds. The path that reaches into Go functions runs on something else. It scans every RET instruction, reads registers directly, combines DWARF with an offset table in two stages, and writes into the buffer of a bufio.Writer. The experience of observing without changing code holds up because an implementation keeps answering each of these hurdles.

Where the work divides between observation from outside and observation from inside is still moving.