Hurdle 4: Context Propagation Across Processes

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/55-hurdle4_propagation.

The first three hurdles stay inside one process. The one that remains crosses the process boundary, and it is context propagation. When it fails, a trace that should be a single piece tears into two unrelated traces at a service boundary.

As Chapter 1 showed, the work in service A and the work in service B become one trace when they carry the same trace ID. The usual way is to attach a traceparent header to the HTTP request. That header carries the trace ID and the sender’s span ID downstream. The downstream service then continues in a new span whose parent is that span ID.

The one line in SDK instrumentation

The instrumentation that an ordinary Go developer writes takes a few lines. Set the tracer and the propagation format up with the OpenTelemetry SDK, and from there you only swap out the HTTP client’s Transport. No propagation format is selected by default, so you have to call otel.SetTextMapPropagator(propagation.TraceContext{}) once.

client := &http.Client{
	Transport: otelhttp.NewTransport(http.DefaultTransport),
}

Inside that, one line does the work.

otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

It formats the trace ID and the span ID that ctx holds as a traceparent value, and writes them into req.Header. The application can write that line because it holds both ctx and req.

Zero-code instrumentation has to write the same line from outside. The application does not even know that traceparent exists, so someone else has to write it. The problem splits in two: how to reproduce what ctx holds from outside, and how to write into req.Header.

Figure 1: The arrows show who writes the header. With SDK instrumentation the application writes it itself. With zero-code instrumentation, OBI writes it from outside instead.

Tracking across goroutines

Even inside one process, the work flows across goroutines. The goroutine that receives a request and the goroutine that sends the downstream request are often different. Passing context.Context along as an argument is the Go way to handle that.

This looks like it contradicts Hurdle 2, which said that OBI can read an argument from a register. If OBI can read an argument, it should be able to read ctx as well. The two are different jobs.

Reading a register takes one value from a fixed place at the instant the uprobe fires. To make use of ctx, OBI would have to keep recognizing the same unit of work. The value passes from function to function, and across goroutines. ctx is an interface value, so the register holds a reference to an object elsewhere, not the contents. Nested calls to context.WithValue change that structure, and the argument position differs from function to function. One read of one pointer does not give you tracking.

Chapter 6 showed that the kernel’s knowledge stops at the thread. A thread ID does not tell you that two events belong to the same work either. Goroutines from different requests run on the same OS thread, and one request can hand its work to another goroutine.

OBI therefore gives up on decoding the value of ctx, and uses goroutine creation relationships as the clue instead.

Figure 2: The arrows show what OBI can and cannot do. The dashed line ending in a circle is the one it cannot. The dotted lines list the reasons; they are not a flow of execution.

OBI hooks goroutine creation itself. First, the names. The side that runs go f() and creates the goroutine is the parent, and the side it creates is the child. go f() ends up calling the runtime function runtime.newproc1, whose signature in go1.26 is as follows.

// Creates and returns the child goroutine. callergp is the parent's g
func newproc1(fn *funcval, callergp *g, callerpc uintptr, parked bool, waitreason waitReason) *g

The second argument is the parent, and the return value is the child. The return value does not exist until the function finishes. The entry alone cannot form the pair, so OBI places hooks on both the entry and the exit. Two lines describe the whole job.

Entry: note down the parent (the child doesn't exist yet)
Exit:  pair the noted parent with the returned child, and record the pair

In the code below, those two lines are all you need to follow.

SEC("uprobe/runtime_newproc1")
int GUARDED_PROG(obi_uprobe_runtime_newproc1, struct pt_regs *, ctx) {
    void *creator_goroutine_addr = GOROUTINE_PTR(ctx);

    new_func_invocation_t invocation = {.parent = (u64)GO_PARAM2(ctx)};
    go_addr_key_t g_key = {};
    go_addr_key_from_id(&g_key, creator_goroutine_addr);

    // Save the registers on invocation to be able to fetch the arguments at return of newproc1
    if (bpf_map_update_elem(&newproc1, &g_key, &invocation, BPF_ANY)) {
        bpf_dbg_printk("can't update map element");
    }

    return 0;
}

GUARDED_PROG wraps the function name, and the macro stops kernel preemption while the body runs. An eBPF program entered from a uprobe runs in task context, so the kernel can switch to another task in the middle of it. Since Linux 6.13, the kernel runs this class of program, the ones that use 64 bytes of stack or more, on a per-CPU dedicated stack. A switch then lets the next task overwrite the saved registers. The macro has nothing to do with the logic traced here.

At the entry, OBI notes the parent from the second argument (GO_PARAM2, that is, BX) in a temporary map called newproc1. The name creator in the code means the side that is running newproc1 right now. It is not the parent itself. It serves as the key that retrieves the note at the exit.

// The exit hook. Skeleton only
int GUARDED_PROG(obi_uprobe_runtime_newproc1_return, struct pt_regs *, ctx) {
    void *creator_goroutine_addr = GOROUTINE_PTR(ctx);      // key to match up with the entry
    void *goroutine_addr = (void *)GO_PARAM1(ctx);          // return value: the address of the child's g

    // Retrieve the parent noted down at the entry
    new_func_invocation_t *invocation = bpf_map_lookup_elem(&newproc1, &c_key);
    void *parent_goroutine = (void *)invocation->parent;

    // Record "child -> parent" in the ongoing_goroutines map
    goroutine_metadata metadata = {.timestamp = bpf_ktime_get_ns(), .parent = p_key};
    bpf_map_update_elem(&ongoing_goroutines, &g_key, &metadata, BPF_ANY);
    return 0;
}

At the exit, the return value (GO_PARAM1, that is, AX) holds the address of the child’s g. OBI pairs it with the parent noted at the entry, and records the mapping from child to parent in the ongoing_goroutines map. From then on it can follow the statement that this goroutine is a child of the goroutine handling that request.

Figure 3: The arrows show the order in time. At the entry only the parent is known, so OBI notes it down temporarily. At the exit, once the child’s address is known, it records the pair as “child → parent”.

SEC("uprobe/runtime_newproc1_return")
int GUARDED_PROG(obi_uprobe_runtime_newproc1_return, struct pt_regs *, ctx) {
    bpf_dbg_printk("=== uprobe/runtime_newproc1_return ===");
    void *creator_goroutine_addr = GOROUTINE_PTR(ctx);
    const u64 pid_tid = bpf_get_current_pid_tgid();
    const u32 pid = pid_from_pid_tgid(pid_tid);
    go_addr_key_t c_key = {.addr = (u64)creator_goroutine_addr, .pid = pid};

    // The result of newproc1 is the new goroutine
    void *goroutine_addr = (void *)GO_PARAM1(ctx);
    go_addr_key_t g_key = {.addr = (u64)goroutine_addr, .pid = pid};

    // Lookup the newproc1 invocation metadata
    new_func_invocation_t *invocation = bpf_map_lookup_elem(&newproc1, &c_key);
    if (invocation == NULL) {
        bpf_dbg_printk("can't read newproc1 invocation metadata");
        goto done;
    }

    // The parent goroutine is the second argument of newproc1
    void *parent_goroutine = (void *)invocation->parent;
    go_addr_key_t p_key = {.addr = (u64)parent_goroutine, .pid = pid};

    goroutine_metadata *g_metadata =
        (goroutine_metadata *)bpf_map_lookup_elem(&ongoing_goroutines, &p_key);

    if (g_metadata) {
        // Don't create cycles at one level on immediate goroutine reuse
        if (g_metadata->parent.addr == (u64)goroutine_addr) {
            bpf_dbg_printk("avoiding cycle %llx -> %llx", parent_goroutine, goroutine_addr);
            goto done;
        }
    }

    goroutine_metadata metadata = {
        .timestamp = bpf_ktime_get_ns(),
        .parent = p_key,
    };

    if (bpf_map_update_elem(&ongoing_goroutines, &g_key, &metadata, BPF_ANY)) {
        bpf_dbg_printk("can't update active goroutine");
    }

done:
    // Delete any stale info on go_trace_map
    bpf_map_delete_elem(&go_trace_map, &g_key);
    bpf_map_delete_elem(&newproc1, &c_key);

    return 0;
}

The skeleton above left out one more branch that the implementation carries.

    // Don't create cycles at one level on immediate goroutine reuse
    if (g_metadata->parent.addr == (u64)goroutine_addr) {
        bpf_dbg_printk("avoiding cycle %llx -> %llx", parent_goroutine, goroutine_addr);
        goto done;
    }

This is the price of the decision in Hurdle 2 to use the address of the g struct as the goroutine identifier. When a goroutine exits and the runtime reuses its g, the same address comes back as a different goroutine. An address recorded as a parent can appear again as a child. Leave that alone, and the parent-child relationships form a cycle that makes the ancestor walk in the next section loop forever.

For the same reason, the function deletes stale entries at the end.

done:
    // Delete any stale info on go_trace_map
    bpf_map_delete_elem(&go_trace_map, &g_key);
    bpf_map_delete_elem(&newproc1, &c_key);

That keeps a reused address from carrying the information of its previous owner.

An address instead of a serial number frees OBI from version tracking, and it makes lifetime management OBI’s own job. Either choice makes you carry something.

The _return suffix in the function name does not mean a uretprobe. As Hurdle 1 showed, OBI disassembles runtime.newproc1, finds every RET, and places an ordinary uprobe on each one. That roundabout procedure is what it takes to do the obvious thing of using an entry and an exit as a pair. The workaround from Hurdle 1 is what makes it possible.

Six lookups and no more

The hook that sends the request downstream is what uses the recorded parent-child relationships. When net/http.(*Transport).roundTrip or a gRPC client entry point fires, OBI walks from the running goroutine toward its ancestors, one parent at a time. It is looking for the server-side receive where this work began. Once it finds that receive, it can carry the trace ID over.

The mark that the walk looks for is an entry in go_trace_map. OBI hooks the server-side receive too: net/http.serverHandler.ServeHTTP for HTTP, and google.golang.org/grpc.(*Server).handleStream for gRPC. At the entry of either, it writes the trace information of the incoming request into this map, keyed by the goroutine that handles it. The receive side only writes the entry, and the send side does the walking.

The job is a repeated lookup, one level at a time, asking whether this goroutine has a trace. The first goroutine that OBI checks is the current one, so a receive and a send on the same goroutine hit on the first try. If the goroutine has no trace, OBI looks up its parent in the parent-child map and repeats.

static __always_inline u64 find_parent_goroutine(go_addr_key_t *current) {
    // ...
    int attempts = 0;
    do {
        tp_info_t *p_inv = bpf_map_lookup_elem(&go_trace_map, parent);
        if (!p_inv) { // not this goroutine running the server request processing
            // Let's find the parent scope
            goroutine_metadata *g_metadata =
                (goroutine_metadata *)bpf_map_lookup_elem(&ongoing_goroutines, parent);
            if (g_metadata) {
                // Lookup now to see if the parent was a request
                // Debug here commented out on purpose to avoid prints in loops.
                // bpf_printk("lookup %llx -> %llx", r_addr, g_metadata->parent.addr);
                r_addr = g_metadata->parent.addr;
                parent = &g_metadata->parent;
            } else {
                break;
            }
        } else {
            bpf_dbg_printk("Found parent, r_addr=%lx", r_addr);
            return r_addr;
        }

        attempts++;
        // We loop far back because some clients, e.g. Kafka Franz-Go really nest the
        // client calls.
    } while (attempts < 6); // Up to 6 levels of goroutine nesting allowed

    return 0;
}

The bpf_printk inside the loop is commented out, and the reason sits beside it: Debug here commented out on purpose to avoid prints in loops. One debug line weighs enough to matter here.

The fixed upper bound is not a shortcut in the implementation. As Chapter 7 showed, the verifier refuses to load a program whose loop has no upper bound. OBI cannot write “walk until you find the parent”. It has to write “walk at most this many times”.

The number 6 itself does not come from safety. The verifier only needs to be able to confirm that the loop ends, so 6 is an implementation choice that weighs instruction count against the depth that real libraries need. The comment records the grounds: the Kafka client franz-go nests its calls deeply. A larger bound might stop passing, but what decides that is the number of instructions after expansion and the volume of states the verifier walks, not the presence of a bound.

The walk can also come up short. Six lookups cover the sending goroutine itself and five generations of parents, so an ancestor with trace information any further away leaves find_parent_goroutine returning 0. The send side is still instrumented, and client_trace_parent builds a new trace ID from random numbers. The downstream request goes into the record as a separate trace, unconnected to the upstream one. That is harder to deal with than a missing trace, because one flow appears as two.

Figure 4: In the upper half the arrows show the flow of the records; in the lower half they show the direction in which OBI follows a child-to-parent reference. The walk starts at the sending goroutine itself, so six lookups cover that goroutine and five generations of parents. The verifier requires an upper bound, but the value 6 is an implementation choice. A send that the walk cuts short is still instrumented; it gets a fresh trace ID and becomes a separate trace.

Injecting the header into the outgoing request

OBI now has the trace context, and it has to write that into the HTTP request that the application is about to send.

The obvious destination is the Header field of http.Request. OBI does not aim there. http.Header is a map[string][]string, so adding a new key from outside means operating on the internal structure of a map inside the target process. OBI would have to compute the hash of the key, find a bucket, and grow the storage when needed, all from outside. That does not work.

OBI aims at the moment just after the headers become a byte sequence and before the blank line that ends them goes out. The write target is the bufio.Writer send buffer, so the write appends to a byte sequence rather than to a map. You can see what an HTTP/1.1 request looks like at that stage with the standard library alone.

• Run it on the Go Playground

package main

import (
	"fmt"
	"net/http"
	"net/http/httputil"
)

func main() {
	req, _ := http.NewRequest("GET", "http://service-b/items", nil)
	req.Header.Set("Traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01")
	dump, _ := httputil.DumpRequestOut(req, false)
	fmt.Printf("%q\n", dump)
}

"GET /items HTTP/1.1\r\nHost: service-b\r\nUser-Agent: Go-http-client/1.1\r\nTraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01\r\nAccept-Encoding: gzip\r\n\r\n"

An HTTP/1.1 request is a single string. One header goes on each line, \r\n separates the lines, and a blank line ends the headers. What SDK instrumentation does in the end is add one Traceparent: line. Zero-code instrumentation adds traceparent the same way, by inserting one line into that string.

When net/http writes the headers out, it passes through Header.writeSubset, which piles a string like the one above into the bufio.Writer buffer behind it. A bufio.Writer is a container that holds writes: buf stores the byte sequence, and n records how many of those bytes are in use. OBI hooks both the entry and the return of this function, and on the return it appends one line to the piled-up string. The probe registration lives on the Go side, in pkg/internal/ebpf/gotracer/gotracer.go.

Figure 5: The vertical arrows show the write-out path, and the arrows from OBI show its write targets. The dashed arrow with a blocked tip points at the target that OBI cannot write. Nothing outside can write into the http.Header map, so OBI writes Traceparent into the bufio.Writer buffer just before serialization and advances n.

	if p.headerPropagationEnabled() {
		m["net/http.Header.writeSubset"] = []*ebpfcommon.ProbeDesc{{
			Start: p.bpfObjects.ObiUprobeWriteSubset,        // http 1.x context propagation
			End:   p.bpfObjects.ObiUprobeWriteSubsetReturns, // inject only if no traceparent present
		}}
		m["golang.org/x/net/http2.(*Framer).WriteHeaders"] = []*ebpfcommon.ProbeDesc{
			{ // http2 context propagation
				Start: p.bpfObjects.ObiUprobeGolangHttp2FramerWriteHeaders,
				End:   p.bpfObjects.ObiUprobeHttp2FramerWriteHeadersReturns,
			},

The key names the symbol to instrument. Start is the entry eBPF program, and End is the exit one. When End is present, the machinery from Hurdle 1 runs, finds every RET, and places a uprobe on each. The comment inject only if no traceparent present says the rest: when the application already attached its own traceparent through SDK instrumentation, OBI does not write. Running alongside SDK instrumentation never produces a duplicate header.

The return hook appends directly to the end of the bufio.Writer buffer.

    unsigned char buf[k_traceparent_len];
    make_tp_string(buf, &inv->tp);

    if (len <
        (size - TP_MAX_VAL_LENGTH - TP_MAX_KEY_LENGTH - 4)) { // 4 = strlen(":_")+strlen("\r\n")
        char key[TP_MAX_KEY_LENGTH + 2] = "Traceparent: ";
        char end[2] = "\r\n";
        bpf_probe_write_user(buf_ptr + (len & 0x0ffff), key, sizeof(key));
        len += TP_MAX_KEY_LENGTH + 2;
        bpf_probe_write_user(buf_ptr + (len & 0x0ffff), buf, sizeof(buf));
        len += TP_MAX_VAL_LENGTH;
        bpf_probe_write_user(buf_ptr + (len & 0x0ffff), end, sizeof(end));
        len += 2;
        bpf_probe_write_user((void *)(io_writer_addr + io_writer_n_pos), &len, sizeof(len));

bpf_probe_write_user is the eBPF helper that rewrites user-space memory in the target process. Three calls pile Traceparent: , the value, and \r\n into the buffer, and the fourth rewrites n in the bufio.Writer. Without that increase in n, the appended bytes stay outside the used range of the buffer and never go out. The fourth write is what makes the Go code accept that a line was added. The expression (len & 0x0ffff), which appears three times, shows the verifier that the repeatedly incremented len stays inside a fixed range. The verifier’s constraints reach every one of those index expressions, not only the upper bound of the walk loop.

offsets.json from Hurdle 3 carries buf, n, and wr of bufio.Writer for this reason. OBI rewrites unexported fields of the Go standard library from outside. The variable name io_writer_n_pos points at the offset of n.

Figure 6: The arrows show the direction of the writes and the direction in which the bytes leave. Path 1 writes into a buffer in the application’s memory, and path 2 injects into the byte sequence on its way out to the socket. Both add the same single line.

Environments that forbid the write, and the second path

bpf_probe_write_user collides with the security mechanisms of the OS. OBI’s support matrix says the following.

On Linux 5.10 and later, OBI requires effective CAP_SYS_ADMIN and kernel lockdown mode [none] to use bpf_probe_write_user.

In an environment with kernel lockdown enabled, or under Secure Boot, this helper is unavailable. The code carries a flag named g_bpf_probe_write_user_enabled, and where the helper is unavailable it skips the whole sequence above.

Context propagation does not disappear there, because OBI has a second path. A comment just below the same function explains it.

        // For Go we support two types of HTTP context propagation for now.
        //   1. The one that this code does, which uses the locked down bpf_probe_write_user.
        //   2. By using a sock_msg program that will extend the packet.
        // If this code ran, we should ensure that the second part doesn't run, therefore
        // we remove the metadata setup in uprobe_persistConnRoundTrip(struct pt_regs *ctx), so
        // that approach 2. skips this packet.

The second one is an sk_msg program. It is a kind of eBPF program that runs where the kernel pushes data toward the socket, not in the application’s memory. It can extend the outgoing byte sequence to inject a header, and it never touches the application’s memory.

The unit here is the sequence of bytes that TCP carries. The kernel decides where to cut that into packets, so one HTTP request is not always one packet. What sk_msg injects into is the byte sequence before those cuts are decided.

Aligning on sk_msg alone, which collides with no security mechanism, looks like the better plan. The largest reason OBI does not is TLS. What arrives at the sk_msg point on HTTPS is the byte sequence after encryption, and no header goes into that. Path 1 writes into the bufio.Writer buffer, which holds the plaintext before encryption. Over HTTPS, path 1 is the only one that can deliver a standard Traceparent header. Under TLS, path 2 switches to an alternative that carries the information in a TCP option instead of a header, and only where the connection multiplexes one exchange at a time, as HTTP/1.1 does. That format is OBI’s own, so the receiving side has to be OBI as well. An L7 proxy or a load balancer in between drops it, because the proxy discards the original packet and builds a new one. HTTP/2 and gRPC cannot use the alternative at all: several streams run in parallel over one connection, and a per-connection TCP option cannot say which stream a context belongs to. OBI’s design documents¹ also position the sk_msg side as a fallback for when path 1 could not write.

This path injects into a byte sequence that is already on its way out, and a miss here breaks the observed system too. v0.13.0 fixed three memory-safety defects in bpf/tpinjector/. In the first, the injection ran against data that a previous request left in the buffer. It wrote traceparent into the middle of a TLS stream and reset the connection (#3257). In the second, a read of the message buffer ran past the mapped range. On a failed fetch it carried unrelated kernel memory out as the contents of a span (#3298). In the third, the code did not invalidate a buffer that failed to fill, so it mistook a previous message for the current one (#3304). A path that never touches the application’s memory still has to be exact, because it interposes in the middle of a transmission.

When the first path runs, OBI deletes the map registration so that the second path skips that send. Two headers with the same name reach the receiver joined into the single value traceparent: A,B under the rules of HTTP, and that no longer matches the format W3C Trace Context defines. The specification tells a receiver that cannot parse the format to start a new trace, so a double injection destroys rather than duplicates. The two paths must never both run against the same send, so OBI has to arbitrate between them.

Context propagation is off by default. OTEL_EBPF_BPF_CONTEXT_PROPAGATION defaults to disabled, and you pick headers, tcp, or all explicitly. The feature rewrites the memory of a process, or the byte sequence being sent, so OBI leaves the decision to enable it to you.

In Hurdle 4, OBI hooks goroutine creation and records the parent-child relationships. Just before a send, it walks back through up to 6 lookups, counting itself, to find the trace ID. It then writes that ID into the send buffer that holds the serialized headers, or into the byte sequence on its way out to the socket.

Figure 7: Solid arrows show the flow of processing, and dashed lines show what gets disabled. The figure covers goroutine parent-child tracking inside a process, and traceparent injection between processes. The tracking gets 6 lookups counting itself, the injection has two paths, and only path 1 drops out when the kernel’s security mechanisms are active.