Linux network namespaces and HTTP requests in Go March 2, 2022 on tomleb's blog

I recently needed to use net/http to make HTTP requests, but within a network namespace. Unfortunately, I couldn’t find examples for this and making it work involved a bit more than just calling netns.Set(namespace).

This blogpost goes into the details of the solution. For those that want to skip the details, you can jump ahead.

Code and linux network namespaces

There are some details to know about linux network namespaces to understand how to use them with code. The interface to linux namespaces is the syscall int setns(int fd, int nstype). From the man page (emphasis mine):

The setns() system call allows the calling thread to move into different namespaces.

The important bit here is that network namespaces are thread-local states. Any code running on an OS thread which was moved to a network namespace will be in that network namespace.

This is simple enough for most languages like C where it is easy to control what code runs in which threads. As you’ll see, this is more difficult in Go.

G’s, M’s and the Go runtime

In Go terminology, a G describes a goroutine and an M describes an OS thread¹. A G is scheduled to run on any M that is available and the same G might run on different M over time. This is all mostly non-deterministic, so we, as programmers, cannot know in advance on which M a G will run.

There are valid cases where we need more control over the scheduling of G’s on M’s. One such case is the one described above, where the network namespace has been changed on a specific OS thread².

Let G1 be a goroutine running on the OS thread M1. Also, let Gy be any other goroutine and Mx be any other OS thread. Then, the following must hold true to safely run G1 in a specific network namespace.

1. After having moved the network namespace of M1, G1 must run on M1 and only on M1. If G1 could be scheduled to another Mx, then we would have undefined behavior because part of G1 could run in Mx’s network namespace.

2. Any goroutine Gy must not run on the OS thread M1. If it did, then we would also have undefined behavior because any Gy could run in M1’s network namespace.

Since Go 1.10, it is possible to meet these requirements with the pair functions runtime.LockOSThread and runtime.UnlockOSThread³. Here are the rules for using those two functions.

The first function locks its G into its current M. It prevents G from running on another M and prevents another G from running on this M. The thread can be unlocked by the second function.

The second function unlocks its G from M. runtime.UnlockOSThread must be called the same number of times as runtime.LockOSThread to unlock the thread. If a G does not unlock its thread before it exits, then there are two scenarios:

1. If M was the main thread, it is parked and no new code can run on it.

2. Otherwise, M is killed.

Both cases prevent other G’s from running on an M that is still locked to avoid potential undefined behavior.

Now we know how to pin a G to an M. There is still one detail we need before we can run an HTTP request in a specific namespace.

http.Client caveats

Due to how net/http implements the HTTP client, we cannot simply run:

func main() {
	runtime.LockOSThread()
	netns.Setns(namespace)
	http.Get(url)
}

This is because the HTTP client creates a separate goroutine for each request. That’s right, the client’s Transport runs its DialContext function in a separate goroutine to establish the connection to the target host, as illustrated in the diagram below.

We must lock the thread and move to the network namespace from the goroutine that runs DialContext.

Putting it all together

The following code uses all that we’ve learned above to properly make HTTP requests from a specified network namespace.

Note that we are using the github.com/vishvananda/netns package, which provides us with utility functions to set and get network namespaces.

package main

import (
	"context"
	"fmt"
	"net"
	"net/http"
	"runtime"

	"github.com/vishvananda/netns"
)

func main() {
	namespace := "my-namespace"

	client := http.Client{
		Transport: &http.Transport{
			DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
				ns, err := netns.GetFromName(namespace)
				if err != nil {
					return nil, fmt.Errorf("get ns '%s': %w", namespace, err)
				}
				defer ns.Close()

				runtime.LockOSThread()
				if err := netns.Set(ns); err != nil {
					return nil, fmt.Errorf("setns '%s': %w", namespace, err)
				}

				return (&net.Dialer{}).DialContext(ctx, network, addr)
			},
		},
	}

	resp, err := client.Get("http://example.com")
	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(resp)
}

As we’ve discussed before, we’re specifying our own DialContext function which will be called in a separate goroutine. This function locks the OS thread (M) before changing the network namespace of the thread. This prevents other goroutines to run on this thread, which would cause undefined behavior due to the network namespace having changed.

Note that go doesn’t provide us a way to call runtime.UnlockOSThread in the dial goroutine once the request has completed. This means that the thread will either be killed by the runtime, or if it was the main thread, it will be forever idle.