Chameth.com - posts like ipv6-docker-routing, understanding-docker-volume-mounts but not modern-css-is-fun, why-you-should-be-using-https

The Case of the Unchanging Config

2026-04-09T00:00:00Z

Last week I was attempting to make it so I could share pictures on IRC directly from my client. This sounds simple, but it involves a bouncer¹ that proxies the request to a standalone image hosting service that I had to modify to be compatible. At one point my testing loop was:

Make a change to the hosting service
Commit it
Tag a new release
Wait for it to build
Update the version I’m running on my server
Reconfigure the bouncer to pass the new parameter or change the URL or whatever
Try uploading a photo from my phone
Realise I’ve overlooked something and go back to step one

I could have set up a local copy of everything and tested it sensibly. I should have set up a local copy of everything and tested it sensibly. But it seemed like such a trivial change, and setting up the whole environment seemed like such a pain. After the third or so iteration of failure I was pretty annoyed with myself, computers, and basically everything.

My biggest annoyance was that my bouncer would not pick up the new URL from the config when I changed it. It’s meant to reload the config when it receives a SIGHUP, and it claimed to in the logs, but I could clearly see it was still hitting the old URL. Restarting the bouncer to update the config is a pain, as it disconnects me from all the IRC networks, and has to reconnect to them all, reauthenticate, etc. It also mildly spams everyone who shares a channel with me².

When I finally got everything working I had a look at the bouncer source, and thought I’d spotted the issue. I raised a bug report, ending in this remark:

It looks like the config is reloaded properly, but the handler for uploads is created once at startup and has its own copy of the uploader, so effectively snapshots the config to whatever it is at startup:

fileUploadHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    cfg := srv.Config()
    h := fileupload.Handler{
        Uploader:    cfg.FileUploader,
        DB:          db,
        Auth:        cfg.Auth,
        HTTPOrigins: cfg.HTTPOrigins,
    }
    h.ServeHTTP(w, r)
})

I’m not quite sure what I thought I saw there, but that’s not right. I want to call it a hallucination, but that feels like a very overloaded word these days. I guess I was just seeing what I wanted to see, and that was a good excuse to stop investigating. Still, even if I misidentified the cause, the bug was still present, right? … Right?

The Science of Deduction

When the developer got back and said he couldn’t reproduce my issue, I went back to my install and immediately reproduced it. With a calmer head, I figured it was probably something with my particular set up. First thing to check: am I actually running the version I think I am?

Unfortunately the bouncer doesn’t actually seem to expose the version anywhere that I can see. It’s not in the logs, it doesn’t have a -version flag, and none of the IRC-based status commands seem to include it. But I know it’s a Go app, and I know Go embeds the version information. go version -m <binary> will dump it all out, but the binary is inside a Docker image, and the Docker image is one of my nice, minimal, artisanal ones so doesn’t ship a go binary. No problem, docker compose cp bouncer:/bnc ./bnc yoinks the binary out of the container, and then dumping the version shows that, yes, I am running the version I thought I was. Hmm.

The next tool I reached for in my toolbox was strace. Maybe it’s not actually reading the file for some reason? I immediately executed strace -p <pid> -e trace=openat,open,read,pread64 -f without having to look any part of that up. Yep. Definitely. Then I edited the config, HUP’d the bouncer, and saw that it was… reading the config file. As it’s meant to. By default strace truncates strings to 32 bytes, so I couldn’t actually see the line I’d changed. Some more definitely-not-RTFMing later, and rerunning it with an extra -s 65536 let me see the full config. Surprise! The config hadn’t changed!

To confirm my findings, I used docker compose cp again, this time yoinking the config file from inside the container. The inside config file was definitely different to the outside config file. What? My hypothesis at this point was “something something Docker nonsense”. I mount the config as read-only, and was wondering if that meant that Docker was doing something weird instead of just bind mounting it. A quick trip to /proc/<pid>/mounts showed that it was, in fact, not doing anything weird, and was just bind mounting it.

If the file is bind mounted, then surely it’s the same file? I ran stat on the file on the host, noted the inode number, then pondered how to actually run stat on the file inside the container, given the aforementioned awkwardly minimal image. The solution was easy: access it via /proc/<pid>/root/. I could’ve saved myself a bunch of docker compose cp if I’d thought about that earlier. Oh well. The inode of that file was different. What?

The answer was DNS. It’s always DNS. Oh, sorry, force of habit. I meant the answer was vim. I was editing the config in vim, and when it saves files, by default, it writes the new content to a temporary file and does an atomic rename. That’s normally a good thing: it prevents corruption if the write fails midway through³. In this case, though, that means the new file has a new inode. Bind mounting a file binds to the inode, so the container just keeps clutching onto the original config from when it was started, blissfully unaware that the party has relocated down the street.

This problem is likely to happen whenever you bind mount a file into a container. When I mentioned this to a friend, he immediately responded “oh yeah, never do that”, and went on to describe the horrible hacks he’s had to add to Ansible to sidestep the issue. The nicer solution is to just bind mount an entire directory if you can, as then it doesn’t matter what happens to the files within it. I really like having the config files sat alongside the Docker compose files, though; having to create a directory just to work around some bind mount weirdness upsets me.

Now I knew what the problem was, I found there was an issue raised against Docker twelve years ago⁴. The first response was “that’s expected”, and I fully agree with the author: “respectfully, that might be expected by you, but it was not expected by me”!⁵ As a result of the issue they added a nice note to the docs:

Note: Many tools used to edit files including vi and sed --in-place may result in an inode change. Since Docker v1.1.0, this will produce an error such as “sed: cannot rename ./sedKdJ9Dy: Device or resource busy”. In the case where you want to edit the mounted file, it is often easiest to instead mount the parent directory.

But in the intervening twelve years, both the note and the functionality described have gone missing. Ho-hum.

I found it interesting how I’ve only just hit this problem, given how long I’ve used Docker. But I realised that almost everything I run I’m happy to just restart. Cattle, not pets, and so on. My IRC bouncer is one of the few exceptions to that. The only other thing I regularly hot reloaded was Centauri, my reverse proxy, but that had a whole config directory mounted because it was shared between containers, so nicely sidestepped the foot-gun.

So lesson learnt: check for weird bind mount issues before raising issues about config hot reloading. It’ll join the esteemed company of “maybe the drive is full and causing completely unrelated problems?”, “perhaps everything is dog slow because the kernel ran out of entropy?”, and “did systemd sneakily take over that functionality while you weren’t looking?” in the troubleshooting checklist.

An IRC bouncer is basically an always-on proxy. It connects to the IRC networks for you, then your clients connect to your bouncer. The bouncer can then send incoming messages to all your different clients, cache them when you’re offline and replay them later, and lots of other nice things people take for granted in their chat apps these days. ↩︎
Join/part/quit spam is part of IRC, and clients have ways of handling it, but I still feel bad about doing it excessively. ↩︎
You can disable this in vim by setting backupcopy=yes. Good luck redoing all the debugging if you ever accidentally remove that from your vimrc, though! ↩︎
Which is weird, because Docker can’t possibly be that old. That would make me much older than I’m prepared to accept. ↩︎
If you read the thread they didn’t actually mean to sound so dismissive, but it’s still pretty funny. “Yes, it’s expected that the foot-gun causes your foot to hurt. Duh. What did you expect?” ↩︎

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Adventures in IPv6 routing in Docker

2022-10-24T00:00:00Z

One of the biggest flaws in Docker’s design is that it wasn’t created with IPv6 in mind. Out of the box Docker assigns each container a private IPv4 address, and they won’t be able to reach IPv6-only services. While incoming connections might work, the containers won’t know the correct remote IP address which can cause problems for some applications. This situation is obviously suboptimal in the current day and age. It’s a bit like not supporting HTTPS on a website — you might not have any issues because of it immediately, but you’re fighting against the currents of progress and are making life worse for your users.

Thankfully, it’s now relatively easy to make Docker behave a lot nicer. The docker-ipv6nat project has been around since 2016, and uses an IPv6 overlay network and some iptables magic to route traffic to and from containers in a sensible fashion. It uses NAT to emulate the behaviour Docker employs for IPv4 traffic; while using NAT with IPv6 is an anathema, I think it makes sense for containers. You could give each container a publicly routable IPv6 address, but that brings with it a lot of headaches: you’re basically going to be forced to implement service discovery and some kind of DNS management to deal with the fact that your containers will be popping up on randomly assigned IP addresses. That is completely overkill for people running a small number of services on one or two physical boxes; and if it’s not overkill for you then you’re probably already looking at more complicated orchestration solutions like Kubernetes.

More recently, similar functionality has been built into the Docker daemon itself. You can now edit the config file to enable ipv6 and each container will be assigned an address in the range specified when it uses the default bridge network. This gives more-or-less the same functionality of docker-ipv6nat — you lose a little flexibility as you can’t disable IPv6 on the default bridge, but that’s a very worthy trade for having the functionality built-in.

So far this all seems very simple. Hardly worthy of being called an “adventure”. Enter stage left: the wicked witch of destination address selection…

Destination address selection and you

When a computer program tries to connect to an address such as google.com, it first resolves it in to an IP address. That’s DNS 101, but what happens if the address resolves to multiple IP addresses? For example, google.com resolves to both 142.250.74.206 and 2a00:1450:4001:82b::200e. You might just assume there’s a simple “prefer IPv4 / prefer IPv6” toggle somewhere that decides, but it’s actually a lot more complicated. With IPv6, devices are likely to have many addresses - a link-local address, a unique local address, a normal public address, various privacy addresses, and so on. To deal with this, a number of RFCs define a series of rules that most implementations follow. These are called the destination address selection rules. RFC6724 gives the rules as:

Avoid unusable destinations
Prefer matching scope
Avoid deprecated addresses
Prefer home addresses
Prefer matching label
Prefer higher precedence
Prefer native transport
Prefer smaller scope
Use the longest matching prefix
Otherwise, leave the order unchanged

Applying these rules will re-order the list of IP addresses such that (in theory) the most likely one to work will be first. Most of them are fairly niche rules; the ones that do the heavy lifting are rules 5 and 6, which rely on a policy table to make their decisions. The policy table ‘SHOULD’ be configurable by system administrators to allow them to tweak how traffic is routed. In the absence of an admin-provided policy table, the RFC gives the following defaults:

Prefix	Precedence	Label	Notes
`::1/128`	50	0	Loopback address
`::/0`	40	1	Any IPv6 address
`::ffff:0:0/96`	35	4	IPv4 addresses mapped as v6 addresses
`2002::/16`	30	2	6-to-4 gateways
`2001::/32`	5	5	Toredo tunnels
`fc00::/7`	3	13	Unique Local Addresses (ULAs)
`::/96`	1	3	IPv4 compatible addresses (deprecated)
`fec0::/10`	1	11	Site-local addresses (deprecated)
`3ffe::/16`	1	12	6bone (deprecated)

The policy table is a bit complex, but you can see that normal IPv6 addresses are preferred (have a higher precedence) over IPv4 addresses, which are preferred over the various tunnels, local addresses, and various deprecated ranges.

We can see the result of this when I run ping google.com on a box that has a native IPv6 connection as well as a native IPv4 connection:

PING google.com(fra07s29-in-x200e.1e100.net (2a00:1450:4001:802::200e)) 56 data bytes
64 bytes from fra24s01-in-x0e.1e100.net (2a00:1450:4001:802::200e): icmp_seq=1 ttl=119 time=5.11 ms
64 bytes from fra24s01-in-x0e.1e100.net (2a00:1450:4001:802::200e): icmp_seq=2 ttl=119 time=5.28 ms
64 bytes from fra07s29-in-x200e.1e100.net (2a00:1450:4001:802::200e): icmp_seq=3 ttl=119 time=5.31 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 5.114/5.235/5.311/0.086 ms

The address selection in this instance has been a result of rule 6: the IPv6 address has a higher precedence (40) than the IPv4 address (35).

However, if I run the same command in an IPv6-enabled Ubuntu container then it seems to prefer the IPv4 address:

PING google.com (172.217.16.206) 56(84) bytes of data.
64 bytes from fra16s08-in-f206.1e100.net (172.217.16.206): icmp_seq=1 ttl=59 time=4.86 ms
64 bytes from fra16s08-in-f14.1e100.net (172.217.16.206): icmp_seq=2 ttl=59 time=4.86 ms
64 bytes from fra16s08-in-f14.1e100.net (172.217.16.206): icmp_seq=3 ttl=59 time=5.03 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.860/4.916/5.027/0.078 ms

If I force ping to use IPv6 with the -6 flag then it works the same as on the host, but when left to its own devices it prefers IPv4. What’s going on there? The key difference between the Docker container and the host is that the host’s network interfaces have public addresses, but the container has a private IPv4 address (172.19.0.7) and a private IPv6 address (fd00:dead:beef::7). My first reaction to this was to think “Ah yes, fc00::/7 has a lower precedence than native IPv4, that makes sense”, but that’s not quite right. These are the destination address selection rules; rule 6 doesn’t care about the source addresses. This is actually rule 5 at work: the label of source IPv6 address is 13, but the label of the destination address is 1; meanwhile both the source and destination IPv4 addresses are 4. This sorts the IPv4 address before the IPv6 one, and rule 6 becomes irrelevant.

Adjusting the policy table

Thankfully, the RFC says the policy table should be configurable by system administrators, so those of us who are doing unorthodox things like NAT’ing IPv6 can customise the behaviour to fit our weird environments. The configuration is done via the /etc/gai.conf file (‘gai’ standing for getaddrinfo, the function in the standard library responsible for dealing with all these rules). The gai.conf file by default (if it exists) will likely just contain comments and examples; if there are no uncommented “label” or “precedence” lines then the library will use its built-in defaults based on the RFC requirements.

To make our container happy, we need to cause fd00::/8 to have the same label as a public IPv6 address. To do this we can uncomment the default labels in gai.conf and then add a single extra line, like so:

 label ::1/128       0
 label ::/0          1
 label 2002::/16     2
 label ::/96         3
 label ::ffff:0:0/96 4
 label fec0::/10     5
 label fc00::/7      6
 label 2001:0::/32   7
+label fd00::/8      1

Because fd00::/8 has a longer prefix than fc00::/7 it will match our addresses and give them a label of 1, the same as a public IPv6 address would get. This makes rule 5 leave it alone, and the default precedence table used by the standard library will put the IPv6 address above the IPv4 address. (They don’t bother giving fc00::/7 a separate precedence as listed in the RFC because rule 5 would have already de-prioritised those addresses, as we discovered.)

Making this small change to the gai.conf file in our Ubuntu container makes it start preferring the IPv6 address for google.com:

PING google.com(fra16s65-in-x0e.1e100.net (2a00:1450:4001:806::200e)) 56 data bytes
64 bytes from fra15s29-in-x0e.1e100.net (2a00:1450:4001:806::200e): icmp_seq=1 ttl=118 time=5.00 ms
64 bytes from fra15s29-in-x0e.1e100.net (2a00:1450:4001:806::200e): icmp_seq=2 ttl=118 time=5.10 ms
64 bytes from fra16s65-in-x0e.1e100.net (2a00:1450:4001:806::200e): icmp_seq=3 ttl=118 time=5.08 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 5.002/5.058/5.098/0.040 ms

A fairly complicated problem, but a simple fix!

But wait, there’s more…

Unfortunately, this fix won’t work on containers that use alpine. And that is a lot of containers. Alpine uses musl as its standard library, rather than the much more common glibc. There is very little documentation on the subject, but if you browse the source code for musl you will see that it doesn’t implement any way at all to configure the policy tables. In network/lookup_name.c we can see the sorting logic:

	/* The following implements a subset of RFC 3484/6724 destination
	 * address selection by generating a single 31-bit sort key for
	 * each address. Rules 3, 4, and 7 are omitted for having
	 * excessive runtime and code size cost and dubious benefit.
	 * So far the label/precedence table cannot be customized. */
	for (i=0; i<cnt; i++) {
		int family = buf[i].family;
		int key = 0;
		struct sockaddr_in6 sa6 = { 0 }, da6 = {
			.sin6_family = AF_INET6,
			.sin6_scope_id = buf[i].scopeid,
			.sin6_port = 65535
		};
		struct sockaddr_in sa4 = { 0 }, da4 = {
			.sin_family = AF_INET,
			.sin_port = 65535
		};
		void *sa, *da;
		socklen_t salen, dalen;
		if (family == AF_INET6) {
			memcpy(da6.sin6_addr.s6_addr, buf[i].addr, 16);
			da = &da6; dalen = sizeof da6;
			sa = &sa6; salen = sizeof sa6;
		} else {
			memcpy(sa6.sin6_addr.s6_addr,
				"\0\0\0\0\0\0\0\0\0\0\xff\xff", 12);
			memcpy(da6.sin6_addr.s6_addr+12, buf[i].addr, 4);
			memcpy(da6.sin6_addr.s6_addr,
				"\0\0\0\0\0\0\0\0\0\0\xff\xff", 12);
			memcpy(da6.sin6_addr.s6_addr+12, buf[i].addr, 4);
			memcpy(&da4.sin_addr, buf[i].addr, 4);
			da = &da4; dalen = sizeof da4;
			sa = &sa4; salen = sizeof sa4;
		}
		const struct policy *dpolicy = policyof(&da6.sin6_addr);
		int dscope = scopeof(&da6.sin6_addr);
		int dlabel = dpolicy->label;
		int dprec = dpolicy->prec;
		int prefixlen = 0;
		int fd = socket(family, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_UDP);
		if (fd >= 0) {
			if (!connect(fd, da, dalen)) {
				key |= DAS_USABLE;
				if (!getsockname(fd, sa, &salen)) {
					if (family == AF_INET) memcpy(
						sa6.sin6_addr.s6_addr+12,
						&sa4.sin_addr, 4);
					if (dscope == scopeof(&sa6.sin6_addr))
						key |= DAS_MATCHINGSCOPE;
					if (dlabel == labelof(&sa6.sin6_addr))
						key |= DAS_MATCHINGLABEL;
					prefixlen = prefixmatch(&sa6.sin6_addr,
						&da6.sin6_addr);
				}
			}
			close(fd);
		}
		key |= dprec << DAS_PREC_SHIFT;
		key |= (15-dscope) << DAS_SCOPE_SHIFT;
		key |= prefixlen << DAS_PREFIX_SHIFT;
		key |= (MAXADDRS-i) << DAS_ORDER_SHIFT;
		buf[i].sortkey = key;
	}
	qsort(buf, cnt, sizeof *buf, addrcmp);

We can see if the address labels match, the sort key is adjusted using the DAS_MATCHINGLABEL constant. But where do the labels come from? For that we need to investigate the policyof func:

static const struct policy {
	unsigned char addr[16];
	unsigned char len, mask;
	unsigned char prec, label;
} defpolicy[] = {
	{ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 15, 0xff, 50, 0 },
	{ "\0\0\0\0\0\0\0\0\0\0\xff\xff", 11, 0xff, 35, 4 },
	{ "\x20\2", 1, 0xff, 30, 2 },
	{ "\x20\1", 3, 0xff, 5, 5 },
	{ "\xfc", 0, 0xfe, 3, 13 },
#if 0
	/* These are deprecated and/or returned to the address
	 * pool, so despite the RFC, treating them as special
	 * is probably wrong. */
	{ "", 11, 0xff, 1, 3 },
	{ "\xfe\xc0", 1, 0xc0, 1, 11 },
	{ "\x3f\xfe", 1, 0xff, 1, 12 },
#endif
	/* Last rule must match all addresses to stop loop. */
	{ "", 0, 0, 40, 1 },
};

static const struct policy *policyof(const struct in6_addr *a)
{
	int i;
	for (i=0; ; i++) {
		if (memcmp(a->s6_addr, defpolicy[i].addr, defpolicy[i].len))
			continue;
		if ((a->s6_addr[defpolicy[i].len] & defpolicy[i].mask)
		    != defpolicy[i].addr[defpolicy[i].len])
			continue;
		return defpolicy+i;
	}
}

So the policyof func simply uses the defpolicy array to find the policy that applies to an address. This defpolicy array contains a number of hardcoded entries which correspond exactly to the ones provided in the RFC. The one causing us problems is { "\xfc", 0, 0xfe, 3, 13 } which matches fc00::/7 addresses.

Unfortunately, without recompiling musl from source there’s not much we can do to address this directly. The only way we can make the containers work as desired is to give them a different address range. Ideally this would be a range that you control and that is otherwise not used, but there’s a good chance you won’t have such a range. One possible alternative is the 2001:db8::/32 range which is reserved for documentation purposes. This doesn’t feature in any of the policy tables, so is treated like a normal public IPv6 address. It doesn’t feel great to misuse a reserved range like that, but it’s probably the least of all evils, at least until musl allows configuring the policy table.

Understanding Docker volume mounts

2019-04-01T00:00:00Z

The Docker project logo

One thing that always confuses me with Docker is how exactly mounting volumes behaves. At a basic level it’s fairly straight forward: you declare a volume in a Dockerfile, and then either explicitly mount something there or docker automatically creates an anonymous volume for you. Done. But it turns out there’s quite a few edge cases…

Changing ownership of the folder

Perhaps the most common operation done on a Docker volume other than simply mounting it is trying to change the ownership of the directory. If your Docker process runs as a certain user you probably want the directory to be writable by that user.

At first we might try something like:

FROM alpine
RUN adduser -D -u 1113 test123
USER test123
VOLUME /testing

But changing the user doesn’t seem to have any effect on the volume. Why? Checking the docs for the USER instruction shows that only affects certain future operations — namely RUN, CMD, and ENTRYPOINT. It doesn’t affect the VOLUME instruction; if it did, you’d probably just get a permission denied error unless the user you switch to had privileges to create mount points.

OK, so instead we might try using the good old chown command:

FROM alpine
RUN adduser -D -u 1113 test123
VOLUME /testing
RUN chown test123 /testing

But again, the directory is just owned by root at runtime. Back to the docs, this time for the VOLUME instruction and towards the bottom is this little tidbit:

Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.

As soon as Docker hits the VOLUME instruction the directory becomes a mount point, and anything we do to the temporary volume mounted there is discarded during the build process. So we have to change the ownership before the instruction, which may seem a little counter-intuitive:

FROM alpine
RUN adduser -D -u 1113 test123
RUN mkdir /testing && chown test123 /testing
VOLUME /testing

Now when the container runs, the /testing directory is owned by the test123 user. It’s not quite over, yet, though. This works if we let Docker create a volume automatically for us, or if we create a named volume and mount that; if we try and mount a host directory, though, it falls flat:

$ docker run --rm -it -v "$PWD/testing:/testing" testing ls -al /testing
total 8
drwxr-xr-x    2 1000     1000          4096 Apr  1 19:39 .
drwxr-xr-x    1 root     root          4096 Apr  1 20:44 ..

Docker handles mounting host directories differently to mounting volumes, even though the syntax is basically the same. Host directories are bind mounted directly into the container, so the permissions and ownership are the same as the directory on your host. The only way to fix them are to either change the permissions on the host, or have the container change them at runtime (assuming it has sufficient privileges).

One final wrinkle in all this happens when you use the same volume in multiple containers. Here we have two images built from the Dockerfile above, one with userid 1113 and one with userid 1114:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/testing testing1113 ls -nal /testing
total 8
drwxr-xr-x    2 1113     0             4096 Apr  1 19:49 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:51 ..

$ docker run --rm -it -v testing:/testing testing1114 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:47 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:52 ..

$ docker run --rm -it -v testing:/testing testing1114 touch /testing/Hello

$ docker run --rm -it -v testing:/testing testing1113 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:52 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:53 ..
-rw-r--r--    1 0        0                0 Apr  1 20:52 Hello

$ docker run --rm -it -v testing:/testing testing1114 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:52 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:52 ..
-rw-r--r--    1 0        0                0 Apr  1 20:52 Hello

Can you see what’s going on? When the volume is empty, the ownership changes based on the mount point in the container. Once it has something in it, the ownership is fixed.

So Docker behaves differently with regard to permissions:

when the folder is mounted from the host vs a volume
when the volume is empty vs having content

Pre-populating mounts with files from the image

One of the more esoteric features of the way Docker handles volume mounts is that in some cases files from the image are copied over into the container. For example:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/etc testing sleep 1

$ docker run --rm -it -v testing:/tmp testing ls -al /tmp
total 184
drwxr-xr-x   15 root     root          4096 Apr  1 20:58 .
drwxr-xr-x    1 root     root          4096 Apr  1 20:59 ..
-rw-r--r--    1 root     root             4 Jun  7  2018 TZ
-rw-r--r--    1 root     root             6 Dec 20 21:31 alpine-release
...

The first container we run mounts the newly created testing volume at /etc. Docker copies all the existing files and folders into the volume; when we then run the second container with the volume mounted at /tmp, we can see all the files that were in the first container’s /etc.

As with permissions, this behaviour is anything but consistent. Say we switch from a volume to a host directory:

$ mkdir testing
$ docker run --rm -it -v "$PWD/testing:/usr/bin" testing sleep 1
$ ls -al testing
total 8
drwxr-xr-x 2 root  root  4096 Apr  1 22:05 .
drwxr-xr-x 3 chris chris 4096 Apr  1 22:05 ..

Nothing is copied in, and inside the container the folder will be empty. Based on our discoveries with permissions, it’s reasonable to assume the same will happen with a non-empty volume too:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/testing testing touch /testing/Hello

$ docker run --rm -it -v testing:/usr/bin testing sleep 1

$ docker run --rm -it -v testing:/tmp testing ls -al /tmp
total 8
drwxr-xr-x    2 root     root          4096 Apr  1 21:09 .
drwxr-xr-x    1 root     root          4096 Apr  1 21:09 ..
-rw-r--r--    1 root     root             0 Apr  1 21:08 Hello

So at least that’s consistent. If you’re very observant, though, you might notice I switched from /etc/ to /usr/bin in the examples. That’s because within the container /etc/ has some files bind-mounted into it, such as /etc/resolv.conf, and these do always result in files being created in the mounted volumes or folders:

$ mkdir testing
$ docker run --rm -it -v "$PWD/testing:/etc" testing sleep 1
$ ls -al testing
total 8
drwxr-xr-x 2 chris chris 4096 Apr  1 22:12 .
drwxr-xr-x 3 chris chris 4096 Apr  1 22:12 ..
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 hostname
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 hosts
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 resolv.conf

Summary

Docker treats mounting host folders and mounting volumes differently. Don’t just assume that you can swap one for another and get the exact same behaviour.
Empty volumes will inherit permissions and files from the image they are mounted in; non-empty volumes and host folders will not.
Relying on Docker copying files into volumes is a very bad idea, as if you change those files in a future version of your image they will not be copied unless the volume is deleted and recreated.

I can’t find anywhere that these points are documented properly; if you know of anywhere, please drop me a message!