Chameth.com - posts like artisanal-docker-images, debugging-beyond-the-debugger, docker-automatic-nginx-proxy but not g15-ram-upgrade, migrating-from-github-to-forgejo, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.

Reproducible Builds and Docker Images

2022-02-18T00:00:00Z

XKCD 2347: Dependency

Reproducible builds are builds which you are able to reproduce byte-for-byte, given the same source input. Your initial reaction to that statement might be “Aren’t nearly all builds ‘reproducible builds’, then? If I give my compiler a source file it will always give me the same binary, won’t it?” It sounds simple, like it’s something that should just be fundamentally true unless we go out of our way to break it, but in reality it’s actually quite a challenge. A group of Debian developers have been working on reproducible packages for the best part of a decade and while they’ve made fantastic progress, Debian still isn’t reproducible. Before we talk about why it’s a hard problem, let’s take a minute to ponder why it’s worth that much effort.

On supply chain attacks

Suppose you want to run some open-source software. One of the many benefits of open-source software is that anyone can look at the source and, in theory, spot bugs or malicious code. Some projects even have sponsored audits or penetration tests to affirm that the software is safe. But how do you actually deploy that software? You’re probably not building from source - more likely you’re using a package manager to install a pre-built version, or downloading a binary archive, or running a docker image. How do you know whoever prepared those binary artifacts did so from an un-doctored copy of the source? How do you know a middle-man hasn’t decided to add malware to the binaries to make money?

Even worse: if the software you’re trying to use includes any dependencies, you have the same issue of trust with them. Maybe your supplier isn’t compromising the software, but that doesn’t mean their supplier isn’t. The beauty-cum-horror of a supply chain attack is that it can target the weakest link anywhere along the supply chain. Even if there aren’t any binary files involved, dependencies can still be attacked: what if npmjs.com or proxy.golang.org or github.com return a different version of a dependency-of-a-dependency when the request comes from your IP address? It doesn’t even need to be a modified dependency, it could be a perfectly un-tampered, properly signed copy of the source, just from an older version with a known vulnerability.

Enter stage left: reproducible builds, here to save the day! If the build process is reproducible then you - or anyone else on the internet - can perform the same build on the same source and validate the output has the same checksum or hash. If Debian publish a binary package and an independent re-builder comes up with the exact same build artifact, there’s a reasonably good chance that the build is good. An attacker would have to compromise both the build machine and the re-build machine to do anything nefarious. The more re-builders there are, the less feasible a supply chain attack is.

So why isn’t software just reproducible?

Compilers

As a bit of an experiment, I asked some friends to run the following for me and report the answer:

echo -e "#include <stdio.h>\nint main() { printf(\"Hello\"); return 0; }" | \
  gcc -x c -o hello.out - && \
  sha256sum hello.out

This compiles a super-simple hello world program and then prints the SHA-256 hash of the resulting binary. Here are the results:

Hash	System	GCC
1f62feab5a06861dc575201d807781926d1ae49fb113da018fde8b670a1346f7	Arch	11.2.0
b8e6f2c7082be69f65ffa5e7a3d749eb47866a1b2e1ec19efb63cc59a8b160cd	Debian	8.3.0
cbad2e47a22c234b5e7fa55e029a8db4d64ac7a962e2176bd2e1373d78954088	Debian	8.3.0
e0f6bbc13b29fea8cfa2a975ba4661e781323298aec166c8311d342e6f93c4a6	Alpine	10.3.1
e379156895e06c7a0bf18ac4d648860edcb2655576b0ab9fab172bd6c8b92075	Debian	10.2.1
7ffdaee4eb64e016b89dc5e54d2c8eebab3cebafe2c7aa97de627b5972ecea46	Debian	11.2.0
8ae52cc166743b6ae1eb3e14179ef33de5061a04237f8f97088c896c41a2f698	Arch	11.1.0
8ae52cc166743b6ae1eb3e14179ef33de5061a04237f8f97088c896c41a2f698	Arch	11.1.0

As you can see, there are barely any duplicates. Even the same version of GCC on the same OS sometimes produces different results. And this is the most basic program I could write! Differences arise from the compiler version, the build flags, the libraries installed, and a whole host of other factors. If you compile a Go application instead of a C one, then by default the compiler will include debug information in the binary. This includes the full path to the source file on disk, so building a project in /home/chris/ will produce a different binary to building the same source in /tmp. Future versions of Go are also going to stamp in other meta-data such as VCS info, so building inside and outside a Git repository will produce different binaries.

Iteration order

So we’ve pinned our build environment, we’re manipulating timestamps when adding files to archives, now what? Imagine part of the build process involves looping through all the files in a directory and doing something. What order do these files get iterated in? Well, that very much depends on the filesystem and perhaps when the files themselves were created. To ensure this is reproducible we need to explicitly sort any such operation so that it’s always consistent. This iteration could be happening in a tool that’s called by another tool that’s called by a build script, so the fix isn’t necessarily straight-forward.

Side note: a bug war story

I’ve personally been victim to this kind of non-determinism. I was working on an Android app, and committed a new test that worked fine on my machine, and worked fine on the CI server. But it failed consistently for a colleague.

We both did fresh checkouts of the source, and ran the tests. Mine passed, his failed. He sent me an archive of his checkout in case there was something weird going on there, and the tests passed on my machine. We compared hashes of our checkouts, and they were the same. It was obviously environmental somehow, but everything else worked fine, and the build system went to great pains to ensure things were the same.

After a lot of debugging, I worked out that his test was running with a different version of a library to me, despite the libraries being defined in the build files and the build files being identical. After even more debugging it turned out there were two versions of the library on the classpath, and the ordering of them was different between my machine and his.

The actual issue turned out to be that the build tool generated the classpath by iterating over the library files, and that iteration was done in order of file creation time. The two libraries were added at different points in the project history, so the creation time in your local cache depended on which versions of the app you’d built in the past. With no cache everything worked as expected but there was a slim range of commits where only one library was in use, and if you had run the tests during that period your cache was effectively poisoned.

We fixed the issue by excluding the older version of the library (which was being pulled in as a transient dependency), and filed a bug against the build tool to make the classpath properly deterministic. I think that stands as the most difficult to diagnose bug I’ve ever dealt with.

Interestingly, if you iterate over a map in Go, the iteration is deliberately non-deterministic. That’s an attempt to defeat Hyrum’s Law and prevent developers from relying on whatever the current behaviour happens to be. This actually makes it easier to make things reproducible as the problem is loud and in-your-face, rather than subtle and hard to spot.

Other sources

There’s an awful lot of other places that non-determinism can come from. If the app pulls in dependencies, their versions have to be pinned, otherwise your build changes depending on the latest release of that dependency. If the build process pulls any information from a website, it’s liable to change. Hopefully the website is under your control so that you can version the resource and pin that version. Obviously, anything to do with dates or the current user will probably cause problems. Timezones and locales can cause subtle differences.

What about Docker?

Docker comes with some good and some bad points for reproducibility. The biggest advantage is that it inherently completely describes the build environment; it should work exactly the same from one system to another, even across different OS families. The biggest drawback is it sprays timestamps around like no-one’s business. Each layer in a container image is a .tar.gz file, meaning each file within it is timestamped as discussed above. Making an image involves a lot of copying of files around, so these timestamps invariably end up causing reproducibility issues.

Even worse than timestamps in the filesystem, the image format also contains some meta-data that includes the timestamp at which each layer was built. That means even if you go out of your way to set the timestamp of every single file in your image, the image itself will be different every time you rebuild it. There is no way to deal with this in Docker, which is a very sad state of affairs. Fortunately, Buildah provides a --timestamp flag for its build commands; this not only sets the layer timestamp but also the creation timestamp of any file within the layer.

The other major issue that affects Docker images is the pinning of packages pulled in by package managers. An awful lot of images are based on Alpine or Debian derivatives, and use apk or apt to install dependencies. These need to have a version specified as otherwise the package manager will just pull in the latest at the time of the build. But this isn’t quite enough: you also need to pin the version of any packages that they depend on, recursively. This means flattening the entire package hierarchy and installing all the packages explicitly and with pinned versions.

One more wrinkle in the package management space is that Alpine don’t keep old packages in their main repositories. If you have a Docker image with pinned alpine packages in, it will stop building if the package is updated. This isn’t necessarily fatal to making a reproducible build — as long as it’s reproducible for its useful lifetime, I don’t really see an issue.

Honestly, though, the biggest issue with making Docker images reproducible is getting people to care. Dockerfiles are a relatively new way of packaging software, and there’s no centralised organisation like you find with Linux distributions. There are enough challenges that most casual packagers aren’t going to bother, and no real incentive for them to. That won’t stop me trying, though!

Artisanal Docker images

2022-02-05T00:00:00Z

Artisanal containers…

I run a fair number of services as docker containers. Recently, I’ve been moving away from pre-built images pulled from Docker Hub in favour of those I’ve hand-crafted myself. If you’re thinking “that sounds like a lot of effort”, you’re right. It also comes with a number of advantages, though, and has been a fairly fun journey.

The problems with Docker Hub and its images

Rate limits

For the last few years, I’ve been getting increasingly unhappy with Docker Hub itself. Docker-the-technology is wonderful, but Docker-the-company has been making some rather large missteps. The biggest and most impactful of these has been introducing “pull rate” limits. At the time of writing, if you want to just pull a public image without logging in then you are limited to 100 pulls every 6 hours. If you log in then you’re limited to 200 pulls per 6 hours, but it’s account wide. This might seem like a big enough number, but I repeatedly hit it and there is no way to actually audit what is causing it. I have various containers that may all pull images at arbitrary times (e.g. continuous integration build agents), and the only information you get back from Docker Hub is the number of pulls remaining.

Obviously, I could start paying Docker Hub for a “Pro” plan. That gets you 5,000 pulls per day for $7/month. The downside is that every docker client would have to be authenticated, which presents a fair annoyance in terms of credential management. I also don’t really like how they positioned the service as a public utility with special treatment in the docker software, and then start tightening the ratchet to make money.

“Bad” images

I’m fairly opinionated about what a container image should look like: most importantly it should run just a single process, and only include the bare minimum dependencies required for that. Other people think differently, and it’s very hard to tell at a glance whether an image on Docker Hub contains just the application you want, or whether it also bundles MySQL, Redis, Elasticsearch, and a partridge in a pear tree. Some people want that kind of thing, but I really don’t. It’s also very hard to tell whether an image is officially endorsed by the upstream project, and where the source Dockerfile is. This used to be better because most projects used Docker Hub’s automatic builds, but they’re now a “pro” feature.

I quite often found that I’d be looking for an image for X, and there would be 5-10 images from different users. None of them looked official, some of them were out-of-date, some bundled the kitchen sink. Even when one looked good, it’s a bit of a gamble whether the author is going to keep it updated or not.

Doijanky

The rate limits and other problems were annoying, but they weren’t really annoying enough to force me to do anything about it. The straw that broke the camel’s back came later: I was looking at the official golang images, and noticed that all the tags were pushed by a random user account called “doijanky”:

An ‘official’ Docker Hub image pushed by user ‘doijanky’

I, perhaps naively, assumed that official images were built on Docker Hub’s own infrastructure. Why would all the Golang images be attributed to this user? Checking out their profile, they’re simply identified as a “Community User” like everyone else, with no repositories of their own. The only thing in the profile is their homepage, which is a link to a Jenkins dashboard: https://doi-janky.infosiftr.net/. It appears legitimate: “Infosiftr” are a container consultancy and the dashboard is linked to from the README in the official images git repository, but I find it baffling that they’re using third-party infrastructure and a normal user account (with a dubious name) to push these images. There doesn’t seem to be a good way to verify what you pull corresponds to the Dockerfile it came from; if infosiftr wanted to inject something into the build they could happily do so, and who knows how good their infosec posture is? If someone got access to the “doijanky” account, how long could they upload malicious images before someone noticed?

This little roller-coaster ride from “are all the official images compromised?!” to “oh, no, they’re not, it’s all just awful” finally convinced me to look at building my own images from scratch.

The implementation: templating with contempt

One of the big issues I needed to tackle was how to deal with updates. I didn’t want to have to go and edit a file every time some minor release was made of some software, or every time there was a security vulnerability in a common library. The official images use a shell-scripting based system to check for updates and generate Dockerfiles, I decided to do something similar but with Go templates. The result is a tool called contempt. It takes a template like:

FROM {{image "golang"}} AS build

ARG TAG="{{github_tag "example/project"}}"

RUN apk add --no-cache \
        {{range $key, $value := alpine_packages "git" -}}
        {{$key}}={{$value}}\
        {{end}}; \
# ...

Contempt has support for getting information from a variety of sources. In this case, it’s getting the latest digest of another Docker image, the latest tag from a Git repository, and the latest version of an alpine package and all its dependencies. The resulting Dockerfile looks something like this:

# Generated from https://github.com/csmith/dockerfiles/blob/master/miniflux/Dockerfile.gotpl
# BOM: {"apk:brotli-libs":"1.0.9-r5","apk:busybox":"1.34.1-r4", <snip> }

FROM reg.c5h.io/golang@sha256:ac8fa5f4078b0a697796b5d741... AS build

ARG TAG="2.0.35"

RUN apk add --no-cache \
        brotli-libs=1.0.9-r5\
        busybox=1.34.1-r4\
        # <snip>
        pcre2=10.39-r0\
        zlib=1.2.11-r3\
        ; \
# ...

I’ve cut out the longer parts for readability. You can see that it pins all versions of the alpine packages in use, as well as the base image. This ensures that if you build from the same Dockerfile at a later time it will build the same image (or will fail entirely, as Alpine doesn’t keep their old packages around indefinitely). It also produces a “bill of materials” as a really long JSON-encoded comment. If you let contempt commit the Dockerfile it uses the BOM to generate useful commit messages like: [project] apk:busybox: 1.34.1-r3->1.34.1-r4, so you can see exactly what changed.

Contempt also has support for building and pushing images whenever it changes the Dockerfile. I use it in a GitHub action that runs daily to check all my images are up-to-date and push those that aren’t. It understands the dependencies between images (by pre-analysing the templates) so it will always check and build base images before ones that require them. This means an update to, say, the “alpine” base image will cause anything that depends on it to get updated at the same time, ensuring security updates are rolled out promptly.

The result

You can see my collection of lovingly hand-crafted Dockerfiles in my dockerfiles repository.

There are a number of advantages to handwriting all the images I use. The obvious one is that they’re all built how I want: there are no extraneous dependencies, they’re all based on the same small set of base images (rather than pulling around 10 different versions of debian), nothing tries to also run a DBMS in its container, etc.

This level of customisation goes further, though. Because I’m packaging the software myself, I can tweak how it’s built to fit my needs. A couple of things I run need their own TLS certificates separate from my normal HTTPS setup, so I bake my certwrapper tool in to manage those; I can even set the build flags on certwrapper to only enable the particular DNS provider I personally need (thus avoiding dragging in clients for AWS, GCP, etc). Some software like Hashicorp Vault has an optional web interface that I don’t need, so I simply don’t enable it in the build. These changes save build time, reduce image sizes, in some cases improve runtime performance, and generally reduce the attack surface of what’s running in the container.

It’s also been a great way to learn more about how software is distributed. Writing a Dockerfile is not that distant from writing a PKGBUILD file for an Arch Linux package, or the equivalent for other distributions. In a couple of instances I’ve googled how to solve a particular issue, and found an Arch or Void linux maintainer asking the upstream project about the exact same issue.

Finally, all the images I build I push to my own registry so there are obviously no rate limiting issues. Standing up a service (assuming the Dockerfile has been written!) is amazingly quick because the base layers are all shared and cached, and the registry is a lot physically closer than Docker Hub. Bootstrapping this whole thing becomes an interesting problem because the image for the registry is stored on the registry, but I’ll leave that discussion for another post…

An introduction to containers

2020-03-01T00:00:00Z

So. Many. Containers.

I’m a huge fan of (software) containers. Most people I know fall in to one of two camps: either they also use, and are fans of, containers, or they haven’t yet really figured them out and view them as some kind of voodoo that they don’t really want or need.

I’m writing this short guide to explain a little how containers work - and how running something in a container isn’t really that much different to running it normally - to hopefully enable more people in that second group to give them a try. It’s aimed at people who have a fairly good grasp of how Linux works.

Containers are often mentioned in the same breath as VMs, which is not a helpful comparison or analogy. Think of containers as standard units of software, much like how Intermodal containers are standard units of freight transport across the world. When a company internationally ships goods in volume there isn’t a question about how they’re packaged - they go in an intermodal container. The same container can be deployed on a freight train, a lorry, or a ship. The haulage company doesn’t need to care what’s in the container because they’re completely standardised. Likewise, with software containers you don’t really need to care about what’s inside: the software you’re deploying could be written in Go, Python2, Python3, Bash, PHP, LOLCODE, or anything¹.

What does a running container look like?

When you run a container, you’re just running a process. In a lot of ways it’s not very different to what happens if you were to start the same process on the host computer.

For example I have a container that just runs cat(1). With no arguments, cat will read from stdin until it receives an EOF, so it’s handy to test with. If I run ps a on my computer, I can see the cat process in amongst everything else I’m currently running:

    PID TTY      STAT   TIME COMMAND
   7199 pts/1    Ss     0:01 /usr/bin/zsh
 323806 pts/0    Ss+    0:00 /bin/cat
 324120 pts/4    R+     0:00 ps a

The /bin/cat process is in a container, and the ps a underneath it is just running like normal on my desktop. They look very similar, right? If I look under /proc/323806 I can see all the usual attributes, the same as any other process running.

If I run ps in a container, though, it’s a different story:

    PID TTY      STAT   TIME COMMAND
      1 pts/0    Rs+    0:00 ps a

So inside the container it looks like there’s only one process running. It can’t see anything running “outside” on my desktop. The secret here is that this isn’t a special container trick: this is just a feature of the Linux kernel called namespacing.

If we go back to procfs and look at the ns/pid node we can see the process in the container is in a separate PID (process ID) namespace to the one on my desktop:

# readlink /proc/323806/ns/pid  
pid:[4026534564]
# readlink /proc/7199/ns/pid
pid:[4026531836]

Almost all² processes running ’normally’ on my desktop have the same PID namespace, whereas each container gets their own by default. PID namespaces are hierarchical: a new process is assigned a PID in its own namespace, and the parent namespace, and the grandparent namespace, and so on. That’s why I can see the process running in the container from my normal shell - the container’s namespace is a child of the main namespace all of my desktop software is running in.

Linux supports - and container software makes use of - a bunch of other namespaces too: mount points, network, UTS³, cgroups, and more. These all play a part in isolating a container from the system it is running on.

You can manually run a process with unshare(1) to “unshare” some namespaces from the parent process. For example if I run unshare -fp --mount-proc ps a, it looks very similar to running ps instead the container:

    PID TTY      STAT   TIME COMMAND
      1 pts/4    R+     0:00 ps a

So: a process running inside a container is just a heavily namespaced process running otherwise normally in the operating system. No voodoo magic here!

What about the filesystem? What are ‘images’?

Containers run in their own mount namespace meaning mount points can be different inside the container to those on the host. This means the container can have a different / mounted to the host, effectively giving it its own filesystem.

The root filesystem of the container is defined in the container’s image. If I use Docker to run a container using the Ubuntu image (docker run ubuntu), the root filesystem inside that container will resemble a minimal ubuntu install. Note that this is just the filesystem: the container doesn’t have its own kernel.

You might be thinking that sounds pretty inefficient. Downloading Ubuntu is definitely not instant, and doing it for every application you run would be insane! Quite. Containers solve this by using filesystem layers. These are stacked on top of one another to create the final filesystem. Each layer can be retrieved and cached independently of all others.

Say (for simplicity) that the Ubuntu image is a single layer. If I run one container with that image, then the layer will be downloaded and cached once. If I run three hundred containers with that image then the layer will be downloaded and cached once. Even better, if I use another image that’s based on Ubuntu but adds some software on top, only the “on top” layer will be downloaded if I already have the relevant “Ubuntu” layer cached.

If all the layers are cached, what happens when you change a file? This is dealt with using the copy-on-write technique: when you modify a file it is copied from the source layer and the changes are only made in a new layer. This is handled by the OverlayFS filesystem which is part of the mainline kernel.

When a container is running, changes made to its filesystem are temporary, and do not persist across container restarts. To persist data - or introduce new data to a container - you can mount volumes. How this works varies depending on how you’re running your container, but at the basic level it is pretty much the same as bind-mounting (mount -o bind).

You may be familiar with using chroots to change the apparent root directory of processes, perhaps with full-blown “jails” built on top. Containers offer much better isolation thanks to the use of namespaces. Instead of being constrained to a portion of the host’s filesystem, they don’t even have it mounted! Containers also get to specify their environment - if they expect in certain places, for example - instead of the sysadmin having to manually set up the chroot. Finally, containers offer much more fine-grained control over what processes can do (if you want it), and allow much more advanced use-cases such as inter-container networking.

Images and filesystems employ a little magic to ensure that layers are reusable and cacheable, but again there’s nothing terribly special about them: a container has a filesystem that appears to it to work the same way as a filesystem on the host, and it’s using a standard filesystem shipped with the kernel.

How about networking?

Again, networking is namespaced, so a container has its own network stack, its own virtual network interface, its own IP address and so on. How that network interacts with your real network depends on how you’re running the container. Docker, for example, can add iptables rules to NAT traffic between containers' networks and the outside world.

Containers can generally be connected into networks, and can communicate amongst themselves without the traffic actually leaving the host machine. This allows you to, for example, run a SQL database and connect it to a web application without ever exposing the database to the outside world. Moreover, as well as being isolated from the outside world, it’s isolated from other containers in other networks. If one of your applications has a crazy bug or is compromised, this significantly limits the damage it can do.

You have to explicitly opt in to “publishing” ports from a container, which exposes them to the outside world (either directly, or via a load balancer or some other middle-man, depending on how you’re running the container). This means you can pick and chose how the outside world sees the app you’re deploying: if it’s a web service that listens on both port 443 and port 80, you can chose to only expose the encrypted port.

If you run some containers and create some networks, you can see the interfaces and bridges on the host:

$ ip l
...
6: br-2405a8cc0445: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:3e:fa:23:62 brd ff:ff:ff:ff:ff:ff
8: veth8ed0735@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2405a8cc0445 state UP mode DEFAULT group default 
    link/ether b2:c1:5d:55:26:9b brd ff:ff:ff:ff:ff:ff link-netnsid 2
10: veth541d84b@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-9d7bc4024c1a state UP mode DEFAULT group default 
    link/ether 86:aa:5f:ee:da:1a brd ff:ff:ff:ff:ff:ff link-netnsid 1

This shows a bridge and two virtual NICs, just the same as if you’d manually created them (albeit with seemingly random names). So: as before, nothing special here.

Docker? Compose? K8s? Floccinaucinihilipilification?

(OK, Floccinaucinihilipilification isn’t actually a container technology, as far as I’m aware.)

Docker is the most popular container engine - that is, the bit of software that actually runs containers. It’s responsible for setting up all those namespaces we found, downloading and caching image layers, and actually starting and stopping the processes. Docker runs as a system-wide daemon - when you run a command like docker run ubuntu it actually just instructs the daemon to do the work.

There are several alternatives to Docker for running containers; one interesting one is Podman which runs containers without a daemon. Container engines have all standardised around the same image format looked after by the Open Container Initiative, so an image you build in Docker can be used in Podman, or pretty much any other engine.

Docker compose is a tool for defining and running multiple-container applications. I mentioned earlier running a database alongside a webapp - in practice to do this you’re going to have to configure a network for them, configure a mount point for the database to persist its data on, pass credentials in to both the database and the application, and so on. Doing all that by hand is tedious and error prone.

Docker compose lets you write “compose files”, which are simple yaml descriptions of the containers you wish to run, their properties, and details about any volumes or networks you may want. Out of the box, docker-compose will create a default network for each compose file you run so the containers within it can communicate.

Kubernetes is a container orchestrator, designed to automate deployment and management of large numbers of containers. It works with Docker under the hood, but provides a huge amount of tooling on top to allow you to deploy applications and manage their dependencies. It runs across multiple physical (or virtual) machines (while still allowing containers to communicate privately), and can support massive workloads by scaling out services (running multiple copies of a container on different hosts) and load balancing. Kubernetes is sometimes shorted to k8s (as in K - 8 elided letters - s) because computer people don’t like long words.

OK, they make sense now. But why bother?

Hopefully if you’ve read this far you’ve already picked up on some of the potential benefits, but this is my personal list:

Isolation. If I run some software in a container, there is very little it can do to upset me. It’s not going to leave bits of itself all over my filesystem, it can’t steal all of the secrets in my home directory, I can even limit its CPU and memory resources if I want. If I decide to stop running it, I just delete the container and it is completely gone: no trace remains.

Ease of use. If you give me a container image I have a very good idea of how to run it already. I might need to do some minor configuration to expose ports or mount volumes, but there’s no question about how to run it, how to make it automatically start, and there’s no “installation” procedure. If I want to then swap it with an alternative (say, move from MySQL to MariaDB), it’s potentially just a case of changing the name of the image I pull.

Dependencies included. Containers just run. Python 2 software includes Python 2 and just work. Python 3 software includes Python 3 and just work. I don’t have a massive headache trying to run both at the same time, because they take care of their own messes. Similarly I’m not going to have to install npm or cargo or composer to pull in dependencies for an application: that’s going to have been done in the build process.

Reproducibility. As a fallout from having dependencies included and being isolated from everything else, containers give you amazing reproducibility. If it “works on your machine” in a container, it’ll almost certainly work in production because it’s the exact same environment.

Standardisation. At the start of this article I called containers standard units of software. One of my favourite advantages of containers is that you basically get an API to list all the software you’re running. Most container engines let you supply labels attached to containers as well, so you can add your own annotations. I use this to annotate services which expose HTTP endpoints, and I have a tool that automatically generates SSL certificates for them and configures haproxy to route traffic to them. I can’t imagine how I’d do this without containers - I imagine it’d involve a lot of manual work.

Hopefully this has helped demystify containers a little. If you feel like I’ve missed something important out, or I’ve left you more confused than when you started, feel free to drop me a note using the feedback form below.

OK, maybe you should care if you’re deploying something written in crazy languages like PHP. ↩︎
Some multi-process apps, such as web browsers, are starting to use namespaces to enhance security, as do certain package systems like Flatpak ↩︎
“Unix timesharing system”; in practice this means having a separate hostname ↩︎