Chameth.com - posts like docker-proxying-redux but not debugging-beyond-the-debugger, migrating-from-github-to-forgejo, tailscale-docker-gotcha, why-you-should-be-using-https

The Case of the Unchanging Config

2026-04-09T00:00:00Z

Last week I was attempting to make it so I could share pictures on IRC directly from my client. This sounds simple, but it involves a bouncer¹ that proxies the request to a standalone image hosting service that I had to modify to be compatible. At one point my testing loop was:

Make a change to the hosting service
Commit it
Tag a new release
Wait for it to build
Update the version I’m running on my server
Reconfigure the bouncer to pass the new parameter or change the URL or whatever
Try uploading a photo from my phone
Realise I’ve overlooked something and go back to step one

I could have set up a local copy of everything and tested it sensibly. I should have set up a local copy of everything and tested it sensibly. But it seemed like such a trivial change, and setting up the whole environment seemed like such a pain. After the third or so iteration of failure I was pretty annoyed with myself, computers, and basically everything.

My biggest annoyance was that my bouncer would not pick up the new URL from the config when I changed it. It’s meant to reload the config when it receives a SIGHUP, and it claimed to in the logs, but I could clearly see it was still hitting the old URL. Restarting the bouncer to update the config is a pain, as it disconnects me from all the IRC networks, and has to reconnect to them all, reauthenticate, etc. It also mildly spams everyone who shares a channel with me².

When I finally got everything working I had a look at the bouncer source, and thought I’d spotted the issue. I raised a bug report, ending in this remark:

It looks like the config is reloaded properly, but the handler for uploads is created once at startup and has its own copy of the uploader, so effectively snapshots the config to whatever it is at startup:

fileUploadHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    cfg := srv.Config()
    h := fileupload.Handler{
        Uploader:    cfg.FileUploader,
        DB:          db,
        Auth:        cfg.Auth,
        HTTPOrigins: cfg.HTTPOrigins,
    }
    h.ServeHTTP(w, r)
})

I’m not quite sure what I thought I saw there, but that’s not right. I want to call it a hallucination, but that feels like a very overloaded word these days. I guess I was just seeing what I wanted to see, and that was a good excuse to stop investigating. Still, even if I misidentified the cause, the bug was still present, right? … Right?

The Science of Deduction

When the developer got back and said he couldn’t reproduce my issue, I went back to my install and immediately reproduced it. With a calmer head, I figured it was probably something with my particular set up. First thing to check: am I actually running the version I think I am?

Unfortunately the bouncer doesn’t actually seem to expose the version anywhere that I can see. It’s not in the logs, it doesn’t have a -version flag, and none of the IRC-based status commands seem to include it. But I know it’s a Go app, and I know Go embeds the version information. go version -m <binary> will dump it all out, but the binary is inside a Docker image, and the Docker image is one of my nice, minimal, artisanal ones so doesn’t ship a go binary. No problem, docker compose cp bouncer:/bnc ./bnc yoinks the binary out of the container, and then dumping the version shows that, yes, I am running the version I thought I was. Hmm.

The next tool I reached for in my toolbox was strace. Maybe it’s not actually reading the file for some reason? I immediately executed strace -p <pid> -e trace=openat,open,read,pread64 -f without having to look any part of that up. Yep. Definitely. Then I edited the config, HUP’d the bouncer, and saw that it was… reading the config file. As it’s meant to. By default strace truncates strings to 32 bytes, so I couldn’t actually see the line I’d changed. Some more definitely-not-RTFMing later, and rerunning it with an extra -s 65536 let me see the full config. Surprise! The config hadn’t changed!

To confirm my findings, I used docker compose cp again, this time yoinking the config file from inside the container. The inside config file was definitely different to the outside config file. What? My hypothesis at this point was “something something Docker nonsense”. I mount the config as read-only, and was wondering if that meant that Docker was doing something weird instead of just bind mounting it. A quick trip to /proc/<pid>/mounts showed that it was, in fact, not doing anything weird, and was just bind mounting it.

If the file is bind mounted, then surely it’s the same file? I ran stat on the file on the host, noted the inode number, then pondered how to actually run stat on the file inside the container, given the aforementioned awkwardly minimal image. The solution was easy: access it via /proc/<pid>/root/. I could’ve saved myself a bunch of docker compose cp if I’d thought about that earlier. Oh well. The inode of that file was different. What?

The answer was DNS. It’s always DNS. Oh, sorry, force of habit. I meant the answer was vim. I was editing the config in vim, and when it saves files, by default, it writes the new content to a temporary file and does an atomic rename. That’s normally a good thing: it prevents corruption if the write fails midway through³. In this case, though, that means the new file has a new inode. Bind mounting a file binds to the inode, so the container just keeps clutching onto the original config from when it was started, blissfully unaware that the party has relocated down the street.

This problem is likely to happen whenever you bind mount a file into a container. When I mentioned this to a friend, he immediately responded “oh yeah, never do that”, and went on to describe the horrible hacks he’s had to add to Ansible to sidestep the issue. The nicer solution is to just bind mount an entire directory if you can, as then it doesn’t matter what happens to the files within it. I really like having the config files sat alongside the Docker compose files, though; having to create a directory just to work around some bind mount weirdness upsets me.

Now I knew what the problem was, I found there was an issue raised against Docker twelve years ago⁴. The first response was “that’s expected”, and I fully agree with the author: “respectfully, that might be expected by you, but it was not expected by me”!⁵ As a result of the issue they added a nice note to the docs:

Note: Many tools used to edit files including vi and sed --in-place may result in an inode change. Since Docker v1.1.0, this will produce an error such as “sed: cannot rename ./sedKdJ9Dy: Device or resource busy”. In the case where you want to edit the mounted file, it is often easiest to instead mount the parent directory.

But in the intervening twelve years, both the note and the functionality described have gone missing. Ho-hum.

I found it interesting how I’ve only just hit this problem, given how long I’ve used Docker. But I realised that almost everything I run I’m happy to just restart. Cattle, not pets, and so on. My IRC bouncer is one of the few exceptions to that. The only other thing I regularly hot reloaded was Centauri, my reverse proxy, but that had a whole config directory mounted because it was shared between containers, so nicely sidestepped the foot-gun.

So lesson learnt: check for weird bind mount issues before raising issues about config hot reloading. It’ll join the esteemed company of “maybe the drive is full and causing completely unrelated problems?”, “perhaps everything is dog slow because the kernel ran out of entropy?”, and “did systemd sneakily take over that functionality while you weren’t looking?” in the troubleshooting checklist.

An IRC bouncer is basically an always-on proxy. It connects to the IRC networks for you, then your clients connect to your bouncer. The bouncer can then send incoming messages to all your different clients, cache them when you’re offline and replay them later, and lots of other nice things people take for granted in their chat apps these days. ↩︎
Join/part/quit spam is part of IRC, and clients have ways of handling it, but I still feel bad about doing it excessively. ↩︎
You can disable this in vim by setting backupcopy=yes. Good luck redoing all the debugging if you ever accidentally remove that from your vimrc, though! ↩︎
Which is weird, because Docker can’t possibly be that old. That would make me much older than I’m prepared to accept. ↩︎
If you read the thread they didn’t actually mean to sound so dismissive, but it’s still pretty funny. “Yes, it’s expected that the foot-gun causes your foot to hurt. Duh. What did you expect?” ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.

Automatic reverse proxying with Docker and nginx

2016-05-21T00:00:00Z

The Docker project logo

Over the past few weeks I’ve gradually been migrating services from running in LXC containers to Docker containers. It takes a while to get into the right mindset for Docker - thinking of containers as basically immutable - especially when you’re coming from a background of running things without containers, or in “full” VM-like containers. Once you’ve got your head around that, though, it opens up a lot of opportunities: Docker doesn’t just provide a container platform, it turns software into discrete units with a defined interface.

With all of your software suddenly having a common interface, it becomes trivial to automate a lot of things that would be tedious or complicated otherwise. You don’t need to manage port forwards because the containers just declare their ports, for example. You can also apply labels to the application containers, and then query the labels through Docker’s API.

Reverse proxying and SSL termination with Nginx and Let’s Encrypt

A fairly significant chunk of the software I run has a web interface. I don’t really want to expose and remember dozens of non-standard ports, so I configure an nginx instance as a reverse proxy. I’m of the opinion that all web traffic should be encrypted, so I also have to provide nginx with trusted certificates to use for each site it reverse proxies. Let’s Encrypt makes the process of obtaining free, trusted certificates approximately a thousand times easier than it was previously, but my workflow still ends up looking like this:

Create a new config file from a template and save it in /etc/nginx/sites-available
Temporarily disable SSL for the site as there’s no valid certificate yet
Enable the site by symlinking to it from /etc/nginx/sites-enabled
Reload nginx
Run the Let’s Encrypt client to obtain certificates
Enable SSL and for the site
Reload nginx

… And that’s not including the extra steps when I miss a semi-colon, accidentally skip a step and have to spend time figuring out why it’s not working, or any of the other human-induced problems that creep in.

I’d been toying with making a script to run through these steps manually for me, if I gave it a domain name and a reverse proxy target, but I never got around to it. Now I’m moving things to Docker, though, there’s an opportunity to automate the entire thing with no human interaction at all.

Existing solutions

It seemed like this probably wasn’t a unique idea, so I had a look around for existing solutions. The most popular by far seems to be nginx-proxy by Jason Wilder. This is based on his docker-gen project that takes a template and populates values from docker containers.

It’s a good solution, but there were a few bits I didn’t like. Firstly, templates don’t really lend themselves well to every step of the process: to request Let’s Encrypt certificates, the container uses a template to create a shell script which it then sources. Each container that generates a template also needs access to the Docker socket. Both of those cause an itch in the back of my head and make me want to say phrases like “attack surface”. I don’t think there’s actually a problem, but it doesn’t really sit well with me.

Secondly, the whole system seems slightly too tightly coupled for my liking. The Let’s Encrypt component needs to modify the nginx config in order to obtain the certificate, while the main nginx component is also making different changes to add and remove sites. It feels like if it doesn’t just work, it’s going to be difficult to debug and pry apart the different components.

Another potential solution is Rancher. This is a complete platform for managing containers, and I’m fairly sure if configured right it can grab certificates from Let’s Encrypt and do SSL termination using haproxy. I tried it for a bit but the whole platform seemed a bit overkill for my purposes, and I didn’t want to invest the time I’d need to fully understand it all.

Rolling my own

In the end I decided to roll my own solution. Here’s a high-level overview of how it all works:

Diagram showing components of a reverse proxy implementation

As you probably noticed, there are quite a few containers involved. Each one performs a small, well-defined task, and its output can easily be inspected in either a volume or a database. I think there’s some similarity to piping commands together on a command line — it’s a lot easier to reason about simpler commands like head, cut and tr than it would be one giant command that combined them. And, if it does go wrong, you can inspect the pipe at each stage to see where the problem is happening.

service-reporter and etcd

The first part of the chain is my service-reporter container. This uses the Docker API to get a list of containers, and store information about them in etcd. Etcd is a distributed key-value store (similar in some ways to redis or memcached). The container also watches for containers that are added and removed, and keeps etcd updated appropriately.

As the service metadata is stored in a database, no other part of the system needs to interact with Docker. If the Docker API changes, or the host configuration changes, then only this container has to be updated.

service-letsencrypt and letsencrypt-lexicon

The left fork of the diagram deals with obtaining SSL certificates. To keep it separate from the nginx configuration, it uses DNS-based challenge to prove that we control the domains. It does this by plumbing together two great open source projects: letsencrypt.sh, a Let’s Encrypt client implemented in bash with support for the dns-01 challenge type, and Lexicon, a python library for updating DNS records using a variety of providers.

My service-letsencrypt container connects to etcd and pulls a list of containers that have a label with the key com.chameth.vhost. It uses this to build a plain text list of certificates we require (in a format understood by letsencrypt.sh), and then monitors etcd for changes and repeats as necessary.

The letsencrypt-lexicon container runs letsencrypt.sh, using Lexicon to perform the required DNS updates, and produces certificates. The nice thing about this is that it can be used in a completely standalone fashion (you can just write a domains.txt yourself). It uses iowait to watch the domains text file for updates, and automatically reruns when there are changes. It also runs once a day to renew any certs that are coming up for expiry.

service-nginx and nginx

The right fork of the diagram is concerned with nginx. My service-nginx container again connects to etcd and pulls a list of containers. It uses a couple of labels to determine the vhost, proxy port, and proxy protocol. It then feeds these values into a template to create a server block for each site, configured with SSL certificates and a reverse proxy setup. The template covers only the very minimal settings, with the expectation that everything else will be done in the global config (things such as SSL ciphers, redirection from HTTP, etc).

This container works completely independently of the Let’s Encrypt side. You can use the Let’s Encrypt containers and mount the certificate volume, or you could just provide your own certificates. It doesn’t really make any difference.

Putting it all together

The only downside to having many small containers is that it’s a bit of a nuisance to get them all set up. Fortunately, Docker has a solution for this in the form of Docker compose. This allows you to write a YAML file defining all of the services you want to run, and bring them up or down in one go. It can handle volumes, dependencies, networking, etc. I’ll be publishing a docker-compose.yml file to get this entire stack up and running soon.