Chameth.com - posts like docker-automatic-nginx-proxy, modern-css-is-fun but not intro-to-containers, why-you-should-be-using-https

Monthly Meanderings: March 2026

2026-04-01T00:00:00Z

Since last month’s update I’ve been unpleasantly reminded that I’m middle aged, through the medium of a dodgy knee. There’s nothing quite like not being able to stand up without groaning to underscore that you’re not young any more. The ongoing game of “will this seemingly mundane activity make my knee go funny again?” is a delight to play, too. I’m great at it.

Website updates

Two new blog posts this month: the longest way to represent a date is a short thought experiment on absurd date formats, and modern CSS is fun goes through some neat new CSS features I’ve used recently for this site.

I did some design tweaks this month: headers now have a blue background behind them¹, and a lot of components have been redesigned to have a consistent style. You can see that component in action in the new “now playing” widget I’ve added:

Alyssa Reid — Talk Me Down The Game Scrobbled 22m ago

Instead of each component defining its own styles (which were mostly copied and pasted), there’s now a single class that handles the drop shadow, border, background colour, and the optional title. It also applies a consistent format to elements that are clickable: they use the blue accent colour for their borders, and have a glow effect shown on hover.

The now playing widget is part of a larger bit of work I did on importing some music stats. I’m pulling in the data from my Navidrome instance. A new music page shows my most listened albums and artists. Navidrome doesn’t store a complete play history, just the last play and the count, so I can’t do “what I listened to last month” stats until next month.

Other projects

Other than a minor bug-fix update to contempt, my Dockerfile templating/updating tool, I’ve not done much on my other open source projects. I’ve got a bit more work to do finish migrating all my repositories from GitHub to my private Forgejo instance. I’ve also been thinking about mirroring my public repositories to Codeberg so there’s a non-GitHub way to access them.

Entertainment

I apparently only watched a single film in March:

The Substance

2026-03-18

I really didn’t get on with this. It’s too long for what it is, and over-the-top in a way that just didn’t work for me.

Demi Moore and Margaret Qualley were great, but nothing else really was. The cinematography and sound design were too affected, the plot was pretty shallow and obvious, most of the body horror fell into the uncanny CGI valley for me.

It’s not completely without merit: there were around 40 minutes in the middle where I was really into it, but that’s not really enough given it’s closing in on a 2.5 hour runtime.

There’s something to be said for the message it’s presenting about women in Hollywood, too, but it was very heavy handed. Dennis Quaid’s character is even called Harvey; it’s not so much a subtle nod as a sledgehammer to the face.

Instead of films, I’ve been watching a bunch of TV. After devouring The Pitt last month, I struggled to find something to fill the hole it left. I settled on catching up on Chicago Fire and Chicago P.D.. They’re both mostly about getting things done without personal drama becoming the primary focus. They’re not quite as grounded as The Pitt, but at least they’ve not completely gone off the deep end like 9-1-1 where the first responders end up in space somehow?

I’ve also watched the first two episodes of SNL UK. I didn’t have very high expectations, and it gave me a pleasant surprise. Both episodes so far have some hilarious sketches, and are obviously written by British comics with the typical darker, more sardonic humour that separates us from the USA.

Last but not least, board games! I spent a weekend at a tabletop/LAN event with friends, so got to physically play some games for a change. I also kept up with some turn-based games on Board Game Arena, as usual.

6 plays
4 plays
4 plays
4 plays
3 plays
3 plays
3 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play

Around the web

Warranty Void if Regenerated

Some really cool speculative fiction about a future where all software is written by LLMs, and the social effects that may have. Unfortunately the story itself is written with an LLM, and that becomes painfully obvious about a third of the way in. By that point I was already hooked, though.

My 2-step process for AI-free blogging

Speaking of LLM-generated text, this short piece by Manuel Moreale made me laugh. I’m by no means anti-LLM, but I really hate people publishing the output like it’s their own work. It breaks the social assumption that the writer put more effort in than the reader, and makes me wonder how much is hallucinated.

25 Years of Eggs

An interesting applied use of coding agents. I’m a sucker for personal statistics. I have no explicit desire to keep my receipts, or generate graphs of my egg consumption, but seeing the graphs makes me question that. The journey to get there is a great read, too.

I made a one-page notebook

The clever paper folding is interesting, but I really like some of the small touches on the page. There’s a nice little “in reply to” callout providing context at the top of the page, you can click to rotate the image further down, and there’s a details element containing a textual version of it. It has that great ‘small web’ vibe of someone who really cares about what they’re doing.

Workers who love ‘synergizing paradigms’ might be bad at their jobs

I haven’t read the full paper, but the summary given by the article is equal parts interesting and amusing. They developed a “Corporate Bullshit Receptivity Scale”, what more is there to say?

It’s meant to resemble painter’s tape, but I’m not sure quite how well it works. ↩︎

Modern CSS is fun

2026-03-17T00:00:00Z

I’ve been doing a bunch of CSS tweaking recently, and keep being surprised by how nice modern CSS is to work with. As someone grey-haired enough to remember writing HTML without CSS, it’s amazing to think how far along web technology has come¹. I wanted to demonstrate some of the handy bits and pieces I’ve used recently.

:has

The :has selector allows you to effectively query for child elements. While a span will match a <span> within an <a>, a:has(span) will match an <a> that contains a <span>. This really shines when combined with more complex selectors, for example:

input {
    border-radius: var(--border-radius);

    &:has(+ .results:not(:empty)) {
        border-bottom-left-radius: 0;
        border-bottom-right-radius: 0;
    }
}

This is part of the styling for my film search box. When the search box is presented initially it has fully rounded corners. But when it is followed by a non-empty results element, it removes the rounding on the bottom corners so that the border continues in a straight line down into the results. You can see it in action below; just enter a few characters (like “the”) to get some results:

You could achieve the same effect by having JavaScript add a class to the <input> element, but I’ll take a CSS solution over a JavaScript solution any day.

Nested rules

You probably spotted this in the example above. It’s what finally made me switch from SCSS to plain CSS. If you have a rule for .foo and a rule for .foo .bar you can just nest them. Not only does it save repeating yourself an awful lot, it keeps everything organised nicely. For example, the CSS for my film list embeds looks like this:

a.film-list {
    display: grid;
    grid-template-areas: "header images" "description images" "meta images";
    grid-template-columns: 1fr auto;
    grid-template-rows: auto 1fr auto;
    row-gap: var(--small-space);

    @media (width <= 800px) {
        grid-template-areas: "header" "description" "images" "meta";
        grid-template-columns: 1fr;
        grid-template-rows: auto auto auto auto;
    }

    & + & {
        margin-top: var(--medium-space);
    }

    h3 {
        font-size: var(--font-size-xxlarge);
    }

    .description {
        /* ... */
    }
}

Everything is wrapped up in a single bundle, including the @media rules for changing the layout on smaller screens, the lovely little & + & rule that adds some extra margin if there are two lists in a row, etc. The & syntax refers to the parent selector, so & + & in this case is the same as a.film-list + a.film-list: it’s very handy! The best part about this is that it’s the exact same syntax as SCSS, so converting is easy². The styled element looks like this:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Media range syntax

Once again, you might have spotted this in the previous snippet. I’m not deliberately teasing things, I promise! Back in the day, you did media queries like so:

@media (min-width: 1000px) and (max-width: 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I always hated this syntax. I always got muddled up as to whether I wanted “min” or “max”, and whether they were inclusive or not. It’s easy to reason through, but it never came naturally. Fortunately you can now just use ranges:

@media (1000px <= width <= 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I find this style so much easier to write and understand.

Anchor positioning

Trying to dynamically position one element next to another used to exclusively fall within the purview of JavaScript. Fortunately anchor positioning fixes all this. I use this in the film search field I showed above:

.film-search {
    input {
        anchor-name: --film-search-box;
    }
    .results {
        position: absolute;
        position-anchor: --film-search-box;
        position-area: bottom center;
        position-visibility: always;
        width: anchor-size();
    }
}

This does lots of fun things. The .results element is anchored to the input field (via the --film-search-box name), and it’s positioned on the bottom, directly under the anchor. The position-area rule describes a 3x3 grid, with the anchor at the center, so top left would put it diagonally adjacent to the anchor. Next, position-visibility keeps the .results element visible when the anchor isn’t; otherwise when you scroll the anchor off the screen the results would immediately vanish. Finally, the special anchor-size() function makes the .results element take up the same exact width as the anchor.

All-in-all, this puts the results exactly where I want them, without having to deal with any JavaScript at all. I can see how it’d also be amazingly useful trying to do tooltips or other forms of ‘floating’ content too. You can even leave the positioning up to the browser, giving it hints about which order to try, or what property to optimise for (e.g. you can say “anchor to the left or right, whichever has more horizontal space”).

attr() function

This one blew my mind a little. In some of the elements on the site, I’ve added a little label to the top. It looks something like this:

It’s meant to just be a visual flourish, not a semantic title or anything, so I initially just added a ::before selector to each element, customising the content to have the right value. I didn’t like the duplication, though. Now, instead, I do this:

&::before {
    content: attr(data-title);
}

Each component defines its title in a data attribute: <div class="raised-box" data-title="an example">, and the attr function plucks it out and puts it in the content rule. It’s worth noting that this is only widely supported for the content rule, you can’t yet use it for colours or dimensions or other things. It still feels a bit magical, though. Like you’re making a reusable, customisable component with just HTML and CSS.

layers

I’d come across layers a few times, but I never found a need for them. It seemed like something you’d only really need with complicated design systems, or something. Then I had a problem and layers were the perfect solution!

The headings on this site have a whole bunch of CSS attached to them. Currently they look a bit like blue painter’s tape with handwriting on them. This style is applied to all headers, and then the various places that don’t want it had to manually reset everything. I got fed up with that, so decided to add a plain-header class which would “disable” the extra styling:

h2,
h3,
h4,
h5,
h6 {
    /* common styles for all headers, regardless */
    font-family: "Chris Hand", sans-serif;
    font-size: var(--font-size-xxlarge);
    color: var(--text-highlight-colour);
    margin: var(--medium-space) 0;
    line-height: 1;

    &:not(.plain-header) {
        /* do fancy stuff if it _doesn't_ have the .plain-header class */
        transform: rotate(-0.15deg);
        padding: var(--smedium-space);
        font-weight: 800;
        filter: drop-shadow(3px 3px 5px rgb(0, 0, 0));

        a {
            background: none;
            text-decoration-skip-ink: auto;
        }
    }
}

This might seem a bit backwards but I want the default to be styled, as those are far more common. I added this change, sprinkled around some “plain-header” classes, tidied up a lot of duplicate CSS, and things mostly worked. Mostly. Some things stopped working. In a few places the headers had slight tweaks, and they stopped working entirely.

It turns out in moving the rules from a plain h2 selector to a h2:not(.plain-header), I’d made them more specific. Previously a selector like h2.special would have been more specific, so its properties would override those from the less specific h2 selector. My first reaction was to try and hack around it. Changing all the overrides to body h2.special would make them more specific again, but I didn’t want to have to remember to do that forever more.

Instead, I defined some custom layers, and put the header definitions in one:

@layer reset, links, headings

@layer headings {
    h2,
    h3 {
        /* etc */
    }
}

The first @layer rule defines some layers that will be handled in a specific order: first the “reset” layer, then the “links” layer, then the “headings” layer, and then anything not in a layer after that. Rules in one layer don’t have to worry about beating the specificity of rules in another layer, because they’re handled separately.

Note that I couldn’t just put the headings in a layer and call it a day: their rules have to come after the CSS reset, and after the normal link styles, as headers have a bit of extra styling for nested links. If the CSS reset weren’t in a layer, then the * { margin: 0; padding: 0; } type reset would apply over the rules from the headings layer.

Using layers is definitely a bit fiddly, and is not necessary for a lot of sites, but it’s a much cleaner alternative to ugly specificity hacking.

Things I’m looking forward to

There are a few things that aren’t (widely) available yet, that particularly interest me:

sibling-index() and sibling-count()

These new functions are available in WebKit and Blink based browsers, but not Firefox. They give you the index of the element within its siblings, or the count of siblings. For the film list component above, I currently have this abomination for the overlapping posters:

&:nth-child(1) {
    z-index: 5;
    left: 0;
}
&:nth-child(2) {
    z-index: 4;
    left: calc(133px * var(--overlap) * 1);
}
&:nth-child(3) {
    z-index: 3;
    left: calc(133px * var(--overlap) * 2);
}
&:nth-child(4) {
    z-index: 2;
    left: calc(133px * var(--overlap) * 3);
}
&:nth-child(5) {
    z-index: 1;
    left: calc(133px * var(--overlap) * 4);
}

These could be replaced with something like:

z-index: calc(1 + sibling-count() - sibling-index());
left: calc(133px * var(--overlap) * (sibling-index() - 1));

As an added bonus, it would scale to any number of posters. index being 1-based not 0-based is a bit unfortunate as it makes using it in calculations awkward (see 1 offset in both of those rules!), but it’s leagues better than writing 5 separate rules.

random()

Currently only available in Safari. There are a few places where I’d like to have slight random variations of the style. Things like elements that are rotated slightly for aesthetics; they look a bit silly if they’re all identically positioned. I also randomise the icons and positions of my rating stars to break up the visual monotony. Here are a few examples:

Currently I do the stars by hardcoding a bunch of classes and having the backend randomly apply one when it generates the markup for stars. Being able to do it in CSS would be great, though.

Mixins

This one’s so far off it’s not even listed on caniuse.com yet. Mixins are another handy feature of SCSS, that allow you to define reusable blocks of rules, then import them when needed:

@mixin fancy-background {
    background: /*...*/;
}

.some-element {
    @include fancy-background;
}

#other-element {
    @include fancy-background;
}

This would be particularly useful if you can’t easily control the markup to add classes to everything. For example, my footnotes are rendered by a markdown plugin, and don’t easily have a way to add extra classes to them. To style it the same as another element, I currently duplicate a bunch of rules between them both. Mixins would allow me to define those rules once, and then import them in both places.

Aside from JavaScript, which seems to just endlessly reinvent new frameworks and ways to make the most sprawling dependency tree possible. ↩︎
It also means that if your website’s syntax highlighting library doesn’t seem to understand nested rules, you can just pretend your perfectly valid CSS is actually SCSS and it’ll magically work. Grumble, grumble. ↩︎

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.

Automatic reverse proxying with Docker and nginx

2016-05-21T00:00:00Z

The Docker project logo

Over the past few weeks I’ve gradually been migrating services from running in LXC containers to Docker containers. It takes a while to get into the right mindset for Docker - thinking of containers as basically immutable - especially when you’re coming from a background of running things without containers, or in “full” VM-like containers. Once you’ve got your head around that, though, it opens up a lot of opportunities: Docker doesn’t just provide a container platform, it turns software into discrete units with a defined interface.

With all of your software suddenly having a common interface, it becomes trivial to automate a lot of things that would be tedious or complicated otherwise. You don’t need to manage port forwards because the containers just declare their ports, for example. You can also apply labels to the application containers, and then query the labels through Docker’s API.

Reverse proxying and SSL termination with Nginx and Let’s Encrypt

A fairly significant chunk of the software I run has a web interface. I don’t really want to expose and remember dozens of non-standard ports, so I configure an nginx instance as a reverse proxy. I’m of the opinion that all web traffic should be encrypted, so I also have to provide nginx with trusted certificates to use for each site it reverse proxies. Let’s Encrypt makes the process of obtaining free, trusted certificates approximately a thousand times easier than it was previously, but my workflow still ends up looking like this:

Create a new config file from a template and save it in /etc/nginx/sites-available
Temporarily disable SSL for the site as there’s no valid certificate yet
Enable the site by symlinking to it from /etc/nginx/sites-enabled
Reload nginx
Run the Let’s Encrypt client to obtain certificates
Enable SSL and for the site
Reload nginx

… And that’s not including the extra steps when I miss a semi-colon, accidentally skip a step and have to spend time figuring out why it’s not working, or any of the other human-induced problems that creep in.

I’d been toying with making a script to run through these steps manually for me, if I gave it a domain name and a reverse proxy target, but I never got around to it. Now I’m moving things to Docker, though, there’s an opportunity to automate the entire thing with no human interaction at all.

Existing solutions

It seemed like this probably wasn’t a unique idea, so I had a look around for existing solutions. The most popular by far seems to be nginx-proxy by Jason Wilder. This is based on his docker-gen project that takes a template and populates values from docker containers.

It’s a good solution, but there were a few bits I didn’t like. Firstly, templates don’t really lend themselves well to every step of the process: to request Let’s Encrypt certificates, the container uses a template to create a shell script which it then sources. Each container that generates a template also needs access to the Docker socket. Both of those cause an itch in the back of my head and make me want to say phrases like “attack surface”. I don’t think there’s actually a problem, but it doesn’t really sit well with me.

Secondly, the whole system seems slightly too tightly coupled for my liking. The Let’s Encrypt component needs to modify the nginx config in order to obtain the certificate, while the main nginx component is also making different changes to add and remove sites. It feels like if it doesn’t just work, it’s going to be difficult to debug and pry apart the different components.

Another potential solution is Rancher. This is a complete platform for managing containers, and I’m fairly sure if configured right it can grab certificates from Let’s Encrypt and do SSL termination using haproxy. I tried it for a bit but the whole platform seemed a bit overkill for my purposes, and I didn’t want to invest the time I’d need to fully understand it all.

Rolling my own

In the end I decided to roll my own solution. Here’s a high-level overview of how it all works:

Diagram showing components of a reverse proxy implementation

As you probably noticed, there are quite a few containers involved. Each one performs a small, well-defined task, and its output can easily be inspected in either a volume or a database. I think there’s some similarity to piping commands together on a command line — it’s a lot easier to reason about simpler commands like head, cut and tr than it would be one giant command that combined them. And, if it does go wrong, you can inspect the pipe at each stage to see where the problem is happening.

service-reporter and etcd

The first part of the chain is my service-reporter container. This uses the Docker API to get a list of containers, and store information about them in etcd. Etcd is a distributed key-value store (similar in some ways to redis or memcached). The container also watches for containers that are added and removed, and keeps etcd updated appropriately.

As the service metadata is stored in a database, no other part of the system needs to interact with Docker. If the Docker API changes, or the host configuration changes, then only this container has to be updated.

service-letsencrypt and letsencrypt-lexicon

The left fork of the diagram deals with obtaining SSL certificates. To keep it separate from the nginx configuration, it uses DNS-based challenge to prove that we control the domains. It does this by plumbing together two great open source projects: letsencrypt.sh, a Let’s Encrypt client implemented in bash with support for the dns-01 challenge type, and Lexicon, a python library for updating DNS records using a variety of providers.

My service-letsencrypt container connects to etcd and pulls a list of containers that have a label with the key com.chameth.vhost. It uses this to build a plain text list of certificates we require (in a format understood by letsencrypt.sh), and then monitors etcd for changes and repeats as necessary.

The letsencrypt-lexicon container runs letsencrypt.sh, using Lexicon to perform the required DNS updates, and produces certificates. The nice thing about this is that it can be used in a completely standalone fashion (you can just write a domains.txt yourself). It uses iowait to watch the domains text file for updates, and automatically reruns when there are changes. It also runs once a day to renew any certs that are coming up for expiry.

service-nginx and nginx

The right fork of the diagram is concerned with nginx. My service-nginx container again connects to etcd and pulls a list of containers. It uses a couple of labels to determine the vhost, proxy port, and proxy protocol. It then feeds these values into a template to create a server block for each site, configured with SSL certificates and a reverse proxy setup. The template covers only the very minimal settings, with the expectation that everything else will be done in the global config (things such as SSL ciphers, redirection from HTTP, etc).

This container works completely independently of the Let’s Encrypt side. You can use the Let’s Encrypt containers and mount the certificate volume, or you could just provide your own certificates. It doesn’t really make any difference.

Putting it all together

The only downside to having many small containers is that it’s a bit of a nuisance to get them all set up. Fortunately, Docker has a solution for this in the form of Docker compose. This allows you to write a YAML file defining all of the services you want to run, and bring them up or down in one go. It can handle volumes, dependencies, networking, etc. I’ll be publishing a docker-compose.yml file to get this entire stack up and running soon.