<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="/feeds.xsl" type="text/xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:base="https://chameth.com/">
    <title>Chameth.com - posts like blogging-and-the-imaginary-quality-bar, docker-automatic-nginx-proxy, why-you-should-be-using-https</title>
    <subtitle>Personal homepage of Chris Smith</subtitle>
    <link href="https://chameth.com/feeds/posts/like/blogging-and-the-imaginary-quality-bar,docker-automatic-nginx-proxy,why-you-should-be-using-https/" rel="self"/>
    <link href="https://chameth.com/"/>
    <icon>https://chameth.com/favicon.png</icon>
    <updated>2026-01-29T00:00:00Z</updated>
    <id>https://chameth.com/</id>
    <author>
        <name>Chris Smith</name>
    </author>
    <entry>
        <title>An interesting Tailscale + Docker gotcha</title>
        <link href="https://chameth.com/tailscale-docker-gotcha/"/>
        <updated>2026-01-29T00:00:00Z</updated>
        <id>https://chameth.com/tailscale-docker-gotcha/</id>
        <content xml:lang="en" type="html">&lt;p&gt;As I’ve &lt;a href=&#34;https://chameth.com/how-i-use-tailscale/&#34;&gt;written&lt;/a&gt; &lt;a href=&#34;https://chameth.com/avoiding-the-consequences-of-dumb-laws-with-tailscale/&#34;&gt;about&lt;/a&gt; &lt;a href=&#34;https://chameth.com/exposing-game-servers-over-tailscale/&#34;&gt;before&lt;/a&gt;, I use &lt;a href=&#34;https://tailscale.com/&#34;&gt;Tailscale&lt;/a&gt; for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.&lt;/p&gt;
&lt;p&gt;I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an &lt;code&gt;app&lt;/code&gt; node can’t access anything via Tailscale, while an &lt;code&gt;integration&lt;/code&gt; or &lt;code&gt;server&lt;/code&gt; node can access things tagged with either &lt;code&gt;app&lt;/code&gt; or &lt;code&gt;integration&lt;/code&gt;. This is expressed pretty simply in the Tailscale ACL JSON:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tagOwners&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Servers that can be SSH&amp;#39;d into
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Applications that are exposed on tailscale but never connect out
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Things which talk to other services over tailscale (connecting to apps etc)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;grants&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Users can access everything
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;autogroup:member&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers and integrations can access integrations and apps
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Over the past week I’ve been setting up a private &lt;a href=&#34;https://forgejo.org/&#34;&gt;Forgejo&lt;/a&gt; instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (&lt;code&gt;forgejo:3000&lt;/code&gt;) instead of using full Tailscale HTTPS URL that I use when accessing it (&lt;code&gt;http://git.example-net.ts.net/&lt;/code&gt;)&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!&lt;/p&gt;
&lt;p&gt;This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my &lt;code&gt;server&lt;/code&gt; tag. That creates a &lt;code&gt;tailscale0&lt;/code&gt; interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the &lt;code&gt;tailscale0&lt;/code&gt; interface.&lt;/p&gt;
&lt;p&gt;I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the &lt;code&gt;tailscale0&lt;/code&gt; interface if it originated from the IP ranges that Docker was configured to use:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;DOCKER-USER&lt;/code&gt; chain is a nice little escape hatch; it comes before the main auto-generated &lt;code&gt;DOCKER&lt;/code&gt; chain, and Docker leaves the rules in it alone.&lt;/p&gt;
&lt;p&gt;So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS&lt;sup id=&#34;fnref:4&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:4&#34; role=&#34;doc-noteref&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;. The server’s DNS resolver is &lt;code&gt;100.100.100.100&lt;/code&gt;, a special Tailscale address. This is so that it can resolve tailnet hostnames via &lt;a href=&#34;https://tailscale.com/kb/1081/magicdns&#34;&gt;MagicDNS&lt;/a&gt; (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.&lt;/p&gt;
&lt;p&gt;Obviously the solution here is to double down and add MORE iptables rules:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use &lt;code&gt;iptables-save&lt;/code&gt;, and then you also get all the rules that both Docker and Tailscale have inserted. I came across &lt;a href=&#34;https://github.com/giangi/iptables-docker-filter&#34;&gt;a script&lt;/a&gt; to filter out the Docker ones, but… yuck&lt;sup id=&#34;fnref:5&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:5&#34; role=&#34;doc-noteref&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;I realised a cleaner idea might just be to lock down what &lt;code&gt;server&lt;/code&gt; tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to &lt;code&gt;curl&lt;/code&gt; services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for &lt;code&gt;infrastructure&lt;/code&gt;, which can be accessed from &lt;code&gt;server&lt;/code&gt; devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the &lt;code&gt;tests&lt;/code&gt; feature of Tailscale’s ACL config to make sure I’d got the rules right:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tests&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers can only access infrastructure
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;   &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;proto&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;allow&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:infrastructure:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;deny&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;100.84.16.43:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;me@example.com:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I discovered later on that logging into the container registry at &lt;code&gt;forgejo:3000&lt;/code&gt; actually issued a redirect to &lt;code&gt;https://git.example-net.ts.net/&lt;/code&gt; so this was all basically for naught… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;
&lt;p&gt;It’s always DNS… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:4&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;
&lt;p&gt;Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:5&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Blogging and the Imaginary Quality Bar</title>
        <link href="https://chameth.com/blogging-and-the-imaginary-quality-bar/"/>
        <updated>2025-09-18T00:00:00Z</updated>
        <id>https://chameth.com/blogging-and-the-imaginary-quality-bar/</id>
        <content xml:lang="en" type="html">&lt;p&gt;Recently I realised that I’ve developed a self-imposed quality bar for blog
posts. They need to be a certain length, and have a certain &lt;em&gt;substance&lt;/em&gt; to them.
They need to be generally useful in some way I can’t quite define, to some
imagined future audience. They need to have images to break up the page, and
opengraph data for when they’re linked to on social media. But… maybe they
don’t? Those things all make sense for longer “article” type posts, but not so
much for a personal blog.&lt;/p&gt;
&lt;p&gt;I’ve done some fiddling so that I can make posts without all that extra stuff.
We’ll see how it goes. The posts won’t be distinguished on the site (yet?), but
I have set up separate RSS feeds for just &lt;a href=&#34;https://chameth.com/short.xml&#34;&gt;short form posts&lt;/a&gt; and
&lt;a href=&#34;https://chameth.com/long.xml&#34;&gt;long form posts&lt;/a&gt; in case subscribers are radically opposed to one
or the other. Now I can start blogging like it’s 2005 again. But hopefully
with a bit less cringe.&lt;/p&gt;
&lt;p&gt;I’ve been staring at this post in my editor for about five minutes. The urge
to make it longer, more thorough, more article-like is really strong. This
entire paragraph is only here as a compromise with myself so I can actually
save and commit the post.&lt;/p&gt;
</content>
    </entry>
    <entry>
        <title>The Curse of Knowledge and Blogging</title>
        <link href="https://chameth.com/curse-of-knowledge-blogging/"/>
        <updated>2025-05-16T00:00:00Z</updated>
        <id>https://chameth.com/curse-of-knowledge-blogging/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/curse-of-knowledge-blogging/ideas.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/curse-of-knowledge-blogging/ideas.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/curse-of-knowledge-blogging/ideas.jpg&#34; alt=&#34;Photo of a sign reading &amp;#34;YOUR IDEAS MATTER … Write them down :)&amp;#34;&#34; loading=&#34;lazy&#34; width=&#34;653&#34; height=&#34;463&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;Finding a picture to go with a blog post about blog post ideas is hard, OK?&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;The single worst part about blogging for me is trying to come up with ideas
for what to write about. Not because they’re hard to come up with, but because
every idea seems too basic or not worth talking about. &lt;em&gt;Seems&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;It’s not just me, either. My friends will occasionally describe some interesting
problem they’ve dealt with, and I’ll chime in with “you should blog about that”.
Almost invariably the response is something like “it’s not that interesting”.&lt;/p&gt;
&lt;p&gt;There’s a cognitive bias called &lt;a href=&#34;https://en.wikipedia.org/wiki/Curse_of_knowledge&#34;&gt;the curse of knowledge&lt;/a&gt;&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.
Basically, once you know something you can’t accurately reconstruct your previous
state of mind from when you didn’t know it. You can struggle to do something
for hours, piecing together scraps of information, using all kinds of specialist
knowledge, and as soon as you’re finished it all seems so &lt;em&gt;basic&lt;/em&gt; why would
anyone want to read about that?&lt;/p&gt;
&lt;!--more--&gt;
&lt;h3 id=&#34;other-mental-obstacles&#34;&gt;Other mental obstacles&lt;/h3&gt;
&lt;p&gt;Of course, nothing involving people is as simple as having one cause and an
obvious solution. Even when I get over the “it’s too basic” road bump, there
are other issues. Looking back over some of my posts, I get a horrible cringe-y
feeling and the question of “why would anyone be interested in reading that?!”
arrives unbidden. Even though I know that &lt;em&gt;I’m&lt;/em&gt; interested in reading similar
posts written by other people. I’m hesitant to attach labels to everything, but
&lt;a href=&#34;https://en.wikipedia.org/wiki/Impostor_syndrome&#34;&gt;imposter syndrome&lt;/a&gt; says hi.&lt;/p&gt;
&lt;p&gt;I think there’s also a cultural issue at play. British people tend to be
somewhat reserved and downplay their accomplishments, especially when compared
to people from the US&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;. When I worked for a certain multinational
&lt;del&gt;ad conglomerate&lt;/del&gt; search engine company we actually had training on how to do
performance reviews and speak about ourselves more like our US counterparts.&lt;/p&gt;
&lt;p&gt;Put all these things into a cauldron, add a lizard’s leg and howlet’s wing,
mix it all up, and you get a brain that despises the idea of blogging about
anything. Great.&lt;/p&gt;
&lt;h3 id=&#34;dealing&#34;&gt;Dealing&lt;/h3&gt;
&lt;p&gt;So, now the problem is well-defined, what do we do about it? Clearly I am the
best person to be offering this advice, given it’s been four months since my
last blog post.&lt;/p&gt;
&lt;p&gt;The best counter to the Curse of Knowledge that I’ve found is keeping notes,
like a &lt;a href=&#34;https://en.wikipedia.org/wiki/Lab_notebook&#34;&gt;lab notebook&lt;/a&gt;. If you
finish something and have dozens of links to different resources that you had
to piece together, it’s probably something interesting and worth talking about
even if it doesn’t seem like it any more. Or maybe you didn’t need a lot of
references, but you had to write a lot down to make sense of something, or
needed to draw diagrams to solve a problem. Again, probably interesting.&lt;/p&gt;
&lt;p&gt;The other issues I’ve mentioned are not so easy to overcome. I’m also not going
to try and perform therapy in the form of a blog post! What I personally do is
keep a list of topics to write about (that have already passed the “probably
interesting even if my brain doesn’t agree” check), and occasionally when the
stars align I’ll feel like plucking one off the list and turning thoughts into
words.&lt;/p&gt;
&lt;h3 id=&#34;a-call-to-action&#34;&gt;A call to action&lt;/h3&gt;
&lt;p&gt;If you’ve got a blog and haven’t written anything recently because you “don’t
do anything interesting”, go write something! I promise that someone will be
interested. Hell, feel free to send me a link and &lt;em&gt;I’ll&lt;/em&gt; be interested for you.&lt;/p&gt;
&lt;p&gt;If you don’t have a blog then why not start one? There are lots of great articles
out there about why-you-should and how-you-can. And as a bonus, blog posts about
how you set up your blog are &lt;em&gt;super&lt;/em&gt; interesting. That’s one post sorted already!&lt;/p&gt;
&lt;p&gt;As the open internet is slowly subsumed by massive corporate interests, and
gummed up with spam and AI slop, the voice of real humans is becoming more and
more important. Maybe we can — one blog at a time — get back to the ’90s
internet that was full of individuality instead of today’s which is full of ads
and spam?&lt;/p&gt;
&lt;hr/&gt;
&lt;p&gt;Photo by &lt;a href=&#34;https://unsplash.com/@kommumikation&#34;&gt;Mika Baumeister&lt;/a&gt; on &lt;a href=&#34;https://unsplash.com/photos/a-sign-on-the-side-of-a-building-LaqL8nxiacc&#34;&gt;Unsplash&lt;/a&gt;&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;There’s also something called “Familiarity Blindness” that people like to
talk about. It’s a much better name. The only problem is that I can’t find any
actual published works that talk about it. No-one authoritative defines or
mentions it. It just popped into existence a few years ago. Maybe it’s an LLM
hallucination? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;My favourite commentary on this is &lt;a href=&#34;https://www.youtube.com/watch?v=8k2AbqTBxao&#34;&gt;Stephen Fry talking about American vs British comedy&lt;/a&gt;. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Docker reverse proxying, redux</title>
        <link href="https://chameth.com/docker-proxying-redux/"/>
        <updated>2022-12-06T00:00:00Z</updated>
        <id>https://chameth.com/docker-proxying-redux/</id>
        <content xml:lang="en" type="html">&lt;p&gt;Six years ago, &lt;a href=&#34;https://chameth.com/docker-automatic-nginx-proxy/&#34;&gt;I described&lt;/a&gt;
my system for configuring a reverse proxy for docker containers.
It involved six containers including a key-value store and a webserver.
Nothing in that system has persisted to this day. Don’t get me wrong – it
worked – but there were a lot of rough edges and areas for improvement.&lt;/p&gt;
&lt;h3 id=&#34;microservices-and-their-limitations&#34;&gt;Microservices and their limitations&lt;/h3&gt;
&lt;p&gt;My goal was to follow the UNIX philosophy of “do one thing and do it well”.
Unfortunately, that doesn’t really work when applied to network services that
have to interact with one and other. UNIX tools are built upon a common file
system and simple data passed over STDIN. Microservices don’t have that
shared foundation. You could make one:
companies that use microservices in anger often have a team that deals with
the “developer experience” of creating and using
microservices. But as a solo developer that’s not something I wanted to
spend my time doing.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;This became very apparent when trying to debug issues. In the UNIX world,
if your series of commands piped together does something strange you can simply
echo the data at various stages. Not so much when that data is flying around
on a network, going into and out of things like etcd. Trying to figure out why
a certificate hadn’t been acquired was a case of searching through logs from
four containers, none of which had particularly good logging.
There are many ways to get insight into what’s
happening with microservices but, again, that’s not something I wanted to do myself.&lt;/p&gt;
&lt;p&gt;Over time, and with experience in other projects, I came to realise that
microservices only really make sense if you’re unable to deploy a monolith.
For tech companies this naturally happens when different teams
contribute to the same service: splitting it out into smaller services that
are wholly owned by individual teams makes sense. For solo developers,
that never happens. You can still gain the other benefits
of microservices – such as code separation and having clearly defined APIs –
by sticking to certain coding standards.&lt;/p&gt;
&lt;h3 id=&#34;proxy-inconveniences&#34;&gt;Proxy inconveniences&lt;/h3&gt;
&lt;p&gt;As well as being unhappy with the microservice nature of the solution,
I wasn’t pleased with nginx. If you requested an unknown domain, nginx
would use the first server block in its config to serve a response, instead
of sending an “unrecognised name” alert as I wanted. It was a minor issue, but
it irked me.&lt;/p&gt;
&lt;p&gt;So from nginx I switched to haproxy. It has a &lt;code&gt;strict-sni&lt;/code&gt; option when configuring
TLS connections which makes it behave properly. It also performs a lot better for
this type of workload than nginx. All was well for a while, but then I started getting alerts
that requests were occasionally failing. I couldn’t reproduce the issue, but
my nightly jobs to build and push containers managed to hit it nearly every
night, causing them to fail.&lt;/p&gt;
&lt;p&gt;After some investigation, I found that the haproxy developers had refactored
the header parsing code, and
neglected to properly reset flags when multiple requests were sent over the same
connection. There was a patch, but it wasn’t released. No problem, I thought,
I’ll just cherry-pick it onto the last release… Except that haproxy use
Git in the most convoluted manner I’ve ever seen – they have one
repository per release. This makes it harder to patch, but it also made me question
whether I trusted them to ship stable software: there were no tests for
the header parsing code (which is both fundamental and finicky,
the perfect target for tests), the source code management was weird, and they didn’t
seem in any rush to patch this bug.&lt;/p&gt;
&lt;p&gt;Not long after that issue, &lt;a href=&#34;https://greg.holmes.name/&#34;&gt;Greg&lt;/a&gt; managed to
encounter another bug where haproxy returned a 500 error whenever the
upstream server replied with a particular, perfectly valid, header.
The die was cast – it was time to move to something else.&lt;/p&gt;
&lt;h3 id=&#34;not-invented-here-syndrome&#34;&gt;Not Invented Here syndrome&lt;/h3&gt;
&lt;p&gt;Looking for a new solution, there were many more options than
back in 2016. I’m still convinced, however, that anything
exposed to the Internet should not have access to run docker containers.
It’s the modern equivalent of running a CGI script as root. That
single requirement eliminates most off-the-shelf solutions.
What do you do when nothing quite meets
your specific requirements? You make something yourself! My new solution has two
components: &lt;a href=&#34;https://github.com/csmith/dotege&#34;&gt;Dotege&lt;/a&gt; and
&lt;a href=&#34;https://github.com/csmith/centauri&#34;&gt;Centauri&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Dotege is a replacement for the
microservices that monitored containers and obtained certificates. It’s fundamentally
a templating engine - whenever the containers change, it evaluates a
template and saves the result to disk. The template has access to details about
the containers, their labels, ports, and so on. Dotege can also obtain certificates
from Let’s Encrypt, and raise a signal against another
process whenever the template or certificates change. I used this to generate
the configuration and certificates used by haproxy for a while, and more recently
changed the template so that it works for Centauri.&lt;/p&gt;
&lt;p&gt;Centauri is my own reverse proxy. It’s configured using a simple text file
and can also obtain certificates from an ACME provider. It doesn’t serve static
content, has no knowledge about docker, and avoids the other bells and
whistles that adorn most reverse proxies. It also has good test
coverage to ensure that I don’t, say, accidentally break header parsing.&lt;/p&gt;
&lt;p&gt;As a software engineer I enjoy writing software, but I also enjoy running
simple, easy to understand software. That’s what I’ve achieved here: it’s
very easy to identify where the problem is if anything goes wrong, both are small
Go programs rather than vast sprawling C
monstrosities, and their interaction is primarily through a file written to disk
that can be inspected or edited as needed.&lt;/p&gt;
</content>
    </entry>
    <entry>
        <title>An introduction to containers</title>
        <link href="https://chameth.com/intro-to-containers/"/>
        <updated>2020-03-01T00:00:00Z</updated>
        <id>https://chameth.com/intro-to-containers/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/intro-to-containers/containers.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/intro-to-containers/containers.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/intro-to-containers/containers.jpg&#34; alt=&#34;Containers in port&#34; loading=&#34;lazy&#34; width=&#34;300&#34; height=&#34;396&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;So. Many. Containers.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;I’m a huge fan of (software) containers. Most people I know fall in to one of two camps:
either they also use, and are fans of, containers, or they haven’t yet really figured them
out and view them as some kind of voodoo that they don’t really want or need.&lt;/p&gt;
&lt;p&gt;I’m writing this short guide to explain a little how containers work - and how running
something in a container isn’t really that much different to running it normally -
to hopefully enable more people in that second group to give them a try. It’s aimed at
people who have a fairly good grasp of how Linux works.&lt;/p&gt;
&lt;p&gt;Containers are often mentioned in the same breath as VMs, which is not a helpful comparison
or analogy. Think of containers as standard units of software, much like how
&lt;a href=&#34;https://en.wikipedia.org/wiki/Intermodal_container&#34;&gt;Intermodal containers&lt;/a&gt; are standard
units of freight transport across the world. When a company internationally ships goods in
volume there isn’t a question about how they’re packaged - they go in an intermodal container.
The same container can be deployed on a freight train, a lorry, or a ship. The haulage company
doesn’t need to care what’s in the container because they’re completely standardised.
Likewise, with software containers you don’t really need to care about what’s inside: the
software you’re deploying could be written in Go, Python2, Python3, Bash, PHP, LOLCODE, or
anything&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;!--more--&gt;
&lt;h3 id=&#34;what-does-a-running-container-look-like&#34;&gt;What does a running container look like?&lt;/h3&gt;
&lt;p&gt;When you run a container, you’re just running a process. In a lot of ways it’s not
very different to what happens if you were to start the same process on the host
computer.&lt;/p&gt;
&lt;p&gt;For example I have a container that just runs &lt;code&gt;cat(1)&lt;/code&gt;. With no arguments, &lt;code&gt;cat&lt;/code&gt;
will read from stdin until it receives an EOF, so it’s handy to test with.
If I run &lt;code&gt;ps a&lt;/code&gt; on my computer, I can see the cat process in amongst everything
else I’m currently running:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;    PID TTY      STAT   TIME COMMAND
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;   7199 pts/1    Ss     0:01 /usr/bin/zsh
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt; 323806 pts/0    Ss+    0:00 /bin/cat
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt; 324120 pts/4    R+     0:00 ps a
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;/bin/cat&lt;/code&gt; process is in a container, and the &lt;code&gt;ps a&lt;/code&gt; underneath it is just running
like normal on my desktop. They look very similar, right? If I look under &lt;code&gt;/proc/323806&lt;/code&gt;
I can see all the usual attributes, the same as any other process running.&lt;/p&gt;
&lt;p&gt;If I run &lt;code&gt;ps&lt;/code&gt; in a container, though, it’s a different story:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;    PID TTY      STAT   TIME COMMAND
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;      1 pts/0    Rs+    0:00 ps a
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So inside the container it looks like there’s only one process running. It can’t see
anything running “outside” on my desktop. The secret here is that this isn’t a special
container trick: this is just a feature of the Linux kernel called &lt;em&gt;namespacing&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;If we go back to procfs and look at the &lt;code&gt;ns/pid&lt;/code&gt; node we can see the process in the
container is in a separate PID (process ID) namespace to the one on my desktop:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;# readlink /proc/323806/ns/pid  
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;pid:[4026534564]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;# readlink /proc/7199/ns/pid
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;pid:[4026531836]
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Almost all&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt; processes running ’normally’ on my desktop have the same PID namespace,
whereas each container gets their own by default. PID namespaces are hierarchical:
a new process is assigned a PID in its own namespace, and the parent namespace, and
the grandparent namespace, and so on. That’s why I can see the process running in
the container from my normal shell - the container’s namespace is a child of the
main namespace all of my desktop software is running in.&lt;/p&gt;
&lt;p&gt;Linux supports - and container software makes use of - a bunch of other namespaces too:
mount points, network, UTS&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;, cgroups, and more. These all play a part in isolating
a container from the system it is running on.&lt;/p&gt;
&lt;p&gt;You can manually run a process with &lt;code&gt;unshare(1)&lt;/code&gt; to “unshare” some namespaces from
the parent process. For example if I run &lt;code&gt;unshare -fp --mount-proc ps a&lt;/code&gt;, it
looks very similar to running ps instead the container:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;    PID TTY      STAT   TIME COMMAND
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;      1 pts/4    R+     0:00 ps a
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So: a process running inside a container is just a heavily namespaced process running
otherwise normally in the operating system. No voodoo magic here!&lt;/p&gt;
&lt;h3 id=&#34;what-about-the-filesystem-what-are-images&#34;&gt;What about the filesystem? What are ‘images’?&lt;/h3&gt;
&lt;p&gt;Containers run in their own mount namespace meaning mount points can be different
inside the container to those on the host. This means the container can have a different
&lt;code&gt;/&lt;/code&gt; mounted to the host, effectively giving it its own filesystem.&lt;/p&gt;
&lt;p&gt;The root filesystem of the container is defined in the container’s image. If I
use Docker to run a container using the Ubuntu image (&lt;code&gt;docker run ubuntu&lt;/code&gt;), the root
filesystem inside that container will resemble a minimal ubuntu install. Note
that this is just the filesystem: the container doesn’t have its own kernel.&lt;/p&gt;
&lt;p&gt;You might be thinking that sounds pretty inefficient. Downloading Ubuntu is definitely
not instant, and doing it for every application you run would be insane! Quite.
Containers solve this by using filesystem &lt;strong&gt;layers&lt;/strong&gt;. These are stacked on top of
one another to create the final filesystem. Each layer can be retrieved and
cached independently of all others.&lt;/p&gt;
&lt;p&gt;Say (for simplicity) that the Ubuntu image is a single layer. If I run one container
with that image, then the layer will be downloaded and cached once. If I run three
hundred containers with that image then the layer will be downloaded and cached once.
Even better, if I use another image that’s based on Ubuntu but adds some software
on top, only the “on top” layer will be downloaded if I already have the relevant
“Ubuntu” layer cached.&lt;/p&gt;
&lt;p&gt;If all the layers are cached, what happens when you change a file? This is dealt
with using the copy-on-write technique: when you modify a file it is copied from
the source layer and the changes are only made in a new layer. This is handled
by the OverlayFS filesystem which is part of the mainline kernel.&lt;/p&gt;
&lt;p&gt;When a container is running, changes made to its filesystem are temporary, and
do not persist across container restarts. To persist data - or introduce new
data to a container - you can mount volumes. How this works varies depending
on how you’re running your container, but at the basic level it is pretty much
the same as bind-mounting (&lt;code&gt;mount -o bind&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;You may be familiar with using &lt;code&gt;chroots&lt;/code&gt; to change the apparent root directory
of processes, perhaps with full-blown “jails” built on top. Containers offer
much better isolation thanks to the use of namespaces. Instead of being
constrained to a portion of the host’s filesystem, they don’t even have
it mounted! Containers also get to specify their environment - if they
expect in certain places, for example - instead of the sysadmin having to
manually set up the chroot. Finally, containers offer much more fine-grained
control over what processes can do (if you want it), and allow much more
advanced use-cases such as inter-container networking.&lt;/p&gt;
&lt;p&gt;Images and filesystems employ a little magic to ensure that layers are reusable
and cacheable, but again there’s nothing terribly special about them: a container
has a filesystem that appears to it to work the same way as a filesystem on
the host, and it’s using a standard filesystem shipped with the kernel.&lt;/p&gt;
&lt;h3 id=&#34;how-about-networking&#34;&gt;How about networking?&lt;/h3&gt;
&lt;p&gt;Again, networking is namespaced, so a container has its own network stack,
its own virtual network interface, its own IP address and so on. How that network
interacts with your real network depends on how you’re running the container.
Docker, for example, can add iptables rules to NAT traffic between containers&amp;#39;
networks and the outside world.&lt;/p&gt;
&lt;p&gt;Containers can generally be connected into networks, and can communicate
amongst themselves without the traffic actually leaving the host machine.
This allows you to, for example, run a SQL database and connect it to
a web application without ever exposing the database to the outside world.
Moreover, as well as being isolated from the outside world, it’s isolated
from other containers in other networks. If one of your applications has a crazy bug
or is compromised, this significantly limits the damage it can do.&lt;/p&gt;
&lt;p&gt;You have to explicitly opt in to “publishing” ports from a container, which
exposes them to the outside world (either directly, or via a load balancer
or some other middle-man, depending on how you’re running the container).
This means you can pick and chose how the outside world sees the app you’re
deploying: if it’s a web service that listens on both port 443 and port 80,
you can chose to only expose the encrypted port.&lt;/p&gt;
&lt;p&gt;If you run some containers and create some networks, you can see the
interfaces and bridges on the host:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;$ ip l
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;...
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;6: br-2405a8cc0445: &amp;lt;BROADCAST,MULTICAST,UP,LOWER_UP&amp;gt; mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;    link/ether 02:42:3e:fa:23:62 brd ff:ff:ff:ff:ff:ff
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;8: veth8ed0735@if7: &amp;lt;BROADCAST,MULTICAST,UP,LOWER_UP&amp;gt; mtu 1500 qdisc noqueue master br-2405a8cc0445 state UP mode DEFAULT group default 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;    link/ether b2:c1:5d:55:26:9b brd ff:ff:ff:ff:ff:ff link-netnsid 2
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;10: veth541d84b@if9: &amp;lt;BROADCAST,MULTICAST,UP,LOWER_UP&amp;gt; mtu 1500 qdisc noqueue master br-9d7bc4024c1a state UP mode DEFAULT group default 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;    link/ether 86:aa:5f:ee:da:1a brd ff:ff:ff:ff:ff:ff link-netnsid 1
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This shows a bridge and two virtual NICs, just the same as if you’d manually created
them (albeit with seemingly random names). So: as before, nothing special here.&lt;/p&gt;
&lt;h3 id=&#34;docker-compose-k8s-floccinaucinihilipilification&#34;&gt;Docker? Compose? K8s? Floccinaucinihilipilification?&lt;/h3&gt;
&lt;p&gt;(OK, Floccinaucinihilipilification isn’t actually a container technology, as far
as I’m aware.)&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Docker&lt;/strong&gt; is the most popular &lt;em&gt;container engine&lt;/em&gt; - that is, the bit of software
that actually runs containers. It’s responsible for setting up all those namespaces
we found, downloading and caching image layers, and actually starting and stopping
the processes. Docker runs as a system-wide daemon - when you run a command like
&lt;code&gt;docker run ubuntu&lt;/code&gt; it actually just instructs the daemon to do the work.&lt;/p&gt;
&lt;p&gt;There are several alternatives to Docker for running containers; one interesting one
is &lt;a href=&#34;https://podman.io/&#34;&gt;Podman&lt;/a&gt; which runs containers without a daemon. Container
engines have all standardised around the same image format looked after by the
&lt;a href=&#34;https://www.opencontainers.org/&#34;&gt;Open Container Initiative&lt;/a&gt;, so an image you build
in Docker can be used in Podman, or pretty much any other engine.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Docker compose&lt;/strong&gt; is a tool for defining and running multiple-container applications.
I mentioned earlier running a database alongside a webapp - in practice to do this
you’re going to have to configure a network for them, configure a mount point for
the database to persist its data on, pass credentials in to both the database and
the application, and so on. Doing all that by hand is tedious and error prone.&lt;/p&gt;
&lt;p&gt;Docker compose lets you write “compose files”, which are simple yaml descriptions
of the containers you wish to run, their properties, and details about any volumes
or networks you may want. Out of the box, docker-compose will create a default
network for each compose file you run so the containers within it can communicate.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Kubernetes&lt;/strong&gt;  is a container &lt;em&gt;orchestrator&lt;/em&gt;, designed to automate deployment
and management of large numbers of containers. It works with Docker under the
hood, but provides a huge amount of tooling on top to allow you to deploy
applications and manage their dependencies. It runs across multiple physical
(or virtual) machines (while still allowing containers to communicate privately),
and can support massive workloads by scaling out services (running multiple copies
of a container on different hosts) and load balancing. Kubernetes is sometimes
shorted to &lt;strong&gt;k8s&lt;/strong&gt; (as in &lt;code&gt;K&lt;/code&gt; - 8 elided letters - &lt;code&gt;s&lt;/code&gt;) because computer people
don’t like long words.&lt;/p&gt;
&lt;h3 id=&#34;ok-they-make-sense-now-but-why-bother&#34;&gt;OK, they make sense now. But why bother?&lt;/h3&gt;
&lt;p&gt;Hopefully if you’ve read this far you’ve already picked up on some of the potential
benefits, but this is my personal list:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Isolation&lt;/strong&gt;. If I run some software in a container, there is very little it
can do to upset me. It’s not going to leave bits of itself all over my filesystem,
it can’t steal all of the secrets in my home directory, I can even limit its CPU
and memory resources if I want. If I decide to stop running it, I just delete
the container and it is completely gone: no trace remains.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ease of use&lt;/strong&gt;. If you give me a container image I have a very good idea of how
to run it already. I might need to do some minor configuration to expose ports
or mount volumes, but there’s no question about how to run it, how to make it
automatically start, and there’s no “installation” procedure. If I want to
then swap it with an alternative (say, move from MySQL to MariaDB), it’s
potentially just a case of changing the name of the image I pull.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Dependencies included&lt;/strong&gt;. Containers just &lt;em&gt;run&lt;/em&gt;. Python 2 software includes
Python 2 and just work. Python 3 software includes Python 3 and just work.
I don’t have a massive headache trying to run both at the same time, because
they take care of their own messes. Similarly I’m not going to have to install
&lt;code&gt;npm&lt;/code&gt; or &lt;code&gt;cargo&lt;/code&gt; or &lt;code&gt;composer&lt;/code&gt; to pull in dependencies for an application:
that’s going to have been done in the build process.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Reproducibility&lt;/strong&gt;. As a fallout from having dependencies included and being
isolated from everything else, containers give you amazing reproducibility.
If it “works on your machine” in a container, it’ll almost certainly work in
production because it’s the exact same environment.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Standardisation&lt;/strong&gt;. At the start of this article I called containers standard
units of software. One of my favourite advantages of containers is that you
basically get an API to list all the software you’re running. Most container
engines let you supply labels attached to containers as well, so you can add
your own annotations. I use this to annotate services which expose HTTP
endpoints, and I have a tool that automatically generates SSL certificates
for them and configures haproxy to route traffic to them. I can’t imagine
how I’d do this without containers - I imagine it’d involve a lot of
manual work.&lt;/p&gt;
&lt;p&gt;Hopefully this has helped demystify containers a little. If you feel like
I’ve missed something important out, or I’ve left you more confused than
when you started, feel free to drop me a note using the feedback form below.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;OK, maybe you should care if you’re deploying something written in crazy languages like PHP. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;Some multi-process apps, such as web browsers, are starting to use namespaces to enhance security,
as do certain package systems like Flatpak &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;“Unix timesharing system”; in practice this means having a separate hostname &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
</feed>
