<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="/feeds.xsl" type="text/xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:base="https://chameth.com/">
    <title>Chameth.com - posts like docker-automatic-nginx-proxy, if-all-you-have-is-a-hammer, thinking-more-about-backups, why-you-should-be-using-https</title>
    <subtitle>Personal homepage of Chris Smith</subtitle>
    <link href="https://chameth.com/feeds/posts/like/docker-automatic-nginx-proxy,if-all-you-have-is-a-hammer,thinking-more-about-backups,why-you-should-be-using-https/" rel="self"/>
    <link href="https://chameth.com/"/>
    <icon>https://chameth.com/favicon.png</icon>
    <updated>2026-01-29T00:00:00Z</updated>
    <id>https://chameth.com/</id>
    <author>
        <name>Chris Smith</name>
    </author>
    <entry>
        <title>An interesting Tailscale + Docker gotcha</title>
        <link href="https://chameth.com/tailscale-docker-gotcha/"/>
        <updated>2026-01-29T00:00:00Z</updated>
        <id>https://chameth.com/tailscale-docker-gotcha/</id>
        <content xml:lang="en" type="html">&lt;p&gt;As I’ve &lt;a href=&#34;https://chameth.com/how-i-use-tailscale/&#34;&gt;written&lt;/a&gt; &lt;a href=&#34;https://chameth.com/avoiding-the-consequences-of-dumb-laws-with-tailscale/&#34;&gt;about&lt;/a&gt; &lt;a href=&#34;https://chameth.com/exposing-game-servers-over-tailscale/&#34;&gt;before&lt;/a&gt;, I use &lt;a href=&#34;https://tailscale.com/&#34;&gt;Tailscale&lt;/a&gt; for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.&lt;/p&gt;
&lt;p&gt;I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an &lt;code&gt;app&lt;/code&gt; node can’t access anything via Tailscale, while an &lt;code&gt;integration&lt;/code&gt; or &lt;code&gt;server&lt;/code&gt; node can access things tagged with either &lt;code&gt;app&lt;/code&gt; or &lt;code&gt;integration&lt;/code&gt;. This is expressed pretty simply in the Tailscale ACL JSON:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tagOwners&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Servers that can be SSH&amp;#39;d into
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Applications that are exposed on tailscale but never connect out
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Things which talk to other services over tailscale (connecting to apps etc)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;grants&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Users can access everything
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;autogroup:member&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers and integrations can access integrations and apps
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Over the past week I’ve been setting up a private &lt;a href=&#34;https://forgejo.org/&#34;&gt;Forgejo&lt;/a&gt; instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (&lt;code&gt;forgejo:3000&lt;/code&gt;) instead of using full Tailscale HTTPS URL that I use when accessing it (&lt;code&gt;http://git.example-net.ts.net/&lt;/code&gt;)&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!&lt;/p&gt;
&lt;p&gt;This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my &lt;code&gt;server&lt;/code&gt; tag. That creates a &lt;code&gt;tailscale0&lt;/code&gt; interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the &lt;code&gt;tailscale0&lt;/code&gt; interface.&lt;/p&gt;
&lt;p&gt;I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the &lt;code&gt;tailscale0&lt;/code&gt; interface if it originated from the IP ranges that Docker was configured to use:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;DOCKER-USER&lt;/code&gt; chain is a nice little escape hatch; it comes before the main auto-generated &lt;code&gt;DOCKER&lt;/code&gt; chain, and Docker leaves the rules in it alone.&lt;/p&gt;
&lt;p&gt;So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS&lt;sup id=&#34;fnref:4&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:4&#34; role=&#34;doc-noteref&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;. The server’s DNS resolver is &lt;code&gt;100.100.100.100&lt;/code&gt;, a special Tailscale address. This is so that it can resolve tailnet hostnames via &lt;a href=&#34;https://tailscale.com/kb/1081/magicdns&#34;&gt;MagicDNS&lt;/a&gt; (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.&lt;/p&gt;
&lt;p&gt;Obviously the solution here is to double down and add MORE iptables rules:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use &lt;code&gt;iptables-save&lt;/code&gt;, and then you also get all the rules that both Docker and Tailscale have inserted. I came across &lt;a href=&#34;https://github.com/giangi/iptables-docker-filter&#34;&gt;a script&lt;/a&gt; to filter out the Docker ones, but… yuck&lt;sup id=&#34;fnref:5&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:5&#34; role=&#34;doc-noteref&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;I realised a cleaner idea might just be to lock down what &lt;code&gt;server&lt;/code&gt; tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to &lt;code&gt;curl&lt;/code&gt; services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for &lt;code&gt;infrastructure&lt;/code&gt;, which can be accessed from &lt;code&gt;server&lt;/code&gt; devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the &lt;code&gt;tests&lt;/code&gt; feature of Tailscale’s ACL config to make sure I’d got the rules right:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tests&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers can only access infrastructure
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;   &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;proto&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;allow&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:infrastructure:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;deny&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;100.84.16.43:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;me@example.com:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I discovered later on that logging into the container registry at &lt;code&gt;forgejo:3000&lt;/code&gt; actually issued a redirect to &lt;code&gt;https://git.example-net.ts.net/&lt;/code&gt; so this was all basically for naught… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;
&lt;p&gt;It’s always DNS… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:4&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;
&lt;p&gt;Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:5&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Thinking more about backups</title>
        <link href="https://chameth.com/thinking-more-about-backups/"/>
        <updated>2025-11-01T00:00:00Z</updated>
        <id>https://chameth.com/thinking-more-about-backups/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/thinking-more-about-backups/backblaze.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/thinking-more-about-backups/backblaze.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/thinking-more-about-backups/backblaze.png&#34; alt=&#34;The Backblaze logo: a stylised flame above the word Backblaze&#34; loading=&#34;lazy&#34; width=&#34;500&#34; height=&#34;320&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;The Backblaze logo&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Almost a year ago I wrote about &lt;a href=&#34;https://chameth.com/simple-backups-restic-hetzner/&#34;&gt;how I do backups with Restic and Hetzner&lt;/a&gt;.
That system has been ticking along well ever since, but recently I had some… thoughts. These backups are
all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are
two glaring issues:&lt;/p&gt;
&lt;p&gt;Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or
my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation
and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse
report get it suspended? What if the UK passes
&lt;a href=&#34;https://www.legislation.gov.uk/ukpga/2023/50/contents&#34;&gt;even more dumb laws&lt;/a&gt; and Hetzner decide it’s easier
just to not do business with people here? This is the same sort of concern I have about Google accounts:
if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube
video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very
likely, but these are things my brain likes to dwell on.&lt;/p&gt;
&lt;p&gt;Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious
gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels
a little silly that the same attack could take down both the originals and the backups. There’s no way to
avoid that with Hetzner’s S3 implementation, as far as I can tell.&lt;/p&gt;
&lt;h3 id=&#34;exploring-options&#34;&gt;Exploring options&lt;/h3&gt;
&lt;p&gt;I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems
would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that,
and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in
the worst case.&lt;/p&gt;
&lt;p&gt;I then looked around at other S3 providers. &lt;a href=&#34;https://aws.amazon.com/s3/storage-classes/glacier/&#34;&gt;Amazon’s glacier offering&lt;/a&gt;
is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything.
There are also lots of weird pricing edge cases around moving data between storage classes, minimum file
sizes, and so on. A much better option is &lt;a href=&#34;https://www.backblaze.com/cloud-storage&#34;&gt;Backblaze’s B2&lt;/a&gt; product.
Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful
in this case: &lt;a href=&#34;https://www.backblaze.com/blog/backblaze-b2-lifecycle-rules/&#34;&gt;lifecycle rules&lt;/a&gt;. Coupled with
the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for
what’s effectively an append-only store.&lt;/p&gt;
&lt;p&gt;This works more-or-less out of the box with Restic. &lt;a href=&#34;https://pricey.uk/blog/restic-backups-without-delete/&#34;&gt;Joseph Price has a guide&lt;/a&gt;
that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during
a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules
decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to
the end of my script: a simple &lt;code&gt;restic copy&lt;/code&gt; and a &lt;code&gt;restic forget&lt;/code&gt;. B2 actually works out cheaper than the
Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month.
Not a bad price for some extra peace of mind!&lt;/p&gt;
</content>
    </entry>
    <entry>
        <title>If all you have is a hammer…</title>
        <link href="https://chameth.com/if-all-you-have-is-a-hammer/"/>
        <updated>2025-06-18T00:00:00Z</updated>
        <id>https://chameth.com/if-all-you-have-is-a-hammer/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/if-all-you-have-is-a-hammer/hammer.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/if-all-you-have-is-a-hammer/hammer.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/if-all-you-have-is-a-hammer/hammer.jpg&#34; alt=&#34;Closeup photo of a well-used hammer with a wooden handle&#34; loading=&#34;lazy&#34; width=&#34;500&#34; height=&#34;411&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;This, dear reader, is a hammer. It is almost entirely irrelevant to the article. Enjoy.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;I presume everyone is familiar with the idiom “if all you have is a hammer,
everything looks like a nail”. If not, well, there it is. It’s generally
used pejoratively about being single-minded, but I think it also gives a glimpse
into something more interesting: mental and perceptual sets.&lt;/p&gt;
&lt;p&gt;Before I explain, let me tell you a story about a person who bought a 3D printer&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.
When they were first thinking about getting one, they weren’t sure if it was
worth it. They could print a few board game accessories, but then what? After
they got it, though, a whole new world opened up: everywhere they looked there
were opportunities to improve things by adding 3D printed plastic. Broken
appliances were repaired with 3D printed parts, all sorts of shelves,
organisers, hooks and other things were made. But when they talked about this
to other people, most often the response was “That’s nice, but I don’t think
I’d use one”. How could they not see the truth in all its glorious layer lines?!&lt;/p&gt;
&lt;p&gt;The answer is in the concept of a ‘set’&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;, and instead of trying to explain it,
I’m just going to quote &lt;a href=&#34;https://en.wikipedia.org/wiki/Set_(psychology)&#34;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In psychology, a set is a group of expectations that shape experience by making
people especially sensitive to specific kinds of information. A perceptual set,
also called perceptual expectancy, is a predisposition to perceive things in a
certain way. […] A mental set is a framework for thinking about a problem.
It can be shaped by habit or by desire. Mental sets can make it easy to solve
a class of problem, but attachment to the wrong mental set can inhibit
problem-solving and creativity.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This perfectly captures what happened. They’d developed a ‘3D printing mental set’:
a predisposition to see problems that can be solved with their &lt;del&gt;hammer&lt;/del&gt; 3D
printer. Once I started noticing this pattern, I started seeing it all over
the place.&lt;/p&gt;
&lt;!--more--&gt;
&lt;h3 id=&#34;sets-at-work&#34;&gt;Sets at work&lt;/h3&gt;
&lt;p&gt;Aside from the “could this be improved with some 3D printed plastic” perceptual
set, I’ve noticed a couple of other sets: DIY, and writing puzzles.&lt;/p&gt;
&lt;p&gt;DIY is probably the most obvious one. Once you become a little familiar with the
basics of what you can fix and how, it seems like you start noticing the
opportunities more. Before, you might have ignored that loose door handle, but
now you notice it and realise with a screwdriver and a sex bolt&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt; you can
fix it and stop it coming loose again anytime soon.&lt;/p&gt;
&lt;p&gt;As you do more things, you build up your set of mental sets as well. You learn
how to attach things to different types of walls, how to safely work with
electricity, and so on, and suddenly installing a new outside light goes from
a seemingly complex job to just exercising some of those concepts you’re already
familiar with.&lt;/p&gt;
&lt;p&gt;This works across ‘disciplines’ as well: I wanted to automatically cool
a room down, using a fan in the window to blow in cool air. The 3D printing
mental set gave me a way to make it physically go where I wanted it to, the
home automation set gave me a plan for how to automate it, and the electricity
set helped me figure out how to wire in the relay. It sounds a bit stilted when
I spell it out like that, but the catch is I didn’t have to sit down and work
any of this out. I was already primed to think in a certain way. I had a whole
selection of hammers, and could pick the right combination of them to beat
this particular nail&lt;sup id=&#34;fnref:4&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:4&#34; role=&#34;doc-noteref&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;Puzzles are a bit different. I enjoy taking part in puzzle hunts&lt;sup id=&#34;fnref:5&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:5&#34; role=&#34;doc-noteref&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;, and at
first I was bewildered at how the authors came up with all the puzzles. After
I started thinking about writing some myself, though, I began to notice all
kinds of things that could maybe be turned into a puzzle: a specialised
database, or a song with repetitive lyrics, or set of things with unique ID
numbers. Writing the actual puzzles is still tricky, but once you’ve equipped
the “look out for puzzle material” perceptual set it’s much easier to gather
a range of ideas.&lt;/p&gt;
&lt;p&gt;For example, one puzzle I wrote came about because I stumbled upon
&lt;a href=&#34;https://archive.org/details/patrick-stewarts-cowboy-classic-sampler&#34;&gt;Sir Patrick Stewart’s Cowboy Classic Sampler&lt;/a&gt;&lt;sup id=&#34;fnref:6&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:6&#34; role=&#34;doc-noteref&#34;&gt;6&lt;/a&gt;&lt;/sup&gt;.
I think everyone will agree that it’s a fairly unique collection. I tied some
samples of the songs together with some screen captures of Sir Patrick as Picard
in Star Trek: The Next Generation, and puzzlers had to find the season and
episode numbers of the pictures, then use those numbers to select a track and a
word from that track in the audio mix. Those words spell out a question, and the
answer to that question is the answer to the puzzle.&lt;/p&gt;
&lt;p&gt;There were other hints mixed in to point you in the right direction, it wasn’t
quite as opaque as it might sound! But the point is without being in that
“puzzle finding” mindset when I came across the album, there’s no way I could
have come up with that puzzle. Like with the fan, I was primed to think a
certain way, and managed to create something I otherwise would have struggled
with.&lt;/p&gt;
&lt;h3 id=&#34;the-curse-of-noticing-things&#34;&gt;The curse of noticing things&lt;/h3&gt;
&lt;p&gt;Figuring out how to make stuff isn’t the only place sets appear, though.
Perceptual sets are about how we perceive things. When you read something
with lots of bullet points, em-dashes, and the word “delve” do you automatically
think it’s LLM generated?&lt;sup id=&#34;fnref:7&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:7&#34; role=&#34;doc-noteref&#34;&gt;7&lt;/a&gt;&lt;/sup&gt; Then you have an LLM text perceptual set! It’s
kind of annoying, as it’s difficult to “turn off” and stop noticing these
things once you’ve seen them. Similarly, I think I have a “bad grammar” set.
I know it doesn’t bother a lot of people, but if I’m reading a book and there’s
a rogue apostrophe it will throw me right out of the moment. It’s like tripping
up when walking and being torn out of a daydream. There’s some use to these —
being able to notice grammar errors easily helps with my writing — but for
the most part they’re a negative input: they point out things are broken or
bad.&lt;/p&gt;
&lt;p&gt;The “edge case set” takes this to another level. After about 20 years of writing
software&lt;sup id=&#34;fnref:8&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:8&#34; role=&#34;doc-noteref&#34;&gt;8&lt;/a&gt;&lt;/sup&gt; I have a finely honed ability to spot problems. Give me something to
implement and my brain will immediately rattle off all the different edge cases
and under-specified parts. Then some other bit of my brain gets the job of
deciding which we actually care about, which are worth raising, and so on.
It’s definitely useful to spot them up-front, rather than when you’re deep in
the weeds implementing something, and tends to make everything go a lot smoother
if you can hammer the major ones out when planning work. It doesn’t turn off
after work though; it comes up when reading board game rule books, or
descriptions of the magic system in fantasy novels, or news reports, and so on.
It’s a lot less useful then.&lt;/p&gt;
&lt;p&gt;Speaking of news reports, here’s a fun one: Gell-Mann amnesia is a cognitive
bias where even when you critically assess media reports about something you’re
knowledgeable about (say, technology), you continue to trust similar reporting
in other areas (say, medicine). But once you know about Gell-Mann amnesia, you
might start to actively notice when it happens. You scoff at the article about
how the Internet is a “series of tubes” and then start reading the next one
about a cancer treatment, but — wait a minute — your Gell-Mann amnesia set
kicks in. I’m not sure what you then do with this information. Dismiss basically
all the media and news as useless? It’s honestly pretty annoying.&lt;/p&gt;
&lt;h3 id=&#34;the-meta-set&#34;&gt;The meta set&lt;/h3&gt;
&lt;p&gt;I’ve had this blog post on my ideas list since I first noticed the “3D printing
mindset”. I didn’t know sets were an actual thing, and I didn’t have most of
the examples I do now. As time went on with this idea rattling around in the
back of my head, I started noticing the patterns come up more and more. There’s
lots of more examples I didn’t include because I didn’t want to just write a
listicle of mental patterns. But the fact that I noticed them all seems to mean
that I’ve developed a set set, or a meta-set, if you will. It’s fascinating
to think about how all this happens inside our mushy bag of neurons, but
also a bit terrifying. If thought processes are that malleable, what stops you
accidentally breaking them? Actually, thinking about things like Q-Anon, flat
earthers, and a lot of contemporary “politics”, maybe that’s quite easy to do?&lt;/p&gt;
&lt;hr/&gt;
&lt;h3 id=&#34;image-credits&#34;&gt;Image credits&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Photo&lt;/th&gt;
&lt;th&gt;Creator&lt;/th&gt;
&lt;th&gt;Licence&lt;/th&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hammer&lt;/td&gt;
&lt;td&gt;&lt;a href=&#34;https://unsplash.com/@iantalmacs&#34;&gt;Ian Talmacs&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Unsplash License&lt;/td&gt;
&lt;td&gt;&lt;a href=&#34;https://unsplash.com/photos/a-close-up-of-a-hammer-with-a-black-background-4kuAvNfHEzI&#34;&gt;Unsplash&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;Surprise! I am that person. Bet you didn’t see that coming! &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;I feel like they should have consulted the mathematicians and/or computer scientists before they picked a name. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;Yes, that’s a real thing. No, I won’t ever use any of the alternative names for it. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;
&lt;p&gt;OK, I’ve strained the metaphor enough. I’m sorry. I’ll stop. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:4&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;
&lt;p&gt;If you’re not familiar, a puzzle hunt is a competition where you try to solve puzzles faster than other teams. The sort of puzzles involved are perfectly described by this quote from Mike Develin: “Imagine a word search. Now imagine you aren’t told what words to look for. Now imagine you aren’t told it’s a word search. Now imagine it isn’t a word search.” &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:5&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:6&#34;&gt;
&lt;p&gt;Go and give it a listen. I’ll wait. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:6&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:7&#34;&gt;
&lt;p&gt;I hereby certify all my em-dashes are hand-crafted. Relatedly, Claude Code keeps writing comments that use hyphens when they should actually be em-dashes. Maybe they’ve trained it to avoid them? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:7&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:8&#34;&gt;
&lt;p&gt;I just made myself sad by working this out. I still feel young at heart! … Or maybe that’s cholesterol? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:8&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>The Curse of Knowledge and Blogging</title>
        <link href="https://chameth.com/curse-of-knowledge-blogging/"/>
        <updated>2025-05-16T00:00:00Z</updated>
        <id>https://chameth.com/curse-of-knowledge-blogging/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/curse-of-knowledge-blogging/ideas.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/curse-of-knowledge-blogging/ideas.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/curse-of-knowledge-blogging/ideas.jpg&#34; alt=&#34;Photo of a sign reading &amp;#34;YOUR IDEAS MATTER … Write them down :)&amp;#34;&#34; loading=&#34;lazy&#34; width=&#34;653&#34; height=&#34;463&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;Finding a picture to go with a blog post about blog post ideas is hard, OK?&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;The single worst part about blogging for me is trying to come up with ideas
for what to write about. Not because they’re hard to come up with, but because
every idea seems too basic or not worth talking about. &lt;em&gt;Seems&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;It’s not just me, either. My friends will occasionally describe some interesting
problem they’ve dealt with, and I’ll chime in with “you should blog about that”.
Almost invariably the response is something like “it’s not that interesting”.&lt;/p&gt;
&lt;p&gt;There’s a cognitive bias called &lt;a href=&#34;https://en.wikipedia.org/wiki/Curse_of_knowledge&#34;&gt;the curse of knowledge&lt;/a&gt;&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.
Basically, once you know something you can’t accurately reconstruct your previous
state of mind from when you didn’t know it. You can struggle to do something
for hours, piecing together scraps of information, using all kinds of specialist
knowledge, and as soon as you’re finished it all seems so &lt;em&gt;basic&lt;/em&gt; why would
anyone want to read about that?&lt;/p&gt;
&lt;!--more--&gt;
&lt;h3 id=&#34;other-mental-obstacles&#34;&gt;Other mental obstacles&lt;/h3&gt;
&lt;p&gt;Of course, nothing involving people is as simple as having one cause and an
obvious solution. Even when I get over the “it’s too basic” road bump, there
are other issues. Looking back over some of my posts, I get a horrible cringe-y
feeling and the question of “why would anyone be interested in reading that?!”
arrives unbidden. Even though I know that &lt;em&gt;I’m&lt;/em&gt; interested in reading similar
posts written by other people. I’m hesitant to attach labels to everything, but
&lt;a href=&#34;https://en.wikipedia.org/wiki/Impostor_syndrome&#34;&gt;imposter syndrome&lt;/a&gt; says hi.&lt;/p&gt;
&lt;p&gt;I think there’s also a cultural issue at play. British people tend to be
somewhat reserved and downplay their accomplishments, especially when compared
to people from the US&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;. When I worked for a certain multinational
&lt;del&gt;ad conglomerate&lt;/del&gt; search engine company we actually had training on how to do
performance reviews and speak about ourselves more like our US counterparts.&lt;/p&gt;
&lt;p&gt;Put all these things into a cauldron, add a lizard’s leg and howlet’s wing,
mix it all up, and you get a brain that despises the idea of blogging about
anything. Great.&lt;/p&gt;
&lt;h3 id=&#34;dealing&#34;&gt;Dealing&lt;/h3&gt;
&lt;p&gt;So, now the problem is well-defined, what do we do about it? Clearly I am the
best person to be offering this advice, given it’s been four months since my
last blog post.&lt;/p&gt;
&lt;p&gt;The best counter to the Curse of Knowledge that I’ve found is keeping notes,
like a &lt;a href=&#34;https://en.wikipedia.org/wiki/Lab_notebook&#34;&gt;lab notebook&lt;/a&gt;. If you
finish something and have dozens of links to different resources that you had
to piece together, it’s probably something interesting and worth talking about
even if it doesn’t seem like it any more. Or maybe you didn’t need a lot of
references, but you had to write a lot down to make sense of something, or
needed to draw diagrams to solve a problem. Again, probably interesting.&lt;/p&gt;
&lt;p&gt;The other issues I’ve mentioned are not so easy to overcome. I’m also not going
to try and perform therapy in the form of a blog post! What I personally do is
keep a list of topics to write about (that have already passed the “probably
interesting even if my brain doesn’t agree” check), and occasionally when the
stars align I’ll feel like plucking one off the list and turning thoughts into
words.&lt;/p&gt;
&lt;h3 id=&#34;a-call-to-action&#34;&gt;A call to action&lt;/h3&gt;
&lt;p&gt;If you’ve got a blog and haven’t written anything recently because you “don’t
do anything interesting”, go write something! I promise that someone will be
interested. Hell, feel free to send me a link and &lt;em&gt;I’ll&lt;/em&gt; be interested for you.&lt;/p&gt;
&lt;p&gt;If you don’t have a blog then why not start one? There are lots of great articles
out there about why-you-should and how-you-can. And as a bonus, blog posts about
how you set up your blog are &lt;em&gt;super&lt;/em&gt; interesting. That’s one post sorted already!&lt;/p&gt;
&lt;p&gt;As the open internet is slowly subsumed by massive corporate interests, and
gummed up with spam and AI slop, the voice of real humans is becoming more and
more important. Maybe we can — one blog at a time — get back to the ’90s
internet that was full of individuality instead of today’s which is full of ads
and spam?&lt;/p&gt;
&lt;hr/&gt;
&lt;p&gt;Photo by &lt;a href=&#34;https://unsplash.com/@kommumikation&#34;&gt;Mika Baumeister&lt;/a&gt; on &lt;a href=&#34;https://unsplash.com/photos/a-sign-on-the-side-of-a-building-LaqL8nxiacc&#34;&gt;Unsplash&lt;/a&gt;&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;There’s also something called “Familiarity Blindness” that people like to
talk about. It’s a much better name. The only problem is that I can’t find any
actual published works that talk about it. No-one authoritative defines or
mentions it. It just popped into existence a few years ago. Maybe it’s an LLM
hallucination? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;My favourite commentary on this is &lt;a href=&#34;https://www.youtube.com/watch?v=8k2AbqTBxao&#34;&gt;Stephen Fry talking about American vs British comedy&lt;/a&gt;. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Simple backups with Restic and Hetzner Cloud</title>
        <link href="https://chameth.com/simple-backups-restic-hetzner/"/>
        <updated>2024-12-06T00:00:00Z</updated>
        <id>https://chameth.com/simple-backups-restic-hetzner/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.png&#34; alt=&#34;The Restic logo — a gopher with two umbrellas.&#34; loading=&#34;lazy&#34; width=&#34;400&#34; height=&#34;400&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;I have a confession: for the past few years I’ve not been backing up any of my
computers. Everyone knows that you &lt;em&gt;should&lt;/em&gt; do backups, but actually getting
around to doing it is another story.&lt;/p&gt;
&lt;p&gt;Don’t get me wrong: most of my important things are “backed up” by virtue of
being committed to remote git repositories, or attached to e-mails, or
re-obtainable from the original source, and so on. I don’t think any machine
failing completely would be a disaster for me, but it would certainly be a pain.&lt;/p&gt;
&lt;p&gt;This week I finally got around to doing something, and it ended up being a lot
more straight forward than my previous forays into backup-land.&lt;/p&gt;
&lt;h3 id=&#34;restic&#34;&gt;Restic&lt;/h3&gt;
&lt;p&gt;After soliciting a few opinions, the choice of backup software came down to
either &lt;a href=&#34;https://www.borgbackup.org/&#34;&gt;Borg&lt;/a&gt; or &lt;a href=&#34;https://restic.net/&#34;&gt;Restic&lt;/a&gt;.
I’m pretty sure either would have done what I want, but I leaned towards Restic
for a few reasons: it has a more informative website, it’s written in Go
rather than Python&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;, and Borg seems to be transitioning between major
releases at the moment&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;The way Restic works is pretty simple: you initialise a ‘repository’, and can
then call &lt;code&gt;restic backup /some/path&lt;/code&gt; and it’ll get backed up to the repository.
Restic handles keeping different backups separate, and only sending data that’s
changed, and deduplicating, and so on. You basically point it at a thing you
don’t want to lose, and it sorts it out for you. Perfect.&lt;/p&gt;
&lt;p&gt;There are equally straight-forward commands for removing old snapshots
(&lt;code&gt;restic forget&lt;/code&gt;) and verifying backups (&lt;code&gt;restic check&lt;/code&gt;). One of the nice things
about modern backup solutions is they support a whole range of backends. I was
originally going to spin up a small VPS to host my backups, but noticed that
Restic supported S3-compatible stores…&lt;/p&gt;
&lt;h3 id=&#34;hetzner-cloud-object-storage&#34;&gt;Hetzner Cloud Object Storage&lt;/h3&gt;
&lt;p&gt;I host my servers with Hetzner, and was going to use them to spin up a VPS as
well. Despite the “Cloud” branding on a bunch of products, they offer reasonable
prices and good service. A couple of months ago, they started offering
&lt;a href=&#34;https://docs.hetzner.com/storage/object-storage/overview&#34;&gt;S3-compatible object storage&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The pricing isn’t totally straight forward, but for continuous use the “free
quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free,
as is traffic within the &lt;code&gt;eu-central&lt;/code&gt; region (where all my servers are). That
quota is only awarded when you pay the “base price”, though, which is €4.99 a
month. So basically it’s €5 a month for 1TB of storage and enough egress to
fully restore every single byte. That’s better value than any VPS I can find,
much cheaper than Amazon S3, and about the same as
&lt;a href=&#34;https://www.backblaze.com/cloud-storage&#34;&gt;Backblaze B2&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&#34;getting-them-to-work-together&#34;&gt;Getting them to work together&lt;/h3&gt;
&lt;p&gt;Now you’d think making the tool that supports S3-compatible object storage
work with your S3-compatible object storage would be easy, right? Well not
quite. The library Restic uses to deal with S3 backends has some
strange logic for figuring out the bucket name given a URL. It doesn’t quite
seem to work right, though…&lt;/p&gt;
&lt;p&gt;Hetzner buckets have URLs like &lt;code&gt;s3://bucketname.hel1.your-objectstorage.com&lt;/code&gt;&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;,
but just passing that to Restic gives an error that the bucket is not specified.
The docs mention there’s an advanced option to make it use the virtual host for
the bucket name: &lt;code&gt;-o s3.bucket-lookup=dns&lt;/code&gt;. But that… also doesn’t work.
What I ended up doing was specifying the URL as
&lt;code&gt;s3://hel1.your-objectstorage.com/bucketname&lt;/code&gt;, and also passing in the &lt;code&gt;dns&lt;/code&gt;
option. The library then seems to muddle its way back to a real, working URL.
I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that
I’m oblivious to?&lt;/p&gt;
&lt;p&gt;The next fun part is that you can configure Restic entirely by using environment
variables, except for that &lt;code&gt;-o s3.bucket-lookup=dns&lt;/code&gt; argument. That has to go
on the command line. I ended up making a little wrapper script to invoke Restic
correctly:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-cp&#34;&gt;#!/bin/sh
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The password restic will use to encrypt your data. You should generate&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# something nice and secure.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;RESTIC_PASSWORD&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;repo-password
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The path to the repository where restic will save the backup. In our case&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# this takes the form `s3:&amp;lt;endpoint&amp;gt;/&amp;lt;bucket&amp;gt;`. Your endpoint might be different&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# to mine depending on the region your bucket is in.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;RESTIC_REPOSITORY&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;s3:hel1.your-objectstorage.com/bucket-name
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The access and secret key generated in the &amp;#39;S3 credentials&amp;#39; section of the&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Hetzner Cloud Console&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;AWS_ACCESS_KEY_ID&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;hetzner-access-key-id
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;AWS_SECRET_ACCESS_KEY&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;hetzner-access-key
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Pass any arguments on to the restic command, along with the magic&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# `s3.bucket-lookup` option we need to resolve the S3 URL properly.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;exec&lt;/span&gt; restic -o s3.bucket-lookup&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;dns &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-nv&#34;&gt;$@&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then in the backup script I just alias &lt;code&gt;restic&lt;/code&gt; to use the script:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-cp&#34;&gt;#!/bin/bash
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;set&lt;/span&gt; -eu
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# This means any time we use `restic` below, we&amp;#39;ll actually execute our special&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# script which supplies all the env vars and arguments needed to find the&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# repository. Make sure the path matches where you saved the script!&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;alias&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;restic&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;chroma-s1&#34;&gt;&amp;#39;~/.bin/restic&amp;#39;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Actually do the backup. Each directory in the list below will be backed up&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# separately.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nv&#34;&gt;dirs&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;/some/path/to/backup/&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;/some/other/path/&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;for&lt;/span&gt; i in &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-si&#34;&gt;${&lt;/span&gt;&lt;span class=&#34;chroma-nv&#34;&gt;dirs&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;[@]&lt;/span&gt;&lt;span class=&#34;chroma-si&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt; 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;do&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nb&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;$i&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-o&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-nb&#34;&gt;cd&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;$i&lt;/span&gt; &lt;span class=&#34;chroma-o&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt; restic --verbose backup .&lt;span class=&#34;chroma-o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;done&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# see what effect any changes would have before actually committing to them. &lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;restic forget --keep-daily &lt;span class=&#34;chroma-m&#34;&gt;7&lt;/span&gt; --keep-weekly &lt;span class=&#34;chroma-m&#34;&gt;10&lt;/span&gt; --keep-monthly &lt;span class=&#34;chroma-m&#34;&gt;24&lt;/span&gt; --keep-yearly &lt;span class=&#34;chroma-m&#34;&gt;10&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The only other notable thing here is that the script changes into the directory
to be backed up. If you give Restic an absolute path, it will create a new
snapshot if the metadata of any folder in the path changes, which is not what
I want.&lt;/p&gt;
&lt;p&gt;If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken
me about half an hour. That’s including setting up the object storage,
installing Restic, and so on. It’s painfully easy. Why didn’t I do this three
years ago?!&lt;/p&gt;
&lt;h3 id=&#34;addendum-a-step-by-step-guide&#34;&gt;Addendum: a step-by-step guide&lt;/h3&gt;
&lt;aside class=&#34;update raised-box&#34;&gt;
  &lt;h5 class=&#34;plain-header&#34;&gt;Update 2025-03-01:&lt;/h5&gt;
  &lt;p&gt;This section was added after the original article was published, following some
helpful feedback. Let me know if you have any problem with these instructions!&lt;/p&gt;
&lt;/aside&gt;
&lt;p&gt;If you want to do this yourself, here’s a quick step-by-step guide:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Log in to the &lt;a href=&#34;https://console.hetzner.cloud&#34;&gt;Hetzner Cloud Console&lt;/a&gt;, and
create a project.&lt;/li&gt;
&lt;li&gt;On the “Object Storage” tab, create a new bucket. Note the name and the
endpoint.&lt;/li&gt;
&lt;li&gt;On the “Security” tab, go to “S3 Credentials” and generate new credentials.
Note down the access key and the secret key.&lt;/li&gt;
&lt;li&gt;Copy the first shell script above, fill in the password (you can pick!),
endpoint, bucket name, access key and secret key.&lt;/li&gt;
&lt;li&gt;Run the script with the &lt;code&gt;init&lt;/code&gt; argument (e.g. &lt;code&gt;~/.bin/restic init&lt;/code&gt;). This
will create a new repository, and only needs to be done once even if you
backup multiple machines.&lt;/li&gt;
&lt;li&gt;Copy the second shell script above, making sure the &lt;code&gt;restic&lt;/code&gt; alias points
at the script you saved in step 4. Change the list of directories to
whatever you want to backup.&lt;/li&gt;
&lt;li&gt;Schedule the script to be run automatically, using crontab or systemd timers
or however you prefer.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;To check everything is working, you can use the &lt;code&gt;snapshots&lt;/code&gt; subcommand to see
a list of saved snapshots. You might also want to try to restore a snapshot
using the &lt;code&gt;restore&lt;/code&gt; subcommand.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I’m not trying to be a language snob, but given the choice between two
otherwise equal projects one in Go and one in Python, I’ll take the Go one.
I know I’m not going to have weird library issues down the line, and I’m much
more comfortable rummaging around the source. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;With a “Don’t use this in production!” notice on the shiny new version. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;Aside: I really hate URLs like that. What is with the trend for completely
generic domains divorced from the service they’re a part of? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
</feed>
