<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="/feeds.xsl" type="text/xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:base="https://chameth.com/">
    <title>Chameth.com - posts like docker-automatic-nginx-proxy, filament-weight-display, g15-ram-upgrade, thinking-more-about-backups, why-you-should-be-using-https</title>
    <subtitle>Personal homepage of Chris Smith</subtitle>
    <link href="https://chameth.com/feeds/posts/like/docker-automatic-nginx-proxy,filament-weight-display,g15-ram-upgrade,thinking-more-about-backups,why-you-should-be-using-https/" rel="self"/>
    <link href="https://chameth.com/"/>
    <icon>https://chameth.com/favicon.png</icon>
    <updated>2026-01-29T00:00:00Z</updated>
    <id>https://chameth.com/</id>
    <author>
        <name>Chris Smith</name>
    </author>
    <entry>
        <title>An interesting Tailscale + Docker gotcha</title>
        <link href="https://chameth.com/tailscale-docker-gotcha/"/>
        <updated>2026-01-29T00:00:00Z</updated>
        <id>https://chameth.com/tailscale-docker-gotcha/</id>
        <content xml:lang="en" type="html">&lt;p&gt;As I’ve &lt;a href=&#34;https://chameth.com/how-i-use-tailscale/&#34;&gt;written&lt;/a&gt; &lt;a href=&#34;https://chameth.com/avoiding-the-consequences-of-dumb-laws-with-tailscale/&#34;&gt;about&lt;/a&gt; &lt;a href=&#34;https://chameth.com/exposing-game-servers-over-tailscale/&#34;&gt;before&lt;/a&gt;, I use &lt;a href=&#34;https://tailscale.com/&#34;&gt;Tailscale&lt;/a&gt; for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.&lt;/p&gt;
&lt;p&gt;I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an &lt;code&gt;app&lt;/code&gt; node can’t access anything via Tailscale, while an &lt;code&gt;integration&lt;/code&gt; or &lt;code&gt;server&lt;/code&gt; node can access things tagged with either &lt;code&gt;app&lt;/code&gt; or &lt;code&gt;integration&lt;/code&gt;. This is expressed pretty simply in the Tailscale ACL JSON:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tagOwners&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Servers that can be SSH&amp;#39;d into
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Applications that are exposed on tailscale but never connect out
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Things which talk to other services over tailscale (connecting to apps etc)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;grants&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Users can access everything
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;autogroup:member&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers and integrations can access integrations and apps
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Over the past week I’ve been setting up a private &lt;a href=&#34;https://forgejo.org/&#34;&gt;Forgejo&lt;/a&gt; instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (&lt;code&gt;forgejo:3000&lt;/code&gt;) instead of using full Tailscale HTTPS URL that I use when accessing it (&lt;code&gt;http://git.example-net.ts.net/&lt;/code&gt;)&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!&lt;/p&gt;
&lt;p&gt;This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my &lt;code&gt;server&lt;/code&gt; tag. That creates a &lt;code&gt;tailscale0&lt;/code&gt; interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the &lt;code&gt;tailscale0&lt;/code&gt; interface.&lt;/p&gt;
&lt;p&gt;I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the &lt;code&gt;tailscale0&lt;/code&gt; interface if it originated from the IP ranges that Docker was configured to use:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;DOCKER-USER&lt;/code&gt; chain is a nice little escape hatch; it comes before the main auto-generated &lt;code&gt;DOCKER&lt;/code&gt; chain, and Docker leaves the rules in it alone.&lt;/p&gt;
&lt;p&gt;So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS&lt;sup id=&#34;fnref:4&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:4&#34; role=&#34;doc-noteref&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;. The server’s DNS resolver is &lt;code&gt;100.100.100.100&lt;/code&gt;, a special Tailscale address. This is so that it can resolve tailnet hostnames via &lt;a href=&#34;https://tailscale.com/kb/1081/magicdns&#34;&gt;MagicDNS&lt;/a&gt; (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.&lt;/p&gt;
&lt;p&gt;Obviously the solution here is to double down and add MORE iptables rules:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use &lt;code&gt;iptables-save&lt;/code&gt;, and then you also get all the rules that both Docker and Tailscale have inserted. I came across &lt;a href=&#34;https://github.com/giangi/iptables-docker-filter&#34;&gt;a script&lt;/a&gt; to filter out the Docker ones, but… yuck&lt;sup id=&#34;fnref:5&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:5&#34; role=&#34;doc-noteref&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;I realised a cleaner idea might just be to lock down what &lt;code&gt;server&lt;/code&gt; tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to &lt;code&gt;curl&lt;/code&gt; services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for &lt;code&gt;infrastructure&lt;/code&gt;, which can be accessed from &lt;code&gt;server&lt;/code&gt; devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the &lt;code&gt;tests&lt;/code&gt; feature of Tailscale’s ACL config to make sure I’d got the rules right:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tests&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers can only access infrastructure
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;   &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;proto&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;allow&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:infrastructure:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;deny&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;100.84.16.43:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;me@example.com:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I discovered later on that logging into the container registry at &lt;code&gt;forgejo:3000&lt;/code&gt; actually issued a redirect to &lt;code&gt;https://git.example-net.ts.net/&lt;/code&gt; so this was all basically for naught… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;
&lt;p&gt;It’s always DNS… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:4&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;
&lt;p&gt;Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:5&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Thinking more about backups</title>
        <link href="https://chameth.com/thinking-more-about-backups/"/>
        <updated>2025-11-01T00:00:00Z</updated>
        <id>https://chameth.com/thinking-more-about-backups/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/thinking-more-about-backups/backblaze.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/thinking-more-about-backups/backblaze.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/thinking-more-about-backups/backblaze.png&#34; alt=&#34;The Backblaze logo: a stylised flame above the word Backblaze&#34; loading=&#34;lazy&#34; width=&#34;500&#34; height=&#34;320&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;The Backblaze logo&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Almost a year ago I wrote about &lt;a href=&#34;https://chameth.com/simple-backups-restic-hetzner/&#34;&gt;how I do backups with Restic and Hetzner&lt;/a&gt;.
That system has been ticking along well ever since, but recently I had some… thoughts. These backups are
all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are
two glaring issues:&lt;/p&gt;
&lt;p&gt;Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or
my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation
and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse
report get it suspended? What if the UK passes
&lt;a href=&#34;https://www.legislation.gov.uk/ukpga/2023/50/contents&#34;&gt;even more dumb laws&lt;/a&gt; and Hetzner decide it’s easier
just to not do business with people here? This is the same sort of concern I have about Google accounts:
if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube
video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very
likely, but these are things my brain likes to dwell on.&lt;/p&gt;
&lt;p&gt;Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious
gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels
a little silly that the same attack could take down both the originals and the backups. There’s no way to
avoid that with Hetzner’s S3 implementation, as far as I can tell.&lt;/p&gt;
&lt;h3 id=&#34;exploring-options&#34;&gt;Exploring options&lt;/h3&gt;
&lt;p&gt;I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems
would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that,
and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in
the worst case.&lt;/p&gt;
&lt;p&gt;I then looked around at other S3 providers. &lt;a href=&#34;https://aws.amazon.com/s3/storage-classes/glacier/&#34;&gt;Amazon’s glacier offering&lt;/a&gt;
is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything.
There are also lots of weird pricing edge cases around moving data between storage classes, minimum file
sizes, and so on. A much better option is &lt;a href=&#34;https://www.backblaze.com/cloud-storage&#34;&gt;Backblaze’s B2&lt;/a&gt; product.
Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful
in this case: &lt;a href=&#34;https://www.backblaze.com/blog/backblaze-b2-lifecycle-rules/&#34;&gt;lifecycle rules&lt;/a&gt;. Coupled with
the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for
what’s effectively an append-only store.&lt;/p&gt;
&lt;p&gt;This works more-or-less out of the box with Restic. &lt;a href=&#34;https://pricey.uk/blog/restic-backups-without-delete/&#34;&gt;Joseph Price has a guide&lt;/a&gt;
that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during
a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules
decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to
the end of my script: a simple &lt;code&gt;restic copy&lt;/code&gt; and a &lt;code&gt;restic forget&lt;/code&gt;. B2 actually works out cheaper than the
Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month.
Not a bad price for some extra peace of mind!&lt;/p&gt;
</content>
    </entry>
    <entry>
        <title>Home Automation Without the Megacorps</title>
        <link href="https://chameth.com/home-automation-without-megacorps/"/>
        <updated>2025-05-21T00:00:00Z</updated>
        <id>https://chameth.com/home-automation-without-megacorps/</id>
        <content xml:lang="en" type="html">&lt;p&gt;I first experimented with home automation in 2016, by picking up a Samsung
“SmartThings” hub. It was terrible. The UI to configure things was slow and
clunky, firmware updates were applied whether you wanted them or not, and
everything stopped working if their cloud services stopped. You were also locked
into whatever integrations they deigned to support, of course. After that broke
for the umpteenth time I scaled back and for years the closest I got to home
automation was a couple of Hue bulbs.&lt;/p&gt;
&lt;p&gt;Recently I’ve been building it out again, though. This time using off-the-shelf
components that interop using Zigbee, open-source software, and some code I
wrote myself. It’s great; it runs entirely locally and has had basically zero
downtime. The Zigbee ecosystem lets me integrate all sorts of things without
having to spend lots of money on “smart” alternatives. I think I’ve spent less
on this incarnation than I did on the original SmartThings hub all those years
ago (even without adjusting for inflation!).&lt;/p&gt;
&lt;h3 id=&#34;my-current-setup&#34;&gt;My current setup&lt;/h3&gt;
&lt;p&gt;I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based
on the CC2652P chipset&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. Interfacing with the Zigbee stack is handled by
&lt;a href=&#34;https://www.zigbee2mqtt.io/&#34;&gt;zigbee2mqtt&lt;/a&gt; (z2m for short), an open-source project that
basically bridges your devices to an MQTT broker&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;. When a device reports some
data, it will send a new message over MQTT; when you want to make a device do
something you just post a message back. It’s incredibly lightweight, but
supports a huge array of devices out of the box. And as it’s just using MQTT,
it’s trivial to integrate with other software or build on top of.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;I know a lot of people building out this kind of home automation use
&lt;a href=&#34;https://www.home-assistant.io/&#34;&gt;Home Assistant&lt;/a&gt;, but I don’t get on with it
terribly well. It felt incredibly sluggish, and the entire project is just a
bit… much. I want a binary or package or docker image I can just run, not an
entire operating system. That’s not how computers are meant to work! In contrast,
z2m is simple to set up, super light weight and responsive.&lt;/p&gt;
&lt;p&gt;Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to
connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond
this now, and I’m skipping some boring bits like error handling and JSON
parsing, but at first I had something like:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;for&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;message&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;topic&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;err&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;:=&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;c&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;chroma-nf&#34;&gt;ReadSlices&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;()&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;    
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;chroma-k&#34;&gt;if&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-nb&#34;&gt;string&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;topic&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;==&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-s&#34;&gt;&amp;#34;zigbee2mqtt/desk-button&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;chroma-k&#34;&gt;if&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;strings&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;chroma-nf&#34;&gt;Contains&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;message&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-s&#34;&gt;&amp;#34;single&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;            &lt;/span&gt;&lt;span class=&#34;chroma-nx&#34;&gt;c&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;chroma-nf&#34;&gt;Publish&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-kc&#34;&gt;nil&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;[]&lt;/span&gt;&lt;span class=&#34;chroma-nb&#34;&gt;byte&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-s&#34;&gt;&amp;#34;{\&amp;#34;state\&amp;#34;: \&amp;#34;TOGGLE\&amp;#34;}&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;),&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;chroma-s&#34;&gt;&amp;#34;zigbee2mqtt/room-lights/set&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;chroma-w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I think I actually made this harder to read by trying to simplify it here, but
hopefully you can follow that it was relatively straight forward to listen
for a particular action to happen (in this case me “single”-pressing on the
device called “desk-button”) and then make another device do something in
response (toggling the state of the “room-lights” device).&lt;/p&gt;
&lt;p&gt;So what actually are these devices? At present in z2m I have the following:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Some generic buttons&lt;/li&gt;
&lt;li&gt;An air quality sensor (USB powered), and a separate temperature sensor (battery powered)&lt;/li&gt;
&lt;li&gt;A light switch&lt;/li&gt;
&lt;li&gt;Some 240V relay modules, that I use to turn non-smart devices on and off&lt;/li&gt;
&lt;li&gt;Some USB relay modules, for the same purpose&lt;/li&gt;
&lt;li&gt;Some “smart” plugs that I mostly use to monitor power usage&lt;/li&gt;
&lt;li&gt;A motorised blind roller&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I think all of these are either Tuya or Moes branded, and were all sourced from
AliExpress. You can get the same things from Amazon or elsewhere, but they tend
to be 3-4x more expensive for the same product.&lt;/p&gt;
&lt;p&gt;One of the cool things about Zigbee devices is that the powered ones work
together to create a mesh network, so you don’t have to worry about network
repeaters or signal strength like you do with Wi-Fi networks. z2m even makes
a map showing the connections:&lt;/p&gt;
&lt;figure class=&#34;image full&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/home-automation-without-megacorps/zigbee-map.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/home-automation-without-megacorps/zigbee-map.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/home-automation-without-megacorps/zigbee-map.png&#34; alt=&#34;A &amp;#34;map&amp;#34; showing how Zigbee devices connect to one another&#34; loading=&#34;lazy&#34; width=&#34;1093&#34; height=&#34;651&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;h3 id=&#34;some-interesting-automations&#34;&gt;Some interesting automations&lt;/h3&gt;
&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/home-automation-without-megacorps/relay.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/home-automation-without-megacorps/relay.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/home-automation-without-megacorps/relay.jpg&#34; alt=&#34;A Zigbee relay and some wago connectors wired up in a project box&#34; loading=&#34;lazy&#34; width=&#34;325&#34; height=&#34;500&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;A Zigbee relay spliced into the power cable for the fan&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;So what do I actually do with all of this? Some of it is just what I’ll call
“laziness automation”: I want to be able to turn on the lights without getting
up and walking to the light switch. So there’s a Zigbee button on my desk that
does it. Then there are some less manual automations: my blinds are
automatically closed at sunset&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;, for example. It feels a bit trivial, but it’s
surprisingly nice just not to have to think about that at all.&lt;/p&gt;
&lt;p&gt;Another nice quality of life automation is the light I have above my monitor.
It’s USB powered, and I now have it going through a Zigbee USB relay that lets
it be toggled on and off. Then I have a small agent running on my desktop that
turns the light on when the computer is unlocked, and off when it locks or
shuts down. I keep meaning to make a “film mode” that detects when I’m watching
a film, and turning all the lights and blinds down, but I haven’t got around
to it yet.&lt;/p&gt;
&lt;p&gt;The most complex automation is probably for a window fan. It’s not smart in
any way, so I cut the power cable and inserted a Zigbee relay. The whole thing
is housed in a little project box to keep it secure. The relay basically acts
as a switch: the live wire leading to the fan runs to the “normally open”&lt;sup id=&#34;fnref:4&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:4&#34; role=&#34;doc-noteref&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;
contact, while the upstream power goes to “common”. When the relay is told to
turn the fan on, the two get connected and the fan gets power. Oh, it’s also
not actually a window fan, but I designed and 3D printed some adapters so that
it sits right in the window.&lt;/p&gt;
&lt;p&gt;So now we have a window fan that can be turned off and on automatically. But I don’t
want to have to press a button to do that. I’m lazy, remember? Instead, I made
it so that the Go code constantly monitors the temperature reported by the air
quality sensor, and queries the free &lt;a href=&#34;https://openweathermap.org/&#34;&gt;OpenWeatherMap API&lt;/a&gt;
to get the rough temperature outside. It can then turn the fan on if the room is
too hot, and outside is cool enough to make a difference (there’s no point in
blowing hotter air in!).&lt;/p&gt;
&lt;figure class=&#34;image left&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/home-automation-without-megacorps/fan-graph.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/home-automation-without-megacorps/fan-graph.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/home-automation-without-megacorps/fan-graph.png&#34; alt=&#34;A graph of room temperature, slowly rising to 23.5 degrees, then sharply dropping to 19, rising to 20, dropping to 19 again, etc&#34; loading=&#34;lazy&#34; width=&#34;540&#34; height=&#34;500&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;A graph of room temperature, showing the effect of the fan being turned on and off&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;I don’t want it to do that all the time though. The room doesn’t need to be
cooled if I’m not around. There are various Zigbee presence sensors you can get,
but the cheaper ones seem to be of dubious quality. Instead, I wrote some code
to guess whether I’m present. It uses the state of the monitor light
as a proxy for “is the computer in use”, and then makes some guesses based on
the last time the computer was used and the current time (if I turn the computer
off at midnight it probably means I’m going to sleep; if I turn it off at 2pm
it probably means I’m going out somewhere).&lt;/p&gt;
&lt;p&gt;You can see in the graph that the fan has a decent impact on temperature. I
coded it to cool the room to 19 degrees, but then not turn on again until it
was above 20. This prevents it flip-flopping on and off constantly. You can
clearly see the pattern in the graph, as it actively cools and then slowly
the room warms back up. This arrangement is much better than being woken up at
5am because it’s painfully cold. Trust me.&lt;/p&gt;
&lt;h3 id=&#34;bonuses-metrics-and-3d-printer&#34;&gt;Bonuses: metrics and 3D printer&lt;/h3&gt;
&lt;p&gt;One of the things my custom Go code does is collate all the various stats
reported by the Zigbee devices, and inserts them into a &lt;a href=&#34;https://victoriametrics.com/&#34;&gt;VictoriaMetrics&lt;/a&gt;
database. I originally hosted this on the Pi itself, and it performed fine, but
I’ve since moved it onto a server so that I can use it for some other things
as well.&lt;/p&gt;
&lt;p&gt;I set up Grafana to point to VM, and can create dashboards showing power usage,
what devices are turned on when, and a bunch of environmental conditions. This
also makes it easy to spot how good the data coming from the devices are. For
example, the air quality sensor reports a figure for the amount of Carbon Dioxide
in the air, as well as the amount of Volatile Organic Compounds (VOCs). The
graphs are basically identical, but on a different scale. It turns out this
particular device has no actual way of detecting CO₂, so it just “calculates”
it from the VOCs figure. Useful to know if you want to actually use that data.&lt;/p&gt;
&lt;p&gt;Recently I came across &lt;a href=&#34;https://github.com/torbenconto/bambulabs_api&#34;&gt;a go library for interacting with Bambu Lab printers&lt;/a&gt;,
so I’ve also hooked that into my automation. It exports metrics about the
printer, so I now have a way of seeing what’s going on when I’m not physically
present&lt;sup id=&#34;fnref:5&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:5&#34; role=&#34;doc-noteref&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;. I even added a HTTP endpoint (exposed over &lt;a href=&#34;https://tailscale.com/&#34;&gt;Tailscale&lt;/a&gt;) that
shows pictures from the built-in camera.&lt;/p&gt;
&lt;p&gt;On the automation front, I’ve made it automatically turn off the printer’s light
when it finishes a print, and also added push notifications via &lt;a href=&#34;https://pushover.net/&#34;&gt;PushOver&lt;/a&gt;
whenever the state changes. No longer will I be sat in another room blissfully
unaware it ran out of filament seconds after I walked out the door!&lt;/p&gt;
&lt;h3 id=&#34;was-it-worth-it&#34;&gt;Was it worth it?&lt;/h3&gt;
&lt;p&gt;These days you could probably just buy some kind of ‘smart home’ hub that works
well enough, and do most of what I’ve done with a lot less effort and no coding.
I’m still of the opinion that for something so essentially &lt;em&gt;local&lt;/em&gt;, it should
itself be managed entirely locally. I don’t trust companies like Google or
Amazon not to kill their products, or change or remove an API I rely on.&lt;/p&gt;
&lt;p&gt;Obviously the “Not Invented Here” approach of coding everything yourself doesn’t
suit everyone, but as someone who enjoys coding and enjoys having things work
&lt;em&gt;just so&lt;/em&gt; it works very well for me.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;It turns out the chipset is important. I initially used a similar Sonoff
dongle that used a CC2531 chipset, and its performance wasn’t great. It often
failed to pair with new devices, and dropped links to existing ones. The
&lt;a href=&#34;https://www.zigbee2mqtt.io/advanced/zigbee/02_improve_network_range_and_stability.html&#34;&gt;zigbee2mqtt docs&lt;/a&gt;
do explicitly advise against the CC2531 chips for that reason. At the time they
recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d
go with whatever their latest recommendation was. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;MQTT is basically a network protocol for publishing and subscribing to
arbitrary messages. A broker is the thing that sits in the middle and routes
the messages. I use &lt;a href=&#34;https://mosquitto.org/&#34;&gt;Mosquitto&lt;/a&gt; but any will do. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;They originally also re-opened at sunrise but that was a terrible
mistake. Who knew the sun rose so early?! &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;
&lt;p&gt;“open” meaning “there’s a gap so it doesn’t work” not
“open for business”. This confusion in terminology also extends to drawbridges. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:4&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;
&lt;p&gt;I use the printer in LAN mode, which means there’s no way to monitor it
from a phone, even if you’re connected to the same network still. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:5&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Simple backups with Restic and Hetzner Cloud</title>
        <link href="https://chameth.com/simple-backups-restic-hetzner/"/>
        <updated>2024-12-06T00:00:00Z</updated>
        <id>https://chameth.com/simple-backups-restic-hetzner/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.png&#34; alt=&#34;The Restic logo — a gopher with two umbrellas.&#34; loading=&#34;lazy&#34; width=&#34;400&#34; height=&#34;400&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;I have a confession: for the past few years I’ve not been backing up any of my
computers. Everyone knows that you &lt;em&gt;should&lt;/em&gt; do backups, but actually getting
around to doing it is another story.&lt;/p&gt;
&lt;p&gt;Don’t get me wrong: most of my important things are “backed up” by virtue of
being committed to remote git repositories, or attached to e-mails, or
re-obtainable from the original source, and so on. I don’t think any machine
failing completely would be a disaster for me, but it would certainly be a pain.&lt;/p&gt;
&lt;p&gt;This week I finally got around to doing something, and it ended up being a lot
more straight forward than my previous forays into backup-land.&lt;/p&gt;
&lt;h3 id=&#34;restic&#34;&gt;Restic&lt;/h3&gt;
&lt;p&gt;After soliciting a few opinions, the choice of backup software came down to
either &lt;a href=&#34;https://www.borgbackup.org/&#34;&gt;Borg&lt;/a&gt; or &lt;a href=&#34;https://restic.net/&#34;&gt;Restic&lt;/a&gt;.
I’m pretty sure either would have done what I want, but I leaned towards Restic
for a few reasons: it has a more informative website, it’s written in Go
rather than Python&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;, and Borg seems to be transitioning between major
releases at the moment&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;The way Restic works is pretty simple: you initialise a ‘repository’, and can
then call &lt;code&gt;restic backup /some/path&lt;/code&gt; and it’ll get backed up to the repository.
Restic handles keeping different backups separate, and only sending data that’s
changed, and deduplicating, and so on. You basically point it at a thing you
don’t want to lose, and it sorts it out for you. Perfect.&lt;/p&gt;
&lt;p&gt;There are equally straight-forward commands for removing old snapshots
(&lt;code&gt;restic forget&lt;/code&gt;) and verifying backups (&lt;code&gt;restic check&lt;/code&gt;). One of the nice things
about modern backup solutions is they support a whole range of backends. I was
originally going to spin up a small VPS to host my backups, but noticed that
Restic supported S3-compatible stores…&lt;/p&gt;
&lt;h3 id=&#34;hetzner-cloud-object-storage&#34;&gt;Hetzner Cloud Object Storage&lt;/h3&gt;
&lt;p&gt;I host my servers with Hetzner, and was going to use them to spin up a VPS as
well. Despite the “Cloud” branding on a bunch of products, they offer reasonable
prices and good service. A couple of months ago, they started offering
&lt;a href=&#34;https://docs.hetzner.com/storage/object-storage/overview&#34;&gt;S3-compatible object storage&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The pricing isn’t totally straight forward, but for continuous use the “free
quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free,
as is traffic within the &lt;code&gt;eu-central&lt;/code&gt; region (where all my servers are). That
quota is only awarded when you pay the “base price”, though, which is €4.99 a
month. So basically it’s €5 a month for 1TB of storage and enough egress to
fully restore every single byte. That’s better value than any VPS I can find,
much cheaper than Amazon S3, and about the same as
&lt;a href=&#34;https://www.backblaze.com/cloud-storage&#34;&gt;Backblaze B2&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&#34;getting-them-to-work-together&#34;&gt;Getting them to work together&lt;/h3&gt;
&lt;p&gt;Now you’d think making the tool that supports S3-compatible object storage
work with your S3-compatible object storage would be easy, right? Well not
quite. The library Restic uses to deal with S3 backends has some
strange logic for figuring out the bucket name given a URL. It doesn’t quite
seem to work right, though…&lt;/p&gt;
&lt;p&gt;Hetzner buckets have URLs like &lt;code&gt;s3://bucketname.hel1.your-objectstorage.com&lt;/code&gt;&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;,
but just passing that to Restic gives an error that the bucket is not specified.
The docs mention there’s an advanced option to make it use the virtual host for
the bucket name: &lt;code&gt;-o s3.bucket-lookup=dns&lt;/code&gt;. But that… also doesn’t work.
What I ended up doing was specifying the URL as
&lt;code&gt;s3://hel1.your-objectstorage.com/bucketname&lt;/code&gt;, and also passing in the &lt;code&gt;dns&lt;/code&gt;
option. The library then seems to muddle its way back to a real, working URL.
I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that
I’m oblivious to?&lt;/p&gt;
&lt;p&gt;The next fun part is that you can configure Restic entirely by using environment
variables, except for that &lt;code&gt;-o s3.bucket-lookup=dns&lt;/code&gt; argument. That has to go
on the command line. I ended up making a little wrapper script to invoke Restic
correctly:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-cp&#34;&gt;#!/bin/sh
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The password restic will use to encrypt your data. You should generate&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# something nice and secure.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;RESTIC_PASSWORD&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;repo-password
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The path to the repository where restic will save the backup. In our case&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# this takes the form `s3:&amp;lt;endpoint&amp;gt;/&amp;lt;bucket&amp;gt;`. Your endpoint might be different&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# to mine depending on the region your bucket is in.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;RESTIC_REPOSITORY&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;s3:hel1.your-objectstorage.com/bucket-name
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The access and secret key generated in the &amp;#39;S3 credentials&amp;#39; section of the&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Hetzner Cloud Console&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;AWS_ACCESS_KEY_ID&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;hetzner-access-key-id
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;AWS_SECRET_ACCESS_KEY&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;hetzner-access-key
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Pass any arguments on to the restic command, along with the magic&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# `s3.bucket-lookup` option we need to resolve the S3 URL properly.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;exec&lt;/span&gt; restic -o s3.bucket-lookup&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;dns &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-nv&#34;&gt;$@&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then in the backup script I just alias &lt;code&gt;restic&lt;/code&gt; to use the script:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-cp&#34;&gt;#!/bin/bash
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;set&lt;/span&gt; -eu
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# This means any time we use `restic` below, we&amp;#39;ll actually execute our special&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# script which supplies all the env vars and arguments needed to find the&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# repository. Make sure the path matches where you saved the script!&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;alias&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;restic&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;chroma-s1&#34;&gt;&amp;#39;~/.bin/restic&amp;#39;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Actually do the backup. Each directory in the list below will be backed up&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# separately.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nv&#34;&gt;dirs&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;/some/path/to/backup/&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;/some/other/path/&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;for&lt;/span&gt; i in &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-si&#34;&gt;${&lt;/span&gt;&lt;span class=&#34;chroma-nv&#34;&gt;dirs&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;[@]&lt;/span&gt;&lt;span class=&#34;chroma-si&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt; 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;do&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nb&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;$i&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-o&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-nb&#34;&gt;cd&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;$i&lt;/span&gt; &lt;span class=&#34;chroma-o&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt; restic --verbose backup .&lt;span class=&#34;chroma-o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;done&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# see what effect any changes would have before actually committing to them. &lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;restic forget --keep-daily &lt;span class=&#34;chroma-m&#34;&gt;7&lt;/span&gt; --keep-weekly &lt;span class=&#34;chroma-m&#34;&gt;10&lt;/span&gt; --keep-monthly &lt;span class=&#34;chroma-m&#34;&gt;24&lt;/span&gt; --keep-yearly &lt;span class=&#34;chroma-m&#34;&gt;10&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The only other notable thing here is that the script changes into the directory
to be backed up. If you give Restic an absolute path, it will create a new
snapshot if the metadata of any folder in the path changes, which is not what
I want.&lt;/p&gt;
&lt;p&gt;If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken
me about half an hour. That’s including setting up the object storage,
installing Restic, and so on. It’s painfully easy. Why didn’t I do this three
years ago?!&lt;/p&gt;
&lt;h3 id=&#34;addendum-a-step-by-step-guide&#34;&gt;Addendum: a step-by-step guide&lt;/h3&gt;
&lt;aside class=&#34;update raised-box&#34;&gt;
  &lt;h5 class=&#34;plain-header&#34;&gt;Update 2025-03-01:&lt;/h5&gt;
  &lt;p&gt;This section was added after the original article was published, following some
helpful feedback. Let me know if you have any problem with these instructions!&lt;/p&gt;
&lt;/aside&gt;
&lt;p&gt;If you want to do this yourself, here’s a quick step-by-step guide:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Log in to the &lt;a href=&#34;https://console.hetzner.cloud&#34;&gt;Hetzner Cloud Console&lt;/a&gt;, and
create a project.&lt;/li&gt;
&lt;li&gt;On the “Object Storage” tab, create a new bucket. Note the name and the
endpoint.&lt;/li&gt;
&lt;li&gt;On the “Security” tab, go to “S3 Credentials” and generate new credentials.
Note down the access key and the secret key.&lt;/li&gt;
&lt;li&gt;Copy the first shell script above, fill in the password (you can pick!),
endpoint, bucket name, access key and secret key.&lt;/li&gt;
&lt;li&gt;Run the script with the &lt;code&gt;init&lt;/code&gt; argument (e.g. &lt;code&gt;~/.bin/restic init&lt;/code&gt;). This
will create a new repository, and only needs to be done once even if you
backup multiple machines.&lt;/li&gt;
&lt;li&gt;Copy the second shell script above, making sure the &lt;code&gt;restic&lt;/code&gt; alias points
at the script you saved in step 4. Change the list of directories to
whatever you want to backup.&lt;/li&gt;
&lt;li&gt;Schedule the script to be run automatically, using crontab or systemd timers
or however you prefer.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;To check everything is working, you can use the &lt;code&gt;snapshots&lt;/code&gt; subcommand to see
a list of saved snapshots. You might also want to try to restore a snapshot
using the &lt;code&gt;restore&lt;/code&gt; subcommand.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I’m not trying to be a language snob, but given the choice between two
otherwise equal projects one in Go and one in Python, I’ll take the Go one.
I know I’m not going to have weird library issues down the line, and I’m much
more comfortable rummaging around the source. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;With a “Don’t use this in production!” notice on the shiny new version. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;Aside: I really hate URLs like that. What is with the trend for completely
generic domains divorced from the service they’re a part of? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Further adventures in 3D printing</title>
        <link href="https://chameth.com/further-adventures-in-3d-printing/"/>
        <updated>2024-08-04T00:00:00Z</updated>
        <id>https://chameth.com/further-adventures-in-3d-printing/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/further-adventures-in-3d-printing/p1s.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/further-adventures-in-3d-printing/p1s.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/further-adventures-in-3d-printing/p1s.png&#34; alt=&#34;Marketing image of the Bambu Labs P1S&#34; loading=&#34;lazy&#34; width=&#34;404&#34; height=&#34;400&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;The Bambu Labs P1S.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Not quite a year ago, I bought a Sovol SV06 3D printer and &lt;a href=&#34;https://chameth.com/adventures-in-3d-printing/&#34;&gt;wrote about&lt;/a&gt;
my initial experiences. At the end I joked:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;I’m sure there will be more problems, but hopefully I get at least a week or
so before anything else goes wrong. Maybe I’ll print an “X days since the
last SV06 problem” counter…&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;That turned out to be more prophetic than I expected. It became a running joke
that I’d flip the imaginary “days since” sign back to 0 almost every time I
printed. I still really enjoyed having a printer, though: it’s so useful to
be able to be able to go from an idea to a CAD drawing to a physical object
in the space of minutes or hours. So when Bambu Labs reduced the price of the
&lt;a href=&#34;https://uk.store.bambulab.com/products/p1s&#34;&gt;P1S&lt;/a&gt; — a printer generally regarded
as about as trouble free as you can get — to £522 including shipping, I splurged on one.&lt;/p&gt;
&lt;h3 id=&#34;concerns&#34;&gt;Concerns&lt;/h3&gt;
&lt;p&gt;I make it sound like that was an easy decision, but I actually ummed and ahed
a lot before pulling the trigger. I had a lot of objections to a Bambu Labs
printer, and wasn’t quite sure if the price made up for them.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;The Bambu Labs printers are designed to be more consumer friendly than most
previous 3D printers. You can start a model printing directly from their
website or mobile app, which employs a cloud-based slicer that then sends the
model down to your printer. This is one of those things that sounds amazing for
consumers, but should terrify anyone who works with computers. There are so many
ways for a “cloud-connected” printer to go wrong, and with such potentially
disastrous results it gives me anxiety just thinking about it.&lt;/p&gt;
&lt;p&gt;This is not just theoretical, either. In a blog post with the rather under-stated
title of &lt;a href=&#34;https://blog.bambulab.com/cloud-temporary-outage-investigation/&#34;&gt;“Initial Investigation in the Bambu Cloud Temporary Outage”&lt;/a&gt;,
Bambu describe how a server-side error caused printers to start printing unprompted.
Users on Reddit reported printers starting in the middle of the night, trying
to print on top of other objects that were left on the build plate, and so on.
3D printers can easily start fires; having them cloud controlled is frankly
insane.&lt;/p&gt;
&lt;p&gt;As a result of that “temporary outage”, Bambu further improved the “LAN mode”
for their printers, which allows you to sever its connection from their servers
and control it over the network. That’s basically how my old printer worked
anyway, so I was happy enough that I could enable LAN mode and ignore all the
cloud nonsense.&lt;/p&gt;
&lt;p&gt;My other objection to Bambu was more nebulous. Almost all 3D printers are built
upon open source software, and some companies like Prusa go out of their way to
make their products more open to give back to the community. Bambu don’t really
do any of that: I’d be very surprised if they &lt;em&gt;hadn’t&lt;/em&gt; incorporated a bunch of
open source software into their printers and just ignored the licences&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. I’d
much rather give my money to a company doing good, but the only equivalent
printer was the &lt;a href=&#34;https://www.prusa3d.com/product/original-prusa-mk4-2/&#34;&gt;Prusa MK4&lt;/a&gt;
which was &lt;em&gt;double&lt;/em&gt; the price and didn’t come with an enclosure. To add an
enclosure kit would have made it £800 more expensive than the P1S, and I just
can’t justify that.&lt;/p&gt;
&lt;h3 id=&#34;experience&#34;&gt;Experience&lt;/h3&gt;
&lt;p&gt;The initial setup of the P1S was a breeze. While the SV06 shipped in parts, the
P1S came fully assembled and just needed some packaging and retaining screws
removed. These were all clearly marked with large coloured arrows. It’s very
much designed so anyone can do it.&lt;/p&gt;
&lt;p&gt;I ran through the initial calibration, firmware update, and then put it
in LAN mode. Then I just started printing and it went fine. No messing around
levelling a bed, or using a sheet of paper to try and set the Z-offset properly;
it just worked. It was &lt;em&gt;really&lt;/em&gt; loud though. After talking a bit with a friend
who’d also bought a P1S, I realised that most of his noise was from the fans,
while most of mine was from the servos. A bit of research later and I found out
they added servo noise compensation in a firmware update, but I hadn’t run the
calibration for it because the initial calibration happened before the update.
After doing that, the printer went from sounding like it was impersonating a
continental police siren to just being a bit fan-y.&lt;/p&gt;
&lt;p&gt;The biggest difference over the SV06 is the speed. I’d swapped the nozzle on the
SV06 to a 0.6mm one, and generally printed at a layer height of 0.4mm. The P1S
still has its 0.4mm nozzle, and I mostly print at a layer height of 0.2mm. So
it’s laying down twice the number of layers, with a nozzle that can extrude
half the amount of plastic… And it’s at least twice as fast! Prints that I
previously wouldn’t have even attempted on the SV06 are now 8-10 hours on the
P1S, without even adjusting the speed or lowering the quality. Here’s a
side-by-side comparison:&lt;/p&gt;
&lt;figure class=&#34;image full&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/further-adventures-in-3d-printing/comparison.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/further-adventures-in-3d-printing/comparison.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/further-adventures-in-3d-printing/comparison.png&#34; alt=&#34;A side-by-side comparison of printing time on the P1S and SV06. The P1S prints 8 objects at higher quality than the SV06 prints 3.&#34; loading=&#34;lazy&#34; width=&#34;1856&#34; height=&#34;944&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;P1S vs SV06.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;The P1S can print eight objects because of its slightly larger bed, while the
SV06 can only manage three. The P1S is also printing at a much higher quality.
Yet somehow it still ends up being faster. It’s mind blowing.&lt;/p&gt;
&lt;p&gt;The other big difference I’ve not mentioned yet is that the P1S is a “core XY”
printer rather than a “bed slinger”. That means in order to move along the
Y-axis (from the front of the bed to the back), the P1S moves the extruder
while the SV06 moves the bed. This makes a big difference when printing tall
objects, as they’re far more likely to wobble and come loose when the bed is
moving back and forth. It also presumably makes the kinematics a lot easier
as the mass of the moving parts is constant, instead of gradually becoming
heavier as more and more plastic gets added.&lt;/p&gt;
&lt;h3 id=&#34;the-only-failures&#34;&gt;The only failures&lt;/h3&gt;
&lt;p&gt;I’ve only had a handful of failures since getting the P1S. I had one weird
issue where the slicer decided to print some parts extremely slowly, and the
print just didn’t work out in those parts; that’s more of a slicer issue,
though: after tweaking the model to stop the weird behaviour it printed fine.&lt;/p&gt;
&lt;p&gt;The only failure I can directly attribute to the printer was when I tried
ironing. This is a process where after printing the top layer, the printer
slowly goes back over it at the same height while extruding a small amount of
plastic. This is meant to fill in the small gaps between extrusions and give
a smooth finish. In my case, though, it just clogged the nozzle and extruder.&lt;/p&gt;
&lt;p&gt;At first I thought it was because I was printing too hot: the profile I was
using was designed for fast printing, so had to melt the filament rapidly, but
there was no way to turn that temperature down just for the ironing pass. I
tried another run at a lower temperature and everything clogged again. I then
swore off ironing, did a normal print, and everything clogged once more.&lt;/p&gt;
&lt;p&gt;At this point I was getting exceedingly good at disassembling the hot end, but
also fairly annoyed. It felt like I was using an SV06 again&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;! I’d previously
read some recommendations that you should leave the door or top of the enclosure
open when printing PLA, but had never really had any issues so hadn’t bothered.
I tried a print with the top propped open and everything worked smoothly again.
In hindsight all these clogging issues happened when it was particularly hot
and humid outside, which was probably a factor.&lt;/p&gt;
&lt;h3 id=&#34;summary&#34;&gt;Summary&lt;/h3&gt;
&lt;p&gt;Upgrading to the P1S has made the 3D printer feel more like a tool and less like
an additional ongoing project. The speed increase continues to amaze me, as
does the increase in quality.&lt;/p&gt;
&lt;p&gt;I still think it was valuable to have the SV06 to learn how printers work — you
definitely don’t get as much of that from using something so consumer-focused
as the P1S — and figure out if I was actually going to use it long-term or
whether it was just a fad for me. So it served its purpose, but I’m rather
glad to see the back of it in favour of something better in pretty much every
way.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;And if they did there’s nothing anyone can do about it because copyright
doesn’t exist in the same way in China. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;Although taking apart the P1S (which is a proprietary machine you might
not expect to be very user-servicable) was orders of magnitude easier than taking
apart the SV06 (which was an open source design). Everything is just much nicer:
the connectors are easier to attach and detach, the screws are all a consistent
size instead of needing a whole suite of allen keys, and so on. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
</feed>
