<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="/feeds.xsl" type="text/xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:base="https://chameth.com/">
    <title>Chameth.com - posts like break-everything-fuzz-testing, debugging-beyond-the-debugger, docker-automatic-nginx-proxy, thinking-more-about-backups, why-you-should-be-using-https but not finding-an-awkward-bug-with-claude-code</title>
    <subtitle>Personal homepage of Chris Smith</subtitle>
    <link href="https://chameth.com/feeds/posts/like/break-everything-fuzz-testing,debugging-beyond-the-debugger,docker-automatic-nginx-proxy,thinking-more-about-backups,why-you-should-be-using-https/unlike/finding-an-awkward-bug-with-claude-code/" rel="self"/>
    <link href="https://chameth.com/"/>
    <icon>https://chameth.com/favicon.png</icon>
    <updated>2026-01-29T00:00:00Z</updated>
    <id>https://chameth.com/</id>
    <author>
        <name>Chris Smith</name>
    </author>
    <entry>
        <title>An interesting Tailscale + Docker gotcha</title>
        <link href="https://chameth.com/tailscale-docker-gotcha/"/>
        <updated>2026-01-29T00:00:00Z</updated>
        <id>https://chameth.com/tailscale-docker-gotcha/</id>
        <content xml:lang="en" type="html">&lt;p&gt;As I’ve &lt;a href=&#34;https://chameth.com/how-i-use-tailscale/&#34;&gt;written&lt;/a&gt; &lt;a href=&#34;https://chameth.com/avoiding-the-consequences-of-dumb-laws-with-tailscale/&#34;&gt;about&lt;/a&gt; &lt;a href=&#34;https://chameth.com/exposing-game-servers-over-tailscale/&#34;&gt;before&lt;/a&gt;, I use &lt;a href=&#34;https://tailscale.com/&#34;&gt;Tailscale&lt;/a&gt; for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.&lt;/p&gt;
&lt;p&gt;I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an &lt;code&gt;app&lt;/code&gt; node can’t access anything via Tailscale, while an &lt;code&gt;integration&lt;/code&gt; or &lt;code&gt;server&lt;/code&gt; node can access things tagged with either &lt;code&gt;app&lt;/code&gt; or &lt;code&gt;integration&lt;/code&gt;. This is expressed pretty simply in the Tailscale ACL JSON:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tagOwners&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Servers that can be SSH&amp;#39;d into
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Applications that are exposed on tailscale but never connect out
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-c1&#34;&gt;// Things which talk to other services over tailscale (connecting to apps etc)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;grants&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Users can access everything
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;autogroup:member&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers and integrations can access integrations and apps
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;dst&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;ip&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;  &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;*&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Over the past week I’ve been setting up a private &lt;a href=&#34;https://forgejo.org/&#34;&gt;Forgejo&lt;/a&gt; instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (&lt;code&gt;forgejo:3000&lt;/code&gt;) instead of using full Tailscale HTTPS URL that I use when accessing it (&lt;code&gt;http://git.example-net.ts.net/&lt;/code&gt;)&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!&lt;/p&gt;
&lt;p&gt;This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my &lt;code&gt;server&lt;/code&gt; tag. That creates a &lt;code&gt;tailscale0&lt;/code&gt; interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the &lt;code&gt;tailscale0&lt;/code&gt; interface.&lt;/p&gt;
&lt;p&gt;I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the &lt;code&gt;tailscale0&lt;/code&gt; interface if it originated from the IP ranges that Docker was configured to use:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;DOCKER-USER&lt;/code&gt; chain is a nice little escape hatch; it comes before the main auto-generated &lt;code&gt;DOCKER&lt;/code&gt; chain, and Docker leaves the rules in it alone.&lt;/p&gt;
&lt;p&gt;So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS&lt;sup id=&#34;fnref:4&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:4&#34; role=&#34;doc-noteref&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;. The server’s DNS resolver is &lt;code&gt;100.100.100.100&lt;/code&gt;, a special Tailscale address. This is so that it can resolve tailnet hostnames via &lt;a href=&#34;https://tailscale.com/kb/1081/magicdns&#34;&gt;MagicDNS&lt;/a&gt; (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.&lt;/p&gt;
&lt;p&gt;Obviously the solution here is to double down and add MORE iptables rules:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use &lt;code&gt;iptables-save&lt;/code&gt;, and then you also get all the rules that both Docker and Tailscale have inserted. I came across &lt;a href=&#34;https://github.com/giangi/iptables-docker-filter&#34;&gt;a script&lt;/a&gt; to filter out the Docker ones, but… yuck&lt;sup id=&#34;fnref:5&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:5&#34; role=&#34;doc-noteref&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;I realised a cleaner idea might just be to lock down what &lt;code&gt;server&lt;/code&gt; tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to &lt;code&gt;curl&lt;/code&gt; services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for &lt;code&gt;infrastructure&lt;/code&gt;, which can be accessed from &lt;code&gt;server&lt;/code&gt; devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the &lt;code&gt;tests&lt;/code&gt; feature of Tailscale’s ACL config to make sure I’d got the rules right:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;tests&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-c1&#34;&gt;// Servers can only access infrastructure
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;src&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt;   &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;proto&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;allow&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:infrastructure:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-nt&#34;&gt;&amp;#34;deny&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;chroma-p&#34;&gt;[&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;100.84.16.43:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;me@example.com:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:server:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:app:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;				&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;tag:integration:8080&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;			&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;		&lt;span class=&#34;chroma-p&#34;&gt;},&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-p&#34;&gt;],&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I discovered later on that logging into the container registry at &lt;code&gt;forgejo:3000&lt;/code&gt; actually issued a redirect to &lt;code&gt;https://git.example-net.ts.net/&lt;/code&gt; so this was all basically for naught… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;
&lt;p&gt;It’s always DNS… &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:4&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;
&lt;p&gt;Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:5&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Thinking more about backups</title>
        <link href="https://chameth.com/thinking-more-about-backups/"/>
        <updated>2025-11-01T00:00:00Z</updated>
        <id>https://chameth.com/thinking-more-about-backups/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/thinking-more-about-backups/backblaze.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/thinking-more-about-backups/backblaze.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/thinking-more-about-backups/backblaze.png&#34; alt=&#34;The Backblaze logo: a stylised flame above the word Backblaze&#34; loading=&#34;lazy&#34; width=&#34;500&#34; height=&#34;320&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;The Backblaze logo&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Almost a year ago I wrote about &lt;a href=&#34;https://chameth.com/simple-backups-restic-hetzner/&#34;&gt;how I do backups with Restic and Hetzner&lt;/a&gt;.
That system has been ticking along well ever since, but recently I had some… thoughts. These backups are
all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are
two glaring issues:&lt;/p&gt;
&lt;p&gt;Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or
my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation
and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse
report get it suspended? What if the UK passes
&lt;a href=&#34;https://www.legislation.gov.uk/ukpga/2023/50/contents&#34;&gt;even more dumb laws&lt;/a&gt; and Hetzner decide it’s easier
just to not do business with people here? This is the same sort of concern I have about Google accounts:
if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube
video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very
likely, but these are things my brain likes to dwell on.&lt;/p&gt;
&lt;p&gt;Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious
gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels
a little silly that the same attack could take down both the originals and the backups. There’s no way to
avoid that with Hetzner’s S3 implementation, as far as I can tell.&lt;/p&gt;
&lt;h3 id=&#34;exploring-options&#34;&gt;Exploring options&lt;/h3&gt;
&lt;p&gt;I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems
would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that,
and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in
the worst case.&lt;/p&gt;
&lt;p&gt;I then looked around at other S3 providers. &lt;a href=&#34;https://aws.amazon.com/s3/storage-classes/glacier/&#34;&gt;Amazon’s glacier offering&lt;/a&gt;
is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything.
There are also lots of weird pricing edge cases around moving data between storage classes, minimum file
sizes, and so on. A much better option is &lt;a href=&#34;https://www.backblaze.com/cloud-storage&#34;&gt;Backblaze’s B2&lt;/a&gt; product.
Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful
in this case: &lt;a href=&#34;https://www.backblaze.com/blog/backblaze-b2-lifecycle-rules/&#34;&gt;lifecycle rules&lt;/a&gt;. Coupled with
the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for
what’s effectively an append-only store.&lt;/p&gt;
&lt;p&gt;This works more-or-less out of the box with Restic. &lt;a href=&#34;https://pricey.uk/blog/restic-backups-without-delete/&#34;&gt;Joseph Price has a guide&lt;/a&gt;
that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during
a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules
decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to
the end of my script: a simple &lt;code&gt;restic copy&lt;/code&gt; and a &lt;code&gt;restic forget&lt;/code&gt;. B2 actually works out cheaper than the
Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month.
Not a bad price for some extra peace of mind!&lt;/p&gt;
</content>
    </entry>
    <entry>
        <title>Simple backups with Restic and Hetzner Cloud</title>
        <link href="https://chameth.com/simple-backups-restic-hetzner/"/>
        <updated>2024-12-06T00:00:00Z</updated>
        <id>https://chameth.com/simple-backups-restic-hetzner/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image right&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/simple-backups-restic-hetzner/restic.png&#34; alt=&#34;The Restic logo — a gopher with two umbrellas.&#34; loading=&#34;lazy&#34; width=&#34;400&#34; height=&#34;400&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;I have a confession: for the past few years I’ve not been backing up any of my
computers. Everyone knows that you &lt;em&gt;should&lt;/em&gt; do backups, but actually getting
around to doing it is another story.&lt;/p&gt;
&lt;p&gt;Don’t get me wrong: most of my important things are “backed up” by virtue of
being committed to remote git repositories, or attached to e-mails, or
re-obtainable from the original source, and so on. I don’t think any machine
failing completely would be a disaster for me, but it would certainly be a pain.&lt;/p&gt;
&lt;p&gt;This week I finally got around to doing something, and it ended up being a lot
more straight forward than my previous forays into backup-land.&lt;/p&gt;
&lt;h3 id=&#34;restic&#34;&gt;Restic&lt;/h3&gt;
&lt;p&gt;After soliciting a few opinions, the choice of backup software came down to
either &lt;a href=&#34;https://www.borgbackup.org/&#34;&gt;Borg&lt;/a&gt; or &lt;a href=&#34;https://restic.net/&#34;&gt;Restic&lt;/a&gt;.
I’m pretty sure either would have done what I want, but I leaned towards Restic
for a few reasons: it has a more informative website, it’s written in Go
rather than Python&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;, and Borg seems to be transitioning between major
releases at the moment&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:2&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;The way Restic works is pretty simple: you initialise a ‘repository’, and can
then call &lt;code&gt;restic backup /some/path&lt;/code&gt; and it’ll get backed up to the repository.
Restic handles keeping different backups separate, and only sending data that’s
changed, and deduplicating, and so on. You basically point it at a thing you
don’t want to lose, and it sorts it out for you. Perfect.&lt;/p&gt;
&lt;p&gt;There are equally straight-forward commands for removing old snapshots
(&lt;code&gt;restic forget&lt;/code&gt;) and verifying backups (&lt;code&gt;restic check&lt;/code&gt;). One of the nice things
about modern backup solutions is they support a whole range of backends. I was
originally going to spin up a small VPS to host my backups, but noticed that
Restic supported S3-compatible stores…&lt;/p&gt;
&lt;h3 id=&#34;hetzner-cloud-object-storage&#34;&gt;Hetzner Cloud Object Storage&lt;/h3&gt;
&lt;p&gt;I host my servers with Hetzner, and was going to use them to spin up a VPS as
well. Despite the “Cloud” branding on a bunch of products, they offer reasonable
prices and good service. A couple of months ago, they started offering
&lt;a href=&#34;https://docs.hetzner.com/storage/object-storage/overview&#34;&gt;S3-compatible object storage&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The pricing isn’t totally straight forward, but for continuous use the “free
quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free,
as is traffic within the &lt;code&gt;eu-central&lt;/code&gt; region (where all my servers are). That
quota is only awarded when you pay the “base price”, though, which is €4.99 a
month. So basically it’s €5 a month for 1TB of storage and enough egress to
fully restore every single byte. That’s better value than any VPS I can find,
much cheaper than Amazon S3, and about the same as
&lt;a href=&#34;https://www.backblaze.com/cloud-storage&#34;&gt;Backblaze B2&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&#34;getting-them-to-work-together&#34;&gt;Getting them to work together&lt;/h3&gt;
&lt;p&gt;Now you’d think making the tool that supports S3-compatible object storage
work with your S3-compatible object storage would be easy, right? Well not
quite. The library Restic uses to deal with S3 backends has some
strange logic for figuring out the bucket name given a URL. It doesn’t quite
seem to work right, though…&lt;/p&gt;
&lt;p&gt;Hetzner buckets have URLs like &lt;code&gt;s3://bucketname.hel1.your-objectstorage.com&lt;/code&gt;&lt;sup id=&#34;fnref:3&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:3&#34; role=&#34;doc-noteref&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;,
but just passing that to Restic gives an error that the bucket is not specified.
The docs mention there’s an advanced option to make it use the virtual host for
the bucket name: &lt;code&gt;-o s3.bucket-lookup=dns&lt;/code&gt;. But that… also doesn’t work.
What I ended up doing was specifying the URL as
&lt;code&gt;s3://hel1.your-objectstorage.com/bucketname&lt;/code&gt;, and also passing in the &lt;code&gt;dns&lt;/code&gt;
option. The library then seems to muddle its way back to a real, working URL.
I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that
I’m oblivious to?&lt;/p&gt;
&lt;p&gt;The next fun part is that you can configure Restic entirely by using environment
variables, except for that &lt;code&gt;-o s3.bucket-lookup=dns&lt;/code&gt; argument. That has to go
on the command line. I ended up making a little wrapper script to invoke Restic
correctly:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-cp&#34;&gt;#!/bin/sh
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The password restic will use to encrypt your data. You should generate&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# something nice and secure.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;RESTIC_PASSWORD&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;repo-password
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The path to the repository where restic will save the backup. In our case&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# this takes the form `s3:&amp;lt;endpoint&amp;gt;/&amp;lt;bucket&amp;gt;`. Your endpoint might be different&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# to mine depending on the region your bucket is in.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;RESTIC_REPOSITORY&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;s3:hel1.your-objectstorage.com/bucket-name
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# The access and secret key generated in the &amp;#39;S3 credentials&amp;#39; section of the&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Hetzner Cloud Console&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;AWS_ACCESS_KEY_ID&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;hetzner-access-key-id
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;AWS_SECRET_ACCESS_KEY&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;hetzner-access-key
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Pass any arguments on to the restic command, along with the magic&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# `s3.bucket-lookup` option we need to resolve the S3 URL properly.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;exec&lt;/span&gt; restic -o s3.bucket-lookup&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;dns &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-nv&#34;&gt;$@&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then in the backup script I just alias &lt;code&gt;restic&lt;/code&gt; to use the script:&lt;/p&gt;
&lt;pre class=&#34;chroma-chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-cp&#34;&gt;#!/bin/bash
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;set&lt;/span&gt; -eu
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# This means any time we use `restic` below, we&amp;#39;ll actually execute our special&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# script which supplies all the env vars and arguments needed to find the&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# repository. Make sure the path matches where you saved the script!&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nb&#34;&gt;alias&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;restic&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;chroma-s1&#34;&gt;&amp;#39;~/.bin/restic&amp;#39;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Actually do the backup. Each directory in the list below will be backed up&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# separately.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-nv&#34;&gt;dirs&lt;/span&gt;&lt;span class=&#34;chroma-o&#34;&gt;=(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;/some/path/to/backup/&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;/some/other/path/&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;for&lt;/span&gt; i in &lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span class=&#34;chroma-si&#34;&gt;${&lt;/span&gt;&lt;span class=&#34;chroma-nv&#34;&gt;dirs&lt;/span&gt;&lt;span class=&#34;chroma-p&#34;&gt;[@]&lt;/span&gt;&lt;span class=&#34;chroma-si&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;chroma-s2&#34;&gt;&amp;#34;&lt;/span&gt; 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;do&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-nb&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;$i&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;	&lt;span class=&#34;chroma-o&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;chroma-nb&#34;&gt;cd&lt;/span&gt; &lt;span class=&#34;chroma-nv&#34;&gt;$i&lt;/span&gt; &lt;span class=&#34;chroma-o&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt; restic --verbose backup .&lt;span class=&#34;chroma-o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-k&#34;&gt;done&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;&lt;span class=&#34;chroma-c1&#34;&gt;# see what effect any changes would have before actually committing to them. &lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;chroma-line&#34;&gt;&lt;span class=&#34;chroma-cl&#34;&gt;restic forget --keep-daily &lt;span class=&#34;chroma-m&#34;&gt;7&lt;/span&gt; --keep-weekly &lt;span class=&#34;chroma-m&#34;&gt;10&lt;/span&gt; --keep-monthly &lt;span class=&#34;chroma-m&#34;&gt;24&lt;/span&gt; --keep-yearly &lt;span class=&#34;chroma-m&#34;&gt;10&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The only other notable thing here is that the script changes into the directory
to be backed up. If you give Restic an absolute path, it will create a new
snapshot if the metadata of any folder in the path changes, which is not what
I want.&lt;/p&gt;
&lt;p&gt;If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken
me about half an hour. That’s including setting up the object storage,
installing Restic, and so on. It’s painfully easy. Why didn’t I do this three
years ago?!&lt;/p&gt;
&lt;h3 id=&#34;addendum-a-step-by-step-guide&#34;&gt;Addendum: a step-by-step guide&lt;/h3&gt;
&lt;aside class=&#34;update raised-box&#34;&gt;
  &lt;h5 class=&#34;plain-header&#34;&gt;Update 2025-03-01:&lt;/h5&gt;
  &lt;p&gt;This section was added after the original article was published, following some
helpful feedback. Let me know if you have any problem with these instructions!&lt;/p&gt;
&lt;/aside&gt;
&lt;p&gt;If you want to do this yourself, here’s a quick step-by-step guide:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Log in to the &lt;a href=&#34;https://console.hetzner.cloud&#34;&gt;Hetzner Cloud Console&lt;/a&gt;, and
create a project.&lt;/li&gt;
&lt;li&gt;On the “Object Storage” tab, create a new bucket. Note the name and the
endpoint.&lt;/li&gt;
&lt;li&gt;On the “Security” tab, go to “S3 Credentials” and generate new credentials.
Note down the access key and the secret key.&lt;/li&gt;
&lt;li&gt;Copy the first shell script above, fill in the password (you can pick!),
endpoint, bucket name, access key and secret key.&lt;/li&gt;
&lt;li&gt;Run the script with the &lt;code&gt;init&lt;/code&gt; argument (e.g. &lt;code&gt;~/.bin/restic init&lt;/code&gt;). This
will create a new repository, and only needs to be done once even if you
backup multiple machines.&lt;/li&gt;
&lt;li&gt;Copy the second shell script above, making sure the &lt;code&gt;restic&lt;/code&gt; alias points
at the script you saved in step 4. Change the list of directories to
whatever you want to backup.&lt;/li&gt;
&lt;li&gt;Schedule the script to be run automatically, using crontab or systemd timers
or however you prefer.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;To check everything is working, you can use the &lt;code&gt;snapshots&lt;/code&gt; subcommand to see
a list of saved snapshots. You might also want to try to restore a snapshot
using the &lt;code&gt;restore&lt;/code&gt; subcommand.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I’m not trying to be a language snob, but given the choice between two
otherwise equal projects one in Go and one in Python, I’ll take the Go one.
I know I’m not going to have weird library issues down the line, and I’m much
more comfortable rummaging around the source. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;With a “Don’t use this in production!” notice on the shiny new version. &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:2&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:3&#34;&gt;
&lt;p&gt;Aside: I really hate URLs like that. What is with the trend for completely
generic domains divorced from the service they’re a part of? &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:3&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
    <entry>
        <title>Docker reverse proxying, redux</title>
        <link href="https://chameth.com/docker-proxying-redux/"/>
        <updated>2022-12-06T00:00:00Z</updated>
        <id>https://chameth.com/docker-proxying-redux/</id>
        <content xml:lang="en" type="html">&lt;p&gt;Six years ago, &lt;a href=&#34;https://chameth.com/docker-automatic-nginx-proxy/&#34;&gt;I described&lt;/a&gt;
my system for configuring a reverse proxy for docker containers.
It involved six containers including a key-value store and a webserver.
Nothing in that system has persisted to this day. Don’t get me wrong – it
worked – but there were a lot of rough edges and areas for improvement.&lt;/p&gt;
&lt;h3 id=&#34;microservices-and-their-limitations&#34;&gt;Microservices and their limitations&lt;/h3&gt;
&lt;p&gt;My goal was to follow the UNIX philosophy of “do one thing and do it well”.
Unfortunately, that doesn’t really work when applied to network services that
have to interact with one and other. UNIX tools are built upon a common file
system and simple data passed over STDIN. Microservices don’t have that
shared foundation. You could make one:
companies that use microservices in anger often have a team that deals with
the “developer experience” of creating and using
microservices. But as a solo developer that’s not something I wanted to
spend my time doing.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;This became very apparent when trying to debug issues. In the UNIX world,
if your series of commands piped together does something strange you can simply
echo the data at various stages. Not so much when that data is flying around
on a network, going into and out of things like etcd. Trying to figure out why
a certificate hadn’t been acquired was a case of searching through logs from
four containers, none of which had particularly good logging.
There are many ways to get insight into what’s
happening with microservices but, again, that’s not something I wanted to do myself.&lt;/p&gt;
&lt;p&gt;Over time, and with experience in other projects, I came to realise that
microservices only really make sense if you’re unable to deploy a monolith.
For tech companies this naturally happens when different teams
contribute to the same service: splitting it out into smaller services that
are wholly owned by individual teams makes sense. For solo developers,
that never happens. You can still gain the other benefits
of microservices – such as code separation and having clearly defined APIs –
by sticking to certain coding standards.&lt;/p&gt;
&lt;h3 id=&#34;proxy-inconveniences&#34;&gt;Proxy inconveniences&lt;/h3&gt;
&lt;p&gt;As well as being unhappy with the microservice nature of the solution,
I wasn’t pleased with nginx. If you requested an unknown domain, nginx
would use the first server block in its config to serve a response, instead
of sending an “unrecognised name” alert as I wanted. It was a minor issue, but
it irked me.&lt;/p&gt;
&lt;p&gt;So from nginx I switched to haproxy. It has a &lt;code&gt;strict-sni&lt;/code&gt; option when configuring
TLS connections which makes it behave properly. It also performs a lot better for
this type of workload than nginx. All was well for a while, but then I started getting alerts
that requests were occasionally failing. I couldn’t reproduce the issue, but
my nightly jobs to build and push containers managed to hit it nearly every
night, causing them to fail.&lt;/p&gt;
&lt;p&gt;After some investigation, I found that the haproxy developers had refactored
the header parsing code, and
neglected to properly reset flags when multiple requests were sent over the same
connection. There was a patch, but it wasn’t released. No problem, I thought,
I’ll just cherry-pick it onto the last release… Except that haproxy use
Git in the most convoluted manner I’ve ever seen – they have one
repository per release. This makes it harder to patch, but it also made me question
whether I trusted them to ship stable software: there were no tests for
the header parsing code (which is both fundamental and finicky,
the perfect target for tests), the source code management was weird, and they didn’t
seem in any rush to patch this bug.&lt;/p&gt;
&lt;p&gt;Not long after that issue, &lt;a href=&#34;https://greg.holmes.name/&#34;&gt;Greg&lt;/a&gt; managed to
encounter another bug where haproxy returned a 500 error whenever the
upstream server replied with a particular, perfectly valid, header.
The die was cast – it was time to move to something else.&lt;/p&gt;
&lt;h3 id=&#34;not-invented-here-syndrome&#34;&gt;Not Invented Here syndrome&lt;/h3&gt;
&lt;p&gt;Looking for a new solution, there were many more options than
back in 2016. I’m still convinced, however, that anything
exposed to the Internet should not have access to run docker containers.
It’s the modern equivalent of running a CGI script as root. That
single requirement eliminates most off-the-shelf solutions.
What do you do when nothing quite meets
your specific requirements? You make something yourself! My new solution has two
components: &lt;a href=&#34;https://github.com/csmith/dotege&#34;&gt;Dotege&lt;/a&gt; and
&lt;a href=&#34;https://github.com/csmith/centauri&#34;&gt;Centauri&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Dotege is a replacement for the
microservices that monitored containers and obtained certificates. It’s fundamentally
a templating engine - whenever the containers change, it evaluates a
template and saves the result to disk. The template has access to details about
the containers, their labels, ports, and so on. Dotege can also obtain certificates
from Let’s Encrypt, and raise a signal against another
process whenever the template or certificates change. I used this to generate
the configuration and certificates used by haproxy for a while, and more recently
changed the template so that it works for Centauri.&lt;/p&gt;
&lt;p&gt;Centauri is my own reverse proxy. It’s configured using a simple text file
and can also obtain certificates from an ACME provider. It doesn’t serve static
content, has no knowledge about docker, and avoids the other bells and
whistles that adorn most reverse proxies. It also has good test
coverage to ensure that I don’t, say, accidentally break header parsing.&lt;/p&gt;
&lt;p&gt;As a software engineer I enjoy writing software, but I also enjoy running
simple, easy to understand software. That’s what I’ve achieved here: it’s
very easy to identify where the problem is if anything goes wrong, both are small
Go programs rather than vast sprawling C
monstrosities, and their interaction is primarily through a file written to disk
that can be inspected or edited as needed.&lt;/p&gt;
</content>
    </entry>
    <entry>
        <title>How to break everything by fuzz testing</title>
        <link href="https://chameth.com/break-everything-fuzz-testing/"/>
        <updated>2020-04-26T00:00:00Z</updated>
        <id>https://chameth.com/break-everything-fuzz-testing/</id>
        <content xml:lang="en" type="html">&lt;figure class=&#34;image left&#34;&gt;
  &lt;picture&gt;
      &lt;source srcset=&#34;https://chameth.com/break-everything-fuzz-testing/chimp.avif&#34; type=&#34;image/avif&#34;/&gt;
      &lt;source srcset=&#34;https://chameth.com/break-everything-fuzz-testing/chimp.webp&#34; type=&#34;image/webp&#34;/&gt;
      &lt;img src=&#34;https://chameth.com/break-everything-fuzz-testing/chimp.jpg&#34; alt=&#34;Chimp sat at a typewriter&#34; loading=&#34;lazy&#34; width=&#34;300&#34; height=&#34;239&#34;/&gt;
  &lt;/picture&gt;
  &lt;figcaption&gt;&lt;p&gt;Fuzz testing is a bit like the infinite monkey theorem, but instead of Shakespeare you get crashes.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Fuzz testing, if you’re not aware, is a form of testing that uses procedurally generated random
inputs to see how a program behaves. For instance, if you were fuzz testing a web page renderer
you might generate a bunch of HTML - some valid, and some not - and make sure the rendering
process didn’t unexpectedly crash.&lt;/p&gt;
&lt;p&gt;Fuzz testing doesn’t readily lend itself to all types of software, but it particularly shines
in cases where some kind of complex user input is accepted and processed in some way - like
the aforementioned web page renderer. I was recently adding a library to parse EXIF data to
images to an Internet-facing service and realised it was a perfect opportunity to do some fuzz
testing. Even if I didn’t find any issues, I’d improve my confidence that the library was safe
enough to expose to the Internet.&lt;/p&gt;
&lt;h3 id=&#34;breaking-my-exif-library&#34;&gt;Breaking my EXIF library&lt;/h3&gt;
&lt;p&gt;I wrote a quick harness to run &lt;a href=&#34;https://github.com/dvyukov/go-fuzz&#34;&gt;go-fuzz&lt;/a&gt; on the library,
and gave it some pre-existing demo files as sample input. The way go-fuzz works is that it
instruments your code and then mutates the inputs to try to improve the coverage. For example,
if I had some sample data that had an EXIF tag with a value of 1 then go-fuzz might change it
to a 2 and see if the code follows a different path. In most cases it won’t but when it does,
they tend to be very interesting cases.&lt;/p&gt;
&lt;!--more--&gt;
&lt;p&gt;One of the first issues that go-fuzz found was that some values in a maker note field would cause
the library to panic (i.e., crash). This happened because there was a check to see if the first
six characters were “Nikon” and a null byte, without first checking to see if there were actually
six characters available. This is a kind of bug that doesn’t happen much with “real” data - as
the field is either not present or completed correctly - but could easily be exploited once this
code is exposed to the Internet.&lt;/p&gt;
&lt;p&gt;Another interesting bug that go-fuzz found was that if a tag had a particularly large count, the
library would try to allocate an obscene amount of memory and die. There was already a check in
the code that was meant to avoid this exact scenario, but go-fuzz managed to find a way around
it. Each tag has a size (for example an integer tag takes a fixed number of bytes) and a count;
the existing check multiplied the two together and made sure that the result wasn’t too large.
For most cases this was fine but go-fuzz found a case where the count was so large that when
multiplied by the size of the tag it overflowed the integer and became negative, thus passing
the sanity check but then subsequently failing when it came around to actually allocating the
memory.&lt;/p&gt;
&lt;p&gt;The final bug of note that go-fuzz found was the most interesting. EXIF data is stored in IFDs
(“Image File Directories”), and each IFD provides what is effectively a pointer (a byte offset)
to the next one. The EXIF library already had a check to make sure that these didn’t loop, but
it only checked the immediately preceding IFD - so if IFD 1 linked to IFD 2, it would catch IFD 2
linking back to IFD 1 and break the loop with an error. Go-fuzz found that having three interlinked
IFDs had the same issue, though, and the guard code wasn’t triggered. This created an infinite
loop, maxing out a CPU core until the process was eventually killed - one of the worst kind of
bugs you could have in an Internet-facing service which doesn’t deal with private data! The fix
for this was fairly straightforward - I just made the library keep a record of the previously
visited IFDs and bail out if it found a loop.&lt;/p&gt;
&lt;h3 id=&#34;breaking-my-ide&#34;&gt;Breaking my IDE&lt;/h3&gt;
&lt;p&gt;When go-fuzz detects an issue it outputs not only the details of the problem (the stack trace,
error message, and so forth) but also the input that generated the problem. This is useful for
reproducing and making sure the issue is fixed, but it also makes it really easy to write
a test to ensure that the behaviour never regresses in the future.&lt;/p&gt;
&lt;p&gt;As I was working through fixing the bugs that go-fuzz found, I dutifully added new tests where
needed. After adding the sample input with looping IFDs to the project, I switched to IDEA to
write a test to use it. I clicked on the input file to copy the file name, and then the entire
IDE hung and had to be restarted. Uh oh! When I restarted IDEA, it immediately began indexing
the project and again hung. It turns out IDEA parses EXIF data (presumably, even if it does
nothing else with the data, to get the rotation property for images), and the library they use -
an independent one written on Java - had the same bug as the Go library I was using.&lt;/p&gt;
&lt;p&gt;In order to stop IDEA from indexing the file and becoming unusable I renamed it from a ‘.tif’
extension to ‘.dat’, and everything went back to normal. I thought I’d best report the bug to
JetBrains, though, so they could put a proper fix in.&lt;/p&gt;
&lt;h3 id=&#34;breaking-youtrack&#34;&gt;Breaking YouTrack&lt;/h3&gt;
&lt;p&gt;JetBrains use their own issue tracker called YouTrack for reporting bugs in IDEA. I dutifully
went over and described the problem, attaching the log files from the IDE, a description of how
the file was malformed, and carefully selected the .dat version of the file to upload so that it
wouldn’t cause anyone else the same immediate problem.&lt;/p&gt;
&lt;p&gt;After trying to upload the file I got a strange error back. Uh oh! I submitted the IDEA issue as
it stood, unable to see if the attachments had even uploaded, and went and wrote up an issue for
YouTrack itself about the error message. While I was doing that, YouTrack seemed to slow down and
become really annoying to use. I had a sinking feeling the exact same thing was probably
happening as with IDEA and my library - but this time YouTrack had content-sniffed the file
instead of relying on the file extension. In hindsight, I should’ve put the file in a passworded
archive to ensure no automated tools got hold of it. I marked the issue as a security problem as in
a service like YouTrack it presents a denial-of-service opportunity&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a class=&#34;footnote-ref&#34; href=&#34;#fn:1&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt; (remember when I said it was
one of the worst kinds of bugs you could have in an Internet-facing service?…)&lt;/p&gt;
&lt;p&gt;Shortly after I raised my YouTrack ticket, a notice appeared at the top of the page saying they
were investigating the current performance issues. Uh oh! I was holding out hope that this was
unrelated to me uploading the buggy dat file, but the timing all seemed a bit suspect. I shot
support an e-mail saying I think I might be the root cause for their performance issues and
linked to the ticket. In the time it took me to e-mail them, the entire site had been
put into maintenance mode. I got an e-mail back a few hours later confirming the outage
was in fact all my fault, as I’d feared. Within the space of days the JetBrains security team
had fixed the issue in YouTrack, which was a pretty nice turnaround.&lt;/p&gt;
&lt;p&gt;So if you were trying to access YouTrack at the start of March and couldn’t - I’m sorry, I didn’t
mean to! Also, if you’re building an Internet-facing service that takes user input you should
really consider running a fuzz tester against it!&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr/&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;“We have a problem”. “Remember, there are no such things as problems,
only opportunities”. “Well then we have a DDoS opportunity.”
– &lt;a href=&#34;https://twitter.com/J4vv4D/status/671090709588496384&#34;&gt;@J4vv4d&lt;/a&gt; &lt;a class=&#34;footnote-backref&#34; href=&#34;#fnref:1&#34; role=&#34;doc-backlink&#34;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</content>
    </entry>
</feed>
