Chameth.com - posts like building-a-new-computer, docker-automatic-nginx-proxy, thinking-more-about-backups, why-you-should-be-using-https but not break-everything-fuzz-testing, debugging-beyond-the-debugger

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Thinking more about backups

2025-11-01T00:00:00Z

The Backblaze logo

Almost a year ago I wrote about how I do backups with Restic and Hetzner. That system has been ticking along well ever since, but recently I had some… thoughts. These backups are all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are two glaring issues:

Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse report get it suspended? What if the UK passes even more dumb laws and Hetzner decide it’s easier just to not do business with people here? This is the same sort of concern I have about Google accounts: if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very likely, but these are things my brain likes to dwell on.

Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels a little silly that the same attack could take down both the originals and the backups. There’s no way to avoid that with Hetzner’s S3 implementation, as far as I can tell.

Exploring options

I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that, and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in the worst case.

I then looked around at other S3 providers. Amazon’s glacier offering is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything. There are also lots of weird pricing edge cases around moving data between storage classes, minimum file sizes, and so on. A much better option is Backblaze’s B2 product. Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful in this case: lifecycle rules. Coupled with the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for what’s effectively an append-only store.

This works more-or-less out of the box with Restic. Joseph Price has a guide that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to the end of my script: a simple restic copy and a restic forget. B2 actually works out cheaper than the Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month. Not a bad price for some extra peace of mind!

Fixing a loud PSU fan without dying

2025-07-30T00:00:00Z

Three months after I built my new computer, it started annoying me. There would occasionally be a noise that sounded like a fan was catching on a cable, but there weren’t any loose cables to be a problem. Over the course of a few weeks, the sound got progressively worse to the extent that I didn’t want to use the computer without headphones on. I measured the sound at 63 dB, which is about the sound of someone talking. That may not sound terrible, but it’s a constant, nasty noise coming from something that sits about 40cm from my head.

After some investigating¹, I identified the PSU fan as the culprit. I have a Cooler Master V750 SFX, which is not super high-end, but wasn’t cheap, either. It shouldn’t be developing issues after three months. Thankfully, it comes with a ten-year warranty, so it should be easy to get sorted, right?

Warranty woes

I looked at Cooler Master’s warranty, and for issues within the first two years you have to deal with the retailer. That would be Amazon in my case. So I looked at Amazon’s information on warranty issues. Their policy is that if it’s more than 30 days since purchase, you have to send it off to a third-party repair center and wait for them to diagnose and try to repair it. Here’s the kicker:

Usually repairs take up to 20 business days (including delivery time), but could take slightly longer

I use the computer for work, have upcoming LAN parties to go to, and generally can’t do without it for an entire month. That’s assuming they reproduce the issue: the PSU fan only turns on when it reaches a certain temperature, so if you just plug it in briefly it won’t exhibit any symptoms.

If I was going to do a warranty return, I’d need to buy a new PSU first to use while this one was away. If I’m spending £100+ on a PSU, then I’d be getting a better one that doesn’t have known issues with the fans. So when the Cooler Master eventually got returned I’d have no use for it. What a waste.

So the warranty is about as much use as a chocolate teapot. Good to know, I guess. If I wanted decent service, I should have probably used a real retailer, not Amazon. Lesson learnt.

Doing some research

If the warranty is useless to me, maybe I should just void it and try to fix it myself. I did some research online. Most of the results went like this:

if you’re not a trained professional you should not be opening up your PSU at all. There’s a lot of current that goes through it which will seriously hurt/main/kill you

it’s time consuming and potentially dangerous

or some other variety of doom and gloom. There were some actually useful results sprinkled in, though. A post on the smallformfactor.net forums details someone swapping out the fan on the 850W version of the same power supply because it was too noisy. It looks like the fan connector in the PSU is a two-pin connector, and they just used an off-the-shelf adapter to convert the standard four-pin case fan connector. It sounded a bit annoying to make it all fit, though.

A similar post on reddit showed basically the same upgrade, but identified the connector as a 2.54mm JST-XH. They crimped their own connector on, which feels a lot better than using a bulky connector. I do have a crimping set for JST connectors, but I absolutely hate doing them because they’re so fiddly and annoying. I’m not sure if I trust one of my crimps inside a power supply, either.

Looking at the pictures in those posts, I identified the fan as an Ong Hua HA9215VH12FD-F00. I searched to see if I could see any specifications for it. I couldn’t. But what I did get back as my first result was another Reddit post, this one titled “Getting rid of a HA9215SH12FD-F00 fan from your PSU”. It says something about the quality of a component when the first result is a post about getting rid of it because it’s annoying².

All of these mods seemed easy enough, and I’m reasonably confident in my ability to both discharge and avoid touching capacitors, so I probably wouldn’t die. I ordered a Noctua NF-A9x14 fan and some pre-made JST-XH connectors and waited for them to arrive.

Upgrade time

The replacement parts arrived one evening, so when I turned the computer off for the night I unplugged it from the mains, held the power button down for a while, and left it overnight for good measure. The next morning I popped open the case, disconnected and removed the PSU, gleefully tore the “warranty void if removed” sticker off, and opened it up.

Removing the fan was simple: unplug the JST connector and undo the screws holding it to the outer panel. I confirmed that my JST connectors were the right size, then started surgery on the Noctua fan. I carefully removed the heatshrink, cut off the normal fan connector, and separated out the four wires. A quick check of their FAQ confirmed the wiring colours: black was ground, yellow +12V, and blue and green used for speed control (which we’re not using in this instance). I clipped the blue and green wires off near the hub so they wouldn’t be in the way, then cut the yellow and black wires down to length, and attached the JST connector with a solder seal:

Old fan, new fan, burnt mouse mat

In the process I discovered my mouse mat doesn’t like being hit with a heat gun. Whoops. The solder seals are great: the solder in the center melts to form an electrical connection, the parts at the end grip the cable to provide mechanical strength, and the whole thing acts like heatshrink and keeps the connection insulated. A lot less fiddly than crimping JST connectors, and a lot smaller and cheaper than an adapter.

I installed the fan and reassembled the PSU. Then undid it all and tried again, as apparently the face the fan mounts to isn’t quite symmetrical and I’d installed the fan upside down. Connected everything back up, plugged the computer in, booted it up, and… nothing. The PSU doesn’t actually power the fan until it hits a certain temperature. I figured putting it under load would probably heat it up, so ran a graphics benchmark a few times. Eventually the fan spun up. Huzzah.

Soon the entire computer will be beige and brown

Peace at last

I measured the sound of the new fan: 43 dB. That’s a decrease of 20 dB. Due to the weird logarithmic properties of the dB scale that no one understands, that means it sounds about 4 times quieter to human ears. Big improvement!

It’s worth pointing out this isn’t a straight one-to-one upgrade. I don’t know the specs of the old fan in terms of airflow, and I’m just assuming that the Noctua will be good enough by virtue of being a decent fan. They also behave slightly differently as the power ramps up: the stock fan starts spinning at a lower voltage than the Noctua, which needs around 7V to get started properly³. I don’t think this will be an issue, as it’s effectively just putting a small step at the lower end of the fan curve. It might shorten the overall lifespan of the PSU, but from my point of view it failed after four months, so any extra time is a bonus.

I recorded some before/after audio, so you can hear the difference. I think there’s some amount of volume normalisation going on, so it’s not quite as stark a difference as in person, but it’s still noticeable.

Audio sample from the old fan

Audio sample from the new fan

And if you couldn’t hear it, here’s a visual representation:

Side-by-side comparison of the audio clips

The whole upgrade took about an hour (less time than writing this post about it!), and I even managed to avoid killing myself.

Opening the case and sticking my ear near to various components. High-tech stuff. ↩︎
On closer inspection it’s actually a slightly different model — there’s an S not a V in the middle there — but I’m going to assume that doesn’t change much on the quality/noise front. ↩︎
It then keeps spinning down to about 5V or so, it just needs enough power to initially get started. ↩︎

Building a new Computer

2025-06-03T00:00:00Z

The finished PC. Cat pen for scale.

I recently¹ built a new computer, after exclusively using a laptop for three years. It’s also the first time I’ve departed from the usual combo of an Intel CPU and Nvidia GPU.

While the form factor of a laptop did make it amazingly handy for travelling and attending LAN events, it was starting to show its age and there was basically no sane upgrade path. The main problem was its 3060 mobile graphics card, which was okay for the first few years and then slowly descended into painful. At the time, the (fairly disappointing) 50-series had just been released, but hadn’t yet made it into laptops. Nvidia had stopped production on the 40-series beforehand, so there also weren’t any compelling options there, either.

That’s not to say there were no laptops at all that I could have upgraded to. There were. Just not really any that ticked all of my perfectly normal boxes like “run Linux”, “have at least 64GB of ram”, and “run modern games well”. Having to upgrade the entire system just because the graphics card was showing its age was a bit of a drag, too. So back to the land of desktop computers I went!

To try to preserve some of that convenience, I opted for a small form factor (SFF) case. The computer and all the accessories can fit inside a hand-luggage-sized flight case. More on that later, though.

Components, Choices, and Cramming

So what, exactly, is in this computer? The case is a Fractal Design Terra, which is a lovely little 10.4L case². My last desktop PC was a normal-sized Fractal Design case, which I liked a lot, and it seems like they’ve upped their game since then. The Terra is both pleasant to look at (just look at that wooden panel!), and a breeze to work in. The sides and top are fully and easily removable, giving you great access to everything inside³. I expected an SFF build to be fiddly, but the case made it feel about the same as a normal full-sized build⁴.

The case with sides and top off, and wires everywhere in desperate need of management.

Another nice feature of the case is the movable spine. I think this is relatively common in SFF cases, but it was new to me. You mount the graphics card on one side of a metal divider, and everything else on the other. The divider (or spine) can be moved side-to-side depending on what exactly you’re putting in the case.

As I mentioned earlier, this was my first departure from the Intel + Nvidia bandwagon. Nvidia’s 50 series was underwhelming, and Intel managed to put out a generation of chips that literally fried themselves. It seemed like a good time to try out AMD’s offering! I went for the newly released Radeon RX 9070XT graphics card, and a Ryzen 7 9800X3D processor⁵. The processor has a Thermalright AXP90 X47 Full heatsink on top of it, with a Noctua NF-A9 fan on top of that.

The motherboard is a Gigabyte B850I Aorus Pro, and it hosts 2 32GB Corsair Vengeance RAM modules, and a 2TB Crucial T705 NVMe drive. Powering it all is a Cooler Master V750 SFX. The PSU is modular to reduce the number of wires that need to be fitted in, but they’re still pretty big and bulky. I’m tempted by custom cables but they’re rather pricey, and you can’t actually see inside with the sides closed…

With the diminutive size of the case and all the components crammed in, there’s actually no room left over for case fans. The PSU, GPU and CPU all have fans attached, but that’s it. There are stories online of people having heat issues, but thankfully everything seems pretty happy in my setup. The components all draw in air from the sides, and the hot air passively vents out the top. If I keep the CPU under maximum load for a while then it starts to thermal throttle slightly, but the only time I’ve actually managed that is when compiling a kernel. Why was I compiling a kernel, you ask? Well, we’ll come to that.

But does it Linux?

I didn’t go into this build completely blind. I did a bit of reading and chatting, and it sounded like AMD drivers were pretty good on Linux these days. I didn’t check specifically if any of the components were supported.

The 9070XT was released on the 6th of March. I built my PC on the 26th of March. The first version of Mesa (the 3D graphics lib) that was generally stable with 9070XT cards landed in Arch on the 20th of March. If I did the build just one week earlier, I would’ve had a much less fun experience.

It still wasn’t quite perfectly smooth, though. When playing games I’d occasionally have issues. In some cases the screen just went blank and came back after switching windows, but in others the graphics drivers couldn’t recover and the PC needed rebooting. Fortunately the drivers are open source, and they have a public issue tracker hosted by the Freedesktop project. Searching through the issues, I found they’d already been reported (by another Arch user, of course), and there were various suggestions for work-arounds, as well as lots of work trying to reproduce the issue.

Within a week, an AMD developer had posted a set of patches that might address the issues, and asked people to test them. That’s fortunately pretty easy on Arch: you just clone the repository containing the PKGBUILD file, drop in the patches, and update the checksums for them. Then you start compiling, realise you didn’t tell it to use more than one CPU core, abort it, and start again but this time running on all cylinders. Soon enough I was running a patched 6.14 kernel and the problems stopped completely. Others reported similar outcomes, and the patches worked their way into the kernel proper.

I spent a few weeks having to compile a custom kernel every time I accidentally updated it (I should’ve just stopped pacman from touching the kernel for a bit…), then one day the patches didn’t apply because they’d been merged upstream. Since then everything has been perfect.

A friend commented that “this wouldn’t happen with Nvidia”. They meant it negatively, but I wholeheartedly agree! Maybe Nvidia have fewer launch bugs than AMD, but you absolutely wouldn’t be able to talk to their driver engineers on a public bug tracker, and get access to test patches that fix your issue. I’ll happily take a few bugs in exchange for a company that operates openly in a way consistent with the rest of the Linux community.

But does it travel?

Travel mode, engage!

So the computer works fine at home, but how do I get it to other places? I just shoved a laptop in a backpack⁶, but you can’t quite do that with a PC even one this size. I’d seen a few people at LANs who packed everything in flight cases, but they always seemed a bit unwieldy. Doing some research, though, I found that I’d just be able to squeeze everything into a flight case sized to be taken onto a plane as cabin baggage.

Peli (or Pelican in the US) are the most well known maker of flight cases, but I chose to go with Nanuk instead. Spec-wise they’re fairly similar. Nanuk are a bit cheaper. The killer difference for me, though, is that Nanuk publish STEP files of their cases. As someone with a 3D printer, giving me CAD models of things so that I can easily mod them is definitely a way to win my favour.

After lots of measuring and thinking and modelling I came up with an arrangement where everything could fit snugly into the case. The case has cutouts for the wheels and handles, which constrain what can go where. I ended up wanting the computer at the handle-end of the case to make best use of space, but that means it needs something to take its weight when the case is vertical. The solution was, of course, 3D printing.

Using the STEP files from Nanuk I designed an insert that spans across the case, with screw holes that line up with the ones in the flight case. I wasn’t sure it would be able to take the weight like that, so I also added in some supports in the other direction (running down to the bottom of the case, if it’s stood up). I ended up printing it in three parts, and it came out perfectly first time. I can’t emphasise how nice it is having CAD models instead of having to measure and guess at things.

With the support in place, I focused on cushioning the computer. I opted for 10mm thick EVA foam as it’s easy to obtain and work with, and doesn’t take up too much room. I used contact adhesive to glue the foam directly to the case and the 3D printed divider. There’s a piece that goes between the computer and the keyboard, so I printed some little 3D supports to hold that more firmly in place, and stuck those in with contact adhesive as well.

So how does it all hold up? It’s been to one LAN event so far, and it’s great. I think the flight case might actually be easier to carry than hefting a heavy backpack: the handles are sturdy and comfortable, and it has chunky wheels for pulling it around. It’s also actually easier to get everything set up when you can see it all laid out in the flight case and pull out what you need, and then the case can be tucked away out the way for the duration of the event.

Recently-ish. It may or may not have taken me a few months to get around to writing this post. ↩︎
Using litres to measure computer cases always feels wrong. Those are the units we use for water. Don’t put them near the computers! ↩︎
The case also has holes to put security screws in to stop all of these nice easily removable parts from being removable. Very handy when you’re leaving it unattended at a LAN. ↩︎
Which for me means it was good fun for about an hour, and then I got fed up and just wanted it to be done already. ↩︎
Three months in and I still can’t remember these random strings of numbers. Someone even explained the versioning scheme to me. ↩︎
Along with a portable monitor, mouse, keyboard, headset, mouse mat, kensington lock, power supply, cables, and controller. It was a heavy bag. ↩︎

Simple backups with Restic and Hetzner Cloud

2024-12-06T00:00:00Z

Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.

I have a confession: for the past few years I’ve not been backing up any of my computers. Everyone knows that you should do backups, but actually getting around to doing it is another story.

Don’t get me wrong: most of my important things are “backed up” by virtue of being committed to remote git repositories, or attached to e-mails, or re-obtainable from the original source, and so on. I don’t think any machine failing completely would be a disaster for me, but it would certainly be a pain.

This week I finally got around to doing something, and it ended up being a lot more straight forward than my previous forays into backup-land.

Restic

After soliciting a few opinions, the choice of backup software came down to either Borg or Restic. I’m pretty sure either would have done what I want, but I leaned towards Restic for a few reasons: it has a more informative website, it’s written in Go rather than Python¹, and Borg seems to be transitioning between major releases at the moment².

The way Restic works is pretty simple: you initialise a ‘repository’, and can then call restic backup /some/path and it’ll get backed up to the repository. Restic handles keeping different backups separate, and only sending data that’s changed, and deduplicating, and so on. You basically point it at a thing you don’t want to lose, and it sorts it out for you. Perfect.

There are equally straight-forward commands for removing old snapshots (restic forget) and verifying backups (restic check). One of the nice things about modern backup solutions is they support a whole range of backends. I was originally going to spin up a small VPS to host my backups, but noticed that Restic supported S3-compatible stores…

Hetzner Cloud Object Storage

I host my servers with Hetzner, and was going to use them to spin up a VPS as well. Despite the “Cloud” branding on a bunch of products, they offer reasonable prices and good service. A couple of months ago, they started offering S3-compatible object storage.

The pricing isn’t totally straight forward, but for continuous use the “free quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free, as is traffic within the eu-central region (where all my servers are). That quota is only awarded when you pay the “base price”, though, which is €4.99 a month. So basically it’s €5 a month for 1TB of storage and enough egress to fully restore every single byte. That’s better value than any VPS I can find, much cheaper than Amazon S3, and about the same as Backblaze B2.

Getting them to work together

Now you’d think making the tool that supports S3-compatible object storage work with your S3-compatible object storage would be easy, right? Well not quite. The library Restic uses to deal with S3 backends has some strange logic for figuring out the bucket name given a URL. It doesn’t quite seem to work right, though…

Hetzner buckets have URLs like s3://bucketname.hel1.your-objectstorage.com³, but just passing that to Restic gives an error that the bucket is not specified. The docs mention there’s an advanced option to make it use the virtual host for the bucket name: -o s3.bucket-lookup=dns. But that… also doesn’t work. What I ended up doing was specifying the URL as s3://hel1.your-objectstorage.com/bucketname, and also passing in the dns option. The library then seems to muddle its way back to a real, working URL. I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that I’m oblivious to?

The next fun part is that you can configure Restic entirely by using environment variables, except for that -o s3.bucket-lookup=dns argument. That has to go on the command line. I ended up making a little wrapper script to invoke Restic correctly:

#!/bin/sh

# The password restic will use to encrypt your data. You should generate
# something nice and secure.
export RESTIC_PASSWORD=repo-password

# The path to the repository where restic will save the backup. In our case
# this takes the form `s3:<endpoint>/<bucket>`. Your endpoint might be different
# to mine depending on the region your bucket is in.
export RESTIC_REPOSITORY=s3:hel1.your-objectstorage.com/bucket-name

# The access and secret key generated in the 'S3 credentials' section of the
# Hetzner Cloud Console
export AWS_ACCESS_KEY_ID=hetzner-access-key-id
export AWS_SECRET_ACCESS_KEY=hetzner-access-key

# Pass any arguments on to the restic command, along with the magic
# `s3.bucket-lookup` option we need to resolve the S3 URL properly.
exec restic -o s3.bucket-lookup=dns "$@"

Then in the backup script I just alias restic to use the script:

#!/bin/bash

set -eu

# This means any time we use `restic` below, we'll actually execute our special
# script which supplies all the env vars and arguments needed to find the
# repository. Make sure the path matches where you saved the script!
alias restic='~/.bin/restic'

# Actually do the backup. Each directory in the list below will be backed up
# separately.
dirs=(
	"/some/path/to/backup/"
	"/some/other/path/"
)
for i in "${dirs[@]}" 
do
	echo $i
	(cd $i && restic --verbose backup .)
done

# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to
# see what effect any changes would have before actually committing to them. 
restic forget --keep-daily 7 --keep-weekly 10 --keep-monthly 24 --keep-yearly 10

The only other notable thing here is that the script changes into the directory to be backed up. If you give Restic an absolute path, it will create a new snapshot if the metadata of any folder in the path changes, which is not what I want.

If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken me about half an hour. That’s including setting up the object storage, installing Restic, and so on. It’s painfully easy. Why didn’t I do this three years ago?!

Addendum: a step-by-step guide

If you want to do this yourself, here’s a quick step-by-step guide:

Log in to the Hetzner Cloud Console, and create a project.
On the “Object Storage” tab, create a new bucket. Note the name and the endpoint.
On the “Security” tab, go to “S3 Credentials” and generate new credentials. Note down the access key and the secret key.
Copy the first shell script above, fill in the password (you can pick!), endpoint, bucket name, access key and secret key.
Run the script with the init argument (e.g. ~/.bin/restic init). This will create a new repository, and only needs to be done once even if you backup multiple machines.
Copy the second shell script above, making sure the restic alias points at the script you saved in step 4. Change the list of directories to whatever you want to backup.
Schedule the script to be run automatically, using crontab or systemd timers or however you prefer.

To check everything is working, you can use the snapshots subcommand to see a list of saved snapshots. You might also want to try to restore a snapshot using the restore subcommand.

I’m not trying to be a language snob, but given the choice between two otherwise equal projects one in Go and one in Python, I’ll take the Go one. I know I’m not going to have weird library issues down the line, and I’m much more comfortable rummaging around the source. ↩︎
With a “Don’t use this in production!” notice on the shiny new version. ↩︎
Aside: I really hate URLs like that. What is with the trend for completely generic domains divorced from the service they’re a part of? ↩︎