Chameth.com - posts like building-a-new-computer, simple-backups-restic-hetzner, why-you-should-be-using-https but not migrating-from-github-to-forgejo

Thinking more about backups

2025-11-01T00:00:00Z

The Backblaze logo

Almost a year ago I wrote about how I do backups with Restic and Hetzner. That system has been ticking along well ever since, but recently I had some… thoughts. These backups are all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are two glaring issues:

Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse report get it suspended? What if the UK passes even more dumb laws and Hetzner decide it’s easier just to not do business with people here? This is the same sort of concern I have about Google accounts: if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very likely, but these are things my brain likes to dwell on.

Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels a little silly that the same attack could take down both the originals and the backups. There’s no way to avoid that with Hetzner’s S3 implementation, as far as I can tell.

Exploring options

I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that, and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in the worst case.

I then looked around at other S3 providers. Amazon’s glacier offering is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything. There are also lots of weird pricing edge cases around moving data between storage classes, minimum file sizes, and so on. A much better option is Backblaze’s B2 product. Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful in this case: lifecycle rules. Coupled with the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for what’s effectively an append-only store.

This works more-or-less out of the box with Restic. Joseph Price has a guide that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to the end of my script: a simple restic copy and a restic forget. B2 actually works out cheaper than the Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month. Not a bad price for some extra peace of mind!

Further Adventures in Music Organisation

2025-09-21T00:00:00Z

I wrote before about how I’d dropped Spotify in favour of locally stored music, but things have advanced a bit since. I had a few issues: Tauon would occasionally manage to lose its database and along with it all my carefully constructed playlists and song ratings¹, and the experience on my phone was not very fun.

I had to manually sync the music by plugging my phone in to the computer, and sometimes it just refused to mount the right partition. I don’t think there’s really a good way to debug an Apple phone not behaving properly when connected to a Linux desktop. Then I started wanting more than one playlist synced², and trying to find a way to make that work just broke me.

I spent a while looking at different ways of hosting the music centrally. Plexamp gets lots of good reviews, and I’ve used Plex a fair bit. I set about spinning up a Plex server, and just could not get it working. The server ran fine, served the web interface, but would neither associate with my account nor run standalone. The docs were contradictory and there was very little useful logging. After a lot of frustration, I stumbled across mentions that Plex block running on Hetzner. I assume that’s the cause of my issues, although I have no way to know for sure. I use Hetzner for all my servers and other hosted services, but Plex have decided I can’t run the self-hosted software that I have a lifetime subscription for there. What the actual fuck?

I could have probably worked around the arbitrary restriction, but I didn’t want to throw more time down the drain. Instead, I set up Navidrome, an open source music server. It supports the Subsonic protocol, which means you can use a whole slew of different clients with it (or even write your own). It also means there’s a nice way to get data in and out of it programmatically, which I recall being a bit of a fight with Plex.

Syncing and organising

All my music lived on my desktop, but now I wanted it on my server. At first I just used rsync to copy everything up. I’d maintain a local “master” copy, and periodically shove the changes to the server³. That quickly got old, so I set up Syncthing to keep the two folders in sync. That worked… for a while.

Concurrently, I was looking at improving the organisation of my music. It was mostly organised by some Go programs I’d thrown together to automate the importing, with no real validation of metadata or anything else. Some albums got split up into multiple folders because the tracks had different artists (and didn’t have an album artist set); some artists ended up with multiple folders with slight spelling or case variations. It was upsetting.

I’d used MusicBrainz Picard before to fix some issues, but I didn’t really like the UI, especially when trying to do bulk actions. The main alternative is beets, which describes itself as “the music geek’s media organiser”, and is entirely command line. I was intrigued.

I started playing around with beets, and quickly noticed a problem. Every time I changed a metadata tag in a music file, Syncthing had to upload the entire file. I was frequently changing tags across the whole library as I got beets set up how I wanted, and Syncthing was handling it by uploading gigabytes of files for every minor change. I was already a bit unhappy with having the content duplicated in two places, so now I was using a command line organiser I figured I could just get rid of my local copy and move everything to the server.

I set up a Docker container for beets, and then wrote some incredibly hacky shell scripts so I could run beet locally on my desktop and it would SSH to the server over Tailscale and exec into the container, passing the arguments along. Then I did one final sync of the music library, got rid of Syncthing, double checked I had a backup and deleted all of my local music.

Bears, Beets, Battlestar Galactica

Beets is amazing. It has a vast array of plugins that can do almost anything you could want with a music library, and the command line workflow works really well for me. It’s very well documented, and all the individual parts are pleasingly simple and easy to understand.

Here’s what it looks like when importing some new music:

Beets importing an album

If it doesn’t get a perfect match then it shows the closest matches, and summarises what’s different about them (missing tracks, different names, etc), and lets you decide what to do. Aside from the metadata matching, it’s doing a lot of things under the hood that aren’t necessarily apparent. It:

Looks up the genre from Last.fm
Analyses all the tracks and writes ReplayGain metadata to them
Fetches album art
Checks all the files to make sure they’re actually playable
Scrubs any existing metadata tags

It also maintains its own database, and you can query against it. The query language is both simple and quite powerful, like the rest of beets. As an example:

$ beet ls artist:"Linkin Park" year:2025 length:2:00..2:30 
Linkin Park - From Zero - Casualty

Almost all the other commands also let you use the query syntax, which makes the whole tool really powerful. Want to delete all country tracks that you added last month? No problem. Redownload all the album art for a certain band? Easy. You can even do smart playlists using the filters.

Actually playing music

With all this organisation, I’ve not actually mentioned one tiny detail: how I actually play music now. Navidrome has a web UI, which is perfectly usable, but I don’t really want my web browser involved as it makes balancing sound levels tricky, and getting media keys working is a pain. Thankfully there are loads of Subsonic clients. I’ve tried several, but eventually settled on Feishin on the desktop. It’s very pretty:

Feishin’s ’now playing’ screen

What attracted me to it, though, was its built-in support for Navidrome’s smart playlists. These aren’t even properly exposed in Navidrome’s web UI yet, but Feishin has a visual editor that lets you create and edit them. I ended up with an identical system to the playlists I had in Tauon: a “blacklist” playlist which are tracks I never want to play, a “favourites” playlist, and then an “everything” playlist which is the entire catalogue minus the blacklist.

On iOS I’ve settled on Arpeggi. It’s not actually on the App Store yet, but available via TestFlight. It’s one of those really nicely polished apps that are obviously a labour of love, not just out to do the minimum possible to get your money. It can cache songs offline, and automatically download entire playlists, and supports all of the standard Subsonic features like rating, reporting plays back to the server, and so on.

One unexpected benefit of using a central server is that it handles reporting plays to Last.fm and ListenBrainz instead of the clients. I don’t think I’ve ever bothered to configure my mobile clients to do that before, and now it just works automagically. I’m relying on those services more for recommendations and discovery, as that’s something you lose out on when self-hosting.

Making bad decisions about bitrates

With all the moving, copying, and organising, I started thinking about how large the collection was. The songs were all in different formats, depending on when and where I’d picked them up, and I couldn’t tell the difference between them. I did a test and transcoded an MP3 file down to 128kbps, and still couldn’t tell the difference. So armed with that sample size of 1, I transcoded the entire library and saved so much space.

I don’t remember which track I did that first test with, but it must have been a very unlucky pick. I quickly started to notice the distortions caused by the low bitrate. It was particularly bad in any song with a lot of treble, and started to really annoy me. I started a painful process of reimporting things in their original format. Beets came in clutch again, both with the import process (if you import a duplicate album, it asks if you want to replace the original, and shows the format and bitrate of them for comparison), and keeping track of what was left to fix (beet ls bitrate:..128000).

Having started paying more attention to the sound quality, I think I’ve been nibbled on by the audiophile bug. I’ve been trying to get new music in FLAC format where possible. I’m pretty sure I can’t tell the difference between a decent bitrate MP3 and a FLAC, but maybe if I got a better DAC and some nice headphones…? Someone please hide my credit card!

As for the library taking up a lot of space, I now realise it’s a price worth paying. I might need to swap out the server to one with more storage at some point in the future, but it’s generally worth upgrading every few years with rented dedicated servers anyway, as you often get more for the same amount of money. Next upgrade I’ll just make sure there’s an appropriately-sized disk instead of focusing entirely on RAM and CPU.

Writing more code

Naturally, this whole process spawned several side projects. When browsing albums in Navidrome, I was a bit upset that the album art wasn’t all a consistent size. I thought about scripting something to crop them consistently, but then I had a better idea, and jewelcase was born. It takes the album art, crops it down to a consistent size, and then renders it inside a jewel case. It applies some slight effects, like adjusting the colours, rounding the corners, and tweaking the edges so it looks a bit more real. I don’t know why⁴, but every time I see the effect it gives me a little spark of joy.

Album art, before and after jewelcase is applied

The other project is BASS, a tool that uses the Subsonic API to grab information about my music catalogue, and then generates a “Daily Mix” playlist. It uses a system of weights to select tracks semi-randomly, biasing towards favourite tracks, those that haven’t been played much, and a few other criteria. This gives me a nice balance between having my favourites on repeat, and exploring the full library at random.

All the weights in BASS are customisable, so I can tweak it to my heart’s desire, and anyone else can also run it and configure it entirely differently to me if they want to. I can’t imagine I would’ve been able to do that if I were using Plex!

I was running a version from git as the stable release had fun dependency issues on Arch, so it’s possible this wouldn’t be an issue in normal use. I also started backing up the database, but it’s still annoying to have to restore it. ↩︎
A “favourites” playlist for me, and a more “family friendly” playlist for when I’m playing music out loud that has a bit less screaming/swearing/etc. ↩︎
One of the reasons for starting this was being annoyed by the periodic sync to my phone, obviously it makes perfect sense to introduce a new, different periodic sync. ↩︎
Nostalgia? A little bit of OCD? Both? ↩︎

Fixing a loud PSU fan without dying

2025-07-30T00:00:00Z

Three months after I built my new computer, it started annoying me. There would occasionally be a noise that sounded like a fan was catching on a cable, but there weren’t any loose cables to be a problem. Over the course of a few weeks, the sound got progressively worse to the extent that I didn’t want to use the computer without headphones on. I measured the sound at 63 dB, which is about the sound of someone talking. That may not sound terrible, but it’s a constant, nasty noise coming from something that sits about 40cm from my head.

After some investigating¹, I identified the PSU fan as the culprit. I have a Cooler Master V750 SFX, which is not super high-end, but wasn’t cheap, either. It shouldn’t be developing issues after three months. Thankfully, it comes with a ten-year warranty, so it should be easy to get sorted, right?

Warranty woes

I looked at Cooler Master’s warranty, and for issues within the first two years you have to deal with the retailer. That would be Amazon in my case. So I looked at Amazon’s information on warranty issues. Their policy is that if it’s more than 30 days since purchase, you have to send it off to a third-party repair center and wait for them to diagnose and try to repair it. Here’s the kicker:

Usually repairs take up to 20 business days (including delivery time), but could take slightly longer

I use the computer for work, have upcoming LAN parties to go to, and generally can’t do without it for an entire month. That’s assuming they reproduce the issue: the PSU fan only turns on when it reaches a certain temperature, so if you just plug it in briefly it won’t exhibit any symptoms.

If I was going to do a warranty return, I’d need to buy a new PSU first to use while this one was away. If I’m spending £100+ on a PSU, then I’d be getting a better one that doesn’t have known issues with the fans. So when the Cooler Master eventually got returned I’d have no use for it. What a waste.

So the warranty is about as much use as a chocolate teapot. Good to know, I guess. If I wanted decent service, I should have probably used a real retailer, not Amazon. Lesson learnt.

Doing some research

If the warranty is useless to me, maybe I should just void it and try to fix it myself. I did some research online. Most of the results went like this:

if you’re not a trained professional you should not be opening up your PSU at all. There’s a lot of current that goes through it which will seriously hurt/main/kill you

it’s time consuming and potentially dangerous

or some other variety of doom and gloom. There were some actually useful results sprinkled in, though. A post on the smallformfactor.net forums details someone swapping out the fan on the 850W version of the same power supply because it was too noisy. It looks like the fan connector in the PSU is a two-pin connector, and they just used an off-the-shelf adapter to convert the standard four-pin case fan connector. It sounded a bit annoying to make it all fit, though.

A similar post on reddit showed basically the same upgrade, but identified the connector as a 2.54mm JST-XH. They crimped their own connector on, which feels a lot better than using a bulky connector. I do have a crimping set for JST connectors, but I absolutely hate doing them because they’re so fiddly and annoying. I’m not sure if I trust one of my crimps inside a power supply, either.

Looking at the pictures in those posts, I identified the fan as an Ong Hua HA9215VH12FD-F00. I searched to see if I could see any specifications for it. I couldn’t. But what I did get back as my first result was another Reddit post, this one titled “Getting rid of a HA9215SH12FD-F00 fan from your PSU”. It says something about the quality of a component when the first result is a post about getting rid of it because it’s annoying².

All of these mods seemed easy enough, and I’m reasonably confident in my ability to both discharge and avoid touching capacitors, so I probably wouldn’t die. I ordered a Noctua NF-A9x14 fan and some pre-made JST-XH connectors and waited for them to arrive.

Upgrade time

The replacement parts arrived one evening, so when I turned the computer off for the night I unplugged it from the mains, held the power button down for a while, and left it overnight for good measure. The next morning I popped open the case, disconnected and removed the PSU, gleefully tore the “warranty void if removed” sticker off, and opened it up.

Removing the fan was simple: unplug the JST connector and undo the screws holding it to the outer panel. I confirmed that my JST connectors were the right size, then started surgery on the Noctua fan. I carefully removed the heatshrink, cut off the normal fan connector, and separated out the four wires. A quick check of their FAQ confirmed the wiring colours: black was ground, yellow +12V, and blue and green used for speed control (which we’re not using in this instance). I clipped the blue and green wires off near the hub so they wouldn’t be in the way, then cut the yellow and black wires down to length, and attached the JST connector with a solder seal:

Old fan, new fan, burnt mouse mat

In the process I discovered my mouse mat doesn’t like being hit with a heat gun. Whoops. The solder seals are great: the solder in the center melts to form an electrical connection, the parts at the end grip the cable to provide mechanical strength, and the whole thing acts like heatshrink and keeps the connection insulated. A lot less fiddly than crimping JST connectors, and a lot smaller and cheaper than an adapter.

I installed the fan and reassembled the PSU. Then undid it all and tried again, as apparently the face the fan mounts to isn’t quite symmetrical and I’d installed the fan upside down. Connected everything back up, plugged the computer in, booted it up, and… nothing. The PSU doesn’t actually power the fan until it hits a certain temperature. I figured putting it under load would probably heat it up, so ran a graphics benchmark a few times. Eventually the fan spun up. Huzzah.

Soon the entire computer will be beige and brown

Peace at last

I measured the sound of the new fan: 43 dB. That’s a decrease of 20 dB. Due to the weird logarithmic properties of the dB scale that no one understands, that means it sounds about 4 times quieter to human ears. Big improvement!

It’s worth pointing out this isn’t a straight one-to-one upgrade. I don’t know the specs of the old fan in terms of airflow, and I’m just assuming that the Noctua will be good enough by virtue of being a decent fan. They also behave slightly differently as the power ramps up: the stock fan starts spinning at a lower voltage than the Noctua, which needs around 7V to get started properly³. I don’t think this will be an issue, as it’s effectively just putting a small step at the lower end of the fan curve. It might shorten the overall lifespan of the PSU, but from my point of view it failed after four months, so any extra time is a bonus.

I recorded some before/after audio, so you can hear the difference. I think there’s some amount of volume normalisation going on, so it’s not quite as stark a difference as in person, but it’s still noticeable.

Audio sample from the old fan

Audio sample from the new fan

And if you couldn’t hear it, here’s a visual representation:

Side-by-side comparison of the audio clips

The whole upgrade took about an hour (less time than writing this post about it!), and I even managed to avoid killing myself.

Opening the case and sticking my ear near to various components. High-tech stuff. ↩︎
On closer inspection it’s actually a slightly different model — there’s an S not a V in the middle there — but I’m going to assume that doesn’t change much on the quality/noise front. ↩︎
It then keeps spinning down to about 5V or so, it just needs enough power to initially get started. ↩︎

Building a new Computer

2025-06-03T00:00:00Z

The finished PC. Cat pen for scale.

I recently¹ built a new computer, after exclusively using a laptop for three years. It’s also the first time I’ve departed from the usual combo of an Intel CPU and Nvidia GPU.

While the form factor of a laptop did make it amazingly handy for travelling and attending LAN events, it was starting to show its age and there was basically no sane upgrade path. The main problem was its 3060 mobile graphics card, which was okay for the first few years and then slowly descended into painful. At the time, the (fairly disappointing) 50-series had just been released, but hadn’t yet made it into laptops. Nvidia had stopped production on the 40-series beforehand, so there also weren’t any compelling options there, either.

That’s not to say there were no laptops at all that I could have upgraded to. There were. Just not really any that ticked all of my perfectly normal boxes like “run Linux”, “have at least 64GB of ram”, and “run modern games well”. Having to upgrade the entire system just because the graphics card was showing its age was a bit of a drag, too. So back to the land of desktop computers I went!

To try to preserve some of that convenience, I opted for a small form factor (SFF) case. The computer and all the accessories can fit inside a hand-luggage-sized flight case. More on that later, though.

Components, Choices, and Cramming

So what, exactly, is in this computer? The case is a Fractal Design Terra, which is a lovely little 10.4L case². My last desktop PC was a normal-sized Fractal Design case, which I liked a lot, and it seems like they’ve upped their game since then. The Terra is both pleasant to look at (just look at that wooden panel!), and a breeze to work in. The sides and top are fully and easily removable, giving you great access to everything inside³. I expected an SFF build to be fiddly, but the case made it feel about the same as a normal full-sized build⁴.

The case with sides and top off, and wires everywhere in desperate need of management.

Another nice feature of the case is the movable spine. I think this is relatively common in SFF cases, but it was new to me. You mount the graphics card on one side of a metal divider, and everything else on the other. The divider (or spine) can be moved side-to-side depending on what exactly you’re putting in the case.

As I mentioned earlier, this was my first departure from the Intel + Nvidia bandwagon. Nvidia’s 50 series was underwhelming, and Intel managed to put out a generation of chips that literally fried themselves. It seemed like a good time to try out AMD’s offering! I went for the newly released Radeon RX 9070XT graphics card, and a Ryzen 7 9800X3D processor⁵. The processor has a Thermalright AXP90 X47 Full heatsink on top of it, with a Noctua NF-A9 fan on top of that.

The motherboard is a Gigabyte B850I Aorus Pro, and it hosts 2 32GB Corsair Vengeance RAM modules, and a 2TB Crucial T705 NVMe drive. Powering it all is a Cooler Master V750 SFX. The PSU is modular to reduce the number of wires that need to be fitted in, but they’re still pretty big and bulky. I’m tempted by custom cables but they’re rather pricey, and you can’t actually see inside with the sides closed…

With the diminutive size of the case and all the components crammed in, there’s actually no room left over for case fans. The PSU, GPU and CPU all have fans attached, but that’s it. There are stories online of people having heat issues, but thankfully everything seems pretty happy in my setup. The components all draw in air from the sides, and the hot air passively vents out the top. If I keep the CPU under maximum load for a while then it starts to thermal throttle slightly, but the only time I’ve actually managed that is when compiling a kernel. Why was I compiling a kernel, you ask? Well, we’ll come to that.

But does it Linux?

I didn’t go into this build completely blind. I did a bit of reading and chatting, and it sounded like AMD drivers were pretty good on Linux these days. I didn’t check specifically if any of the components were supported.

The 9070XT was released on the 6th of March. I built my PC on the 26th of March. The first version of Mesa (the 3D graphics lib) that was generally stable with 9070XT cards landed in Arch on the 20th of March. If I did the build just one week earlier, I would’ve had a much less fun experience.

It still wasn’t quite perfectly smooth, though. When playing games I’d occasionally have issues. In some cases the screen just went blank and came back after switching windows, but in others the graphics drivers couldn’t recover and the PC needed rebooting. Fortunately the drivers are open source, and they have a public issue tracker hosted by the Freedesktop project. Searching through the issues, I found they’d already been reported (by another Arch user, of course), and there were various suggestions for work-arounds, as well as lots of work trying to reproduce the issue.

Within a week, an AMD developer had posted a set of patches that might address the issues, and asked people to test them. That’s fortunately pretty easy on Arch: you just clone the repository containing the PKGBUILD file, drop in the patches, and update the checksums for them. Then you start compiling, realise you didn’t tell it to use more than one CPU core, abort it, and start again but this time running on all cylinders. Soon enough I was running a patched 6.14 kernel and the problems stopped completely. Others reported similar outcomes, and the patches worked their way into the kernel proper.

I spent a few weeks having to compile a custom kernel every time I accidentally updated it (I should’ve just stopped pacman from touching the kernel for a bit…), then one day the patches didn’t apply because they’d been merged upstream. Since then everything has been perfect.

A friend commented that “this wouldn’t happen with Nvidia”. They meant it negatively, but I wholeheartedly agree! Maybe Nvidia have fewer launch bugs than AMD, but you absolutely wouldn’t be able to talk to their driver engineers on a public bug tracker, and get access to test patches that fix your issue. I’ll happily take a few bugs in exchange for a company that operates openly in a way consistent with the rest of the Linux community.

But does it travel?

Travel mode, engage!

So the computer works fine at home, but how do I get it to other places? I just shoved a laptop in a backpack⁶, but you can’t quite do that with a PC even one this size. I’d seen a few people at LANs who packed everything in flight cases, but they always seemed a bit unwieldy. Doing some research, though, I found that I’d just be able to squeeze everything into a flight case sized to be taken onto a plane as cabin baggage.

Peli (or Pelican in the US) are the most well known maker of flight cases, but I chose to go with Nanuk instead. Spec-wise they’re fairly similar. Nanuk are a bit cheaper. The killer difference for me, though, is that Nanuk publish STEP files of their cases. As someone with a 3D printer, giving me CAD models of things so that I can easily mod them is definitely a way to win my favour.

After lots of measuring and thinking and modelling I came up with an arrangement where everything could fit snugly into the case. The case has cutouts for the wheels and handles, which constrain what can go where. I ended up wanting the computer at the handle-end of the case to make best use of space, but that means it needs something to take its weight when the case is vertical. The solution was, of course, 3D printing.

Using the STEP files from Nanuk I designed an insert that spans across the case, with screw holes that line up with the ones in the flight case. I wasn’t sure it would be able to take the weight like that, so I also added in some supports in the other direction (running down to the bottom of the case, if it’s stood up). I ended up printing it in three parts, and it came out perfectly first time. I can’t emphasise how nice it is having CAD models instead of having to measure and guess at things.

With the support in place, I focused on cushioning the computer. I opted for 10mm thick EVA foam as it’s easy to obtain and work with, and doesn’t take up too much room. I used contact adhesive to glue the foam directly to the case and the 3D printed divider. There’s a piece that goes between the computer and the keyboard, so I printed some little 3D supports to hold that more firmly in place, and stuck those in with contact adhesive as well.

So how does it all hold up? It’s been to one LAN event so far, and it’s great. I think the flight case might actually be easier to carry than hefting a heavy backpack: the handles are sturdy and comfortable, and it has chunky wheels for pulling it around. It’s also actually easier to get everything set up when you can see it all laid out in the flight case and pull out what you need, and then the case can be tucked away out the way for the duration of the event.

Recently-ish. It may or may not have taken me a few months to get around to writing this post. ↩︎
Using litres to measure computer cases always feels wrong. Those are the units we use for water. Don’t put them near the computers! ↩︎
The case also has holes to put security screws in to stop all of these nice easily removable parts from being removable. Very handy when you’re leaving it unattended at a LAN. ↩︎
Which for me means it was good fun for about an hour, and then I got fed up and just wanted it to be done already. ↩︎
Three months in and I still can’t remember these random strings of numbers. Someone even explained the versioning scheme to me. ↩︎
Along with a portable monitor, mouse, keyboard, headset, mouse mat, kensington lock, power supply, cables, and controller. It was a heavy bag. ↩︎

Simple backups with Restic and Hetzner Cloud

2024-12-06T00:00:00Z

Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.

I have a confession: for the past few years I’ve not been backing up any of my computers. Everyone knows that you should do backups, but actually getting around to doing it is another story.

Don’t get me wrong: most of my important things are “backed up” by virtue of being committed to remote git repositories, or attached to e-mails, or re-obtainable from the original source, and so on. I don’t think any machine failing completely would be a disaster for me, but it would certainly be a pain.

This week I finally got around to doing something, and it ended up being a lot more straight forward than my previous forays into backup-land.

Restic

After soliciting a few opinions, the choice of backup software came down to either Borg or Restic. I’m pretty sure either would have done what I want, but I leaned towards Restic for a few reasons: it has a more informative website, it’s written in Go rather than Python¹, and Borg seems to be transitioning between major releases at the moment².

The way Restic works is pretty simple: you initialise a ‘repository’, and can then call restic backup /some/path and it’ll get backed up to the repository. Restic handles keeping different backups separate, and only sending data that’s changed, and deduplicating, and so on. You basically point it at a thing you don’t want to lose, and it sorts it out for you. Perfect.

There are equally straight-forward commands for removing old snapshots (restic forget) and verifying backups (restic check). One of the nice things about modern backup solutions is they support a whole range of backends. I was originally going to spin up a small VPS to host my backups, but noticed that Restic supported S3-compatible stores…

Hetzner Cloud Object Storage

I host my servers with Hetzner, and was going to use them to spin up a VPS as well. Despite the “Cloud” branding on a bunch of products, they offer reasonable prices and good service. A couple of months ago, they started offering S3-compatible object storage.

The pricing isn’t totally straight forward, but for continuous use the “free quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free, as is traffic within the eu-central region (where all my servers are). That quota is only awarded when you pay the “base price”, though, which is €4.99 a month. So basically it’s €5 a month for 1TB of storage and enough egress to fully restore every single byte. That’s better value than any VPS I can find, much cheaper than Amazon S3, and about the same as Backblaze B2.

Getting them to work together

Now you’d think making the tool that supports S3-compatible object storage work with your S3-compatible object storage would be easy, right? Well not quite. The library Restic uses to deal with S3 backends has some strange logic for figuring out the bucket name given a URL. It doesn’t quite seem to work right, though…

Hetzner buckets have URLs like s3://bucketname.hel1.your-objectstorage.com³, but just passing that to Restic gives an error that the bucket is not specified. The docs mention there’s an advanced option to make it use the virtual host for the bucket name: -o s3.bucket-lookup=dns. But that… also doesn’t work. What I ended up doing was specifying the URL as s3://hel1.your-objectstorage.com/bucketname, and also passing in the dns option. The library then seems to muddle its way back to a real, working URL. I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that I’m oblivious to?

The next fun part is that you can configure Restic entirely by using environment variables, except for that -o s3.bucket-lookup=dns argument. That has to go on the command line. I ended up making a little wrapper script to invoke Restic correctly:

#!/bin/sh

# The password restic will use to encrypt your data. You should generate
# something nice and secure.
export RESTIC_PASSWORD=repo-password

# The path to the repository where restic will save the backup. In our case
# this takes the form `s3:<endpoint>/<bucket>`. Your endpoint might be different
# to mine depending on the region your bucket is in.
export RESTIC_REPOSITORY=s3:hel1.your-objectstorage.com/bucket-name

# The access and secret key generated in the 'S3 credentials' section of the
# Hetzner Cloud Console
export AWS_ACCESS_KEY_ID=hetzner-access-key-id
export AWS_SECRET_ACCESS_KEY=hetzner-access-key

# Pass any arguments on to the restic command, along with the magic
# `s3.bucket-lookup` option we need to resolve the S3 URL properly.
exec restic -o s3.bucket-lookup=dns "$@"

Then in the backup script I just alias restic to use the script:

#!/bin/bash

set -eu

# This means any time we use `restic` below, we'll actually execute our special
# script which supplies all the env vars and arguments needed to find the
# repository. Make sure the path matches where you saved the script!
alias restic='~/.bin/restic'

# Actually do the backup. Each directory in the list below will be backed up
# separately.
dirs=(
	"/some/path/to/backup/"
	"/some/other/path/"
)
for i in "${dirs[@]}" 
do
	echo $i
	(cd $i && restic --verbose backup .)
done

# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to
# see what effect any changes would have before actually committing to them. 
restic forget --keep-daily 7 --keep-weekly 10 --keep-monthly 24 --keep-yearly 10

The only other notable thing here is that the script changes into the directory to be backed up. If you give Restic an absolute path, it will create a new snapshot if the metadata of any folder in the path changes, which is not what I want.

If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken me about half an hour. That’s including setting up the object storage, installing Restic, and so on. It’s painfully easy. Why didn’t I do this three years ago?!

Addendum: a step-by-step guide

If you want to do this yourself, here’s a quick step-by-step guide:

Log in to the Hetzner Cloud Console, and create a project.
On the “Object Storage” tab, create a new bucket. Note the name and the endpoint.
On the “Security” tab, go to “S3 Credentials” and generate new credentials. Note down the access key and the secret key.
Copy the first shell script above, fill in the password (you can pick!), endpoint, bucket name, access key and secret key.
Run the script with the init argument (e.g. ~/.bin/restic init). This will create a new repository, and only needs to be done once even if you backup multiple machines.
Copy the second shell script above, making sure the restic alias points at the script you saved in step 4. Change the list of directories to whatever you want to backup.
Schedule the script to be run automatically, using crontab or systemd timers or however you prefer.

To check everything is working, you can use the snapshots subcommand to see a list of saved snapshots. You might also want to try to restore a snapshot using the restore subcommand.

I’m not trying to be a language snob, but given the choice between two otherwise equal projects one in Go and one in Python, I’ll take the Go one. I know I’m not going to have weird library issues down the line, and I’m much more comfortable rummaging around the source. ↩︎
With a “Don’t use this in production!” notice on the shiny new version. ↩︎
Aside: I really hate URLs like that. What is with the trend for completely generic domains divorced from the service they’re a part of? ↩︎