Chameth.com - posts like thinking-more-about-backups, travel-toolkit but not migrating-from-github-to-forgejo, why-you-should-be-using-https

Thinking more about backups

2025-11-01T00:00:00Z

The Backblaze logo

Almost a year ago I wrote about how I do backups with Restic and Hetzner. That system has been ticking along well ever since, but recently I had some… thoughts. These backups are all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are two glaring issues:

Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse report get it suspended? What if the UK passes even more dumb laws and Hetzner decide it’s easier just to not do business with people here? This is the same sort of concern I have about Google accounts: if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very likely, but these are things my brain likes to dwell on.

Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels a little silly that the same attack could take down both the originals and the backups. There’s no way to avoid that with Hetzner’s S3 implementation, as far as I can tell.

Exploring options

I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that, and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in the worst case.

I then looked around at other S3 providers. Amazon’s glacier offering is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything. There are also lots of weird pricing edge cases around moving data between storage classes, minimum file sizes, and so on. A much better option is Backblaze’s B2 product. Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful in this case: lifecycle rules. Coupled with the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for what’s effectively an append-only store.

This works more-or-less out of the box with Restic. Joseph Price has a guide that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to the end of my script: a simple restic copy and a restic forget. B2 actually works out cheaper than the Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month. Not a bad price for some extra peace of mind!

Building a travel toolkit

2025-01-02T00:00:00Z

The container for my toolkit, a UGREEN “travel accessories” bag

For a while now, whenever I’ve traveled anywhere I’ve dumped a selection of tools into my backpack just in case I need them. Originally this was mostly focused around being able to open and fix my laptop if anything went wrong, but over time has slowly expanded to include other useful tools.

Repeatedly packing these tools and then returning them to where they belonged was tedious, though. I couldn’t just leave them in the backpack as I often needed them for non-travelling purposes, and I didn’t want to just buy duplicate tools that spent most of their life a few meters away from each other, so I ignored the problem and just dealt with the tedium.

Eventually I acquired enough tools that — while not quite duplicates — I wouldn’t miss often if they were kept in the backpack. Building on these I’ve created a little travel toolkit that contains everything I might want (and more) while away.

The base

The biggest challenge I had before was finding a screwdriver set. I have an iFixit Mako set that did the in-and-out-of-the-bag routine for several years. Nothing else covered the range of drivers I wanted for generic electronic tinkering and more chunky work. I finally found one in the form of a Wera Tool-Check PLUS TR-Edition. “TR” stands for “Tool Rebel” and is Wera’s “fan club” brand. Its signature features over the normal Tool-Check PLUS are a fancy fabric case and a 152mm bit extender; but it also comes with a much different set of bits, which is why I picked it up.

Here’s a comparison

Type	Standard version	Tool Rebel edition
Phillips	2x PH1 3x PH2 1x PH3	1x PH00 1x PH0 1x PH1 1x PH2 1x PH3
Pozidrive	None	1x PZ1 3x PZ2
Slotted	1x 5.5mm	1x 4mm 1x 4.5mm 2x 5.5mm 1x 6.5mm
Torx	1x T10 1x T15 1x T20 1x T25 1x T30 1x T40	1x T10 1x T15 1x T20 1x T25 1x T30 1x T40
Security Torx	1x T10 1x T15 1x T20 1x T25 1x T30	1x T25
Hex	1x H3 1x H4 1x H5 1x H6 1x H8	1x H1.5 1x H2 1x H2.5 1x H3 1x H4 1x H6 1x H8

There are only five bits present in the standard version that are missing in the TR edition: 5mm hex (pretty uncommon) and four sizes of security torx (rare). In exchange for those and the removal of a bunch of duplicates, you get two extra phillips sizes (0 and 00, both very useful for electronics and spectacles), some pozidrive bits (extremely common on random hardware/furniture/etc), smaller hex bits (for electronics, 3D printers, and so on), and a few other bits and pieces.

Overall it’s a vastly superior selection for my needs. And while it doesn’t have the full range of bits provided by the iFixit kit, it does cover almost everything I’m ever actually likely to need while travelling, and it’s probably around a third of the size.

The other core part of the toolkit is the bag. I had a UGREEN “travel accessory” bag hanging around unused, and noticed it was almost the exact right size for the Tool-Check. Now, how many other tools could I stuff in there?

All the tools

The array of tools that make up the travel toolkit

The kit can be broken down into a few sections:

Screwdriving

The aforementioned Wera Tool-Check
Its 152mm bit extender

I’ve talked enough about this already, I think, so I’ll spare you any additional commentary on screwdrivers and bits.

Splitting, gripping and cutting

Engineer tweezers
Engineer mini snips
Engineer mini screw removal pliers
Knipex mini waterpump pliers
Two iFixit spudgers (one metal, one plastic)
iFixit jimmy

I have bigger or slightly different versions of these that stay in my normal toolkit. The tweezers and snips are useful in a variety of situations, and the spudgers are handy for getting inside laptops and other devices. The pliers are less useful, but they’re one of those things where when you do need them there aren’t any good alternatives.

Sewing

Two needles
Some black thread
Set of mini clips

You never know when you’re going to rip a seam in something. I find the clips a lot less fiddly than using pins for keeping material in place when I’m trying to sew.

Computering

Retractable ethernet cable
RJ45 coupler
USB-C to RJ45 adapter
Dual USB-A and USB-C thumb drive, with an Arch ISO on it

Sometimes Wi-Fi just doesn’t cut it, or you need to fiddle with a router that doesn’t expose its admin UI over Wi-Fi. The USB-C adapter is for use with my phone, as it’s often easier to move that get the laptop where it needs to be.

I can boot from the thumb drive into an Arch environment, and from there reinstall my laptop from scratch if needed. There’s enough space left over so it can be used to transfer files as well.

Miscellaneousing

Fabric tape measure
Spirit level
Sharpie
Superglue
Spare batteries: 2 CR2032s, 2 AA, 2 AAA

I’m almost certain I won’t use the spirit level, but it’s tiny and I had a spare so it went in. Everything else is just in the “generally handy to have” category.

Putting it all together

The nicely-organised side of the toolkit

The Tool-Check PLUS fits almost perfectly into one side of the UGREEN bag, leaving a little wiggle room along the side for the longer parts like spudgers. The middle pockets can’t fit much in, but keep the batteries and sewing bits out of the way. Everything else is in a big zip-up compartment in the other side, which is slightly messy but still easy enough to dig through to find whatever I need.

The complete kit weighs in at around 1.2KG, so it’s not exactly lightweight, but it goes in a bag with a heavy laptop and all of its peripherals so it’s not as noticeable as it might otherwise be.

It remains to be seen how much use I’ll actually get out of any of this. Maybe I’ll put a bit of paper in so that I can track how often I use each tool…

Simple backups with Restic and Hetzner Cloud

2024-12-06T00:00:00Z

Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.

I have a confession: for the past few years I’ve not been backing up any of my computers. Everyone knows that you should do backups, but actually getting around to doing it is another story.

Don’t get me wrong: most of my important things are “backed up” by virtue of being committed to remote git repositories, or attached to e-mails, or re-obtainable from the original source, and so on. I don’t think any machine failing completely would be a disaster for me, but it would certainly be a pain.

This week I finally got around to doing something, and it ended up being a lot more straight forward than my previous forays into backup-land.

Restic

After soliciting a few opinions, the choice of backup software came down to either Borg or Restic. I’m pretty sure either would have done what I want, but I leaned towards Restic for a few reasons: it has a more informative website, it’s written in Go rather than Python¹, and Borg seems to be transitioning between major releases at the moment².

The way Restic works is pretty simple: you initialise a ‘repository’, and can then call restic backup /some/path and it’ll get backed up to the repository. Restic handles keeping different backups separate, and only sending data that’s changed, and deduplicating, and so on. You basically point it at a thing you don’t want to lose, and it sorts it out for you. Perfect.

There are equally straight-forward commands for removing old snapshots (restic forget) and verifying backups (restic check). One of the nice things about modern backup solutions is they support a whole range of backends. I was originally going to spin up a small VPS to host my backups, but noticed that Restic supported S3-compatible stores…

Hetzner Cloud Object Storage

I host my servers with Hetzner, and was going to use them to spin up a VPS as well. Despite the “Cloud” branding on a bunch of products, they offer reasonable prices and good service. A couple of months ago, they started offering S3-compatible object storage.

The pricing isn’t totally straight forward, but for continuous use the “free quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free, as is traffic within the eu-central region (where all my servers are). That quota is only awarded when you pay the “base price”, though, which is €4.99 a month. So basically it’s €5 a month for 1TB of storage and enough egress to fully restore every single byte. That’s better value than any VPS I can find, much cheaper than Amazon S3, and about the same as Backblaze B2.

Getting them to work together

Now you’d think making the tool that supports S3-compatible object storage work with your S3-compatible object storage would be easy, right? Well not quite. The library Restic uses to deal with S3 backends has some strange logic for figuring out the bucket name given a URL. It doesn’t quite seem to work right, though…

Hetzner buckets have URLs like s3://bucketname.hel1.your-objectstorage.com³, but just passing that to Restic gives an error that the bucket is not specified. The docs mention there’s an advanced option to make it use the virtual host for the bucket name: -o s3.bucket-lookup=dns. But that… also doesn’t work. What I ended up doing was specifying the URL as s3://hel1.your-objectstorage.com/bucketname, and also passing in the dns option. The library then seems to muddle its way back to a real, working URL. I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that I’m oblivious to?

The next fun part is that you can configure Restic entirely by using environment variables, except for that -o s3.bucket-lookup=dns argument. That has to go on the command line. I ended up making a little wrapper script to invoke Restic correctly:

#!/bin/sh

# The password restic will use to encrypt your data. You should generate
# something nice and secure.
export RESTIC_PASSWORD=repo-password

# The path to the repository where restic will save the backup. In our case
# this takes the form `s3:<endpoint>/<bucket>`. Your endpoint might be different
# to mine depending on the region your bucket is in.
export RESTIC_REPOSITORY=s3:hel1.your-objectstorage.com/bucket-name

# The access and secret key generated in the 'S3 credentials' section of the
# Hetzner Cloud Console
export AWS_ACCESS_KEY_ID=hetzner-access-key-id
export AWS_SECRET_ACCESS_KEY=hetzner-access-key

# Pass any arguments on to the restic command, along with the magic
# `s3.bucket-lookup` option we need to resolve the S3 URL properly.
exec restic -o s3.bucket-lookup=dns "$@"

Then in the backup script I just alias restic to use the script:

#!/bin/bash

set -eu

# This means any time we use `restic` below, we'll actually execute our special
# script which supplies all the env vars and arguments needed to find the
# repository. Make sure the path matches where you saved the script!
alias restic='~/.bin/restic'

# Actually do the backup. Each directory in the list below will be backed up
# separately.
dirs=(
	"/some/path/to/backup/"
	"/some/other/path/"
)
for i in "${dirs[@]}" 
do
	echo $i
	(cd $i && restic --verbose backup .)
done

# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to
# see what effect any changes would have before actually committing to them. 
restic forget --keep-daily 7 --keep-weekly 10 --keep-monthly 24 --keep-yearly 10

The only other notable thing here is that the script changes into the directory to be backed up. If you give Restic an absolute path, it will create a new snapshot if the metadata of any folder in the path changes, which is not what I want.

If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken me about half an hour. That’s including setting up the object storage, installing Restic, and so on. It’s painfully easy. Why didn’t I do this three years ago?!

Addendum: a step-by-step guide

If you want to do this yourself, here’s a quick step-by-step guide:

Log in to the Hetzner Cloud Console, and create a project.
On the “Object Storage” tab, create a new bucket. Note the name and the endpoint.
On the “Security” tab, go to “S3 Credentials” and generate new credentials. Note down the access key and the secret key.
Copy the first shell script above, fill in the password (you can pick!), endpoint, bucket name, access key and secret key.
Run the script with the init argument (e.g. ~/.bin/restic init). This will create a new repository, and only needs to be done once even if you backup multiple machines.
Copy the second shell script above, making sure the restic alias points at the script you saved in step 4. Change the list of directories to whatever you want to backup.
Schedule the script to be run automatically, using crontab or systemd timers or however you prefer.

To check everything is working, you can use the snapshots subcommand to see a list of saved snapshots. You might also want to try to restore a snapshot using the restore subcommand.

I’m not trying to be a language snob, but given the choice between two otherwise equal projects one in Go and one in Python, I’ll take the Go one. I know I’m not going to have weird library issues down the line, and I’m much more comfortable rummaging around the source. ↩︎
With a “Don’t use this in production!” notice on the shiny new version. ↩︎
Aside: I really hate URLs like that. What is with the trend for completely generic domains divorced from the service they’re a part of? ↩︎