Chameth.com - posts like debugging-beyond-the-debugger, infinite-avatars, tailscale-docker-gotcha, thinking-more-about-backups, why-you-should-be-using-https but not migrating-from-github-to-forgejo

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Exposing game servers over Tailscale

2025-12-07T00:00:00Z

I’ve recently been playing a lot of Factorio with a friend. I’ve been hosting, but my desktop computer is behind far too many layers of NAT, and I can’t be bothered dealing with setting up port forwards. Up until today we made do with Steam’s networking support, which in our case ended up relaying the connection via one of their servers. This is amazing as a free, no-hassle service, but the performance was so-so. We’d get random lag spikes, or the initial map download would crawl along at about 1/100th of the speed we should be able to get. We’re both fans of Tailscale though, so maybe there’s an easy solution there?

Tailscale even have an article on sharing a Factorio server but it’s a ten minute long video¹, and focused on setting up an actual dedicated server in Docker rather than just clicking the convenient “Host” button in the game client. What we did instead was:

I shared my desktop’s Tailscale node with my friend, by creating a sharing link in the Admin console
My friend accepted the link, and could then see my machine in their admin panel
I used tailscale lock to sign my friend’s desktop’s node key, as I have tailnet lock enabled
I added a grant to my tailnet’s ACL to permit access from my friend to the specific IP/port on my tailnet:

		{
			"src": ["friend@example.com"],
			"dst": ["100.0.0.42", "fd7a::42"],
			"ip":  ["udp:34197"],
		}

After this, my friend could connect to Factorio by entering the Tailscale IP address, or the Tailscale hostname (as they use MagicDNS). Running tailscale status shows that Tailscale managed to establish a direct connection despite the many layers of NAT involved, and the performance improvements were significant and immediate.

We did a bit of testing, and confirmed that no other services on my machine are accessible. And sharing a machine by default only allows it to accept incoming connections, so I can’t get unwanted access to anything on my friend’s Tailnet, either.

I’ve written before about all the different ways I use Tailscale; this is yet another new way it’s solved a problem for me.

I’m not a fan of video tutorials at the best of times, but especially not when I want to do something as quickly as possible so I can get back to ~~my addiction~~ growing the factory. ↩︎

Thinking more about backups

2025-11-01T00:00:00Z

The Backblaze logo

Almost a year ago I wrote about how I do backups with Restic and Hetzner. That system has been ticking along well ever since, but recently I had some… thoughts. These backups are all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are two glaring issues:

Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse report get it suspended? What if the UK passes even more dumb laws and Hetzner decide it’s easier just to not do business with people here? This is the same sort of concern I have about Google accounts: if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very likely, but these are things my brain likes to dwell on.

Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels a little silly that the same attack could take down both the originals and the backups. There’s no way to avoid that with Hetzner’s S3 implementation, as far as I can tell.

Exploring options

I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that, and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in the worst case.

I then looked around at other S3 providers. Amazon’s glacier offering is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything. There are also lots of weird pricing edge cases around moving data between storage classes, minimum file sizes, and so on. A much better option is Backblaze’s B2 product. Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful in this case: lifecycle rules. Coupled with the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for what’s effectively an append-only store.

This works more-or-less out of the box with Restic. Joseph Price has a guide that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to the end of my script: a simple restic copy and a restic forget. B2 actually works out cheaper than the Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month. Not a bad price for some extra peace of mind!

Avoiding the Consequences of Dumb Laws with Tailscale

2025-09-30T00:00:00Z

More and more sites are implementing privacy-invading age checks or just completely blocking the UK thanks to the Online Safety Act.

Protecting kids from some content online is certainly a noble goal, but the asinine guidance from Ofcom, threats of absolutely disproportionate fines, and the stupidly broad categories of content have resulted in companies just giving up or going through a tick-box exercise that offers very little protection but lots of inconvenience and a complete invasion of privacy.

Instead of uploading my ID to some third party company, I’ve taken to proxying my traffic through to a country that doesn’t have such stupid laws. Thankfully, Tailscale makes this really easy. I’ve discussed how I use Tailscale before, but not really covered app connectors. I find Tailscale’s description of these pretty confusing, but they basically amount to automatic, DNS-based subnet routing configurations (or, to put it another way, a per-website exit node). You can safely ignore all references to ‘SaaS apps’ in their docs.

I create a custom app connector, and give it the domains to be included:

App configuration in the Tailscale admin console

Tailscale then magically resolves those domains, and has the ‘connector’ advertise routes for them. Any client that accepts routes will start sending requests to the connector, which passes them onto the Internet at large. Any other traffic is left alone, unlike when you use an exit node.

The special bit here is how you can specify wildcard domains. Tailscale proxies the DNS requests from clients (so it can inject responses for nodes on your tailnet), which means it can dynamically update the routes as you resolve new domains. I tried to set this up more manually, and quickly came unstuck: despite using the same DNS servers, my server and my desktop would get different responses for the same query as it varied by geography. Trying to get the full set of IPs (and keeping them updated) would have been a nightmare. Tailscale expanding the wildcards nicely sidesteps all of that.

At first I was just proxying the traffic to one of my servers, but just today I added a new connector for Imgur and found I was still blocked, just for different reasons. They not only block my entire country but also a load of known datacenter IP ranges. Hmph. I fixed this by hacking up a new side project: tsv. It’s a simple Go app that accepts traffic from the tailnet (advertising itself as both an app connector and an exit node), and passes it on to another VPN.

There are lots of other ways you could accomplish this, but this makes it so all my devices can still access services without any additional configuration. As long as Tailscale is installed, the Internet will still work as it’s meant to, without all the nonsense. If I come across a site that doesn’t work, adding it is trivial: I just make a new app connector in Tailscale.

Obvious disclaimer: the laws in the UK are binding on the service providers, not the end user. Doing this sort of thing in other countries might be illegal. I don’t know; do your own research! Also all of this is a workaround for something that should be fixed at a legislative level, but I’m not holding my breath.

Making a font of my handwriting

2025-08-08T00:00:00Z

Recently I’ve been on a small campaign to try to make my personal website more… personal. Little ways to make it obvious it’s mine and personal, not just another piece of the boring corporate dystopia that is most of the web these days. I don’t quite want to fully regress to the Geocities era and fill the screen with animated under construction GIFs, but I do want to capture some of that vibe.

I’d added some bits and pieces along those lines: floating images in articles now look like they’re stuck to the page with sellotape, related post links have a wavy border that animates when you hover over them, and so on. Next, I wanted to change the heading fonts from a monospace font to something cursive, to resemble handwriting. Less terminal output, more handwritten letter. I couldn’t find one I liked, though. So why not make my own? It can’t be that hard, right?

Failing to do it myself

I set out to try to make the font myself using open source tools. After doing a bit of research, it seemed like the general approach was to create vectors of each character and then import them into a font editor. That seems to mean either Adobe Illustrator and FontLab (if you have too much money) or Inkscape and FontForge (if you like open source). I fall firmly into the latter category, so I grabbed my graphics tablet and opened Inkscape.

I wrote out my first three letters: capital A, B and C. Saved them in Inkscape, and attempted to import them into FontForge. Then I remembered one crucial thing that had slipped my mind: I absolutely loathe using FontForge. It’s a bit like when you open an old version of GIMP and get a bunch of weird looking windows floating all over the place; it feels like you’re fighting against the tool to do even the most basic operations. The difference is I have cause to edit images a lot more than I edit fonts, and GIMP has actually significantly improved their UI over the years.

Here are the rough steps I went through with FontForge:

Launch Font Forge. It shows a weird bit of art in one window, and an open file dialog in another.
I don’t want to open a file, so I close that dialog. The program exits.
Relaunch Font Forge, and realise that within the “Open Font” dialog is a “New” button. Click it.
Get to the standard font-editing UI. Right-click on the “A” looking for a way to import an SVG. Don’t see one.
Click around a bit, exploring the menus. Everything feels a bit off. You can’t open one menu then hover over the next to see its content, like basically every UI toolkit in existence. I think FontForge has eschewed QT and GTK in favour of doing things itself.
Find the “Import” option in the File menu. Hope it’s for a single glyph not the whole font.
A file picker opens. Again it’s all a bit off from normal desktop conventions. Try to resize it, and just get blank grey space at the bottom.
Type the absolute path I want to go to in the text field.
Get a dialog saying “Not a bdf file /home/chris/etc”. Press OK.
Get a dialog saying “Could not find a bitmap font in”. Press OK.
Press Ctrl+L to see if that lets me enter a path. Click everything in the dialog to try to find a way to enter a path. Get annoyed. Give up. Click through folder-by-folder to get to where I want to be.
Get to the folder and don’t see any files. Change the format to “SVG”. Double-click the newly-visible SVG file.
Get a dialog saying “You must select a glyph before you can import an image into it”. Press OK.
The import dialog goes away, having not imported.
Select the glyph in the main tool area, then repeat the File→Import dance.
It’s actually there now! Open the glyph in the editor and see it’s a complete mess of Bézier curves. I can’t click what I want without accidentally moving a handle for an adjacent curve.
Rage-quit.

I’m sure FontForge is less anger inducing once you’re used to it. And you definitely could use it to build a font like this if you had much more patience than me. I’d had enough of death-by-a-thousand-paper-cuts though.

I briefly tried Inkscape’s built-in support for making an SVG font. It annoyed me a lot less, but it’s fiddly: it seemed like each glyph had to be a single path, so you had to convert the glyphs to paths, then merge them correctly. If you merge them incorrectly then the wrong bits of your letters end up filled (like the inside of the ‘B’). Path manipulation is getting towards the limit of my knowledge of vector editing, and it took a bit of trial and error for each letter that had more than a single stroke. I didn’t fancy doing that for every letter.

I’m usually a big advocate of open source, but this was one of those painful times when it feels like it just falls short. Clunky, painful UI and processes where commercial tools just let you get on with your work.

You can exchange money for goods and services

When I’d been looking for open source tutorials, I found many mentions of a closed source, hosted tool: Calligraphr. It has a free version with limitations (no ligatures, no variations, 75 glyphs per font), and a pro version for £8/month. I’d normally balk at the idea of a subscription for this, but they have the perfect answer: you can make a one-time payment, and your account automatically downgrades back to free after a month. It’s not a hidden option, either, it’s the most prominent button on the upgrade page. That made me happy to give them £8 to play around with the service for a month.

Calligraphr works by having you print templates, write out the letters, then scan them in. It does some magical processing to extract the glyphs, provides tools to tidy them up, align them, etc, and then produces a TTF file for you. You can see some of my completed templates here:

Most of the templates I used for the font

Calligraphr has a nice UI to generate the templates, allowing you to select which glyphs to include. I added the “minimal English”, “basic punctuation” and “Ligatures” sets. That gave me four pages to fill out, and I did them all twice. That let me filter out versions that didn’t work well, and have variants for some letters so the font wasn’t too repetitive. Later on, I went back and added some custom ligatures based on blog post titles that didn’t look quite right: “Re”, “To”, “ers”, “ey”, “hy”, “ra”, “re” and “ty”. Ligatures like this help it look more natural: when we write we don’t just stamp out identical letters regardless of their surroundings, instead they will connect to their neighbours, or overlap slightly, or even share a stroke.

I filled these templates in with a Sharpie, as I wanted a fairly informal, scrap-booky look, and it would also give good solid shapes that should be easy to pick out of the template. I scanned them with the “Scan Document” function on my iPhone, and uploaded the PDFs to Calligraphr.

Iterating and tweaking

The Calligraphr UI allows you to preview the font, but I found it a lot more useful to just download a copy and use it on a local copy of my website. That let me test it with real text, and see how it’d look at the different font sizes I use on the site.

The first version was not great. Despite the guidelines on the template, I apparently wasn’t good at sticking to them. Some letters were floating way off the baseline, and some were sunken below. When those opposites met it looked terrible. Fortunately Calligraphr has a pretty easy tool to slide each letter up and down, and scale it up or down if needed, and you can see it next to other letters as you do it. It took a little bit of time to go through all the variants of all the letters, but the next version looked a lot better.

Another tweak I ended up doing was reducing the spacing between letters. The defaults Calligraphr uses are probably good for a blocky font, but I wanted to put the letters close together to give it more of a joined-up look. Again, this is an easy tool to use, you just drag the sides in or out as desired. While these tweaking steps were probably as fiddly as some of the Inkscape steps I refused to do earlier, they’re a lot more rewarding as you see things improving with each one. It’s a lot easier for me to commit time and effort to improving something that’s already working reasonably, than put that time and energy into an unknown.

Later, I noticed that occasionally there would be a huge gap in a title. Not “the kerning is slightly off” but “there’s enough room to park a bus”. It took me a while to figure out what was happening: a couple of glyphs hadn’t been isolated perfectly and had picked up a few pixels from the template lines at the edge of their boxes. That meant the glyph had a width that covered the actual written glyph, a big gap, and then the rogue marks. At first, I fixed this by just adjusting the width, but that left the little pixels floating awkwardly down-sentence. The proper fix was to use the editing tool and simply delete them, and then Calligraphr snapped the width back to what it should be.

These iterations took a while to do, but I just dipped in and out occasionally over the course of a week, so it didn’t actually feel like too much work. I quite enjoy the process of refining things, too.

Result and a surprise

If you’re viewing this post on my website¹, you can see the font in the headers, captions, and a few other places. Here’s how it compares to my actual handwriting:

My handwriting vs my handwriting font

It’s not close enough to forge documents, but I think it definitely gets across my style, and that’s exactly what I wanted. It’s surprisingly legible even at smaller font sizes — I think the weight of the Sharpie helps here — and at £8 and a bit of manual work was a lot more economical than spending days wresting with open source tools.

A few weeks after I put the finishing touches on the font, I got an e-mail from Calligraphr. As my account had lapsed back to the free version, I was no longer eligible for the “server-side backup” feature. So what did they do? They e-mailed me an exported copy! It’s a JSON file with the properties of each glyph and a base64 encoded image. Not only can I re-upload this to Calligraphr if I resubscribe, I can probably hook something up to edit it should I ever need to. I’m blown away by how pro-user Calligraphr’s business practices are. They’re up-front about pricing, don’t try to get you stuck on an auto-renewing subscription, and automatically export your data. It’s like a breath of fresh air compared to the barrage of dark patterns that other websites foist on us. If you want to make this kind of font, I’d definitely recommend them just because of how nice they are.

And I haven’t changed everything since writing this post… ↩︎