Chameth.com - posts like curse-of-knowledge-blogging, docker-automatic-nginx-proxy but not debugging-beyond-the-debugger, migrating-from-github-to-forgejo, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Blogging and the Imaginary Quality Bar

2025-09-18T00:00:00Z

Recently I realised that I’ve developed a self-imposed quality bar for blog posts. They need to be a certain length, and have a certain substance to them. They need to be generally useful in some way I can’t quite define, to some imagined future audience. They need to have images to break up the page, and opengraph data for when they’re linked to on social media. But… maybe they don’t? Those things all make sense for longer “article” type posts, but not so much for a personal blog.

I’ve done some fiddling so that I can make posts without all that extra stuff. We’ll see how it goes. The posts won’t be distinguished on the site (yet?), but I have set up separate RSS feeds for just short form posts and long form posts in case subscribers are radically opposed to one or the other. Now I can start blogging like it’s 2005 again. But hopefully with a bit less cringe.

I’ve been staring at this post in my editor for about five minutes. The urge to make it longer, more thorough, more article-like is really strong. This entire paragraph is only here as a compromise with myself so I can actually save and commit the post.

If all you have is a hammer…

2025-06-18T00:00:00Z

This, dear reader, is a hammer. It is almost entirely irrelevant to the article. Enjoy.

I presume everyone is familiar with the idiom “if all you have is a hammer, everything looks like a nail”. If not, well, there it is. It’s generally used pejoratively about being single-minded, but I think it also gives a glimpse into something more interesting: mental and perceptual sets.

Before I explain, let me tell you a story about a person who bought a 3D printer¹. When they were first thinking about getting one, they weren’t sure if it was worth it. They could print a few board game accessories, but then what? After they got it, though, a whole new world opened up: everywhere they looked there were opportunities to improve things by adding 3D printed plastic. Broken appliances were repaired with 3D printed parts, all sorts of shelves, organisers, hooks and other things were made. But when they talked about this to other people, most often the response was “That’s nice, but I don’t think I’d use one”. How could they not see the truth in all its glorious layer lines?!

The answer is in the concept of a ‘set’², and instead of trying to explain it, I’m just going to quote Wikipedia:

In psychology, a set is a group of expectations that shape experience by making people especially sensitive to specific kinds of information. A perceptual set, also called perceptual expectancy, is a predisposition to perceive things in a certain way. […] A mental set is a framework for thinking about a problem. It can be shaped by habit or by desire. Mental sets can make it easy to solve a class of problem, but attachment to the wrong mental set can inhibit problem-solving and creativity.

This perfectly captures what happened. They’d developed a ‘3D printing mental set’: a predisposition to see problems that can be solved with their ~~hammer~~ 3D printer. Once I started noticing this pattern, I started seeing it all over the place.

Sets at work

Aside from the “could this be improved with some 3D printed plastic” perceptual set, I’ve noticed a couple of other sets: DIY, and writing puzzles.

DIY is probably the most obvious one. Once you become a little familiar with the basics of what you can fix and how, it seems like you start noticing the opportunities more. Before, you might have ignored that loose door handle, but now you notice it and realise with a screwdriver and a sex bolt³ you can fix it and stop it coming loose again anytime soon.

As you do more things, you build up your set of mental sets as well. You learn how to attach things to different types of walls, how to safely work with electricity, and so on, and suddenly installing a new outside light goes from a seemingly complex job to just exercising some of those concepts you’re already familiar with.

This works across ‘disciplines’ as well: I wanted to automatically cool a room down, using a fan in the window to blow in cool air. The 3D printing mental set gave me a way to make it physically go where I wanted it to, the home automation set gave me a plan for how to automate it, and the electricity set helped me figure out how to wire in the relay. It sounds a bit stilted when I spell it out like that, but the catch is I didn’t have to sit down and work any of this out. I was already primed to think in a certain way. I had a whole selection of hammers, and could pick the right combination of them to beat this particular nail⁴.

Puzzles are a bit different. I enjoy taking part in puzzle hunts⁵, and at first I was bewildered at how the authors came up with all the puzzles. After I started thinking about writing some myself, though, I began to notice all kinds of things that could maybe be turned into a puzzle: a specialised database, or a song with repetitive lyrics, or set of things with unique ID numbers. Writing the actual puzzles is still tricky, but once you’ve equipped the “look out for puzzle material” perceptual set it’s much easier to gather a range of ideas.

For example, one puzzle I wrote came about because I stumbled upon Sir Patrick Stewart’s Cowboy Classic Sampler⁶. I think everyone will agree that it’s a fairly unique collection. I tied some samples of the songs together with some screen captures of Sir Patrick as Picard in Star Trek: The Next Generation, and puzzlers had to find the season and episode numbers of the pictures, then use those numbers to select a track and a word from that track in the audio mix. Those words spell out a question, and the answer to that question is the answer to the puzzle.

There were other hints mixed in to point you in the right direction, it wasn’t quite as opaque as it might sound! But the point is without being in that “puzzle finding” mindset when I came across the album, there’s no way I could have come up with that puzzle. Like with the fan, I was primed to think a certain way, and managed to create something I otherwise would have struggled with.

The curse of noticing things

Figuring out how to make stuff isn’t the only place sets appear, though. Perceptual sets are about how we perceive things. When you read something with lots of bullet points, em-dashes, and the word “delve” do you automatically think it’s LLM generated?⁷ Then you have an LLM text perceptual set! It’s kind of annoying, as it’s difficult to “turn off” and stop noticing these things once you’ve seen them. Similarly, I think I have a “bad grammar” set. I know it doesn’t bother a lot of people, but if I’m reading a book and there’s a rogue apostrophe it will throw me right out of the moment. It’s like tripping up when walking and being torn out of a daydream. There’s some use to these — being able to notice grammar errors easily helps with my writing — but for the most part they’re a negative input: they point out things are broken or bad.

The “edge case set” takes this to another level. After about 20 years of writing software⁸ I have a finely honed ability to spot problems. Give me something to implement and my brain will immediately rattle off all the different edge cases and under-specified parts. Then some other bit of my brain gets the job of deciding which we actually care about, which are worth raising, and so on. It’s definitely useful to spot them up-front, rather than when you’re deep in the weeds implementing something, and tends to make everything go a lot smoother if you can hammer the major ones out when planning work. It doesn’t turn off after work though; it comes up when reading board game rule books, or descriptions of the magic system in fantasy novels, or news reports, and so on. It’s a lot less useful then.

Speaking of news reports, here’s a fun one: Gell-Mann amnesia is a cognitive bias where even when you critically assess media reports about something you’re knowledgeable about (say, technology), you continue to trust similar reporting in other areas (say, medicine). But once you know about Gell-Mann amnesia, you might start to actively notice when it happens. You scoff at the article about how the Internet is a “series of tubes” and then start reading the next one about a cancer treatment, but — wait a minute — your Gell-Mann amnesia set kicks in. I’m not sure what you then do with this information. Dismiss basically all the media and news as useless? It’s honestly pretty annoying.

The meta set

I’ve had this blog post on my ideas list since I first noticed the “3D printing mindset”. I didn’t know sets were an actual thing, and I didn’t have most of the examples I do now. As time went on with this idea rattling around in the back of my head, I started noticing the patterns come up more and more. There’s lots of more examples I didn’t include because I didn’t want to just write a listicle of mental patterns. But the fact that I noticed them all seems to mean that I’ve developed a set set, or a meta-set, if you will. It’s fascinating to think about how all this happens inside our mushy bag of neurons, but also a bit terrifying. If thought processes are that malleable, what stops you accidentally breaking them? Actually, thinking about things like Q-Anon, flat earthers, and a lot of contemporary “politics”, maybe that’s quite easy to do?

Image credits

Photo	Creator	Licence	Source
Hammer	Ian Talmacs	Unsplash License	Unsplash

Surprise! I am that person. Bet you didn’t see that coming! ↩︎
I feel like they should have consulted the mathematicians and/or computer scientists before they picked a name. ↩︎
Yes, that’s a real thing. No, I won’t ever use any of the alternative names for it. ↩︎
OK, I’ve strained the metaphor enough. I’m sorry. I’ll stop. ↩︎
If you’re not familiar, a puzzle hunt is a competition where you try to solve puzzles faster than other teams. The sort of puzzles involved are perfectly described by this quote from Mike Develin: “Imagine a word search. Now imagine you aren’t told what words to look for. Now imagine you aren’t told it’s a word search. Now imagine it isn’t a word search.” ↩︎
Go and give it a listen. I’ll wait. ↩︎
I hereby certify all my em-dashes are hand-crafted. Relatedly, Claude Code keeps writing comments that use hyphens when they should actually be em-dashes. Maybe they’ve trained it to avoid them? ↩︎
I just made myself sad by working this out. I still feel young at heart! … Or maybe that’s cholesterol? ↩︎

The Curse of Knowledge and Blogging

2025-05-16T00:00:00Z

Finding a picture to go with a blog post about blog post ideas is hard, OK?

The single worst part about blogging for me is trying to come up with ideas for what to write about. Not because they’re hard to come up with, but because every idea seems too basic or not worth talking about. Seems.

It’s not just me, either. My friends will occasionally describe some interesting problem they’ve dealt with, and I’ll chime in with “you should blog about that”. Almost invariably the response is something like “it’s not that interesting”.

There’s a cognitive bias called the curse of knowledge¹. Basically, once you know something you can’t accurately reconstruct your previous state of mind from when you didn’t know it. You can struggle to do something for hours, piecing together scraps of information, using all kinds of specialist knowledge, and as soon as you’re finished it all seems so basic why would anyone want to read about that?

Other mental obstacles

Of course, nothing involving people is as simple as having one cause and an obvious solution. Even when I get over the “it’s too basic” road bump, there are other issues. Looking back over some of my posts, I get a horrible cringe-y feeling and the question of “why would anyone be interested in reading that?!” arrives unbidden. Even though I know that I’m interested in reading similar posts written by other people. I’m hesitant to attach labels to everything, but imposter syndrome says hi.

I think there’s also a cultural issue at play. British people tend to be somewhat reserved and downplay their accomplishments, especially when compared to people from the US². When I worked for a certain multinational ~~ad conglomerate~~ search engine company we actually had training on how to do performance reviews and speak about ourselves more like our US counterparts.

Put all these things into a cauldron, add a lizard’s leg and howlet’s wing, mix it all up, and you get a brain that despises the idea of blogging about anything. Great.

Dealing

So, now the problem is well-defined, what do we do about it? Clearly I am the best person to be offering this advice, given it’s been four months since my last blog post.

The best counter to the Curse of Knowledge that I’ve found is keeping notes, like a lab notebook. If you finish something and have dozens of links to different resources that you had to piece together, it’s probably something interesting and worth talking about even if it doesn’t seem like it any more. Or maybe you didn’t need a lot of references, but you had to write a lot down to make sense of something, or needed to draw diagrams to solve a problem. Again, probably interesting.

The other issues I’ve mentioned are not so easy to overcome. I’m also not going to try and perform therapy in the form of a blog post! What I personally do is keep a list of topics to write about (that have already passed the “probably interesting even if my brain doesn’t agree” check), and occasionally when the stars align I’ll feel like plucking one off the list and turning thoughts into words.

A call to action

If you’ve got a blog and haven’t written anything recently because you “don’t do anything interesting”, go write something! I promise that someone will be interested. Hell, feel free to send me a link and I’ll be interested for you.

If you don’t have a blog then why not start one? There are lots of great articles out there about why-you-should and how-you-can. And as a bonus, blog posts about how you set up your blog are super interesting. That’s one post sorted already!

As the open internet is slowly subsumed by massive corporate interests, and gummed up with spam and AI slop, the voice of real humans is becoming more and more important. Maybe we can — one blog at a time — get back to the ’90s internet that was full of individuality instead of today’s which is full of ads and spam?

Photo by Mika Baumeister on Unsplash

There’s also something called “Familiarity Blindness” that people like to talk about. It’s a much better name. The only problem is that I can’t find any actual published works that talk about it. No-one authoritative defines or mentions it. It just popped into existence a few years ago. Maybe it’s an LLM hallucination? ↩︎
My favourite commentary on this is Stephen Fry talking about American vs British comedy. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.