Chameth.com - posts like apple-google-aligned-incentives, debugging-beyond-the-debugger, tailscale-docker-gotcha, understanding-docker-volume-mounts, why-you-should-be-using-https

The Case of the Unchanging Config

2026-04-09T00:00:00Z

Last week I was attempting to make it so I could share pictures on IRC directly from my client. This sounds simple, but it involves a bouncer¹ that proxies the request to a standalone image hosting service that I had to modify to be compatible. At one point my testing loop was:

Make a change to the hosting service
Commit it
Tag a new release
Wait for it to build
Update the version I’m running on my server
Reconfigure the bouncer to pass the new parameter or change the URL or whatever
Try uploading a photo from my phone
Realise I’ve overlooked something and go back to step one

I could have set up a local copy of everything and tested it sensibly. I should have set up a local copy of everything and tested it sensibly. But it seemed like such a trivial change, and setting up the whole environment seemed like such a pain. After the third or so iteration of failure I was pretty annoyed with myself, computers, and basically everything.

My biggest annoyance was that my bouncer would not pick up the new URL from the config when I changed it. It’s meant to reload the config when it receives a SIGHUP, and it claimed to in the logs, but I could clearly see it was still hitting the old URL. Restarting the bouncer to update the config is a pain, as it disconnects me from all the IRC networks, and has to reconnect to them all, reauthenticate, etc. It also mildly spams everyone who shares a channel with me².

When I finally got everything working I had a look at the bouncer source, and thought I’d spotted the issue. I raised a bug report, ending in this remark:

It looks like the config is reloaded properly, but the handler for uploads is created once at startup and has its own copy of the uploader, so effectively snapshots the config to whatever it is at startup:

fileUploadHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    cfg := srv.Config()
    h := fileupload.Handler{
        Uploader:    cfg.FileUploader,
        DB:          db,
        Auth:        cfg.Auth,
        HTTPOrigins: cfg.HTTPOrigins,
    }
    h.ServeHTTP(w, r)
})

I’m not quite sure what I thought I saw there, but that’s not right. I want to call it a hallucination, but that feels like a very overloaded word these days. I guess I was just seeing what I wanted to see, and that was a good excuse to stop investigating. Still, even if I misidentified the cause, the bug was still present, right? … Right?

The Science of Deduction

When the developer got back and said he couldn’t reproduce my issue, I went back to my install and immediately reproduced it. With a calmer head, I figured it was probably something with my particular set up. First thing to check: am I actually running the version I think I am?

Unfortunately the bouncer doesn’t actually seem to expose the version anywhere that I can see. It’s not in the logs, it doesn’t have a -version flag, and none of the IRC-based status commands seem to include it. But I know it’s a Go app, and I know Go embeds the version information. go version -m <binary> will dump it all out, but the binary is inside a Docker image, and the Docker image is one of my nice, minimal, artisanal ones so doesn’t ship a go binary. No problem, docker compose cp bouncer:/bnc ./bnc yoinks the binary out of the container, and then dumping the version shows that, yes, I am running the version I thought I was. Hmm.

The next tool I reached for in my toolbox was strace. Maybe it’s not actually reading the file for some reason? I immediately executed strace -p <pid> -e trace=openat,open,read,pread64 -f without having to look any part of that up. Yep. Definitely. Then I edited the config, HUP’d the bouncer, and saw that it was… reading the config file. As it’s meant to. By default strace truncates strings to 32 bytes, so I couldn’t actually see the line I’d changed. Some more definitely-not-RTFMing later, and rerunning it with an extra -s 65536 let me see the full config. Surprise! The config hadn’t changed!

To confirm my findings, I used docker compose cp again, this time yoinking the config file from inside the container. The inside config file was definitely different to the outside config file. What? My hypothesis at this point was “something something Docker nonsense”. I mount the config as read-only, and was wondering if that meant that Docker was doing something weird instead of just bind mounting it. A quick trip to /proc/<pid>/mounts showed that it was, in fact, not doing anything weird, and was just bind mounting it.

If the file is bind mounted, then surely it’s the same file? I ran stat on the file on the host, noted the inode number, then pondered how to actually run stat on the file inside the container, given the aforementioned awkwardly minimal image. The solution was easy: access it via /proc/<pid>/root/. I could’ve saved myself a bunch of docker compose cp if I’d thought about that earlier. Oh well. The inode of that file was different. What?

The answer was DNS. It’s always DNS. Oh, sorry, force of habit. I meant the answer was vim. I was editing the config in vim, and when it saves files, by default, it writes the new content to a temporary file and does an atomic rename. That’s normally a good thing: it prevents corruption if the write fails midway through³. In this case, though, that means the new file has a new inode. Bind mounting a file binds to the inode, so the container just keeps clutching onto the original config from when it was started, blissfully unaware that the party has relocated down the street.

This problem is likely to happen whenever you bind mount a file into a container. When I mentioned this to a friend, he immediately responded “oh yeah, never do that”, and went on to describe the horrible hacks he’s had to add to Ansible to sidestep the issue. The nicer solution is to just bind mount an entire directory if you can, as then it doesn’t matter what happens to the files within it. I really like having the config files sat alongside the Docker compose files, though; having to create a directory just to work around some bind mount weirdness upsets me.

Now I knew what the problem was, I found there was an issue raised against Docker twelve years ago⁴. The first response was “that’s expected”, and I fully agree with the author: “respectfully, that might be expected by you, but it was not expected by me”!⁵ As a result of the issue they added a nice note to the docs:

Note: Many tools used to edit files including vi and sed --in-place may result in an inode change. Since Docker v1.1.0, this will produce an error such as “sed: cannot rename ./sedKdJ9Dy: Device or resource busy”. In the case where you want to edit the mounted file, it is often easiest to instead mount the parent directory.

But in the intervening twelve years, both the note and the functionality described have gone missing. Ho-hum.

I found it interesting how I’ve only just hit this problem, given how long I’ve used Docker. But I realised that almost everything I run I’m happy to just restart. Cattle, not pets, and so on. My IRC bouncer is one of the few exceptions to that. The only other thing I regularly hot reloaded was Centauri, my reverse proxy, but that had a whole config directory mounted because it was shared between containers, so nicely sidestepped the foot-gun.

So lesson learnt: check for weird bind mount issues before raising issues about config hot reloading. It’ll join the esteemed company of “maybe the drive is full and causing completely unrelated problems?”, “perhaps everything is dog slow because the kernel ran out of entropy?”, and “did systemd sneakily take over that functionality while you weren’t looking?” in the troubleshooting checklist.

An IRC bouncer is basically an always-on proxy. It connects to the IRC networks for you, then your clients connect to your bouncer. The bouncer can then send incoming messages to all your different clients, cache them when you’re offline and replay them later, and lots of other nice things people take for granted in their chat apps these days. ↩︎
Join/part/quit spam is part of IRC, and clients have ways of handling it, but I still feel bad about doing it excessively. ↩︎
You can disable this in vim by setting backupcopy=yes. Good luck redoing all the debugging if you ever accidentally remove that from your vimrc, though! ↩︎
Which is weird, because Docker can’t possibly be that old. That would make me much older than I’m prepared to accept. ↩︎
If you read the thread they didn’t actually mean to sound so dismissive, but it’s still pretty funny. “Yes, it’s expected that the foot-gun causes your foot to hurt. Duh. What did you expect?” ↩︎

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Exposing game servers over Tailscale

2025-12-07T00:00:00Z

I’ve recently been playing a lot of Factorio with a friend. I’ve been hosting, but my desktop computer is behind far too many layers of NAT, and I can’t be bothered dealing with setting up port forwards. Up until today we made do with Steam’s networking support, which in our case ended up relaying the connection via one of their servers. This is amazing as a free, no-hassle service, but the performance was so-so. We’d get random lag spikes, or the initial map download would crawl along at about 1/100th of the speed we should be able to get. We’re both fans of Tailscale though, so maybe there’s an easy solution there?

Tailscale even have an article on sharing a Factorio server but it’s a ten minute long video¹, and focused on setting up an actual dedicated server in Docker rather than just clicking the convenient “Host” button in the game client. What we did instead was:

I shared my desktop’s Tailscale node with my friend, by creating a sharing link in the Admin console
My friend accepted the link, and could then see my machine in their admin panel
I used tailscale lock to sign my friend’s desktop’s node key, as I have tailnet lock enabled
I added a grant to my tailnet’s ACL to permit access from my friend to the specific IP/port on my tailnet:

		{
			"src": ["friend@example.com"],
			"dst": ["100.0.0.42", "fd7a::42"],
			"ip":  ["udp:34197"],
		}

After this, my friend could connect to Factorio by entering the Tailscale IP address, or the Tailscale hostname (as they use MagicDNS). Running tailscale status shows that Tailscale managed to establish a direct connection despite the many layers of NAT involved, and the performance improvements were significant and immediate.

We did a bit of testing, and confirmed that no other services on my machine are accessible. And sharing a machine by default only allows it to accept incoming connections, so I can’t get unwanted access to anything on my friend’s Tailnet, either.

I’ve written before about all the different ways I use Tailscale; this is yet another new way it’s solved a problem for me.

I’m not a fan of video tutorials at the best of times, but especially not when I want to do something as quickly as possible so I can get back to ~~my addiction~~ growing the factory. ↩︎

Avoiding the Consequences of Dumb Laws with Tailscale

2025-09-30T00:00:00Z

More and more sites are implementing privacy-invading age checks or just completely blocking the UK thanks to the Online Safety Act.

Protecting kids from some content online is certainly a noble goal, but the asinine guidance from Ofcom, threats of absolutely disproportionate fines, and the stupidly broad categories of content have resulted in companies just giving up or going through a tick-box exercise that offers very little protection but lots of inconvenience and a complete invasion of privacy.

Instead of uploading my ID to some third party company, I’ve taken to proxying my traffic through to a country that doesn’t have such stupid laws. Thankfully, Tailscale makes this really easy. I’ve discussed how I use Tailscale before, but not really covered app connectors. I find Tailscale’s description of these pretty confusing, but they basically amount to automatic, DNS-based subnet routing configurations (or, to put it another way, a per-website exit node). You can safely ignore all references to ‘SaaS apps’ in their docs.

I create a custom app connector, and give it the domains to be included:

App configuration in the Tailscale admin console

Tailscale then magically resolves those domains, and has the ‘connector’ advertise routes for them. Any client that accepts routes will start sending requests to the connector, which passes them onto the Internet at large. Any other traffic is left alone, unlike when you use an exit node.

The special bit here is how you can specify wildcard domains. Tailscale proxies the DNS requests from clients (so it can inject responses for nodes on your tailnet), which means it can dynamically update the routes as you resolve new domains. I tried to set this up more manually, and quickly came unstuck: despite using the same DNS servers, my server and my desktop would get different responses for the same query as it varied by geography. Trying to get the full set of IPs (and keeping them updated) would have been a nightmare. Tailscale expanding the wildcards nicely sidesteps all of that.

At first I was just proxying the traffic to one of my servers, but just today I added a new connector for Imgur and found I was still blocked, just for different reasons. They not only block my entire country but also a load of known datacenter IP ranges. Hmph. I fixed this by hacking up a new side project: tsv. It’s a simple Go app that accepts traffic from the tailnet (advertising itself as both an app connector and an exit node), and passes it on to another VPN.

There are lots of other ways you could accomplish this, but this makes it so all my devices can still access services without any additional configuration. As long as Tailscale is installed, the Internet will still work as it’s meant to, without all the nonsense. If I come across a site that doesn’t work, adding it is trivial: I just make a new app connector in Tailscale.

Obvious disclaimer: the laws in the UK are binding on the service providers, not the end user. Doing this sort of thing in other countries might be illegal. I don’t know; do your own research! Also all of this is a workaround for something that should be fixed at a legislative level, but I’m not holding my breath.

How tech companies failed to build the Star Trek computer

2025-07-16T00:00:00Z

A computer core room on the Enterprise-D

In most Star Trek series, the ship or station computer is ever-present in the background, waiting to be called on by the main characters¹. It nearly always does exactly the right thing, and there’s little limit to the functions it can perform. Take this mundane example from DS9:

KIRA: Computer, establish link with the Bajoran Medical Index for the Northwestern District.
COMPUTER: Link established.
KIRA: Access all information on Doctor Surmak Ren.
COMPUTER: There are no records matching that name.
KIRA: Try the Northeastern District, same search.
COMPUTER: Doctor Surmak Ren, currently serving as Chief Administrator of the Ilvian Medical Complex.
KIRA: Computer, open a channel to the Ilvian Medical Complex. Administrator’s office.

The computer is doing some kind of networking to a database only identified by name. It does a search and summarises the lack of results. It then repeats the process with another database, and succinctly announces the results. Finally, it opens a communication channel to a specific room in a facility, based only on its name.

This whole interaction is remarkably boring². Kira doesn’t have to know any URLs or API endpoints, or what protocol she wants to use. She doesn’t have to open a specific app and then login and then try the query again. She just says what she wants and the computer does it.

It seems like this should be one of the most easily obtainable bits of sci-fi wizardry with our current technology. We have multiple massive companies throwing lots of money at digital assistants, LLMs that are improving at an insane rate, but we’re somehow not even close to the usability or usefulness of the Trek computers. What gives?

Boring is, well, boring.

Larry Page once said something that might help explain it:

The Star Trek computer doesn’t seem that interesting. They ask it random questions, it thinks for a while. I think we can do better than that.

This is the same Larry Page that founded Google, whose mission statement is “to organize the world’s information and make it universally accessible and useful”. Of all people, surely he should find an omnipresent computer that can answer ‘random questions’ interesting?! It seems like it should be the epitome of Google’s mission!

Google’s “better than that” seems to have been to stuff LLMs into every product they can, even when you don’t want them there. Even when they’re worse than the normal content they displace. These things look exciting when they’re part of a scripted demo at Google I/O, but they fall flat and just get in the way when they’re exposed to the reality of day-to-day use.

The Star Trek computer is the opposite: it isn’t snazzy, but it is genuinely useful. That means it’s not an attractive target for the company execs who want marketing opportunities, and it’s not appealing for engineers who need to demonstrate “impact”. But even if Google did try to make the Trek computer, there are other problems…

Assistants need to be free

A significant amount of tech companies’ business models currently revolves around trapping users in walled gardens. They want you using their ecosystem; that way they get more data from you, and you’re more likely to spend more money on their other offerings that work together. There’s barely any incentive to allow any kind of interoperability with other platforms outside carefully contracted integrations.

I remember trying to help a family member move their photos from iCloud to Google Photos. At one point they turned around and said, exasperated, “why is this so hard? Aren’t they both in the cloud?!”. It’s easy to dismiss that as someone who hasn’t quite grasped the fundamental idea that “the cloud” is just someone else’s computers, but that’s not the whole story. There’s no reason why there shouldn’t be a quick and easy transfer: both services already allow uploading and downloading, there’s just no incentive for the companies involved to make it so³.

These kinds of misaligned incentives and walled garden business models cause even more problems when it comes to digital assistants. Siri is basically never going to be able to interact with, say, your Google Drive; ~~Bard~~ Gemini is never going to be able to send a message via iMessage. Even when there are appropriately blessed interactions, they’re so clunky. Can you imagine Captain Picard saying “Computer, ask the turbolift skill to take me to deck 5”?

Someone else’s computer

Software issues aside, there’s still a key difference between the Star Trek computers and our current batch of digital assistants: where they run. The Trek computers are all housed within the ship or station they serve; they can connect elsewhere to gather information, but they run entirely independently. If they go wrong, a local engineer can go in and fix things. While some of our assistants may have physical hardware in your home, they don’t work without a vast cloud apparatus behind them. If your Internet connection fails, they become paperweights. If the company running them decide to remove some functionality you depend on, you have no recourse.

That kind of helplessness isn’t limited to assistants, either. There’s a rapidly growing trend of being unable to modify or repair hardware you fully own and control. Part of this is just that they’re becoming more complex: it’s a lot harder to replace a microchip than a gear, but companies are also going out of their way to make it more difficult for users through draconian DRM regimes⁴ and aggressive intellectual property enforcement. If the US Navy can’t repair their own equipment because a corporation says so, what hope do consumers have?

We’re approaching a point where you don’t actually own anything. Software is cloud and subscription based, hardware is unrepairable. Even cars can be remotely updated and have features added or removed. The Federation wouldn’t allow a third party control over their ships⁵, so why are we so happy to put up with it in everything we consume?

A small ray of hope?

The most promising way of tackling all of these problems is through legislation. The EU’s Digital Market Act is an attempt to force ‘gatekeepers’ like Google, Apple and Meta, to allow third-party access to their services. It seems like a pretty reasonable approach, but the tech companies are unsurprisingly resisting it. Apple in particular have gone out of their way to refuse to comply, and when forced to do so have limited the functionality to people in Europe. Still, the DMA is a promising start, and if similar legislation is introduced (and robustly enforced) elsewhere it might start forcing companies to behave a bit better.

There are also smaller companies that actually do the right thing. Framework make laptops that are user-serviceable; Fairphone do the same for mobile phones. Smaller software companies provide useful, open APIs. The average person on the street will probably have never heard of these, unfortunately, but they do still exist. Maybe as the bigger tech companies tighten the screws more, people will turn to alternatives like this? Or maybe we’ll just keep accepting that our computers work for everyone but us?

Unless, of course, the computer is playing the role of the episode’s MacGuffin and has contracted space-computer-COVID or something, then it’s a lot less in-the-background. ↩︎
It’s almost like it only exists to move the plot along. ↩︎
You can generally export your data, thanks to a combination of legislation and efforts like Google’s “Data Liberation Front”, but I’ve never seen an export format that could then just be imported into an equivalent commercial product. ↩︎
Oh, you’ve changed the screen on your iPhone? Better hope it can do the secret handshake with the Apple hardware. ↩︎
I think there might actually have been an episode where that did in fact happen. We’ll just ignore that as a plot contrivance. ↩︎