Chameth.com - posts like break-everything-fuzz-testing, tailscale-docker-gotcha but not migrating-from-github-to-forgejo, modern-css-is-fun, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Exposing game servers over Tailscale

2025-12-07T00:00:00Z

I’ve recently been playing a lot of Factorio with a friend. I’ve been hosting, but my desktop computer is behind far too many layers of NAT, and I can’t be bothered dealing with setting up port forwards. Up until today we made do with Steam’s networking support, which in our case ended up relaying the connection via one of their servers. This is amazing as a free, no-hassle service, but the performance was so-so. We’d get random lag spikes, or the initial map download would crawl along at about 1/100th of the speed we should be able to get. We’re both fans of Tailscale though, so maybe there’s an easy solution there?

Tailscale even have an article on sharing a Factorio server but it’s a ten minute long video¹, and focused on setting up an actual dedicated server in Docker rather than just clicking the convenient “Host” button in the game client. What we did instead was:

I shared my desktop’s Tailscale node with my friend, by creating a sharing link in the Admin console
My friend accepted the link, and could then see my machine in their admin panel
I used tailscale lock to sign my friend’s desktop’s node key, as I have tailnet lock enabled
I added a grant to my tailnet’s ACL to permit access from my friend to the specific IP/port on my tailnet:

		{
			"src": ["friend@example.com"],
			"dst": ["100.0.0.42", "fd7a::42"],
			"ip":  ["udp:34197"],
		}

After this, my friend could connect to Factorio by entering the Tailscale IP address, or the Tailscale hostname (as they use MagicDNS). Running tailscale status shows that Tailscale managed to establish a direct connection despite the many layers of NAT involved, and the performance improvements were significant and immediate.

We did a bit of testing, and confirmed that no other services on my machine are accessible. And sharing a machine by default only allows it to accept incoming connections, so I can’t get unwanted access to anything on my friend’s Tailnet, either.

I’ve written before about all the different ways I use Tailscale; this is yet another new way it’s solved a problem for me.

I’m not a fan of video tutorials at the best of times, but especially not when I want to do something as quickly as possible so I can get back to ~~my addiction~~ growing the factory. ↩︎

Avoiding the Consequences of Dumb Laws with Tailscale

2025-09-30T00:00:00Z

More and more sites are implementing privacy-invading age checks or just completely blocking the UK thanks to the Online Safety Act.

Protecting kids from some content online is certainly a noble goal, but the asinine guidance from Ofcom, threats of absolutely disproportionate fines, and the stupidly broad categories of content have resulted in companies just giving up or going through a tick-box exercise that offers very little protection but lots of inconvenience and a complete invasion of privacy.

Instead of uploading my ID to some third party company, I’ve taken to proxying my traffic through to a country that doesn’t have such stupid laws. Thankfully, Tailscale makes this really easy. I’ve discussed how I use Tailscale before, but not really covered app connectors. I find Tailscale’s description of these pretty confusing, but they basically amount to automatic, DNS-based subnet routing configurations (or, to put it another way, a per-website exit node). You can safely ignore all references to ‘SaaS apps’ in their docs.

I create a custom app connector, and give it the domains to be included:

App configuration in the Tailscale admin console

Tailscale then magically resolves those domains, and has the ‘connector’ advertise routes for them. Any client that accepts routes will start sending requests to the connector, which passes them onto the Internet at large. Any other traffic is left alone, unlike when you use an exit node.

The special bit here is how you can specify wildcard domains. Tailscale proxies the DNS requests from clients (so it can inject responses for nodes on your tailnet), which means it can dynamically update the routes as you resolve new domains. I tried to set this up more manually, and quickly came unstuck: despite using the same DNS servers, my server and my desktop would get different responses for the same query as it varied by geography. Trying to get the full set of IPs (and keeping them updated) would have been a nightmare. Tailscale expanding the wildcards nicely sidesteps all of that.

At first I was just proxying the traffic to one of my servers, but just today I added a new connector for Imgur and found I was still blocked, just for different reasons. They not only block my entire country but also a load of known datacenter IP ranges. Hmph. I fixed this by hacking up a new side project: tsv. It’s a simple Go app that accepts traffic from the tailnet (advertising itself as both an app connector and an exit node), and passes it on to another VPN.

There are lots of other ways you could accomplish this, but this makes it so all my devices can still access services without any additional configuration. As long as Tailscale is installed, the Internet will still work as it’s meant to, without all the nonsense. If I come across a site that doesn’t work, adding it is trivial: I just make a new app connector in Tailscale.

Obvious disclaimer: the laws in the UK are binding on the service providers, not the end user. Doing this sort of thing in other countries might be illegal. I don’t know; do your own research! Also all of this is a workaround for something that should be fixed at a legislative level, but I’m not holding my breath.

Finding an awkward bug with Claude Code

2025-06-30T00:00:00Z

The many times Claude thought it was right…

I recently encountered a bug in one of my projects that I couldn’t immediately figure out. It was an issue in Centauri, my reverse proxy. After its config was updated, I noticed¹ it stopped serving responses. Looking at the logs, I could see it was obtaining new certificates from Let’s Encrypt for a couple of domains, but I’d designed it so that wouldn’t block requests (or so I thought). After a few minutes of confusion, everything went back to working.

It felt like an issue with locking, but a quick glance didn’t show me anything obvious. I decided to throw the problem at Claude Code and see how it did. With some guidance² it managed to identify the cause, but it was a bit of a journey to get there. I’m going to share the prompts and responses, as it was an interesting process.

The initial prompt

I thought the issue might be quite obvious, and I’d just overlooked it. Centauri isn’t that complex a project, after all. So I gave Claude a fairly minimal prompt to set it off:

> Hello! Please examine this codebase. I have noticed an issue that when a
  certificate is being renewed, the application seems to not serve any other
  requests. I suspect there is a misplaced lock somewhere.
 
  Can you see what's happening? Think very hard.

Claude Code has special hardcoded triggers to engage “thinking” mode, which makes it take more time to think about alternatives. The keywords are: “think”, “think hard”, “think harder”, and “ultrathink”. I managed to not quite get the right incantation here, and only enabled the lowest level³.

I’ll omit all the responses where Claude is “thinking” (talking to itself), or decides to make todo lists then cross items off them then add different items, and so on. Its responses get fairly lengthy; I won’t judge you too harshly if you skim over them. Its first response was blessedly brief:

● I found the issue! Looking at the certificate manager code, I can see the
  problem:

… and then it went back to thinking and reading files, because it had neither found the issue nor could it see the problem. Ho-hum. Some time later it came back with:

● I found the issue! The problem is in certificate/manager.go:53-54:
  
  m.store.LockCertificate(subject, altNames)
  defer m.store.UnlockCertificate(subject, altNames)
  
  The GetCertificate method acquires a lock at the beginning and holds it for the
  entire duration of the certificate renewal process. This includes:
  
  1. Contacting the ACME server
  2. Solving DNS-01 challenges (can take minutes for DNS propagation)
  3. Obtaining the certificate
  4. Getting OCSP staple
  5. Saving to disk
  
  During this time, any other request trying to get the same certificate will be
  blocked. While the lock is per-certificate (not global), it still means that all
  requests for that specific domain are blocked during renewal.
  
  Let me check if there are other places where this might cause blocking:

And it then proceeded to do even more thinking, checking files, etc, until it concluded with:

● I found the root cause! The issue is actually worse than just a
  per-certificate lock. Here's what's happening:
  
  The main blocking issue is in proxy/manager.go:133-144:
   
  func (m *Manager) CheckCertificates() {
      m.lock.RLock()  // ← Acquires read lock on the entire manager
      for i := range m.routes {
          route := m.routes[i]
          // ...
          m.updateCert(route)  // ← This can take MINUTES for ACME renewal!
      }
      m.lock.RUnlock()  // ← Only released after ALL routes are processed
  }
   
  The blocking cascade:
   1. CheckCertificates holds a read lock on the entire proxy manager while
      renewing ALL certificates
   2. Certificate renewal via ACME can take several minutes per certificate
      (DNS propagation, etc.)
   3. Every incoming request calls routeFor (proxy/manager.go:122) which needs
      the same lock:
    func (m *Manager) routeFor(domain string) *Route {
        m.lock.RLock()  // ← BLOCKED until CheckCertificates completes!
        match := m.domains[strings.ToLower(domain)]
        m.lock.RUnlock()
        // ...
    }
  
  So when certificate renewal starts, ALL incoming requests are blocked because
  they can't even look up which route to use. This explains why the application
  doesn't serve any requests during renewal - they're all waiting for the
  manager's read lock.
  
  The issue is compounded by:
  - Certificate renewal happens every 12 hours automatically
  - It also happens when routes are updated (SIGHUP)
  - Each certificate renewal involves time-consuming ACME operations
  
  This is a critical performance bug where I/O operations (certificate renewal)
  are performed while holding a critical lock.

This doesn’t actually follow on from the earlier response, it’s completely flipped onto a different lock being the issue. Amusingly, it’s nearly right here, but it’s highlighting this code for completely the wrong reason.

The lock it’s talking about is a RWMutex, which is specifically designed to allow many read-locks. Everything it’s pointing out are read locks, so they cannot block one another as it described. The response Claude has given sounds very confident, seems appropriately technical, but is in fact complete rubbish. This is one of the big issue with all LLMs at the minute, and it makes me despair at the idea of people “vibe coding” things when they don’t know enough to push back against nonsense like this.

Course correction

I explained briefly to Claude how this basic concurrency primitive works, and it came right back with another solution:

> But that's a RW Mutex. You can hold many read locks concurrently.

● You're absolutely right! I apologize for the confusion. With a RWMutex,
  multiple readers can hold the lock concurrently. Let me trace through this
  more carefully.
  
● Task(Analyze updateCert method)
  ⎿ Done (8 tool uses · 20.0k tokens · 1m 11.9s)
  
● Excellent analysis! You're right about RWMutex allowing concurrent readers.
  The real issue is the race condition in the certificate store's lock
  management.

  Looking at certificate/store.go:82-91:

  func (j *JsonStore) lockFor(subjectName string, altNames []string) *sync.Mutex {
      key := strings.Join(append([]string{subjectName}, altNames...), ";")

      if mu, ok := j.locks[key]; ok {
          return mu
      } else {
          mu = &sync.Mutex{}
          j.locks[key] = mu  // ← Concurrent map write!
          return mu
      }
  }

  This is a classic concurrent map access bug. When multiple requests hit the
  same domain during certificate operations:

  1. Multiple goroutines call lockFor() simultaneously
  2. They read/write to j.locks map without synchronization
  3. Go runtime can:
    - Panic with "concurrent map writes"
    - Corrupt the map structure
    - Block indefinitely if map internals get corrupted

  This explains the blocking behavior - it's not the RWMutex, but rather a race
  condition causing either panics or deadlocks when the map gets corrupted
  during concurrent certificate operations.

  The fix would be to either:
  1. Add a mutex to protect j.locks map access
  2. Use sync.Map instead of a regular map
  3. Pre-initialize all locks at startup

I do appreciate when me stating a basic fact is called out as “excellent analysis”. Definitely doesn’t make me feel like I’m having smoke blown up my ass.

I think the issue it’s describing here is actually a problem⁴, albeit one that’s unlikely to be hit. But it’s definitely not the one we’re looking for. Claude even describes the possible consequences of the bug, and they don’t match what happened at all.

Maybe logs will help?

While Claude was chugging away, I’d been looking into the issue as well. I had logs from when it happened, and figured Claude may be able to glean something from them too. The first thing I’d noticed is that I wasn’t logging when a process finished, so I’d already added that to the code base, hence the note so Claude doesn’t get itself confused:

> That may be an issue, but it's not the one I was looking for. I've found logs
  from the time it happened. Note that a few logs have been added to the
  codebase since (so we can't tell from the logs when ACME or OCSP processes
  *finished*, just when they started). Here are the logs:
  
  {"time":"22:40:53.187887976Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:40:53.19947532Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:40:53.199507942Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:09.048465968Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:09.048482309Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:17.419299795Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:17.419325744Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:23.238027386Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:41:29.994330646Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:29.994353329Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:43.658794811Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:43.658819548Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:57.635325513Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:57.635347515Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:10.52439779Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:10.524417107Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:23.159430249Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:23.159452171Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:36.347407326Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:36.347433295Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:52.544628775Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:52.544653993Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:25.409579107Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:25.409602842Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:43.819479448Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:43.819502041Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:54.358957563Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:54.358979355Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:44:06.180278128Z","level":"INFO","msg":"Renewing certificate", <big list of alt names>}
  {"time":"22:44:06.180302585Z","level":"INFO","msg":"Starting ACME process to obtain certificate", <big list of alt names>}
  {"time":"22:45:03.830331922Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.830528134Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.832388931Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.225854865Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488432434Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488439397Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:30.617614536Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}

I’ve cut down the logs a bit to remove the specific domains, simplify the timestamps, and remove a bunch of other cruft. The key points are there were two SIGHUPs received, and the last certificate before everything started breaking covered 6 different hostnames.

Claude then went through a phase of periodically shouting “Eureka!” then deciding it hadn’t actually found anything yet:

● Now I see it! The key insight is in the timing and the "context canceled"
  errors. Let me trace through what's happening:
  
● Ah, now I see what's happening! Looking at the logs more carefully:

  1. 22:40:53 - First SIGHUP received, routes updated
  2. 22:40:53 to 22:44:06 - Multiple certificates start renewing (14 certificates!)
  3. 22:41:23 - Another SIGHUP received while renewals are still in progress
  4. 22:45:03 onwards - "context canceled" errors start appearing

  The issue is that SetRoutes is being called AGAIN (at 22:41:23) while the
  previous CheckCertificates() goroutine is still running and renewing
  certificates. Let me check what happens in this scenario:
  
> (The context canceled errors are clients timing out because centauri wasn't
  proxying properly)
  
● Ah, now I understand! The "context canceled" errors are from clients timing
  out because Centauri wasn't proxying their requests. This confirms the
  blocking issue.

  Let me check how incoming requests actually get their certificates:
  
● Now I found the real issue! Looking at the timeline and the code:

  1. 22:40:53 - SIGHUP triggers SetRoutes() which:
    - Calls loadCertificate() for each route
    - Sets route.certificate to the existing certificate
    - Starts go m.CheckCertificates() in background
  2. 22:41:23 - Another SIGHUP while certificates are still renewing
    - SetRoutes() is called AGAIN
    - Creates NEW Route objects with route.certificate = nil initially
    - Replaces the old routes in m.routes
    - The old Route objects (with valid certificates) are replaced!

  The critical issue is in proxy/manager.go:42-65:

  func (m *Manager) SetRoutes(newRoutes []*Route, fallback *Route) error {
      // ... validation ...

      for i := range newRoutes {
          route := newRoutes[i]  // These are NEW Route objects!
          // ...
          m.loadCertificate(route)  // Try to load certificate
      }

      m.lock.Lock()
      m.routes = newRoutes  // REPLACES all routes!
      m.lock.Unlock()
  }

  When the second SIGHUP arrives:
  1. New Route objects are created
  2. For domains currently being renewed, GetExistingCertificate might fail or
     return expired certs
  3. These routes get route.certificate = nil and CertificateMissing status
  4. CertificateForClient returns nil for these domains
  5. TLS handshake fails → clients can't connect → "context canceled" errors

  The fix would be to either:
  1. Prevent concurrent CheckCertificates runs
  2. Copy certificates from existing routes before replacing them
  3. Make certificate renewal truly independent of route management

Midway through it had started searching for “context canceled” errors in the codebase, which doesn’t make much sense. That’s what happens when a client gives up waiting, so the corresponding backend request is terminated. You won’t actually find “context canceled” anywhere in Centauri, because it’s a feature of the Go standard library. I gave it a little nudge to try to stop it rabbit holing.

As far as I can tell, what it’s talking about in this response isn’t an issue. I guess it’s possible that there’s a race condition whereby a route would be replaced and end up with an old version of a certificate that was just being renewed, but it wouldn’t cause an issue in practice; it’d just get updated next time the certificates were refreshed.

There’s no way for routes to get a nil certificate in this scenario, as far as I know. But I don’t have the unfounded confidence of an LLM, so I simply said:

> I don't follow why the certificate would be nil?

Attempt the third

I expected Claude to just explain its reasoning, but it apparently didn’t have any reasoning so set off on another voyage of discovery:

● You're right to question that. Let me trace through this more carefully.

● Good point! Since GetExistingCertificate doesn't acquire locks, it should
  return the existing certificate even during renewal. Let me dig deeper:
  
● Now I see a potential race condition! Here's what could happen:

● I found it! There's a critical race condition that explains the blocking:

  The Race Condition

  1. Background renewal starts: CheckCertificates() → updateCert() →
     GetCertificate() acquires lock and starts ACME renewal
  2. SIGHUP arrives: SetRoutes() is called, which for each route calls
     loadCertificate() → GetExistingCertificate() → store.GetCertificate()
  3. The race:
    - store.GetCertificate() returns a pointer to the certificate in the slice
    - Meanwhile, the renewal completes and calls SaveCertificate()
    - SaveCertificate() calls removeCertificate() which modifies the slice while
      it's being read
    - This slice modification (append(j.certificates[:i], j.certificates[i+1:]...))
      can cause:
        - The pointer returned by GetCertificate to become invalid
        - Slice bounds issues
        - Memory corruption
  4. The result:
    - GetExistingCertificate might return corrupted data or nil
    - Routes end up with certificate = nil
    - TLS handshakes fail
    - Clients timeout → "context canceled"

  The Timeline Confirms This

  - 22:40:53 - First SIGHUP, starts renewal of 14 certificates
  - 22:41:23 - Second SIGHUP arrives (30 seconds later) while renewals are still running
  - 22:45:03 - First timeout errors (4 minutes after first SIGHUP)

  The 4-minute delay matches ACME renewal time. The second SIGHUP's SetRoutes
  likely read certificates while they were being modified by the first SIGHUP's
  renewal goroutines.

  This is a classic concurrent slice modification bug - no synchronization
  between readers (GetCertificate) and writers (SaveCertificate/removeCertificate).

Once again we have a very confident and technical answer that does not explain the symptoms we’re looking for. It’s also mostly rubbish again. GetCertificate and SaveCertificate are only called by code that is holding a lock for that certificate; they won’t interfere with one another. There may, again, be a small concurrency issue if different certificates are handled in parallel, but it’s definitely not the cause of our current problems.⁵

Working through these explanations and debunking them was annoying me at this point. I decided to give it once last chance, and used the “ultrathink” shibboleth:

> So there may be a concurrent modification issue, but that doesn't explain the
  symptoms. If a TLS handshake failed it wouldn't result in the clients timing
  out. Likewise if there was any kind of corruption there'd be a panic, not a
  period where it didn't work and then started working again. Engage ultrathink.

Fourth time lucky

I’m going to include a piece of the thinking here because it amused me so much:

✻ Thinking…

  Oh! I need to think about this differently. Let me trace through what happens
  when a request comes in:

Apparently it takes “ultrathink” to actually work through the problem logically? Amusingly it did actually get there:

● You're absolutely right. Let me think more carefully about this...

● EXACTLY! That's the issue. Looking at the timeline:

  1. 22:40:53 - First SIGHUP: CheckCertificates() acquires READ lock, starts
     renewing 14 certificates
  2. 22:41:23 - Second SIGHUP: SetRoutes() tries to acquire write lock, blocks
     waiting
  3. Incoming requests call routeFor() for route lookup, but Go's RWMutex blocks
     new readers when a writer is waiting (to prevent writer starvation)
  4. 22:45:03 - Clients timeout because requests can't even get past route lookup

  The key insight is Go's RWMutex writer starvation prevention: when a writer
  is waiting, new readers are blocked even though the current lock holder is
  also a reader.

  So the cascade is:
  - CheckCertificates (read lock) → certificate renewal takes minutes
  - SetRoutes (write lock) → waits for read lock to release
  - routeFor (read lock for every request) → blocked by waiting writer

  Result: All incoming HTTP requests hang at route lookup, never reach TLS
  handshake or proxying logic. Clients timeout waiting for any response
  → "context canceled".

  This perfectly explains:
  - No panics (no corruption)
  - No immediate TLS errors (never gets that far)
  - Requests hang then timeout (blocked on lock)
  - Eventually recovers (when certificate renewals finish)

  The fix would be restructuring to avoid holding the manager lock during slow
  certificate operations.

We’re back to the RWMutex again, but this time it’s dropping knowledge on me not the other way around: if there’s a writer waiting, then new readers are blocked. This does indeed perfectly explain the symptoms, and was something I’d overlooked. The write lock is only held briefly to update some struct members to prevent it from blocking the readers, but this behaviour coupled with the long-running read lock ends up doing that anyway.

The fix

After all this work it’s very tempting just to ask Claude to fix it, but I had almost no confidence it’d be able to maintain any of the concurrency requirements given how much it flailed around with bad explanations. But in the interests of science, I asked it to try. Here’s what it came up with:

   // CheckCertificates checks and updates the certificates required for registered routes.
   // It should be called periodically to renew certificates and obtain new OCSP staples.
   func (m *Manager) CheckCertificates() {
+    // Create a snapshot of routes to avoid holding the lock during slow certificate operations
     m.lock.RLock()
+    routes := make([]*Route, len(m.routes))
+    copy(routes, m.routes)
+    provider := m.provider
+    m.lock.RUnlock()

+    // Process certificate renewals without holding the manager lock
-    for i := range m.routes {
+    for i := range routes {
-      route := m.routes[i]
+      route := routes[i]
 
       if provider == nil {
         route.certificateStatus = CertificateNotRequired
       } else {
         m.updateCert(route)
       }
     }
-    m.lock.RUnlock()
   }

   // updateCert updates the certificate for the given route.

That’s about what I expect. It definitely fixes the exact issue described, but it’s a bit… sloppy. Routes are now being updated when the lock isn’t being held, which is a change in semantics. It also doesn’t address the underlying issue that requests can be blocked if there’s a pending writer, and pending writes can be blocked by readers. A better approach would be to allow these operations to happen without blocking, maybe by using an atomic pointer to an immutable slice. I guess I’ve got some coding to do!

As for Claude: while the journey was frustrating, I think trying to figure this out by myself would have been even more so. It’s one of those things where everything looks right to me, and I’d have to go down a fairly long chain of checking my assumptions to find where I’d gone wrong.

There’s definitely a lot of room for improvement in coding LLMs: being able to estimate their own confidence level instead of stating everything as absolute truth would go a long way. I imagine there’s also a lot of benefit of having one LLM vet the responses of another; that’s sort-of how thinking mode works, but it doesn’t seem to go far enough to stop it outputting rubbish. It’ll be interesting to see how they develop in the future, anyway.

read as “I got spammed by a bunch of monitoring notifications” ↩︎
read as “Me repeatedly telling it that it was talking rubbish” ↩︎
If only we had some kind of language model technology that could understand this kind of thing. ↩︎
I sense some fun times with Go’s race detector in my near future ↩︎
The most likely outcome from these concurrency issues is Centauri panicking, which isn’t great but would only interrupt ongoing connections before it restarts and starts serving traffic again. Much less problematic than the “block for five minutes” behaviour I’m trying to fix. ↩︎

How I use Tailscale

2025-06-25T00:00:00Z

The Tailscale logo

I’ve been using Tailscale for around four years to connect my disparate devices, servers and apps together. I wanted to talk a bit about how I use it, some cool features you might not know about, and some stumbling blocks I encountered.

I’m not sure Tailscale needs an introduction for the likely audience of this blog, but I’ll give one anyway. Tailscale is basically a WireGuard¹ orchestration service, with lots of nice features sprinkled on top. It’s a subscription product, but it has an insanely generous free tier that covers basically anything you’d ever want to do as an individual. They also open source all their client software, and there’s a third party control server implementation called Headscale if you want to avoid the hosted system entirely.

Basic connectivity

At its core, Tailscale lets you easily connect from one device to another, even if they’re not directly exposed to the Internet. You install the Tailscale client wherever you like (on your phone, computer, servers, Raspberry Pi, etc), authenticate the machine with the control server, and it can then talk to all the other machines on the tailnet using their private Tailscale IP addresses.

That isn’t anything revolutionary: it’s the basic premise behind VPNs. But Tailscale makes it so easy. You don’t have to bother with any networking configuration. You don’t have to distribute keys. You just install the client², and login.

For example, my home automation service runs on a Raspberry Pi that sits behind two different routers. I installed Tailscale on it³, logged in, and immediately I could SSH into it from my computer or my phone even when they’re on different networks.

Speaking of SSH, Tailscale has special support for it whereby it handles any incoming connection to port 22 from the Tailscale network, and deals with authentication itself. No public keys or passwords: if you’re logged into Tailscale you can be logged into the machine. This is particularly handy when you SSH from a phone, as proper credential management is a bit of a nightmare there.

Exposing entire machines is just the start, though. If you run multiple services on the machine you need to remember which port they’re all on. Yuck. Instead, you can expose individual services as their own node on a tailnet. There are lots of options for doing so: there’s an official Docker image, a Go library, and many third-party tools (such as my own Centauri and tsp).

Not just a VPN

Exposing individual services wouldn’t be much use if you still had to remember their IP addresses. For a long time I manually added DNS entries for Tailscale nodes, allowing me to connect to http://grafana/ instead of http://100.73.190.100. Recently I switched to Tailscale’s MagicDNS, though, which takes care of that for you. Every machine on the tailnet gets an automatic DNS entry based on its name.

I initially shied off of this because it changes the DNS resolver on every machine it’s installed on, and it all seemed a bit too “magic” for me⁴. I got fed up with manually adding DNS entries though, so I spent a bit of time learning about it and got over my knee-jerk “what on earth is going on?!” reaction. The DNS controls also let you force a specific upstream DNS server. I use NextDNS everywhere, so having Tailscale automatically configure this for me on all my devices was a bonus⁵.

As well as the short names, you can resolve nodes on the tailnet using a slightly longer form: machine.your-tailnet.ts.net. The “your-tailnet” bit is randomly generated, and you can reroll it in the control panel to find a fun one. But why waste time say lot word, when few word do trick? Because Tailscale can make these addresses globally routable, and issue TLS certs for them too.

Say you’re developing something on your local machine, and want to get feedback from someone. You can use Tailscale’s “funnel” feature to expose it publicly:

tailscale funnel 127.0.0.1:8080

With no other options, this will expose the service over HTTPS on port 443. You can then invite people to visit https://machine.your-tailnet.ts.net and they’ll connect through to port 8080 on your machine. They don’t even need Tailscale installed. I don’t use this terribly often, but it’s a great quality-of-life improvement on the occasions I do want to share something.

Tailscale also has a “serve” command, which does much the same but only exposes it over your tailnet. This is how you’d use the Tailscale Docker image to expose a normal service onto your tailnet, but it’s also useful for similar development scenarios. Occasionally I’ll want to test something on my phone, and rather than make sure it’s connected to the right Wi-Fi network, recompiling the app to not bind to localhost, figuring out my machine’s IP, and so on, and so forth, I just run tailscale serve and access it over Tailscale.

Authentication, two ways

One of my minor frustrations with Tailscale was how cumbersome the authentication process was. I logged in using GitHub, and every time I tried to login to Tailscale, I had to bounce through the GitHub OAuth flow and then select which organisation I wanted to use. Even if I was logged in to GitHub, it still shows the permission page every time. It’s not a dealbreaker, but it’s definitely not smooth. This is what it looks like:

Logging in to Tailscale with a GitHub account

Tailscale have since added the ability to sign in with a custom OIDC provider, so I set up a KeyCloak instance, and my login flow now looks like this:

Logging in to Tailscale with KeyCloak as an OIDC provider

It’s a significantly shorter flow, and because KeyCloak is entirely under my control I can configure how long I stay logged in for. It also reduces my reliance on third parties: if GitHub goes down, or decides to ban my account⁶ or something, I can still access Tailscale.

That’s how you log in to Tailscale, but Tailscale can also assist with logging in to other services. If you connect to a HTTP service over Tailscale, it will add in the following headers:

Tailscale-User-Login: alice@example.com
Tailscale-User-Name: Alice Smith
Tailscale-User-Profile-Pic: https://example.com/photo.jpg

A lot of software has support for accepting login credentials provided by a reverse proxy, which is basically what’s happening here. For example, I run a Grafana instance that’s only exposed over Tailscale. Its config (set in environment variables) is:

GF_USERS_AUTO_ASSIGN_ORG: "true"
GF_USERS_AUTO_ASSIGN_ORG_ROLE: "Admin"
GF_AUTH_PROXY_ENABLED: "true"
GF_AUTH_PROXY_HEADER_NAME: "Tailscale-User-Login"
GF_AUTH_PROXY_HEADER_PROPERTY: "email"
GF_AUTH_PROXY_AUTO_SIGN_UP: "true"

So anyone who presents a Tailscale-User-Login header gets an admin account created with that as their e-mail address. It’s super convenient, but obviously make sure it’s only exposed via Tailscale if you do that!

Other services I run that support this include Miniflux (an RSS reader), Jelu (a book tracker), Linkding (a bookmark manager), and Seafile (a file storage system). If you’re hosting something yourself, look out for “proxy auth” or “remote users” in the documentation or configuration. It’s amazingly convenient to be able to access things without entering credentials.

ACLs, tags and mistakes

For a long time I didn’t bother with any kind of ACLs within my Tailscale network. Every machine could access every other machine. Then one day a thought hit me. The default Tailscale SSH config re-prompts you for user authentication, but I’d disabled that by tweaking the config… That means anyone with access to any of my machines (or who managed to get Tailscale credentials out of one of my apps) would be able to SSH into anywhere else on my network. Even as root! Compromising a small bit of my network would immediately lead to a complete take-over. What’s the point in defence-in-depth if you roll out a red carpet over the drawbridge for any attackers?

The first and most obvious fix for this was to restrict root access over SSH. I changed my Tailscale ACL to prompt for reauthentication for root access, but still allow regular SSH unprompted:

{
  "ssh": [
    {
      "action": "accept",
      "src":    ["autogroup:member"],
      "dst":    ["autogroup:self"],
      "users":  ["autogroup:nonroot"],
    },
    {
      "action": "check",
      "src":    ["autogroup:member"],
      "dst":    ["autogroup:self"],
      "users":  ["root"],
    },
  ],
}

This still didn’t sit right with me, though. Why should a random webserver be able to (theoretically) SSH into my desktop computer unchallenged? Tailscale allows you to do more advanced ACLs by applying tags to devices. I dutifully created a bunch of tags, and went through and tagged every device: user, server, app, container, and so on. This, it turns out, was a mistake. I could no longer SSH to anywhere. If I’d read the docs properly, I might have noticed how they describe tags:

Tailscale tags are how you authenticate and identify non-user devices, such as servers and ephemeral nodes. They serve two primary purposes: to provide an identity to non-user devices and to let you manage access control policies based on purpose.

By applying a nice user tag to all of my devices (computer, phone, tablet and so on), I’d effectively removed my own identity from them. They no longer counted for the autogroup:member source in the SSH config. Whoops. And it’s not easy to fix either:

You cannot remove all tags from a device. A device with a tag-based identity must have at least one tag.

So I’d have to remove all the affected devices, then re-add them. At the time I wanted to roll back all the tags, which would mean reconnecting dozens of nodes to the Tailnet. I did end up doing that, but I also took it as an opportunity to switch to KeyCloak as I mentioned earlier. Setting everything up again was a bit less painful when it wasn’t just correcting a dumb mistake.

I now have a much more sensible tag setup: servers are tagged server, apps are tagged app, and a few special apps are tagged integration. I then use Tailscale’s grants to limit access within the tailnet:

{
  "grants": [
    {
      // Users can access everything
      "src": ["autogroup:member"],
      "dst": ["*"],
      "ip":  ["*"],
    },
    {
      // Servers and integrations can access integrations and apps
      "src": ["tag:server", "tag:integration"],
      "dst": ["tag:integration", "tag:app"],
      "ip":  ["*"],
    },
  ],
}

This stops anyone but a user device from accessing servers, and stops plain apps from accessing anything at all. So for example my Grafana instance is tagged as an app; it doesn’t connect out over Tailscale for anything, it only exposes its interface over Tailscale. My home automation software is tagged as an integration, as it reaches out to other services over Tailscale (e.g., to push metrics it receives from devices).

You could go even further and limit which specific machines talk to which others, but I think this tier system sets reasonable boundaries for my use without getting too much in the way.

Even more

There are lots of other cool features offered by Tailscale that I’ve not covered: exit nodes (allowing you to route normal Internet traffic over Tailscale; very handy when you’re behind a router you don’t trust, or if you want to appear as being in another country for totally legitimate reasons); Tailscale lock (a system where existing nodes have to verify any new ones, guarding against both unauthorised access and Tailscale themselves inserting nodes into your network); file sharing, and more I’ve probably forgotten.

As I mentioned before, an awful lot is available on the free personal plan. It’s definitely worth checking out if you manage servers, or self-host anything. I can’t imagine going back to managing servers and private apps without Tailscale, and I’m excited to see what else they do going forward.

If you’re not familiar with it, WireGuard is a modern VPN included in the Linux kernel. It’s basically the only VPN that isn’t terrible. ↩︎
They have installation instructions and packaged versions of the client for just about every device you can imagine: Windows, Mac, every major Linux distribution, iOS, Android, etc, etc ↩︎
They even have an apt repository specifically for raspbian builds, so it’s super easy. ↩︎
Lots of people seem to treat “magic” as a good thing in software. I think it’s the absolutely opposite. I want to know how something works, not for it to be inscrutable! ↩︎
This is entirely optional, of course, you can just let each device do their own thing as well. ↩︎
I don’t think this is likely, but it’s one of my biggest concerns about using “social” logins. More-or-less all of these services can arbitrarily ban you with no recourse; why would you trust them as keyholders to other things? ↩︎