Chameth.com - posts like break-everything-fuzz-testing, filament-weight-display, just-a-nod, why-you-should-be-using-https

Monthly Meanderings: February 2026

2026-03-01T00:00:00Z

It doesn’t feel like a whole month has gone past since I wrote the last instalment of Monthly Meanderings, even allowing for how short a month February is. For more context on this series, you can check out the introduction to the first edition.

Website updates

I only wrote one new blog post this month: Just a nod, which is about the “nod” button I added to the bottom of most pages. I did a lot of other little bits and pieces, though:

My blog posts are now automatically posted to a dedicated Bluesky account, and posts now include syndication links at the bottom.
The site now tracks how long each request takes to generate along with how many SQL queries were used. You can see the details for the current request by hovering/tapping over the pi symbol at the bottom right of a page. To understand why it’s a pi symbol, you should watch The Net!
Speaking of films, I redid the star rating rendering so instead of unicode stars it uses some hand-drawn pixel art stickers. They’re somewhat randomised, so they don’t look repetitive. This is what it now looks like: .
I spent some time tweaking various styles and layouts so everything works nicely on smaller screens. I try to make sure these things work when I make them, but it’s easy to miss a particular interaction, or make an innocent-seeming change that breaks tiny screens.
I added a new page that shows stats about my walks, based on my Apple Health data. I had a private view of this for a while, but decided there’s no reason it couldn’t be more public.
The films page has gained a search box, to directly find my review of a particular film. I used CSS anchor positioning for the results panel. Modern CSS is fun!
I fixed the 3D prints page being utterly broken in numerous ways. It showed up on my shiny new “how many SQL queries does each page do?” dashboard, as it was doing multiple queries for each individual print instead of just getting all the data in one go. Then I noticed a badly written redirect rule had broken all the images and download links. Whoops!
I did a lot of under-the-hood cleaning, most of which won’t be visible. Besides just making me happy, this means the contact form and nod buttons now work with Javascript disabled.
Finally, I added a page that shows my board game plays. The data comes from exports from bgstats. I have a bunch of ideas for other things to do with that data, but haven’t got around to it yet.

Phew. After writing all that out it feels like a really long month, not a short one!

Other projects

Not much going on with my other projects. The only big thing was goplum, my self-hosted monitoring/alerting tool, getting a 1.0 release. It adds a reminder feature so you can get alerted repeatedly if a service is down. It also reworks how plugins work so they’re compile-time instead of trying to be clever and dynamic, as that never really gave any benefits.

I’m still continuing my migration of projects to a private Forgejo instance. All of my personal repos are now sourced there, and mirrored out to GitHub, except one: my Dockerfiles. I’m not sure how I want to approach that yet, as the action that does the updating is pretty involved. I also have some repositories split off into their own GitHub organisations which I haven’t got around to mirroring yet.

Entertainment

Unlike the past few months, I didn’t touch many video games in February. The ones I’d been dabbling with didn’t really hook me enough to make me keep going back. I made up for it with lots of other media forms though. First up, a whole bunch of films:

I won’t write too much about them here — you can click through to read my reviews if you want — other than to call out the three excellent horror films I hadn’t seen before: Black Swan, The Cabin in the Woods, and Get Out. Not so long ago I would’ve said that I didn’t enjoy horror films, now they’re some my favourites.

I don’t have any TV show tracking on my website (yet?), as I haven’t really watched much recently. This month I started watching The Pitt though, and it rapidly become one of my favourite shows. By “started watching” I mean “watched every aired episode of both season 1 and season 2”. I highly recommend it. The only problem is that it’s hard to find anything to watch afterwards, as nothing else can quite measure up.

I’ve also been binging a book series: Beneath the Dragoneye Moons. It’s a LitRPG series: basically a fantasy setting that borrows systems from an RPG, like classes and levelling. Something about the genre really works for me, and Beneath the Dragoneye Moons is one of the more interesting series I’ve come across in the genre. I’ve been going through the series at the rate of about a book a day.

Finally, board games! Nothing particularly notable, but now I can embed a pretty display of what I actually played:

6 plays
4 plays
4 plays
4 plays
2 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play

All of these were online via Board Game Arena. I haven’t really had time for in-person games with all the TV, films and reading!

Around the web

Start all of your commands with a comma

An old post that was doing the rounds again on Hacker News. It’s an interesting idea. I recently aliased cc to launch Claude Code with some custom parameters, and it horribly backfired when I tried to compile a C program and remembered that cc was part of GCC. I haven’t transitioned all my personal commands to use a prefix yet, but I might.

Tactical tornado is the new default

An interesting post about software developers who act as “tactical tornadoes”, and how that’s basically the role that all LLMs fill. Whenever I use coding agents, I often find myself following behind them in my IDE cleaning up bits and pieces. It’s often faster than prompting them to do it. I definitely get the “cleaning up after a tornado” metaphor.

Amusingly, the agents will sometimes notice that you’re tidying up after them and ascribe it to an automated process, outputting something like “the automated linter hasn’t run yet” if you fall behind. There’s probably a deeper philosophical discussion about what it means when the agent thinks that the human is the automated part of the system…

My smart sleep mask broadcasts users’ brainwaves to an open MQTT broker

Another one for the “the ’s’ in ‘IoT’ stands for security” books. I find it hard to actually comprehend how these things get to market. Does nobody actually think about it, or do they know full well how bad it is and just launch it anyway?

The IndieWeb and Small web

Ask three people to define ‘IndieWeb’ and ‘Small web’ and you’ll get about 20 different answers. This post by Christiano almost exactly nails my own views on the terms, and explains them well. I think most personal sites (my own included) straddle both terms to some extent, but having the labels is useful in discussions.

Just a nod

2026-02-11T00:00:00Z

Last month I added a new feature to the bottom of almost all pages on this site: a “nod” button. It’s a bit like the upvote buttons you see on bear blog posts, or the ubiquitous like button you get on social media, with one key difference: it doesn’t track or count anything. Before I get into the whys and hows, this is what it currently looks like:

The nod button in all its glory. If you want a clickable version you’ll need to scroll a bit more.

So why bother? I like hearing from people who have looked at my website. I have a contact form that’s also embedded at the bottom of all my blog posts; it’s always a delight when people reach out. Actually writing a message is a pretty high bar that most visitors won’t clear, though. A like/upvote/whatever-you-want-to-call-it is a much less demanding interaction.

And why no counting? I’d been thinking about some kind of upvote button for a while, and counting was always the thing that put me off. Having a number attached to a post like that feels like I’d end up invoking Goodhart’s law and start aiming for a certain number of upvotes or likes or whatever. I don’t want to be anxiously watching numbers, or feel pressured to make a post “good enough” (I already self-filter too much…).

As for not tracking people, that feels like table stakes to me: I’m an individual concerned about privacy, not a giant mega-corporation who wants to slurp up all the data to serve ads or engagement bait feeds. I don’t really need to know someone’s identity to receive a message from them. In the olden days of the internet, you just knew everyone by screen names which could change on a whim and didn’t reveal anything about the real you. On the internet, nobody knows you’re a dog, as Peter Steiner said.

All these constraints felt insurmountable for a while. I considered just making a button that sent a contact form submission with a pre-filled message, but those end up in my e-mail inbox. I don’t really want to have to process them, especially if there’s a big traffic spike like when Hacker News notices one of my posts. I could store the button presses in a database, but then how would I actually know they happened, especially if I didn’t want to watch a number go up?

An Amazon Dash button

So I did what I usually do with these types of problems: ignored it for a bit. The idea rattles around at the back of my head, and sometimes I’ll come up with an answer. I’ve seen this called diffuse thinking. Eventually the idea bounced to the forefront of my head while I was looking at my IRC client, and the solution suddenly clicked. If you’ve not heard of it, IRC is a chat protocol that’s almost as old as I am. Daniel Stenberg has compiled a nice history of IRC that I won’t repeat here. I use IRC to stay in touch with friends, hang around the channels for some open source projects, and I also run a private server with just me and some notification bots.

The notifications I have on IRC are all “transient”: they’re handy to see in the moment, but it doesn’t matter if I miss them. I have some high-volume news feeds, log messages from some self-hosted services, details about git commits, and so on. Basically just background noise of things that are happening that don’t need action. Perfect for the kind of interaction I want!

Now that I’d finally figured out that the transient-notification-shape goes in the transient-notification-hole, I just needed to actually implement it. As part of that private IRC setup I already have a bot that accepts messages over HTTP and sends them to a channel, so the backend was pretty simple. For the frontend I was imagining something like the ill-fated Amazon Dash buttons. My first attempt was… underwhelming:

Flat UIs are in right now, right?… Right?

I guess I shouldn’t give up the day job to become a graphic designer. Fortunately, I came across this codepen from Thierry Michel with some very fun skeuomorphic arcade buttons. I spent some time and LLM tokens on converting the SCSS to CSS, changing the colour, and making a few other tweaks. I really love how this looks and feels.

This is what it looks like in my IRC client when someone nods (and when containers get pushed to the GitHub Container Registry for my projects…):

Receiving some nods on IRC via a bot

When I added the button I wasn’t sure if I’d keep it, but I’ve very much enjoyed seeing the “nods” come in every now and then. I’ve been surprised by how many come in for the various snippets and other non-blog pages on the site. It’s a nice reminder that people are out there browsing the site, without all the baggage of monitoring analytics or tracking likes.

Monthly Meanderings: January 2026

2026-02-01T00:00:00Z

Welcome to the second edition of my monthly meanderings. For a bit of context, you can check out the introduction to the first edition.

Website updates

It’s been a pretty busy month for chameth.com. Three blog posts: The Meaning of Life — an entry into the IndieWeb carnival where I mostly review a book on Stoicism — Surge Protectors: Marketing vs Reality which is a dump of a research rabbit hole I fell into, and An Interesting Tailscale + Docker Gotcha which documents a fun problem I encountered.

I’ve also made a new films section, which is basically my Letterboxd data but free range instead of cooped up in a silo. I imported my 200-odd reviews, and for the last few I wrote the review on my site and then copied it over to Letterboxd. That’s what the IndieWeb folks call POSSE: publish on your own site, syndicate elsewhere. I like the idea that my website is the canonical source of truth for all the public things I generate. I want to expand this to some other areas in the future: music, video games, board games, etc.

I made most of the film components into “shortcodes”, so I can drop them into arbitrary pages or posts, like so:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Finally, I added a new widget to the bottom of most pages: a “nod” button. It’s effectively a “like” or “kudos” button but deliberately does not track how many times it’s been pressed, has no social network connotations, and won’t flood my inbox or hit rate limits if it gets spammed. If you click the button, it simply sends a message to a private IRC channel that says “Someone nodded at $URL”. I’ll probably write a bit more about it in a blog post in the future.

Other projects

I only have one small non-chameth.com update: a new project called pigeonhole which provides a really basic web service that lets you leave messages in a named cubby, and retrieve them at a later date. I’m using it to automate adding things to my Obsidian notes from my Apple Watch. There’s lots of reasons why that isn’t as simple as it sounds. I went through a bunch of iterations to try and make it work before just writing code to fix it.

I wanted to do this because I’m trying to replace Todoist with just having tasks in Obsidian. I already used Obsidian for longer form notes, but I’ve now also started using it for drafting blog posts (I’m typing this in Obsidian right now!), and trying to keep more useful day-to-day notes. I’ll probably write a blog post with more details once I’ve figured everything out.

Finally, I’ve been moving a bunch of projects to a private Forgejo instance, with the ultimate goal of having all of my projects live there. I’m going to mirror the contents out to GitHub and maybe other places, but Forgejo will be the single source of truth, and where all the workflows/dependency wrangling/etc happens. I’ve got Renovate configured to do dependency bumps, and I’m very much liking the whole setup so far. I’m saying it a lot but again: I’ll blog about it in more detail when it’s a bit further along.

Entertainment

I mentioned last month that I was playing Hogwarts: Legacy. I finished that off, and have been struggling to find something to scratch the same itch. I’ve tried a whole bunch of open world games, and basically none of them work for me. They have cumbersome control schemes (yay console ports), really awkward or sluggish movement, and so on. I think I’m just too fussy, but if I’m going to sink dozens of hours into a game it shouldn’t feel annoying or like a chore!

I did play through Thank Goodness You’re Here, which is delightfully funny but sadly only took 3.6 hours to finish with 100% of the achievements. Still highly recommend picking it up, especially when it’s on sale for under ‘10 bob’. I’ve also started playing through Control, and I’m enjoying the ambience. There’s an insanely sharp difficulty spike for the first boss, though, which nearly made me put it down.

In other entertainment, I’ve been playing a load of turn-based games on Board Game Arena , and I’ve watched a bunch of films after barely watching any last month. Here’s another of those shortcodes to show them off:

Around the web

Remotely unlocking an encrypted hard disk

A fun post documenting using Tailscale in a very weird way to solve a relatable problem. Very much my cup of tea.

I charged $18,000 for a static HTML page

This one’s actually from 2019, but it’s the first time I’ve come across it, and it’s a fun story. As a contractor, deciding what to bill for and what not to can be a bit messy. Downtime between meetings? The ten minutes in the shower you spent thinking through a bug? It’s a lot more clear cut if the client wants you to sit in an office, twiddling your thumbs, though.

Oxide RFD: Using LLMs at Oxide

A wonderfully well thought out piece about using LLMs in a respectful, sensible manner. I’d been thinking about writing something similar, but it wouldn’t have been anywhere near as well crafted as this. While I basically agree with it all, the part about LLM-generated text really resonates:

LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion.

Your app subscription is now my weekend project

Roberto points out that monthly SaaS subscriptions start looking even less appealing when you can just prompt an alternative into existence. I talked a bit about using LLMs for one-person software before, and it becomes even more appealing when the alternative is paying monthly for a service that keeps shoe-horning in features you don’t want, making you agree to share data with 329 “trusted partners”, and/or repeatedly ratcheting up the price.

Ekul’s Internet Space

I love the aesthetics of this personal site, and the data tracking that goes along with it. I’m also slightly aghast at the amount of Monster consumed! Makes me want to track more things and expose them on my own site.

Monthly Meanderings: December 2025

2026-01-01T00:00:00Z

For a while I’ve been idly thinking about a way to get smaller bits of content onto my website without it being too annoying for me, or too hard to consume. Things like interesting links, small project updates, and so on.

I didn’t immediately come up with anything I was happy with, so just sat on it. Ignoring the problem seems to work surprisingly well for things like this. I’d been meaning to redesign the about page for a while, but wasn’t sure what to do with all the interests I wanted to list. After ignoring the problem for a while I came across the interests directory and realised I could just bundle all those things into their own page.

This seems amazingly obvious in hindsight, but I think I needed a bit of a push to accept that a page just about my interests was… well… interesting. Anyway, as the end of December rolled around, I got a similar dose of inspiration: a flurry of monthly round-up posts from various blogs I follow. If I bundle up all the small bits and bobs, it’ll make something vaguely blog post shaped, and writing and consuming those is already a solved problem. Again, pretty obvious in hindsight, and again I think I just needed a nudge to shut up the little voice in my head saying “but those people all have interesting things to say… do you, really?”

So without further ado, welcome to the inaugural post in my new monthly meanderings series. If you have any feedback on the format or contents, please drop me a note with the contact form at the bottom!

Website updates

As already mentioned, I reworked the about page and split out a new interests page. The old about page had a few lists of interests show up in thought bubbles when you hovered over things, which was a cool effect on desktop, but barely worked at all on mobile. The new page gives me a lot more room for waffle, and I do like waffling.

Only a single new blog post this month: exposing game servers over Tailscale. Just a quick how-to I wrote up, having not found it documented usefully anywhere before. I have a bunch of ideas for posts I want to write, but none of them have really sparked joy quite yet, so I’m once again employing the “sit on them for a bit and maybe inspiration will strike” approach.

Other projects

I created a small collection of Docker related projects this month: containuum is a go library that helps with listing containers, and keeping that list up-to-date. On top of that I built centauri-docker-confd which generates a config for centauri (my reverse proxy) based on running Docker containers. I already had a way of doing that, but containuum is designed to be a bit more robust, and I made it so the config can be passed over a network connection instead of having to write it to disk and then signal the proxy to reload.

The next project has a bit of backstory: after reading I got hacked, my server started mining Monero this morning [warning: the article is interesting but a bit LLM-sloppy] I realised that I, too, was running a vulnerable version of Umami. Luckily, as far as I can tell, it wasn’t exploited. But it got me looking into automated vulnerability scanning. I don’t put much stock into it normally, as you get so many false alerts, but this is one case where it would have definitely helped. There didn’t seem to be any good solution for my “middle-of-the-road” approach to containers (lots of containers, but not needing Kubernetes), so I wrote purser. It takes every image used by a running container, runs them through Trivy, and generates a combined report. I’ll add some form of active notifications to it in the future.

Entertainment

This month I’ve been playing a lot of Hogwarts: Legacy, after picking it up for free on Epic Games. I grew up with the Harry Potter books coming out, and while I’m not a super-fan, it still has a special appeal to me. Obviously she-who-must-not-be-named is doing her best to out-evil Voldemort, but I think you can still enjoy the universe despite that. Especially when you don’t have to contribute any money that might go towards her.

Anyway, the game is surprisingly fun. They did a really good job of making it feel like an open world. You want to walk around from place to place to see everything, find all the various collectables, and so on. Compared to Far Cry or Cyberpunk the world is a lot smaller but that means it’s far more densely packed with stuff to do. I initially didn’t vibe with the way combat works: it’s pretty heavily reliant on dodging or shielding against incoming attacks, which reminded me a lot of Clair Obscur: Expedition 33. Fortunately, it’s a lot more forgiving in Hogwarts: Legacy and I got used to it after not too long.

It’s been a relatively sedate month for other forms of entertainment. I saw Fackham Hall and Wicked: For Good at the cinema. The former was a funny romp, vaguely reminiscent of Airplane or The Naked Gun, as you’d expect from something penned by Jimmy Carr. Wicked was… fine? I wasn’t overly enamoured with part one, so it’s not a surprise I didn’t vibe with the follow-up. If you smushed them into a single normal-length feature I think they would have made a really good film, but two films both over two hours long is a bit much for what actually happens.

Around the web

Wikipedia: List of citogenesis incidents

A list of incidents where (often false) information on Wikipedia has ended up becoming its own source. That led to Mark Dominus’ blog post titled Imaginary Albanian eggplant festivals… IN SPACE where he discovers that a mountain on Ceres got named after a made-up festival (and then got it fixed!)

Not relying on Wikipedia as a source was drummed into us at school, but it feels like this sort of thing is only going to get worse and worse now LLMs are on the scene. I assume we’re already at the point where LLM hallucinations are prevalent enough that they’ll find their way into the training data for future iterations.

Simon Collison’s website

I really like the design here. Both the homepage and how well every journal entry is formatted. It makes me feel bad for the many walls of text that adorn this site. The archived v4 design is also wonderful.

My Dinner with Skinner

A 43-minute-long feature film that is “just” the Steamed Hams gag from The Simpsons done in the style of My Dinner with Andre. There’s a (much shorter) behind-the-scenes video that goes over the extreme amount of effort that was put into it, too.

I’m still trying to wrap my head around this. It’s… something.

Delete Spotify? Sure, But Don’t Just Replace it With Another Subscription

A similar take to my own escaping Spotify the hard way, but it avoids getting lost in the technical weeds like I did. (In my defence, they’re very fun weeds to get lost in.) Stephanie is also one of the people that does monthly wrap-ups, and I was pleasantly surprised when the latest one popped up in my RSS reader and I saw my own site linked.

Finding an awkward bug with Claude Code

2025-06-30T00:00:00Z

The many times Claude thought it was right…

I recently encountered a bug in one of my projects that I couldn’t immediately figure out. It was an issue in Centauri, my reverse proxy. After its config was updated, I noticed¹ it stopped serving responses. Looking at the logs, I could see it was obtaining new certificates from Let’s Encrypt for a couple of domains, but I’d designed it so that wouldn’t block requests (or so I thought). After a few minutes of confusion, everything went back to working.

It felt like an issue with locking, but a quick glance didn’t show me anything obvious. I decided to throw the problem at Claude Code and see how it did. With some guidance² it managed to identify the cause, but it was a bit of a journey to get there. I’m going to share the prompts and responses, as it was an interesting process.

The initial prompt

I thought the issue might be quite obvious, and I’d just overlooked it. Centauri isn’t that complex a project, after all. So I gave Claude a fairly minimal prompt to set it off:

> Hello! Please examine this codebase. I have noticed an issue that when a
  certificate is being renewed, the application seems to not serve any other
  requests. I suspect there is a misplaced lock somewhere.
 
  Can you see what's happening? Think very hard.

Claude Code has special hardcoded triggers to engage “thinking” mode, which makes it take more time to think about alternatives. The keywords are: “think”, “think hard”, “think harder”, and “ultrathink”. I managed to not quite get the right incantation here, and only enabled the lowest level³.

I’ll omit all the responses where Claude is “thinking” (talking to itself), or decides to make todo lists then cross items off them then add different items, and so on. Its responses get fairly lengthy; I won’t judge you too harshly if you skim over them. Its first response was blessedly brief:

● I found the issue! Looking at the certificate manager code, I can see the
  problem:

… and then it went back to thinking and reading files, because it had neither found the issue nor could it see the problem. Ho-hum. Some time later it came back with:

● I found the issue! The problem is in certificate/manager.go:53-54:
  
  m.store.LockCertificate(subject, altNames)
  defer m.store.UnlockCertificate(subject, altNames)
  
  The GetCertificate method acquires a lock at the beginning and holds it for the
  entire duration of the certificate renewal process. This includes:
  
  1. Contacting the ACME server
  2. Solving DNS-01 challenges (can take minutes for DNS propagation)
  3. Obtaining the certificate
  4. Getting OCSP staple
  5. Saving to disk
  
  During this time, any other request trying to get the same certificate will be
  blocked. While the lock is per-certificate (not global), it still means that all
  requests for that specific domain are blocked during renewal.
  
  Let me check if there are other places where this might cause blocking:

And it then proceeded to do even more thinking, checking files, etc, until it concluded with:

● I found the root cause! The issue is actually worse than just a
  per-certificate lock. Here's what's happening:
  
  The main blocking issue is in proxy/manager.go:133-144:
   
  func (m *Manager) CheckCertificates() {
      m.lock.RLock()  // ← Acquires read lock on the entire manager
      for i := range m.routes {
          route := m.routes[i]
          // ...
          m.updateCert(route)  // ← This can take MINUTES for ACME renewal!
      }
      m.lock.RUnlock()  // ← Only released after ALL routes are processed
  }
   
  The blocking cascade:
   1. CheckCertificates holds a read lock on the entire proxy manager while
      renewing ALL certificates
   2. Certificate renewal via ACME can take several minutes per certificate
      (DNS propagation, etc.)
   3. Every incoming request calls routeFor (proxy/manager.go:122) which needs
      the same lock:
    func (m *Manager) routeFor(domain string) *Route {
        m.lock.RLock()  // ← BLOCKED until CheckCertificates completes!
        match := m.domains[strings.ToLower(domain)]
        m.lock.RUnlock()
        // ...
    }
  
  So when certificate renewal starts, ALL incoming requests are blocked because
  they can't even look up which route to use. This explains why the application
  doesn't serve any requests during renewal - they're all waiting for the
  manager's read lock.
  
  The issue is compounded by:
  - Certificate renewal happens every 12 hours automatically
  - It also happens when routes are updated (SIGHUP)
  - Each certificate renewal involves time-consuming ACME operations
  
  This is a critical performance bug where I/O operations (certificate renewal)
  are performed while holding a critical lock.

This doesn’t actually follow on from the earlier response, it’s completely flipped onto a different lock being the issue. Amusingly, it’s nearly right here, but it’s highlighting this code for completely the wrong reason.

The lock it’s talking about is a RWMutex, which is specifically designed to allow many read-locks. Everything it’s pointing out are read locks, so they cannot block one another as it described. The response Claude has given sounds very confident, seems appropriately technical, but is in fact complete rubbish. This is one of the big issue with all LLMs at the minute, and it makes me despair at the idea of people “vibe coding” things when they don’t know enough to push back against nonsense like this.

Course correction

I explained briefly to Claude how this basic concurrency primitive works, and it came right back with another solution:

> But that's a RW Mutex. You can hold many read locks concurrently.

● You're absolutely right! I apologize for the confusion. With a RWMutex,
  multiple readers can hold the lock concurrently. Let me trace through this
  more carefully.
  
● Task(Analyze updateCert method)
  ⎿ Done (8 tool uses · 20.0k tokens · 1m 11.9s)
  
● Excellent analysis! You're right about RWMutex allowing concurrent readers.
  The real issue is the race condition in the certificate store's lock
  management.

  Looking at certificate/store.go:82-91:

  func (j *JsonStore) lockFor(subjectName string, altNames []string) *sync.Mutex {
      key := strings.Join(append([]string{subjectName}, altNames...), ";")

      if mu, ok := j.locks[key]; ok {
          return mu
      } else {
          mu = &sync.Mutex{}
          j.locks[key] = mu  // ← Concurrent map write!
          return mu
      }
  }

  This is a classic concurrent map access bug. When multiple requests hit the
  same domain during certificate operations:

  1. Multiple goroutines call lockFor() simultaneously
  2. They read/write to j.locks map without synchronization
  3. Go runtime can:
    - Panic with "concurrent map writes"
    - Corrupt the map structure
    - Block indefinitely if map internals get corrupted

  This explains the blocking behavior - it's not the RWMutex, but rather a race
  condition causing either panics or deadlocks when the map gets corrupted
  during concurrent certificate operations.

  The fix would be to either:
  1. Add a mutex to protect j.locks map access
  2. Use sync.Map instead of a regular map
  3. Pre-initialize all locks at startup

I do appreciate when me stating a basic fact is called out as “excellent analysis”. Definitely doesn’t make me feel like I’m having smoke blown up my ass.

I think the issue it’s describing here is actually a problem⁴, albeit one that’s unlikely to be hit. But it’s definitely not the one we’re looking for. Claude even describes the possible consequences of the bug, and they don’t match what happened at all.

Maybe logs will help?

While Claude was chugging away, I’d been looking into the issue as well. I had logs from when it happened, and figured Claude may be able to glean something from them too. The first thing I’d noticed is that I wasn’t logging when a process finished, so I’d already added that to the code base, hence the note so Claude doesn’t get itself confused:

> That may be an issue, but it's not the one I was looking for. I've found logs
  from the time it happened. Note that a few logs have been added to the
  codebase since (so we can't tell from the logs when ACME or OCSP processes
  *finished*, just when they started). Here are the logs:
  
  {"time":"22:40:53.187887976Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:40:53.19947532Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:40:53.199507942Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:09.048465968Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:09.048482309Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:17.419299795Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:17.419325744Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:23.238027386Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:41:29.994330646Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:29.994353329Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:43.658794811Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:43.658819548Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:57.635325513Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:57.635347515Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:10.52439779Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:10.524417107Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:23.159430249Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:23.159452171Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:36.347407326Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:36.347433295Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:52.544628775Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:52.544653993Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:25.409579107Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:25.409602842Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:43.819479448Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:43.819502041Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:54.358957563Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:54.358979355Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:44:06.180278128Z","level":"INFO","msg":"Renewing certificate", <big list of alt names>}
  {"time":"22:44:06.180302585Z","level":"INFO","msg":"Starting ACME process to obtain certificate", <big list of alt names>}
  {"time":"22:45:03.830331922Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.830528134Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.832388931Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.225854865Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488432434Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488439397Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:30.617614536Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}

I’ve cut down the logs a bit to remove the specific domains, simplify the timestamps, and remove a bunch of other cruft. The key points are there were two SIGHUPs received, and the last certificate before everything started breaking covered 6 different hostnames.

Claude then went through a phase of periodically shouting “Eureka!” then deciding it hadn’t actually found anything yet:

● Now I see it! The key insight is in the timing and the "context canceled"
  errors. Let me trace through what's happening:
  
● Ah, now I see what's happening! Looking at the logs more carefully:

  1. 22:40:53 - First SIGHUP received, routes updated
  2. 22:40:53 to 22:44:06 - Multiple certificates start renewing (14 certificates!)
  3. 22:41:23 - Another SIGHUP received while renewals are still in progress
  4. 22:45:03 onwards - "context canceled" errors start appearing

  The issue is that SetRoutes is being called AGAIN (at 22:41:23) while the
  previous CheckCertificates() goroutine is still running and renewing
  certificates. Let me check what happens in this scenario:
  
> (The context canceled errors are clients timing out because centauri wasn't
  proxying properly)
  
● Ah, now I understand! The "context canceled" errors are from clients timing
  out because Centauri wasn't proxying their requests. This confirms the
  blocking issue.

  Let me check how incoming requests actually get their certificates:
  
● Now I found the real issue! Looking at the timeline and the code:

  1. 22:40:53 - SIGHUP triggers SetRoutes() which:
    - Calls loadCertificate() for each route
    - Sets route.certificate to the existing certificate
    - Starts go m.CheckCertificates() in background
  2. 22:41:23 - Another SIGHUP while certificates are still renewing
    - SetRoutes() is called AGAIN
    - Creates NEW Route objects with route.certificate = nil initially
    - Replaces the old routes in m.routes
    - The old Route objects (with valid certificates) are replaced!

  The critical issue is in proxy/manager.go:42-65:

  func (m *Manager) SetRoutes(newRoutes []*Route, fallback *Route) error {
      // ... validation ...

      for i := range newRoutes {
          route := newRoutes[i]  // These are NEW Route objects!
          // ...
          m.loadCertificate(route)  // Try to load certificate
      }

      m.lock.Lock()
      m.routes = newRoutes  // REPLACES all routes!
      m.lock.Unlock()
  }

  When the second SIGHUP arrives:
  1. New Route objects are created
  2. For domains currently being renewed, GetExistingCertificate might fail or
     return expired certs
  3. These routes get route.certificate = nil and CertificateMissing status
  4. CertificateForClient returns nil for these domains
  5. TLS handshake fails → clients can't connect → "context canceled" errors

  The fix would be to either:
  1. Prevent concurrent CheckCertificates runs
  2. Copy certificates from existing routes before replacing them
  3. Make certificate renewal truly independent of route management

Midway through it had started searching for “context canceled” errors in the codebase, which doesn’t make much sense. That’s what happens when a client gives up waiting, so the corresponding backend request is terminated. You won’t actually find “context canceled” anywhere in Centauri, because it’s a feature of the Go standard library. I gave it a little nudge to try to stop it rabbit holing.

As far as I can tell, what it’s talking about in this response isn’t an issue. I guess it’s possible that there’s a race condition whereby a route would be replaced and end up with an old version of a certificate that was just being renewed, but it wouldn’t cause an issue in practice; it’d just get updated next time the certificates were refreshed.

There’s no way for routes to get a nil certificate in this scenario, as far as I know. But I don’t have the unfounded confidence of an LLM, so I simply said:

> I don't follow why the certificate would be nil?

Attempt the third

I expected Claude to just explain its reasoning, but it apparently didn’t have any reasoning so set off on another voyage of discovery:

● You're right to question that. Let me trace through this more carefully.

● Good point! Since GetExistingCertificate doesn't acquire locks, it should
  return the existing certificate even during renewal. Let me dig deeper:
  
● Now I see a potential race condition! Here's what could happen:

● I found it! There's a critical race condition that explains the blocking:

  The Race Condition

  1. Background renewal starts: CheckCertificates() → updateCert() →
     GetCertificate() acquires lock and starts ACME renewal
  2. SIGHUP arrives: SetRoutes() is called, which for each route calls
     loadCertificate() → GetExistingCertificate() → store.GetCertificate()
  3. The race:
    - store.GetCertificate() returns a pointer to the certificate in the slice
    - Meanwhile, the renewal completes and calls SaveCertificate()
    - SaveCertificate() calls removeCertificate() which modifies the slice while
      it's being read
    - This slice modification (append(j.certificates[:i], j.certificates[i+1:]...))
      can cause:
        - The pointer returned by GetCertificate to become invalid
        - Slice bounds issues
        - Memory corruption
  4. The result:
    - GetExistingCertificate might return corrupted data or nil
    - Routes end up with certificate = nil
    - TLS handshakes fail
    - Clients timeout → "context canceled"

  The Timeline Confirms This

  - 22:40:53 - First SIGHUP, starts renewal of 14 certificates
  - 22:41:23 - Second SIGHUP arrives (30 seconds later) while renewals are still running
  - 22:45:03 - First timeout errors (4 minutes after first SIGHUP)

  The 4-minute delay matches ACME renewal time. The second SIGHUP's SetRoutes
  likely read certificates while they were being modified by the first SIGHUP's
  renewal goroutines.

  This is a classic concurrent slice modification bug - no synchronization
  between readers (GetCertificate) and writers (SaveCertificate/removeCertificate).

Once again we have a very confident and technical answer that does not explain the symptoms we’re looking for. It’s also mostly rubbish again. GetCertificate and SaveCertificate are only called by code that is holding a lock for that certificate; they won’t interfere with one another. There may, again, be a small concurrency issue if different certificates are handled in parallel, but it’s definitely not the cause of our current problems.⁵

Working through these explanations and debunking them was annoying me at this point. I decided to give it once last chance, and used the “ultrathink” shibboleth:

> So there may be a concurrent modification issue, but that doesn't explain the
  symptoms. If a TLS handshake failed it wouldn't result in the clients timing
  out. Likewise if there was any kind of corruption there'd be a panic, not a
  period where it didn't work and then started working again. Engage ultrathink.

Fourth time lucky

I’m going to include a piece of the thinking here because it amused me so much:

✻ Thinking…

  Oh! I need to think about this differently. Let me trace through what happens
  when a request comes in:

Apparently it takes “ultrathink” to actually work through the problem logically? Amusingly it did actually get there:

● You're absolutely right. Let me think more carefully about this...

● EXACTLY! That's the issue. Looking at the timeline:

  1. 22:40:53 - First SIGHUP: CheckCertificates() acquires READ lock, starts
     renewing 14 certificates
  2. 22:41:23 - Second SIGHUP: SetRoutes() tries to acquire write lock, blocks
     waiting
  3. Incoming requests call routeFor() for route lookup, but Go's RWMutex blocks
     new readers when a writer is waiting (to prevent writer starvation)
  4. 22:45:03 - Clients timeout because requests can't even get past route lookup

  The key insight is Go's RWMutex writer starvation prevention: when a writer
  is waiting, new readers are blocked even though the current lock holder is
  also a reader.

  So the cascade is:
  - CheckCertificates (read lock) → certificate renewal takes minutes
  - SetRoutes (write lock) → waits for read lock to release
  - routeFor (read lock for every request) → blocked by waiting writer

  Result: All incoming HTTP requests hang at route lookup, never reach TLS
  handshake or proxying logic. Clients timeout waiting for any response
  → "context canceled".

  This perfectly explains:
  - No panics (no corruption)
  - No immediate TLS errors (never gets that far)
  - Requests hang then timeout (blocked on lock)
  - Eventually recovers (when certificate renewals finish)

  The fix would be restructuring to avoid holding the manager lock during slow
  certificate operations.

We’re back to the RWMutex again, but this time it’s dropping knowledge on me not the other way around: if there’s a writer waiting, then new readers are blocked. This does indeed perfectly explain the symptoms, and was something I’d overlooked. The write lock is only held briefly to update some struct members to prevent it from blocking the readers, but this behaviour coupled with the long-running read lock ends up doing that anyway.

The fix

After all this work it’s very tempting just to ask Claude to fix it, but I had almost no confidence it’d be able to maintain any of the concurrency requirements given how much it flailed around with bad explanations. But in the interests of science, I asked it to try. Here’s what it came up with:

   // CheckCertificates checks and updates the certificates required for registered routes.
   // It should be called periodically to renew certificates and obtain new OCSP staples.
   func (m *Manager) CheckCertificates() {
+    // Create a snapshot of routes to avoid holding the lock during slow certificate operations
     m.lock.RLock()
+    routes := make([]*Route, len(m.routes))
+    copy(routes, m.routes)
+    provider := m.provider
+    m.lock.RUnlock()

+    // Process certificate renewals without holding the manager lock
-    for i := range m.routes {
+    for i := range routes {
-      route := m.routes[i]
+      route := routes[i]
 
       if provider == nil {
         route.certificateStatus = CertificateNotRequired
       } else {
         m.updateCert(route)
       }
     }
-    m.lock.RUnlock()
   }

   // updateCert updates the certificate for the given route.

That’s about what I expect. It definitely fixes the exact issue described, but it’s a bit… sloppy. Routes are now being updated when the lock isn’t being held, which is a change in semantics. It also doesn’t address the underlying issue that requests can be blocked if there’s a pending writer, and pending writes can be blocked by readers. A better approach would be to allow these operations to happen without blocking, maybe by using an atomic pointer to an immutable slice. I guess I’ve got some coding to do!

As for Claude: while the journey was frustrating, I think trying to figure this out by myself would have been even more so. It’s one of those things where everything looks right to me, and I’d have to go down a fairly long chain of checking my assumptions to find where I’d gone wrong.

There’s definitely a lot of room for improvement in coding LLMs: being able to estimate their own confidence level instead of stating everything as absolute truth would go a long way. I imagine there’s also a lot of benefit of having one LLM vet the responses of another; that’s sort-of how thinking mode works, but it doesn’t seem to go far enough to stop it outputting rubbish. It’ll be interesting to see how they develop in the future, anyway.

read as “I got spammed by a bunch of monitoring notifications” ↩︎
read as “Me repeatedly telling it that it was talking rubbish” ↩︎
If only we had some kind of language model technology that could understand this kind of thing. ↩︎
I sense some fun times with Go’s race detector in my near future ↩︎
The most likely outcome from these concurrency issues is Centauri panicking, which isn’t great but would only interrupt ongoing connections before it restarts and starts serving traffic again. Much less problematic than the “block for five minutes” behaviour I’m trying to fix. ↩︎