Chameth.com - posts like blogging-and-the-imaginary-quality-bar, break-everything-fuzz-testing, modern-css-is-fun, why-you-should-be-using-https

Monthly Meanderings: March 2026

2026-04-01T00:00:00Z

Since last month’s update I’ve been unpleasantly reminded that I’m middle aged, through the medium of a dodgy knee. There’s nothing quite like not being able to stand up without groaning to underscore that you’re not young any more. The ongoing game of “will this seemingly mundane activity make my knee go funny again?” is a delight to play, too. I’m great at it.

Website updates

Two new blog posts this month: the longest way to represent a date is a short thought experiment on absurd date formats, and modern CSS is fun goes through some neat new CSS features I’ve used recently for this site.

I did some design tweaks this month: headers now have a blue background behind them¹, and a lot of components have been redesigned to have a consistent style. You can see that component in action in the new “now playing” widget I’ve added:

Alien Ant Farm — Flesh and Bone ANThology Scrobbled 2m ago

Instead of each component defining its own styles (which were mostly copied and pasted), there’s now a single class that handles the drop shadow, border, background colour, and the optional title. It also applies a consistent format to elements that are clickable: they use the blue accent colour for their borders, and have a glow effect shown on hover.

The now playing widget is part of a larger bit of work I did on importing some music stats. I’m pulling in the data from my Navidrome instance. A new music page shows my most listened albums and artists. Navidrome doesn’t store a complete play history, just the last play and the count, so I can’t do “what I listened to last month” stats until next month.

Other projects

Other than a minor bug-fix update to contempt, my Dockerfile templating/updating tool, I’ve not done much on my other open source projects. I’ve got a bit more work to do finish migrating all my repositories from GitHub to my private Forgejo instance. I’ve also been thinking about mirroring my public repositories to Codeberg so there’s a non-GitHub way to access them.

Entertainment

I apparently only watched a single film in March:

The Substance

2026-03-18

I really didn’t get on with this. It’s too long for what it is, and over-the-top in a way that just didn’t work for me.

Demi Moore and Margaret Qualley were great, but nothing else really was. The cinematography and sound design were too affected, the plot was pretty shallow and obvious, most of the body horror fell into the uncanny CGI valley for me.

It’s not completely without merit: there were around 40 minutes in the middle where I was really into it, but that’s not really enough given it’s closing in on a 2.5 hour runtime.

There’s something to be said for the message it’s presenting about women in Hollywood, too, but it was very heavy handed. Dennis Quaid’s character is even called Harvey; it’s not so much a subtle nod as a sledgehammer to the face.

Instead of films, I’ve been watching a bunch of TV. After devouring The Pitt last month, I struggled to find something to fill the hole it left. I settled on catching up on Chicago Fire and Chicago P.D.. They’re both mostly about getting things done without personal drama becoming the primary focus. They’re not quite as grounded as The Pitt, but at least they’ve not completely gone off the deep end like 9-1-1 where the first responders end up in space somehow?

I’ve also watched the first two episodes of SNL UK. I didn’t have very high expectations, and it gave me a pleasant surprise. Both episodes so far have some hilarious sketches, and are obviously written by British comics with the typical darker, more sardonic humour that separates us from the USA.

Last but not least, board games! I spent a weekend at a tabletop/LAN event with friends, so got to physically play some games for a change. I also kept up with some turn-based games on Board Game Arena, as usual.

6 plays
4 plays
4 plays
4 plays
3 plays
3 plays
3 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play

Around the web

Warranty Void if Regenerated

Some really cool speculative fiction about a future where all software is written by LLMs, and the social effects that may have. Unfortunately the story itself is written with an LLM, and that becomes painfully obvious about a third of the way in. By that point I was already hooked, though.

My 2-step process for AI-free blogging

Speaking of LLM-generated text, this short piece by Manuel Moreale made me laugh. I’m by no means anti-LLM, but I really hate people publishing the output like it’s their own work. It breaks the social assumption that the writer put more effort in than the reader, and makes me wonder how much is hallucinated.

25 Years of Eggs

An interesting applied use of coding agents. I’m a sucker for personal statistics. I have no explicit desire to keep my receipts, or generate graphs of my egg consumption, but seeing the graphs makes me question that. The journey to get there is a great read, too.

I made a one-page notebook

The clever paper folding is interesting, but I really like some of the small touches on the page. There’s a nice little “in reply to” callout providing context at the top of the page, you can click to rotate the image further down, and there’s a details element containing a textual version of it. It has that great ‘small web’ vibe of someone who really cares about what they’re doing.

Workers who love ‘synergizing paradigms’ might be bad at their jobs

I haven’t read the full paper, but the summary given by the article is equal parts interesting and amusing. They developed a “Corporate Bullshit Receptivity Scale”, what more is there to say?

It’s meant to resemble painter’s tape, but I’m not sure quite how well it works. ↩︎

Modern CSS is fun

2026-03-17T00:00:00Z

I’ve been doing a bunch of CSS tweaking recently, and keep being surprised by how nice modern CSS is to work with. As someone grey-haired enough to remember writing HTML without CSS, it’s amazing to think how far along web technology has come¹. I wanted to demonstrate some of the handy bits and pieces I’ve used recently.

:has

The :has selector allows you to effectively query for child elements. While a span will match a <span> within an <a>, a:has(span) will match an <a> that contains a <span>. This really shines when combined with more complex selectors, for example:

input {
    border-radius: var(--border-radius);

    &:has(+ .results:not(:empty)) {
        border-bottom-left-radius: 0;
        border-bottom-right-radius: 0;
    }
}

This is part of the styling for my film search box. When the search box is presented initially it has fully rounded corners. But when it is followed by a non-empty results element, it removes the rounding on the bottom corners so that the border continues in a straight line down into the results. You can see it in action below; just enter a few characters (like “the”) to get some results:

You could achieve the same effect by having JavaScript add a class to the <input> element, but I’ll take a CSS solution over a JavaScript solution any day.

Nested rules

You probably spotted this in the example above. It’s what finally made me switch from SCSS to plain CSS. If you have a rule for .foo and a rule for .foo .bar you can just nest them. Not only does it save repeating yourself an awful lot, it keeps everything organised nicely. For example, the CSS for my film list embeds looks like this:

a.film-list {
    display: grid;
    grid-template-areas: "header images" "description images" "meta images";
    grid-template-columns: 1fr auto;
    grid-template-rows: auto 1fr auto;
    row-gap: var(--small-space);

    @media (width <= 800px) {
        grid-template-areas: "header" "description" "images" "meta";
        grid-template-columns: 1fr;
        grid-template-rows: auto auto auto auto;
    }

    & + & {
        margin-top: var(--medium-space);
    }

    h3 {
        font-size: var(--font-size-xxlarge);
    }

    .description {
        /* ... */
    }
}

Everything is wrapped up in a single bundle, including the @media rules for changing the layout on smaller screens, the lovely little & + & rule that adds some extra margin if there are two lists in a row, etc. The & syntax refers to the parent selector, so & + & in this case is the same as a.film-list + a.film-list: it’s very handy! The best part about this is that it’s the exact same syntax as SCSS, so converting is easy². The styled element looks like this:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Media range syntax

Once again, you might have spotted this in the previous snippet. I’m not deliberately teasing things, I promise! Back in the day, you did media queries like so:

@media (min-width: 1000px) and (max-width: 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I always hated this syntax. I always got muddled up as to whether I wanted “min” or “max”, and whether they were inclusive or not. It’s easy to reason through, but it never came naturally. Fortunately you can now just use ranges:

@media (1000px <= width <= 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I find this style so much easier to write and understand.

Anchor positioning

Trying to dynamically position one element next to another used to exclusively fall within the purview of JavaScript. Fortunately anchor positioning fixes all this. I use this in the film search field I showed above:

.film-search {
    input {
        anchor-name: --film-search-box;
    }
    .results {
        position: absolute;
        position-anchor: --film-search-box;
        position-area: bottom center;
        position-visibility: always;
        width: anchor-size();
    }
}

This does lots of fun things. The .results element is anchored to the input field (via the --film-search-box name), and it’s positioned on the bottom, directly under the anchor. The position-area rule describes a 3x3 grid, with the anchor at the center, so top left would put it diagonally adjacent to the anchor. Next, position-visibility keeps the .results element visible when the anchor isn’t; otherwise when you scroll the anchor off the screen the results would immediately vanish. Finally, the special anchor-size() function makes the .results element take up the same exact width as the anchor.

All-in-all, this puts the results exactly where I want them, without having to deal with any JavaScript at all. I can see how it’d also be amazingly useful trying to do tooltips or other forms of ‘floating’ content too. You can even leave the positioning up to the browser, giving it hints about which order to try, or what property to optimise for (e.g. you can say “anchor to the left or right, whichever has more horizontal space”).

attr() function

This one blew my mind a little. In some of the elements on the site, I’ve added a little label to the top. It looks something like this:

It’s meant to just be a visual flourish, not a semantic title or anything, so I initially just added a ::before selector to each element, customising the content to have the right value. I didn’t like the duplication, though. Now, instead, I do this:

&::before {
    content: attr(data-title);
}

Each component defines its title in a data attribute: <div class="raised-box" data-title="an example">, and the attr function plucks it out and puts it in the content rule. It’s worth noting that this is only widely supported for the content rule, you can’t yet use it for colours or dimensions or other things. It still feels a bit magical, though. Like you’re making a reusable, customisable component with just HTML and CSS.

layers

I’d come across layers a few times, but I never found a need for them. It seemed like something you’d only really need with complicated design systems, or something. Then I had a problem and layers were the perfect solution!

The headings on this site have a whole bunch of CSS attached to them. Currently they look a bit like blue painter’s tape with handwriting on them. This style is applied to all headers, and then the various places that don’t want it had to manually reset everything. I got fed up with that, so decided to add a plain-header class which would “disable” the extra styling:

h2,
h3,
h4,
h5,
h6 {
    /* common styles for all headers, regardless */
    font-family: "Chris Hand", sans-serif;
    font-size: var(--font-size-xxlarge);
    color: var(--text-highlight-colour);
    margin: var(--medium-space) 0;
    line-height: 1;

    &:not(.plain-header) {
        /* do fancy stuff if it _doesn't_ have the .plain-header class */
        transform: rotate(-0.15deg);
        padding: var(--smedium-space);
        font-weight: 800;
        filter: drop-shadow(3px 3px 5px rgb(0, 0, 0));

        a {
            background: none;
            text-decoration-skip-ink: auto;
        }
    }
}

This might seem a bit backwards but I want the default to be styled, as those are far more common. I added this change, sprinkled around some “plain-header” classes, tidied up a lot of duplicate CSS, and things mostly worked. Mostly. Some things stopped working. In a few places the headers had slight tweaks, and they stopped working entirely.

It turns out in moving the rules from a plain h2 selector to a h2:not(.plain-header), I’d made them more specific. Previously a selector like h2.special would have been more specific, so its properties would override those from the less specific h2 selector. My first reaction was to try and hack around it. Changing all the overrides to body h2.special would make them more specific again, but I didn’t want to have to remember to do that forever more.

Instead, I defined some custom layers, and put the header definitions in one:

@layer reset, links, headings

@layer headings {
    h2,
    h3 {
        /* etc */
    }
}

The first @layer rule defines some layers that will be handled in a specific order: first the “reset” layer, then the “links” layer, then the “headings” layer, and then anything not in a layer after that. Rules in one layer don’t have to worry about beating the specificity of rules in another layer, because they’re handled separately.

Note that I couldn’t just put the headings in a layer and call it a day: their rules have to come after the CSS reset, and after the normal link styles, as headers have a bit of extra styling for nested links. If the CSS reset weren’t in a layer, then the * { margin: 0; padding: 0; } type reset would apply over the rules from the headings layer.

Using layers is definitely a bit fiddly, and is not necessary for a lot of sites, but it’s a much cleaner alternative to ugly specificity hacking.

Things I’m looking forward to

There are a few things that aren’t (widely) available yet, that particularly interest me:

sibling-index() and sibling-count()

These new functions are available in WebKit and Blink based browsers, but not Firefox. They give you the index of the element within its siblings, or the count of siblings. For the film list component above, I currently have this abomination for the overlapping posters:

&:nth-child(1) {
    z-index: 5;
    left: 0;
}
&:nth-child(2) {
    z-index: 4;
    left: calc(133px * var(--overlap) * 1);
}
&:nth-child(3) {
    z-index: 3;
    left: calc(133px * var(--overlap) * 2);
}
&:nth-child(4) {
    z-index: 2;
    left: calc(133px * var(--overlap) * 3);
}
&:nth-child(5) {
    z-index: 1;
    left: calc(133px * var(--overlap) * 4);
}

These could be replaced with something like:

z-index: calc(1 + sibling-count() - sibling-index());
left: calc(133px * var(--overlap) * (sibling-index() - 1));

As an added bonus, it would scale to any number of posters. index being 1-based not 0-based is a bit unfortunate as it makes using it in calculations awkward (see 1 offset in both of those rules!), but it’s leagues better than writing 5 separate rules.

random()

Currently only available in Safari. There are a few places where I’d like to have slight random variations of the style. Things like elements that are rotated slightly for aesthetics; they look a bit silly if they’re all identically positioned. I also randomise the icons and positions of my rating stars to break up the visual monotony. Here are a few examples:

Currently I do the stars by hardcoding a bunch of classes and having the backend randomly apply one when it generates the markup for stars. Being able to do it in CSS would be great, though.

Mixins

This one’s so far off it’s not even listed on caniuse.com yet. Mixins are another handy feature of SCSS, that allow you to define reusable blocks of rules, then import them when needed:

@mixin fancy-background {
    background: /*...*/;
}

.some-element {
    @include fancy-background;
}

#other-element {
    @include fancy-background;
}

This would be particularly useful if you can’t easily control the markup to add classes to everything. For example, my footnotes are rendered by a markdown plugin, and don’t easily have a way to add extra classes to them. To style it the same as another element, I currently duplicate a bunch of rules between them both. Mixins would allow me to define those rules once, and then import them in both places.

Aside from JavaScript, which seems to just endlessly reinvent new frameworks and ways to make the most sprawling dependency tree possible. ↩︎
It also means that if your website’s syntax highlighting library doesn’t seem to understand nested rules, you can just pretend your perfectly valid CSS is actually SCSS and it’ll magically work. Grumble, grumble. ↩︎

Blogging and the Imaginary Quality Bar

2025-09-18T00:00:00Z

Recently I realised that I’ve developed a self-imposed quality bar for blog posts. They need to be a certain length, and have a certain substance to them. They need to be generally useful in some way I can’t quite define, to some imagined future audience. They need to have images to break up the page, and opengraph data for when they’re linked to on social media. But… maybe they don’t? Those things all make sense for longer “article” type posts, but not so much for a personal blog.

I’ve done some fiddling so that I can make posts without all that extra stuff. We’ll see how it goes. The posts won’t be distinguished on the site (yet?), but I have set up separate RSS feeds for just short form posts and long form posts in case subscribers are radically opposed to one or the other. Now I can start blogging like it’s 2005 again. But hopefully with a bit less cringe.

I’ve been staring at this post in my editor for about five minutes. The urge to make it longer, more thorough, more article-like is really strong. This entire paragraph is only here as a compromise with myself so I can actually save and commit the post.

Finding an awkward bug with Claude Code

2025-06-30T00:00:00Z

The many times Claude thought it was right…

I recently encountered a bug in one of my projects that I couldn’t immediately figure out. It was an issue in Centauri, my reverse proxy. After its config was updated, I noticed¹ it stopped serving responses. Looking at the logs, I could see it was obtaining new certificates from Let’s Encrypt for a couple of domains, but I’d designed it so that wouldn’t block requests (or so I thought). After a few minutes of confusion, everything went back to working.

It felt like an issue with locking, but a quick glance didn’t show me anything obvious. I decided to throw the problem at Claude Code and see how it did. With some guidance² it managed to identify the cause, but it was a bit of a journey to get there. I’m going to share the prompts and responses, as it was an interesting process.

The initial prompt

I thought the issue might be quite obvious, and I’d just overlooked it. Centauri isn’t that complex a project, after all. So I gave Claude a fairly minimal prompt to set it off:

> Hello! Please examine this codebase. I have noticed an issue that when a
  certificate is being renewed, the application seems to not serve any other
  requests. I suspect there is a misplaced lock somewhere.
 
  Can you see what's happening? Think very hard.

Claude Code has special hardcoded triggers to engage “thinking” mode, which makes it take more time to think about alternatives. The keywords are: “think”, “think hard”, “think harder”, and “ultrathink”. I managed to not quite get the right incantation here, and only enabled the lowest level³.

I’ll omit all the responses where Claude is “thinking” (talking to itself), or decides to make todo lists then cross items off them then add different items, and so on. Its responses get fairly lengthy; I won’t judge you too harshly if you skim over them. Its first response was blessedly brief:

● I found the issue! Looking at the certificate manager code, I can see the
  problem:

… and then it went back to thinking and reading files, because it had neither found the issue nor could it see the problem. Ho-hum. Some time later it came back with:

● I found the issue! The problem is in certificate/manager.go:53-54:
  
  m.store.LockCertificate(subject, altNames)
  defer m.store.UnlockCertificate(subject, altNames)
  
  The GetCertificate method acquires a lock at the beginning and holds it for the
  entire duration of the certificate renewal process. This includes:
  
  1. Contacting the ACME server
  2. Solving DNS-01 challenges (can take minutes for DNS propagation)
  3. Obtaining the certificate
  4. Getting OCSP staple
  5. Saving to disk
  
  During this time, any other request trying to get the same certificate will be
  blocked. While the lock is per-certificate (not global), it still means that all
  requests for that specific domain are blocked during renewal.
  
  Let me check if there are other places where this might cause blocking:

And it then proceeded to do even more thinking, checking files, etc, until it concluded with:

● I found the root cause! The issue is actually worse than just a
  per-certificate lock. Here's what's happening:
  
  The main blocking issue is in proxy/manager.go:133-144:
   
  func (m *Manager) CheckCertificates() {
      m.lock.RLock()  // ← Acquires read lock on the entire manager
      for i := range m.routes {
          route := m.routes[i]
          // ...
          m.updateCert(route)  // ← This can take MINUTES for ACME renewal!
      }
      m.lock.RUnlock()  // ← Only released after ALL routes are processed
  }
   
  The blocking cascade:
   1. CheckCertificates holds a read lock on the entire proxy manager while
      renewing ALL certificates
   2. Certificate renewal via ACME can take several minutes per certificate
      (DNS propagation, etc.)
   3. Every incoming request calls routeFor (proxy/manager.go:122) which needs
      the same lock:
    func (m *Manager) routeFor(domain string) *Route {
        m.lock.RLock()  // ← BLOCKED until CheckCertificates completes!
        match := m.domains[strings.ToLower(domain)]
        m.lock.RUnlock()
        // ...
    }
  
  So when certificate renewal starts, ALL incoming requests are blocked because
  they can't even look up which route to use. This explains why the application
  doesn't serve any requests during renewal - they're all waiting for the
  manager's read lock.
  
  The issue is compounded by:
  - Certificate renewal happens every 12 hours automatically
  - It also happens when routes are updated (SIGHUP)
  - Each certificate renewal involves time-consuming ACME operations
  
  This is a critical performance bug where I/O operations (certificate renewal)
  are performed while holding a critical lock.

This doesn’t actually follow on from the earlier response, it’s completely flipped onto a different lock being the issue. Amusingly, it’s nearly right here, but it’s highlighting this code for completely the wrong reason.

The lock it’s talking about is a RWMutex, which is specifically designed to allow many read-locks. Everything it’s pointing out are read locks, so they cannot block one another as it described. The response Claude has given sounds very confident, seems appropriately technical, but is in fact complete rubbish. This is one of the big issue with all LLMs at the minute, and it makes me despair at the idea of people “vibe coding” things when they don’t know enough to push back against nonsense like this.

Course correction

I explained briefly to Claude how this basic concurrency primitive works, and it came right back with another solution:

> But that's a RW Mutex. You can hold many read locks concurrently.

● You're absolutely right! I apologize for the confusion. With a RWMutex,
  multiple readers can hold the lock concurrently. Let me trace through this
  more carefully.
  
● Task(Analyze updateCert method)
  ⎿ Done (8 tool uses · 20.0k tokens · 1m 11.9s)
  
● Excellent analysis! You're right about RWMutex allowing concurrent readers.
  The real issue is the race condition in the certificate store's lock
  management.

  Looking at certificate/store.go:82-91:

  func (j *JsonStore) lockFor(subjectName string, altNames []string) *sync.Mutex {
      key := strings.Join(append([]string{subjectName}, altNames...), ";")

      if mu, ok := j.locks[key]; ok {
          return mu
      } else {
          mu = &sync.Mutex{}
          j.locks[key] = mu  // ← Concurrent map write!
          return mu
      }
  }

  This is a classic concurrent map access bug. When multiple requests hit the
  same domain during certificate operations:

  1. Multiple goroutines call lockFor() simultaneously
  2. They read/write to j.locks map without synchronization
  3. Go runtime can:
    - Panic with "concurrent map writes"
    - Corrupt the map structure
    - Block indefinitely if map internals get corrupted

  This explains the blocking behavior - it's not the RWMutex, but rather a race
  condition causing either panics or deadlocks when the map gets corrupted
  during concurrent certificate operations.

  The fix would be to either:
  1. Add a mutex to protect j.locks map access
  2. Use sync.Map instead of a regular map
  3. Pre-initialize all locks at startup

I do appreciate when me stating a basic fact is called out as “excellent analysis”. Definitely doesn’t make me feel like I’m having smoke blown up my ass.

I think the issue it’s describing here is actually a problem⁴, albeit one that’s unlikely to be hit. But it’s definitely not the one we’re looking for. Claude even describes the possible consequences of the bug, and they don’t match what happened at all.

Maybe logs will help?

While Claude was chugging away, I’d been looking into the issue as well. I had logs from when it happened, and figured Claude may be able to glean something from them too. The first thing I’d noticed is that I wasn’t logging when a process finished, so I’d already added that to the code base, hence the note so Claude doesn’t get itself confused:

> That may be an issue, but it's not the one I was looking for. I've found logs
  from the time it happened. Note that a few logs have been added to the
  codebase since (so we can't tell from the logs when ACME or OCSP processes
  *finished*, just when they started). Here are the logs:
  
  {"time":"22:40:53.187887976Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:40:53.19947532Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:40:53.199507942Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:09.048465968Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:09.048482309Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:17.419299795Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:17.419325744Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:23.238027386Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:41:29.994330646Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:29.994353329Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:43.658794811Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:43.658819548Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:57.635325513Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:57.635347515Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:10.52439779Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:10.524417107Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:23.159430249Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:23.159452171Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:36.347407326Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:36.347433295Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:52.544628775Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:52.544653993Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:25.409579107Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:25.409602842Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:43.819479448Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:43.819502041Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:54.358957563Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:54.358979355Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:44:06.180278128Z","level":"INFO","msg":"Renewing certificate", <big list of alt names>}
  {"time":"22:44:06.180302585Z","level":"INFO","msg":"Starting ACME process to obtain certificate", <big list of alt names>}
  {"time":"22:45:03.830331922Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.830528134Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.832388931Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.225854865Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488432434Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488439397Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:30.617614536Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}

I’ve cut down the logs a bit to remove the specific domains, simplify the timestamps, and remove a bunch of other cruft. The key points are there were two SIGHUPs received, and the last certificate before everything started breaking covered 6 different hostnames.

Claude then went through a phase of periodically shouting “Eureka!” then deciding it hadn’t actually found anything yet:

● Now I see it! The key insight is in the timing and the "context canceled"
  errors. Let me trace through what's happening:
  
● Ah, now I see what's happening! Looking at the logs more carefully:

  1. 22:40:53 - First SIGHUP received, routes updated
  2. 22:40:53 to 22:44:06 - Multiple certificates start renewing (14 certificates!)
  3. 22:41:23 - Another SIGHUP received while renewals are still in progress
  4. 22:45:03 onwards - "context canceled" errors start appearing

  The issue is that SetRoutes is being called AGAIN (at 22:41:23) while the
  previous CheckCertificates() goroutine is still running and renewing
  certificates. Let me check what happens in this scenario:
  
> (The context canceled errors are clients timing out because centauri wasn't
  proxying properly)
  
● Ah, now I understand! The "context canceled" errors are from clients timing
  out because Centauri wasn't proxying their requests. This confirms the
  blocking issue.

  Let me check how incoming requests actually get their certificates:
  
● Now I found the real issue! Looking at the timeline and the code:

  1. 22:40:53 - SIGHUP triggers SetRoutes() which:
    - Calls loadCertificate() for each route
    - Sets route.certificate to the existing certificate
    - Starts go m.CheckCertificates() in background
  2. 22:41:23 - Another SIGHUP while certificates are still renewing
    - SetRoutes() is called AGAIN
    - Creates NEW Route objects with route.certificate = nil initially
    - Replaces the old routes in m.routes
    - The old Route objects (with valid certificates) are replaced!

  The critical issue is in proxy/manager.go:42-65:

  func (m *Manager) SetRoutes(newRoutes []*Route, fallback *Route) error {
      // ... validation ...

      for i := range newRoutes {
          route := newRoutes[i]  // These are NEW Route objects!
          // ...
          m.loadCertificate(route)  // Try to load certificate
      }

      m.lock.Lock()
      m.routes = newRoutes  // REPLACES all routes!
      m.lock.Unlock()
  }

  When the second SIGHUP arrives:
  1. New Route objects are created
  2. For domains currently being renewed, GetExistingCertificate might fail or
     return expired certs
  3. These routes get route.certificate = nil and CertificateMissing status
  4. CertificateForClient returns nil for these domains
  5. TLS handshake fails → clients can't connect → "context canceled" errors

  The fix would be to either:
  1. Prevent concurrent CheckCertificates runs
  2. Copy certificates from existing routes before replacing them
  3. Make certificate renewal truly independent of route management

Midway through it had started searching for “context canceled” errors in the codebase, which doesn’t make much sense. That’s what happens when a client gives up waiting, so the corresponding backend request is terminated. You won’t actually find “context canceled” anywhere in Centauri, because it’s a feature of the Go standard library. I gave it a little nudge to try to stop it rabbit holing.

As far as I can tell, what it’s talking about in this response isn’t an issue. I guess it’s possible that there’s a race condition whereby a route would be replaced and end up with an old version of a certificate that was just being renewed, but it wouldn’t cause an issue in practice; it’d just get updated next time the certificates were refreshed.

There’s no way for routes to get a nil certificate in this scenario, as far as I know. But I don’t have the unfounded confidence of an LLM, so I simply said:

> I don't follow why the certificate would be nil?

Attempt the third

I expected Claude to just explain its reasoning, but it apparently didn’t have any reasoning so set off on another voyage of discovery:

● You're right to question that. Let me trace through this more carefully.

● Good point! Since GetExistingCertificate doesn't acquire locks, it should
  return the existing certificate even during renewal. Let me dig deeper:
  
● Now I see a potential race condition! Here's what could happen:

● I found it! There's a critical race condition that explains the blocking:

  The Race Condition

  1. Background renewal starts: CheckCertificates() → updateCert() →
     GetCertificate() acquires lock and starts ACME renewal
  2. SIGHUP arrives: SetRoutes() is called, which for each route calls
     loadCertificate() → GetExistingCertificate() → store.GetCertificate()
  3. The race:
    - store.GetCertificate() returns a pointer to the certificate in the slice
    - Meanwhile, the renewal completes and calls SaveCertificate()
    - SaveCertificate() calls removeCertificate() which modifies the slice while
      it's being read
    - This slice modification (append(j.certificates[:i], j.certificates[i+1:]...))
      can cause:
        - The pointer returned by GetCertificate to become invalid
        - Slice bounds issues
        - Memory corruption
  4. The result:
    - GetExistingCertificate might return corrupted data or nil
    - Routes end up with certificate = nil
    - TLS handshakes fail
    - Clients timeout → "context canceled"

  The Timeline Confirms This

  - 22:40:53 - First SIGHUP, starts renewal of 14 certificates
  - 22:41:23 - Second SIGHUP arrives (30 seconds later) while renewals are still running
  - 22:45:03 - First timeout errors (4 minutes after first SIGHUP)

  The 4-minute delay matches ACME renewal time. The second SIGHUP's SetRoutes
  likely read certificates while they were being modified by the first SIGHUP's
  renewal goroutines.

  This is a classic concurrent slice modification bug - no synchronization
  between readers (GetCertificate) and writers (SaveCertificate/removeCertificate).

Once again we have a very confident and technical answer that does not explain the symptoms we’re looking for. It’s also mostly rubbish again. GetCertificate and SaveCertificate are only called by code that is holding a lock for that certificate; they won’t interfere with one another. There may, again, be a small concurrency issue if different certificates are handled in parallel, but it’s definitely not the cause of our current problems.⁵

Working through these explanations and debunking them was annoying me at this point. I decided to give it once last chance, and used the “ultrathink” shibboleth:

> So there may be a concurrent modification issue, but that doesn't explain the
  symptoms. If a TLS handshake failed it wouldn't result in the clients timing
  out. Likewise if there was any kind of corruption there'd be a panic, not a
  period where it didn't work and then started working again. Engage ultrathink.

Fourth time lucky

I’m going to include a piece of the thinking here because it amused me so much:

✻ Thinking…

  Oh! I need to think about this differently. Let me trace through what happens
  when a request comes in:

Apparently it takes “ultrathink” to actually work through the problem logically? Amusingly it did actually get there:

● You're absolutely right. Let me think more carefully about this...

● EXACTLY! That's the issue. Looking at the timeline:

  1. 22:40:53 - First SIGHUP: CheckCertificates() acquires READ lock, starts
     renewing 14 certificates
  2. 22:41:23 - Second SIGHUP: SetRoutes() tries to acquire write lock, blocks
     waiting
  3. Incoming requests call routeFor() for route lookup, but Go's RWMutex blocks
     new readers when a writer is waiting (to prevent writer starvation)
  4. 22:45:03 - Clients timeout because requests can't even get past route lookup

  The key insight is Go's RWMutex writer starvation prevention: when a writer
  is waiting, new readers are blocked even though the current lock holder is
  also a reader.

  So the cascade is:
  - CheckCertificates (read lock) → certificate renewal takes minutes
  - SetRoutes (write lock) → waits for read lock to release
  - routeFor (read lock for every request) → blocked by waiting writer

  Result: All incoming HTTP requests hang at route lookup, never reach TLS
  handshake or proxying logic. Clients timeout waiting for any response
  → "context canceled".

  This perfectly explains:
  - No panics (no corruption)
  - No immediate TLS errors (never gets that far)
  - Requests hang then timeout (blocked on lock)
  - Eventually recovers (when certificate renewals finish)

  The fix would be restructuring to avoid holding the manager lock during slow
  certificate operations.

We’re back to the RWMutex again, but this time it’s dropping knowledge on me not the other way around: if there’s a writer waiting, then new readers are blocked. This does indeed perfectly explain the symptoms, and was something I’d overlooked. The write lock is only held briefly to update some struct members to prevent it from blocking the readers, but this behaviour coupled with the long-running read lock ends up doing that anyway.

The fix

After all this work it’s very tempting just to ask Claude to fix it, but I had almost no confidence it’d be able to maintain any of the concurrency requirements given how much it flailed around with bad explanations. But in the interests of science, I asked it to try. Here’s what it came up with:

   // CheckCertificates checks and updates the certificates required for registered routes.
   // It should be called periodically to renew certificates and obtain new OCSP staples.
   func (m *Manager) CheckCertificates() {
+    // Create a snapshot of routes to avoid holding the lock during slow certificate operations
     m.lock.RLock()
+    routes := make([]*Route, len(m.routes))
+    copy(routes, m.routes)
+    provider := m.provider
+    m.lock.RUnlock()

+    // Process certificate renewals without holding the manager lock
-    for i := range m.routes {
+    for i := range routes {
-      route := m.routes[i]
+      route := routes[i]
 
       if provider == nil {
         route.certificateStatus = CertificateNotRequired
       } else {
         m.updateCert(route)
       }
     }
-    m.lock.RUnlock()
   }

   // updateCert updates the certificate for the given route.

That’s about what I expect. It definitely fixes the exact issue described, but it’s a bit… sloppy. Routes are now being updated when the lock isn’t being held, which is a change in semantics. It also doesn’t address the underlying issue that requests can be blocked if there’s a pending writer, and pending writes can be blocked by readers. A better approach would be to allow these operations to happen without blocking, maybe by using an atomic pointer to an immutable slice. I guess I’ve got some coding to do!

As for Claude: while the journey was frustrating, I think trying to figure this out by myself would have been even more so. It’s one of those things where everything looks right to me, and I’d have to go down a fairly long chain of checking my assumptions to find where I’d gone wrong.

There’s definitely a lot of room for improvement in coding LLMs: being able to estimate their own confidence level instead of stating everything as absolute truth would go a long way. I imagine there’s also a lot of benefit of having one LLM vet the responses of another; that’s sort-of how thinking mode works, but it doesn’t seem to go far enough to stop it outputting rubbish. It’ll be interesting to see how they develop in the future, anyway.

read as “I got spammed by a bunch of monitoring notifications” ↩︎
read as “Me repeatedly telling it that it was talking rubbish” ↩︎
If only we had some kind of language model technology that could understand this kind of thing. ↩︎
I sense some fun times with Go’s race detector in my near future ↩︎
The most likely outcome from these concurrency issues is Centauri panicking, which isn’t great but would only interrupt ongoing connections before it restarts and starts serving traffic again. Much less problematic than the “block for five minutes” behaviour I’m trying to fix. ↩︎

The Curse of Knowledge and Blogging

2025-05-16T00:00:00Z

Finding a picture to go with a blog post about blog post ideas is hard, OK?

The single worst part about blogging for me is trying to come up with ideas for what to write about. Not because they’re hard to come up with, but because every idea seems too basic or not worth talking about. Seems.

It’s not just me, either. My friends will occasionally describe some interesting problem they’ve dealt with, and I’ll chime in with “you should blog about that”. Almost invariably the response is something like “it’s not that interesting”.

There’s a cognitive bias called the curse of knowledge¹. Basically, once you know something you can’t accurately reconstruct your previous state of mind from when you didn’t know it. You can struggle to do something for hours, piecing together scraps of information, using all kinds of specialist knowledge, and as soon as you’re finished it all seems so basic why would anyone want to read about that?

Other mental obstacles

Of course, nothing involving people is as simple as having one cause and an obvious solution. Even when I get over the “it’s too basic” road bump, there are other issues. Looking back over some of my posts, I get a horrible cringe-y feeling and the question of “why would anyone be interested in reading that?!” arrives unbidden. Even though I know that I’m interested in reading similar posts written by other people. I’m hesitant to attach labels to everything, but imposter syndrome says hi.

I think there’s also a cultural issue at play. British people tend to be somewhat reserved and downplay their accomplishments, especially when compared to people from the US². When I worked for a certain multinational ~~ad conglomerate~~ search engine company we actually had training on how to do performance reviews and speak about ourselves more like our US counterparts.

Put all these things into a cauldron, add a lizard’s leg and howlet’s wing, mix it all up, and you get a brain that despises the idea of blogging about anything. Great.

Dealing

So, now the problem is well-defined, what do we do about it? Clearly I am the best person to be offering this advice, given it’s been four months since my last blog post.

The best counter to the Curse of Knowledge that I’ve found is keeping notes, like a lab notebook. If you finish something and have dozens of links to different resources that you had to piece together, it’s probably something interesting and worth talking about even if it doesn’t seem like it any more. Or maybe you didn’t need a lot of references, but you had to write a lot down to make sense of something, or needed to draw diagrams to solve a problem. Again, probably interesting.

The other issues I’ve mentioned are not so easy to overcome. I’m also not going to try and perform therapy in the form of a blog post! What I personally do is keep a list of topics to write about (that have already passed the “probably interesting even if my brain doesn’t agree” check), and occasionally when the stars align I’ll feel like plucking one off the list and turning thoughts into words.

A call to action

If you’ve got a blog and haven’t written anything recently because you “don’t do anything interesting”, go write something! I promise that someone will be interested. Hell, feel free to send me a link and I’ll be interested for you.

If you don’t have a blog then why not start one? There are lots of great articles out there about why-you-should and how-you-can. And as a bonus, blog posts about how you set up your blog are super interesting. That’s one post sorted already!

As the open internet is slowly subsumed by massive corporate interests, and gummed up with spam and AI slop, the voice of real humans is becoming more and more important. Maybe we can — one blog at a time — get back to the ’90s internet that was full of individuality instead of today’s which is full of ads and spam?

Photo by Mika Baumeister on Unsplash

There’s also something called “Familiarity Blindness” that people like to talk about. It’s a much better name. The only problem is that I can’t find any actual published works that talk about it. No-one authoritative defines or mentions it. It just popped into existence a few years ago. Maybe it’s an LLM hallucination? ↩︎
My favourite commentary on this is Stephen Fry talking about American vs British comedy. ↩︎