Chameth.com - posts like a-year-of-boardgames, break-everything-fuzz-testing, debugging-beyond-the-debugger, sense-api, why-you-should-be-using-https

10 Weeks with an Apple Watch 10

2025-09-09T00:00:00Z

My watch. Yes, I am available for wrist modelling opportunities.

Around ten weeks ago¹ I picked up an Apple Watch 10, and have been wearing it almost constantly since. It’s not my first Apple Watch — I had a Series 5 for a bit back in 2020 — but it’s the first time I’ve actually stuck with it. Ten weeks seems like an apt time to reflect on it.

Firstly, why did I even bother? Well, for a couple of years I’d been wearing a Xiaomi Smart Band 7, mainly to monitor my sleep stats and set alarms that won’t wake up everyone else nearby. Its battery life was fantastic — with notifications and other things turned off, I got about a month of use between charges — but actually using it felt like trying to order food via the medium of interpretive dance.

My biggest gripe was the screen lock. If I didn’t have the screen locked then I’d periodically trigger it during the night when I moved around and it came in contact with my chest or leg. With the lock enabled you had to deliberately swipe up from bottom to top to enable interaction, but it didn’t work reliably. When I wanted to adjust an alarm, I’d be stood swiping repeatedly trying to get it to respond. When you finally get it unlocked, the whole interface is just fiddly.

The other issue was the data quality. There were some nights when I’d been woken up, sometimes even getting up and moving around, and it just didn’t show it in the data. If it can’t even get whether I’m asleep right, can I trust anything else it says?

I spent a while researching the best devices for sleep tracking. The Oura ring came highly recommended, but it was expensive and required a subscription to do anything useful. No thanks! The Apple Watch was consistently rated pretty well, and I reasoned I could pick up a refurbished older unit. I’ve been using an iPhone as my daily driver for a while, so it’d fit right into my begrudged walled garden.

The Series 10 has a significant advantage, though: it charges much quicker than all the previous generations. On a 30-minute charge, the Series 10 can go from 0 to 60%; the 9 can only make it to 40%, and my old 5 a measly 30%. Shorter charge times means I’m far less likely to leave it on charge and wander off without it. In some ways the daily charging is more convenient than monthly: the wireless charger sits on my desk, and I plop the watch on it for a little while in the evening; I don’t need to dig out the weird pogo-pin connector that has vanished sometime in the last four weeks, then carefully arrange it so it stays attached.

How a watch maybe saved my life

One of the big features of the Apple Watch, like many other wearable devices, is health and fitness tracking. I didn’t think much about this, beyond the sleep data I wanted, at first. I’ve never had a particularly good relationship with exercise², but I do like some good statistics. I started going for walks more often to get more data and see the graphs of VO2 max and HR recovery gradually inch up. That wasn’t the most profound effect on my health, though…

The recent versions of the Apple Watch have a feature that monitors for sleep apnea, a disorder where you don’t breathe properly during your sleep. I knew I wasn’t sleeping great — that’s why I was paying attention to sleep tracking data — but was still a bit surprised to get a notification from Apple Health after wearing the watch for 30 days. It gives you a graph to print and take to your doctor. So I diligently booked an appointment and a few weeks later went to see my GP.

The appointment went about as you’d expect: talking about referral to a sleep centre for a study, and so on. Towards the end, the doctor took my blood pressure (sleep apnea can be caused by, and can cause high blood pressure, in a lovely little vicious cycle). I don’t think either of us expected anything; it was just one of those standard checks for a related problem. After taking the reading, the doctor looked at me and said, “I can’t let you leave with a BP like this”. Yikes!

Blood pressure readings are split into stages: normal is under 120 mmHg³ over 80 mmHg⁴, stage 1 is up to 140/90, stage 2 is up to 180/120, and above is simply called “crisis”. Guess where I was? Also, fun fact: depending on how exactly you count and attribute things, hypertension comes out as the single largest cause of death in humans. It doesn’t kill you outright, but contributes to strokes, coronary artery disease, heart failure, and lots of other lovely things you don’t want on your CV.

After doing a few more readings, it settled down to just under the “crisis” stage and into the “holy crap, start treatment immediately” stage instead. I won’t labour on much more about this, but things are definitely moving in the right direction now⁵.

While the Apple Watch didn’t literally save my life, it triggered the chain of events that led to this diagnosis and treatment. Who knows what would have happened had it remained undetected? Probably nothing good. Also, go check your blood pressure! It’s simple to do and simple to treat if there is an issue, but so many people are walking around with hypertension and not even realising.

Building gates in the walled garden

Even though the watch is arguably a life-saver, not everything is rosy. It’s an Apple product, so you’re firmly locked in a walled garden, jealously guarded by people wearing black turtlenecks. Fortunately, there are a few ways to make it less painful.

All the health and fitness data is stored in Apple Health. You can export data as a big messy file, but it’s a very manual process to do so and the data format is gnarly. Luckily, there’s an app for that! Health Auto Export can, well, automatically export health data. It does what it says on the tin. It can send the data to Home Assistant, over MQTT, or dump it in some cloud file services, but I just have it post it all to a REST endpoint on a service I wrote. Then I dump it all in a database and can do whatever I want with it!⁶

Another tool that’s more useful on the watch itself is Apple Shortcuts. This is their no-code “if-this-then-that”-type thing. You can make automations or shortcuts that run a number of tasks. I have a whole slew of them I access via a complication⁷ on my watch face: one which prompts for input and adds an item to my to-do list (swiping to write letters is surprisingly not horrible), one which lets me select from a bunch of pre-written ones (“charge kindle”, “take laundry out in 1 hour”, type things), one to log my weight into Apple Health, one which can open and close the blinds in my room, and so on. It’s a surprisingly robust and easy-to-use system and offers just enough freedom that I’m not constantly grating on the edge of the walled garden.

Shortcuts being able to initiate arbitrary web requests is the real killer feature for me. Anything I can’t do on the watch itself, I can just farm off to a web server and connect it up with a shortcut. No need to learn Swift or pay Apple for the privilege of being a developer! For a lot of things, like controlling the blinds or adding to-do items, I already had a HTTP endpoint available and exposed over Tailscale⁸. Adding it to the watch was just a case of entering the right things in the Shortcuts app.

Daily nitty-gritty

There are lots of other little bits and pieces that come up when using the watch daily. I don’t think I can bundle them up into a pleasing narrative arc, so instead please enjoy some disjointed paragraphs of observations.

The Apple Watch has a lot of nagging abilities. It can notify you about your fitness “rings”, prompt you to stand up every hour, count how many seconds you wash your hands for, etc. I think I’d dislike these just on general principle, but the way it does them is so condescending it’s painful. I think there’s probably a cultural divide issue here, but there is no way in British English to say “Great job! You washed your hands for 30 seconds!” without it sounding like you’re being amazingly sarcastic or like you’re talking to a young child. So I turned all of that nonsense off. It’s meant to be a tool not a wannabe life coach.

You can access Maps directly on the watch and even do navigation. It works really well. The navigation mode has some nice haptic feedback: it does a short pulse as you’re approaching a turn, and then a long pulse at the actual turn. I like it a lot more than having to dig out my phone or have the directions read out. You get one pulse, glance down and see where you need to go, then it reminds you a little later when it’s time to do it. It’s a delightful user experience.

Watchfaces aren’t quite so delightful. There’s a limited number of built-in ones, and some are “exclusive” to the Ultra Series, and you can’t use them on a peasant watch like a Series 10. Annoyingly, there’s not one that does exactly what I want: a plain analogue clock with an inset date and four complication slots around the outside. Instead, I have to use one of the slots to show the date. There are third-party watchfaces, but they have issues. Firstly, there’s no actual API for making watchfaces⁹, so what they do is bodge it horribly by using a photo background that has fake widgets on it. On top of that they’re almost universally subscription-based. Again, no thanks¹⁰.

The issue I had with the Smart Band triggering when I was sleeping is solved trivially on an Apple Watch, by virtue of it having a physical button in the crown. When you put it in sleep mode, you have to double-press the crown to unlock it before it’ll do anything else. It hasn’t misfired once while I’ve been using it.

Sleep data; spot when I swapped devices!

Finally, a special mention for the gesture controls. If you raise the watch it reliably wakes up (switching from a low refresh rate, dimmed screen to an active, brighter one); you can then double-tap your index finger and thumb together and it will scroll down or page through whatever you’re looking at. The killer feature for this is navigating recipes: you can advance to the next step in a recipe while your hands are covered in flour. It’s also handy for reading notifications: when one pops up, you can double-tap to expand it, then, when it gets to the bottom, it’ll outline the default button (often “Dismiss”) and you can double-tap again to click it.

The verdict

I normally don’t like writing an actual labelled conclusion, but it feels like one is needed here! Overall, I’m happy with the watch. The daily charging doesn’t bother me, the data gathered seems reliable, the health monitoring has obviously paid dividends already, and the walled garden isn’t too chafing. It’s a straight upgrade over my old Smart Band, and I think it was worth the cost.

The original reason for getting the watch was better sleep tracking, though, so how well did it do? I’m much happier with the data: it seems to more accurately represent when I was awake in the night, and overall the sleep phases just seem to make more sense. You can see in the graph that the old data switched frequently between phases, and they didn’t quite line up for some reason; towards the right when the Apple Watch is supplying the data instead there’s a much more consistent pattern of sleep phases that repeat over the course of the night.

I’m not going to advocate that you go out and buy one, though. I know my requirements and usage aren’t typical, and I’ve also not got experience with any recent Android Wear alternatives or the new version of the Pebble watch that’s coming soon. You should definitely get your blood pressure checked, though!

OK, it’s more like 14 now, it’s taken a while for this post to make its way from my brain into text. ↩︎
Exercise for the sake of exercising just seems so overwhelmingly tedious and boring to me. And other types of exercise generally require social interaction, coordinating with people, and so on. ↩︎
Who decided to use “millimetres of mercury displaced” as a unit? You can’t just put random chemical symbols in units! That’s not how this works! ↩︎
Blood pressure readings have two parts: systolic (the pressure when the heart is beating) and diastolic (the pressure between those beats). They’re generally presented with the systolic reading on top and the diastolic reading below, and read as “X over Y”. Now you know what some of the random numbers they shout in medical shows mean! Yay learning! ↩︎
No thanks to how much salt is in everything. I’m pretty sure I’ve had at least twice the recommended daily amount of salt in a single serving before. Don’t even get me started on the things that are “low salt” but are still full of sodium from other sources. I don’t have a problem with ionic compounds, I have a problem with sodium! ↩︎
This mainly looks like drawing graphs that are slightly different to the graphs in the Apple Health graphs, for reasons I’m not sure I can explain. Making graphs is fun, OK? ↩︎
Complications are basically just home screen widgets, but with a fancy name because they’re on a watch. ↩︎
Tailscale actually causes me some problems here: everything works fine when the watch is connected to my phone, as the phone handles the Tailscale part, but if I’m not carrying my phone the watch will try to connect over WiFi directly and doesn’t understand anything about Tailscale. It happens infrequently enough that I’ll just live with it; it’s not much worse than having no signal on a phone. ↩︎
Yay walled gardens… ↩︎
I don’t object to subscribing to things in general, but it has to be something that’s worth the ongoing cost and offers something in return for the subscription. A watchface doesn’t need enough ongoing maintenance to justify subscribing to it, it’s just a cash grab. ↩︎

Finding an awkward bug with Claude Code

2025-06-30T00:00:00Z

The many times Claude thought it was right…

I recently encountered a bug in one of my projects that I couldn’t immediately figure out. It was an issue in Centauri, my reverse proxy. After its config was updated, I noticed¹ it stopped serving responses. Looking at the logs, I could see it was obtaining new certificates from Let’s Encrypt for a couple of domains, but I’d designed it so that wouldn’t block requests (or so I thought). After a few minutes of confusion, everything went back to working.

It felt like an issue with locking, but a quick glance didn’t show me anything obvious. I decided to throw the problem at Claude Code and see how it did. With some guidance² it managed to identify the cause, but it was a bit of a journey to get there. I’m going to share the prompts and responses, as it was an interesting process.

The initial prompt

I thought the issue might be quite obvious, and I’d just overlooked it. Centauri isn’t that complex a project, after all. So I gave Claude a fairly minimal prompt to set it off:

> Hello! Please examine this codebase. I have noticed an issue that when a
  certificate is being renewed, the application seems to not serve any other
  requests. I suspect there is a misplaced lock somewhere.
 
  Can you see what's happening? Think very hard.

Claude Code has special hardcoded triggers to engage “thinking” mode, which makes it take more time to think about alternatives. The keywords are: “think”, “think hard”, “think harder”, and “ultrathink”. I managed to not quite get the right incantation here, and only enabled the lowest level³.

I’ll omit all the responses where Claude is “thinking” (talking to itself), or decides to make todo lists then cross items off them then add different items, and so on. Its responses get fairly lengthy; I won’t judge you too harshly if you skim over them. Its first response was blessedly brief:

● I found the issue! Looking at the certificate manager code, I can see the
  problem:

… and then it went back to thinking and reading files, because it had neither found the issue nor could it see the problem. Ho-hum. Some time later it came back with:

● I found the issue! The problem is in certificate/manager.go:53-54:
  
  m.store.LockCertificate(subject, altNames)
  defer m.store.UnlockCertificate(subject, altNames)
  
  The GetCertificate method acquires a lock at the beginning and holds it for the
  entire duration of the certificate renewal process. This includes:
  
  1. Contacting the ACME server
  2. Solving DNS-01 challenges (can take minutes for DNS propagation)
  3. Obtaining the certificate
  4. Getting OCSP staple
  5. Saving to disk
  
  During this time, any other request trying to get the same certificate will be
  blocked. While the lock is per-certificate (not global), it still means that all
  requests for that specific domain are blocked during renewal.
  
  Let me check if there are other places where this might cause blocking:

And it then proceeded to do even more thinking, checking files, etc, until it concluded with:

● I found the root cause! The issue is actually worse than just a
  per-certificate lock. Here's what's happening:
  
  The main blocking issue is in proxy/manager.go:133-144:
   
  func (m *Manager) CheckCertificates() {
      m.lock.RLock()  // ← Acquires read lock on the entire manager
      for i := range m.routes {
          route := m.routes[i]
          // ...
          m.updateCert(route)  // ← This can take MINUTES for ACME renewal!
      }
      m.lock.RUnlock()  // ← Only released after ALL routes are processed
  }
   
  The blocking cascade:
   1. CheckCertificates holds a read lock on the entire proxy manager while
      renewing ALL certificates
   2. Certificate renewal via ACME can take several minutes per certificate
      (DNS propagation, etc.)
   3. Every incoming request calls routeFor (proxy/manager.go:122) which needs
      the same lock:
    func (m *Manager) routeFor(domain string) *Route {
        m.lock.RLock()  // ← BLOCKED until CheckCertificates completes!
        match := m.domains[strings.ToLower(domain)]
        m.lock.RUnlock()
        // ...
    }
  
  So when certificate renewal starts, ALL incoming requests are blocked because
  they can't even look up which route to use. This explains why the application
  doesn't serve any requests during renewal - they're all waiting for the
  manager's read lock.
  
  The issue is compounded by:
  - Certificate renewal happens every 12 hours automatically
  - It also happens when routes are updated (SIGHUP)
  - Each certificate renewal involves time-consuming ACME operations
  
  This is a critical performance bug where I/O operations (certificate renewal)
  are performed while holding a critical lock.

This doesn’t actually follow on from the earlier response, it’s completely flipped onto a different lock being the issue. Amusingly, it’s nearly right here, but it’s highlighting this code for completely the wrong reason.

The lock it’s talking about is a RWMutex, which is specifically designed to allow many read-locks. Everything it’s pointing out are read locks, so they cannot block one another as it described. The response Claude has given sounds very confident, seems appropriately technical, but is in fact complete rubbish. This is one of the big issue with all LLMs at the minute, and it makes me despair at the idea of people “vibe coding” things when they don’t know enough to push back against nonsense like this.

Course correction

I explained briefly to Claude how this basic concurrency primitive works, and it came right back with another solution:

> But that's a RW Mutex. You can hold many read locks concurrently.

● You're absolutely right! I apologize for the confusion. With a RWMutex,
  multiple readers can hold the lock concurrently. Let me trace through this
  more carefully.
  
● Task(Analyze updateCert method)
  ⎿ Done (8 tool uses · 20.0k tokens · 1m 11.9s)
  
● Excellent analysis! You're right about RWMutex allowing concurrent readers.
  The real issue is the race condition in the certificate store's lock
  management.

  Looking at certificate/store.go:82-91:

  func (j *JsonStore) lockFor(subjectName string, altNames []string) *sync.Mutex {
      key := strings.Join(append([]string{subjectName}, altNames...), ";")

      if mu, ok := j.locks[key]; ok {
          return mu
      } else {
          mu = &sync.Mutex{}
          j.locks[key] = mu  // ← Concurrent map write!
          return mu
      }
  }

  This is a classic concurrent map access bug. When multiple requests hit the
  same domain during certificate operations:

  1. Multiple goroutines call lockFor() simultaneously
  2. They read/write to j.locks map without synchronization
  3. Go runtime can:
    - Panic with "concurrent map writes"
    - Corrupt the map structure
    - Block indefinitely if map internals get corrupted

  This explains the blocking behavior - it's not the RWMutex, but rather a race
  condition causing either panics or deadlocks when the map gets corrupted
  during concurrent certificate operations.

  The fix would be to either:
  1. Add a mutex to protect j.locks map access
  2. Use sync.Map instead of a regular map
  3. Pre-initialize all locks at startup

I do appreciate when me stating a basic fact is called out as “excellent analysis”. Definitely doesn’t make me feel like I’m having smoke blown up my ass.

I think the issue it’s describing here is actually a problem⁴, albeit one that’s unlikely to be hit. But it’s definitely not the one we’re looking for. Claude even describes the possible consequences of the bug, and they don’t match what happened at all.

Maybe logs will help?

While Claude was chugging away, I’d been looking into the issue as well. I had logs from when it happened, and figured Claude may be able to glean something from them too. The first thing I’d noticed is that I wasn’t logging when a process finished, so I’d already added that to the code base, hence the note so Claude doesn’t get itself confused:

> That may be an issue, but it's not the one I was looking for. I've found logs
  from the time it happened. Note that a few logs have been added to the
  codebase since (so we can't tell from the logs when ACME or OCSP processes
  *finished*, just when they started). Here are the logs:
  
  {"time":"22:40:53.187887976Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:40:53.19947532Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:40:53.199507942Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:09.048465968Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:09.048482309Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:17.419299795Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:17.419325744Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:23.238027386Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:41:29.994330646Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:29.994353329Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:43.658794811Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:43.658819548Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:57.635325513Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:57.635347515Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:10.52439779Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:10.524417107Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:23.159430249Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:23.159452171Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:36.347407326Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:36.347433295Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:52.544628775Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:52.544653993Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:25.409579107Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:25.409602842Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:43.819479448Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:43.819502041Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:54.358957563Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:54.358979355Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:44:06.180278128Z","level":"INFO","msg":"Renewing certificate", <big list of alt names>}
  {"time":"22:44:06.180302585Z","level":"INFO","msg":"Starting ACME process to obtain certificate", <big list of alt names>}
  {"time":"22:45:03.830331922Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.830528134Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.832388931Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.225854865Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488432434Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488439397Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:30.617614536Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}

I’ve cut down the logs a bit to remove the specific domains, simplify the timestamps, and remove a bunch of other cruft. The key points are there were two SIGHUPs received, and the last certificate before everything started breaking covered 6 different hostnames.

Claude then went through a phase of periodically shouting “Eureka!” then deciding it hadn’t actually found anything yet:

● Now I see it! The key insight is in the timing and the "context canceled"
  errors. Let me trace through what's happening:
  
● Ah, now I see what's happening! Looking at the logs more carefully:

  1. 22:40:53 - First SIGHUP received, routes updated
  2. 22:40:53 to 22:44:06 - Multiple certificates start renewing (14 certificates!)
  3. 22:41:23 - Another SIGHUP received while renewals are still in progress
  4. 22:45:03 onwards - "context canceled" errors start appearing

  The issue is that SetRoutes is being called AGAIN (at 22:41:23) while the
  previous CheckCertificates() goroutine is still running and renewing
  certificates. Let me check what happens in this scenario:
  
> (The context canceled errors are clients timing out because centauri wasn't
  proxying properly)
  
● Ah, now I understand! The "context canceled" errors are from clients timing
  out because Centauri wasn't proxying their requests. This confirms the
  blocking issue.

  Let me check how incoming requests actually get their certificates:
  
● Now I found the real issue! Looking at the timeline and the code:

  1. 22:40:53 - SIGHUP triggers SetRoutes() which:
    - Calls loadCertificate() for each route
    - Sets route.certificate to the existing certificate
    - Starts go m.CheckCertificates() in background
  2. 22:41:23 - Another SIGHUP while certificates are still renewing
    - SetRoutes() is called AGAIN
    - Creates NEW Route objects with route.certificate = nil initially
    - Replaces the old routes in m.routes
    - The old Route objects (with valid certificates) are replaced!

  The critical issue is in proxy/manager.go:42-65:

  func (m *Manager) SetRoutes(newRoutes []*Route, fallback *Route) error {
      // ... validation ...

      for i := range newRoutes {
          route := newRoutes[i]  // These are NEW Route objects!
          // ...
          m.loadCertificate(route)  // Try to load certificate
      }

      m.lock.Lock()
      m.routes = newRoutes  // REPLACES all routes!
      m.lock.Unlock()
  }

  When the second SIGHUP arrives:
  1. New Route objects are created
  2. For domains currently being renewed, GetExistingCertificate might fail or
     return expired certs
  3. These routes get route.certificate = nil and CertificateMissing status
  4. CertificateForClient returns nil for these domains
  5. TLS handshake fails → clients can't connect → "context canceled" errors

  The fix would be to either:
  1. Prevent concurrent CheckCertificates runs
  2. Copy certificates from existing routes before replacing them
  3. Make certificate renewal truly independent of route management

Midway through it had started searching for “context canceled” errors in the codebase, which doesn’t make much sense. That’s what happens when a client gives up waiting, so the corresponding backend request is terminated. You won’t actually find “context canceled” anywhere in Centauri, because it’s a feature of the Go standard library. I gave it a little nudge to try to stop it rabbit holing.

As far as I can tell, what it’s talking about in this response isn’t an issue. I guess it’s possible that there’s a race condition whereby a route would be replaced and end up with an old version of a certificate that was just being renewed, but it wouldn’t cause an issue in practice; it’d just get updated next time the certificates were refreshed.

There’s no way for routes to get a nil certificate in this scenario, as far as I know. But I don’t have the unfounded confidence of an LLM, so I simply said:

> I don't follow why the certificate would be nil?

Attempt the third

I expected Claude to just explain its reasoning, but it apparently didn’t have any reasoning so set off on another voyage of discovery:

● You're right to question that. Let me trace through this more carefully.

● Good point! Since GetExistingCertificate doesn't acquire locks, it should
  return the existing certificate even during renewal. Let me dig deeper:
  
● Now I see a potential race condition! Here's what could happen:

● I found it! There's a critical race condition that explains the blocking:

  The Race Condition

  1. Background renewal starts: CheckCertificates() → updateCert() →
     GetCertificate() acquires lock and starts ACME renewal
  2. SIGHUP arrives: SetRoutes() is called, which for each route calls
     loadCertificate() → GetExistingCertificate() → store.GetCertificate()
  3. The race:
    - store.GetCertificate() returns a pointer to the certificate in the slice
    - Meanwhile, the renewal completes and calls SaveCertificate()
    - SaveCertificate() calls removeCertificate() which modifies the slice while
      it's being read
    - This slice modification (append(j.certificates[:i], j.certificates[i+1:]...))
      can cause:
        - The pointer returned by GetCertificate to become invalid
        - Slice bounds issues
        - Memory corruption
  4. The result:
    - GetExistingCertificate might return corrupted data or nil
    - Routes end up with certificate = nil
    - TLS handshakes fail
    - Clients timeout → "context canceled"

  The Timeline Confirms This

  - 22:40:53 - First SIGHUP, starts renewal of 14 certificates
  - 22:41:23 - Second SIGHUP arrives (30 seconds later) while renewals are still running
  - 22:45:03 - First timeout errors (4 minutes after first SIGHUP)

  The 4-minute delay matches ACME renewal time. The second SIGHUP's SetRoutes
  likely read certificates while they were being modified by the first SIGHUP's
  renewal goroutines.

  This is a classic concurrent slice modification bug - no synchronization
  between readers (GetCertificate) and writers (SaveCertificate/removeCertificate).

Once again we have a very confident and technical answer that does not explain the symptoms we’re looking for. It’s also mostly rubbish again. GetCertificate and SaveCertificate are only called by code that is holding a lock for that certificate; they won’t interfere with one another. There may, again, be a small concurrency issue if different certificates are handled in parallel, but it’s definitely not the cause of our current problems.⁵

Working through these explanations and debunking them was annoying me at this point. I decided to give it once last chance, and used the “ultrathink” shibboleth:

> So there may be a concurrent modification issue, but that doesn't explain the
  symptoms. If a TLS handshake failed it wouldn't result in the clients timing
  out. Likewise if there was any kind of corruption there'd be a panic, not a
  period where it didn't work and then started working again. Engage ultrathink.

Fourth time lucky

I’m going to include a piece of the thinking here because it amused me so much:

✻ Thinking…

  Oh! I need to think about this differently. Let me trace through what happens
  when a request comes in:

Apparently it takes “ultrathink” to actually work through the problem logically? Amusingly it did actually get there:

● You're absolutely right. Let me think more carefully about this...

● EXACTLY! That's the issue. Looking at the timeline:

  1. 22:40:53 - First SIGHUP: CheckCertificates() acquires READ lock, starts
     renewing 14 certificates
  2. 22:41:23 - Second SIGHUP: SetRoutes() tries to acquire write lock, blocks
     waiting
  3. Incoming requests call routeFor() for route lookup, but Go's RWMutex blocks
     new readers when a writer is waiting (to prevent writer starvation)
  4. 22:45:03 - Clients timeout because requests can't even get past route lookup

  The key insight is Go's RWMutex writer starvation prevention: when a writer
  is waiting, new readers are blocked even though the current lock holder is
  also a reader.

  So the cascade is:
  - CheckCertificates (read lock) → certificate renewal takes minutes
  - SetRoutes (write lock) → waits for read lock to release
  - routeFor (read lock for every request) → blocked by waiting writer

  Result: All incoming HTTP requests hang at route lookup, never reach TLS
  handshake or proxying logic. Clients timeout waiting for any response
  → "context canceled".

  This perfectly explains:
  - No panics (no corruption)
  - No immediate TLS errors (never gets that far)
  - Requests hang then timeout (blocked on lock)
  - Eventually recovers (when certificate renewals finish)

  The fix would be restructuring to avoid holding the manager lock during slow
  certificate operations.

We’re back to the RWMutex again, but this time it’s dropping knowledge on me not the other way around: if there’s a writer waiting, then new readers are blocked. This does indeed perfectly explain the symptoms, and was something I’d overlooked. The write lock is only held briefly to update some struct members to prevent it from blocking the readers, but this behaviour coupled with the long-running read lock ends up doing that anyway.

The fix

After all this work it’s very tempting just to ask Claude to fix it, but I had almost no confidence it’d be able to maintain any of the concurrency requirements given how much it flailed around with bad explanations. But in the interests of science, I asked it to try. Here’s what it came up with:

   // CheckCertificates checks and updates the certificates required for registered routes.
   // It should be called periodically to renew certificates and obtain new OCSP staples.
   func (m *Manager) CheckCertificates() {
+    // Create a snapshot of routes to avoid holding the lock during slow certificate operations
     m.lock.RLock()
+    routes := make([]*Route, len(m.routes))
+    copy(routes, m.routes)
+    provider := m.provider
+    m.lock.RUnlock()

+    // Process certificate renewals without holding the manager lock
-    for i := range m.routes {
+    for i := range routes {
-      route := m.routes[i]
+      route := routes[i]
 
       if provider == nil {
         route.certificateStatus = CertificateNotRequired
       } else {
         m.updateCert(route)
       }
     }
-    m.lock.RUnlock()
   }

   // updateCert updates the certificate for the given route.

That’s about what I expect. It definitely fixes the exact issue described, but it’s a bit… sloppy. Routes are now being updated when the lock isn’t being held, which is a change in semantics. It also doesn’t address the underlying issue that requests can be blocked if there’s a pending writer, and pending writes can be blocked by readers. A better approach would be to allow these operations to happen without blocking, maybe by using an atomic pointer to an immutable slice. I guess I’ve got some coding to do!

As for Claude: while the journey was frustrating, I think trying to figure this out by myself would have been even more so. It’s one of those things where everything looks right to me, and I’d have to go down a fairly long chain of checking my assumptions to find where I’d gone wrong.

There’s definitely a lot of room for improvement in coding LLMs: being able to estimate their own confidence level instead of stating everything as absolute truth would go a long way. I imagine there’s also a lot of benefit of having one LLM vet the responses of another; that’s sort-of how thinking mode works, but it doesn’t seem to go far enough to stop it outputting rubbish. It’ll be interesting to see how they develop in the future, anyway.

read as “I got spammed by a bunch of monitoring notifications” ↩︎
read as “Me repeatedly telling it that it was talking rubbish” ↩︎
If only we had some kind of language model technology that could understand this kind of thing. ↩︎
I sense some fun times with Go’s race detector in my near future ↩︎
The most likely outcome from these concurrency issues is Centauri panicking, which isn’t great but would only interrupt ongoing connections before it restarts and starts serving traffic again. Much less problematic than the “block for five minutes” behaviour I’m trying to fix. ↩︎

If all you have is a hammer…

2025-06-18T00:00:00Z

This, dear reader, is a hammer. It is almost entirely irrelevant to the article. Enjoy.

I presume everyone is familiar with the idiom “if all you have is a hammer, everything looks like a nail”. If not, well, there it is. It’s generally used pejoratively about being single-minded, but I think it also gives a glimpse into something more interesting: mental and perceptual sets.

Before I explain, let me tell you a story about a person who bought a 3D printer¹. When they were first thinking about getting one, they weren’t sure if it was worth it. They could print a few board game accessories, but then what? After they got it, though, a whole new world opened up: everywhere they looked there were opportunities to improve things by adding 3D printed plastic. Broken appliances were repaired with 3D printed parts, all sorts of shelves, organisers, hooks and other things were made. But when they talked about this to other people, most often the response was “That’s nice, but I don’t think I’d use one”. How could they not see the truth in all its glorious layer lines?!

The answer is in the concept of a ‘set’², and instead of trying to explain it, I’m just going to quote Wikipedia:

In psychology, a set is a group of expectations that shape experience by making people especially sensitive to specific kinds of information. A perceptual set, also called perceptual expectancy, is a predisposition to perceive things in a certain way. […] A mental set is a framework for thinking about a problem. It can be shaped by habit or by desire. Mental sets can make it easy to solve a class of problem, but attachment to the wrong mental set can inhibit problem-solving and creativity.

This perfectly captures what happened. They’d developed a ‘3D printing mental set’: a predisposition to see problems that can be solved with their ~~hammer~~ 3D printer. Once I started noticing this pattern, I started seeing it all over the place.

Sets at work

Aside from the “could this be improved with some 3D printed plastic” perceptual set, I’ve noticed a couple of other sets: DIY, and writing puzzles.

DIY is probably the most obvious one. Once you become a little familiar with the basics of what you can fix and how, it seems like you start noticing the opportunities more. Before, you might have ignored that loose door handle, but now you notice it and realise with a screwdriver and a sex bolt³ you can fix it and stop it coming loose again anytime soon.

As you do more things, you build up your set of mental sets as well. You learn how to attach things to different types of walls, how to safely work with electricity, and so on, and suddenly installing a new outside light goes from a seemingly complex job to just exercising some of those concepts you’re already familiar with.

This works across ‘disciplines’ as well: I wanted to automatically cool a room down, using a fan in the window to blow in cool air. The 3D printing mental set gave me a way to make it physically go where I wanted it to, the home automation set gave me a plan for how to automate it, and the electricity set helped me figure out how to wire in the relay. It sounds a bit stilted when I spell it out like that, but the catch is I didn’t have to sit down and work any of this out. I was already primed to think in a certain way. I had a whole selection of hammers, and could pick the right combination of them to beat this particular nail⁴.

Puzzles are a bit different. I enjoy taking part in puzzle hunts⁵, and at first I was bewildered at how the authors came up with all the puzzles. After I started thinking about writing some myself, though, I began to notice all kinds of things that could maybe be turned into a puzzle: a specialised database, or a song with repetitive lyrics, or set of things with unique ID numbers. Writing the actual puzzles is still tricky, but once you’ve equipped the “look out for puzzle material” perceptual set it’s much easier to gather a range of ideas.

For example, one puzzle I wrote came about because I stumbled upon Sir Patrick Stewart’s Cowboy Classic Sampler⁶. I think everyone will agree that it’s a fairly unique collection. I tied some samples of the songs together with some screen captures of Sir Patrick as Picard in Star Trek: The Next Generation, and puzzlers had to find the season and episode numbers of the pictures, then use those numbers to select a track and a word from that track in the audio mix. Those words spell out a question, and the answer to that question is the answer to the puzzle.

There were other hints mixed in to point you in the right direction, it wasn’t quite as opaque as it might sound! But the point is without being in that “puzzle finding” mindset when I came across the album, there’s no way I could have come up with that puzzle. Like with the fan, I was primed to think a certain way, and managed to create something I otherwise would have struggled with.

The curse of noticing things

Figuring out how to make stuff isn’t the only place sets appear, though. Perceptual sets are about how we perceive things. When you read something with lots of bullet points, em-dashes, and the word “delve” do you automatically think it’s LLM generated?⁷ Then you have an LLM text perceptual set! It’s kind of annoying, as it’s difficult to “turn off” and stop noticing these things once you’ve seen them. Similarly, I think I have a “bad grammar” set. I know it doesn’t bother a lot of people, but if I’m reading a book and there’s a rogue apostrophe it will throw me right out of the moment. It’s like tripping up when walking and being torn out of a daydream. There’s some use to these — being able to notice grammar errors easily helps with my writing — but for the most part they’re a negative input: they point out things are broken or bad.

The “edge case set” takes this to another level. After about 20 years of writing software⁸ I have a finely honed ability to spot problems. Give me something to implement and my brain will immediately rattle off all the different edge cases and under-specified parts. Then some other bit of my brain gets the job of deciding which we actually care about, which are worth raising, and so on. It’s definitely useful to spot them up-front, rather than when you’re deep in the weeds implementing something, and tends to make everything go a lot smoother if you can hammer the major ones out when planning work. It doesn’t turn off after work though; it comes up when reading board game rule books, or descriptions of the magic system in fantasy novels, or news reports, and so on. It’s a lot less useful then.

Speaking of news reports, here’s a fun one: Gell-Mann amnesia is a cognitive bias where even when you critically assess media reports about something you’re knowledgeable about (say, technology), you continue to trust similar reporting in other areas (say, medicine). But once you know about Gell-Mann amnesia, you might start to actively notice when it happens. You scoff at the article about how the Internet is a “series of tubes” and then start reading the next one about a cancer treatment, but — wait a minute — your Gell-Mann amnesia set kicks in. I’m not sure what you then do with this information. Dismiss basically all the media and news as useless? It’s honestly pretty annoying.

The meta set

I’ve had this blog post on my ideas list since I first noticed the “3D printing mindset”. I didn’t know sets were an actual thing, and I didn’t have most of the examples I do now. As time went on with this idea rattling around in the back of my head, I started noticing the patterns come up more and more. There’s lots of more examples I didn’t include because I didn’t want to just write a listicle of mental patterns. But the fact that I noticed them all seems to mean that I’ve developed a set set, or a meta-set, if you will. It’s fascinating to think about how all this happens inside our mushy bag of neurons, but also a bit terrifying. If thought processes are that malleable, what stops you accidentally breaking them? Actually, thinking about things like Q-Anon, flat earthers, and a lot of contemporary “politics”, maybe that’s quite easy to do?

Image credits

Photo	Creator	Licence	Source
Hammer	Ian Talmacs	Unsplash License	Unsplash

Surprise! I am that person. Bet you didn’t see that coming! ↩︎
I feel like they should have consulted the mathematicians and/or computer scientists before they picked a name. ↩︎
Yes, that’s a real thing. No, I won’t ever use any of the alternative names for it. ↩︎
OK, I’ve strained the metaphor enough. I’m sorry. I’ll stop. ↩︎
If you’re not familiar, a puzzle hunt is a competition where you try to solve puzzles faster than other teams. The sort of puzzles involved are perfectly described by this quote from Mike Develin: “Imagine a word search. Now imagine you aren’t told what words to look for. Now imagine you aren’t told it’s a word search. Now imagine it isn’t a word search.” ↩︎
Go and give it a listen. I’ll wait. ↩︎
I hereby certify all my em-dashes are hand-crafted. Relatedly, Claude Code keeps writing comments that use hyphens when they should actually be em-dashes. Maybe they’ve trained it to avoid them? ↩︎
I just made myself sad by working this out. I still feel young at heart! … Or maybe that’s cholesterol? ↩︎

Home Automation Without the Megacorps

2025-05-21T00:00:00Z

I first experimented with home automation in 2016, by picking up a Samsung “SmartThings” hub. It was terrible. The UI to configure things was slow and clunky, firmware updates were applied whether you wanted them or not, and everything stopped working if their cloud services stopped. You were also locked into whatever integrations they deigned to support, of course. After that broke for the umpteenth time I scaled back and for years the closest I got to home automation was a couple of Hue bulbs.

Recently I’ve been building it out again, though. This time using off-the-shelf components that interop using Zigbee, open-source software, and some code I wrote myself. It’s great; it runs entirely locally and has had basically zero downtime. The Zigbee ecosystem lets me integrate all sorts of things without having to spend lots of money on “smart” alternatives. I think I’ve spent less on this incarnation than I did on the original SmartThings hub all those years ago (even without adjusting for inflation!).

My current setup

I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based on the CC2652P chipset¹. Interfacing with the Zigbee stack is handled by zigbee2mqtt (z2m for short), an open-source project that basically bridges your devices to an MQTT broker². When a device reports some data, it will send a new message over MQTT; when you want to make a device do something you just post a message back. It’s incredibly lightweight, but supports a huge array of devices out of the box. And as it’s just using MQTT, it’s trivial to integrate with other software or build on top of.

I know a lot of people building out this kind of home automation use Home Assistant, but I don’t get on with it terribly well. It felt incredibly sluggish, and the entire project is just a bit… much. I want a binary or package or docker image I can just run, not an entire operating system. That’s not how computers are meant to work! In contrast, z2m is simple to set up, super light weight and responsive.

Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond this now, and I’m skipping some boring bits like error handling and JSON parsing, but at first I had something like:

for {
    message, topic, err := c.ReadSlices()
    
    if string(topic) == "zigbee2mqtt/desk-button" {
        if strings.Contains(message, "single") {
            c.Publish(nil, []byte("{\"state\": \"TOGGLE\"}"), "zigbee2mqtt/room-lights/set")
        }
    }
}

I think I actually made this harder to read by trying to simplify it here, but hopefully you can follow that it was relatively straight forward to listen for a particular action to happen (in this case me “single”-pressing on the device called “desk-button”) and then make another device do something in response (toggling the state of the “room-lights” device).

So what actually are these devices? At present in z2m I have the following:

Some generic buttons
An air quality sensor (USB powered), and a separate temperature sensor (battery powered)
A light switch
Some 240V relay modules, that I use to turn non-smart devices on and off
Some USB relay modules, for the same purpose
Some “smart” plugs that I mostly use to monitor power usage
A motorised blind roller

I think all of these are either Tuya or Moes branded, and were all sourced from AliExpress. You can get the same things from Amazon or elsewhere, but they tend to be 3-4x more expensive for the same product.

One of the cool things about Zigbee devices is that the powered ones work together to create a mesh network, so you don’t have to worry about network repeaters or signal strength like you do with Wi-Fi networks. z2m even makes a map showing the connections:

The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.

Some interesting automations

A Zigbee relay spliced into the power cable for the fan

So what do I actually do with all of this? Some of it is just what I’ll call “laziness automation”: I want to be able to turn on the lights without getting up and walking to the light switch. So there’s a Zigbee button on my desk that does it. Then there are some less manual automations: my blinds are automatically closed at sunset³, for example. It feels a bit trivial, but it’s surprisingly nice just not to have to think about that at all.

Another nice quality of life automation is the light I have above my monitor. It’s USB powered, and I now have it going through a Zigbee USB relay that lets it be toggled on and off. Then I have a small agent running on my desktop that turns the light on when the computer is unlocked, and off when it locks or shuts down. I keep meaning to make a “film mode” that detects when I’m watching a film, and turning all the lights and blinds down, but I haven’t got around to it yet.

The most complex automation is probably for a window fan. It’s not smart in any way, so I cut the power cable and inserted a Zigbee relay. The whole thing is housed in a little project box to keep it secure. The relay basically acts as a switch: the live wire leading to the fan runs to the “normally open”⁴ contact, while the upstream power goes to “common”. When the relay is told to turn the fan on, the two get connected and the fan gets power. Oh, it’s also not actually a window fan, but I designed and 3D printed some adapters so that it sits right in the window.

So now we have a window fan that can be turned off and on automatically. But I don’t want to have to press a button to do that. I’m lazy, remember? Instead, I made it so that the Go code constantly monitors the temperature reported by the air quality sensor, and queries the free OpenWeatherMap API to get the rough temperature outside. It can then turn the fan on if the room is too hot, and outside is cool enough to make a difference (there’s no point in blowing hotter air in!).

A graph of room temperature, showing the effect of the fan being turned on and off

I don’t want it to do that all the time though. The room doesn’t need to be cooled if I’m not around. There are various Zigbee presence sensors you can get, but the cheaper ones seem to be of dubious quality. Instead, I wrote some code to guess whether I’m present. It uses the state of the monitor light as a proxy for “is the computer in use”, and then makes some guesses based on the last time the computer was used and the current time (if I turn the computer off at midnight it probably means I’m going to sleep; if I turn it off at 2pm it probably means I’m going out somewhere).

You can see in the graph that the fan has a decent impact on temperature. I coded it to cool the room to 19 degrees, but then not turn on again until it was above 20. This prevents it flip-flopping on and off constantly. You can clearly see the pattern in the graph, as it actively cools and then slowly the room warms back up. This arrangement is much better than being woken up at 5am because it’s painfully cold. Trust me.

Bonuses: metrics and 3D printer

One of the things my custom Go code does is collate all the various stats reported by the Zigbee devices, and inserts them into a VictoriaMetrics database. I originally hosted this on the Pi itself, and it performed fine, but I’ve since moved it onto a server so that I can use it for some other things as well.

I set up Grafana to point to VM, and can create dashboards showing power usage, what devices are turned on when, and a bunch of environmental conditions. This also makes it easy to spot how good the data coming from the devices are. For example, the air quality sensor reports a figure for the amount of Carbon Dioxide in the air, as well as the amount of Volatile Organic Compounds (VOCs). The graphs are basically identical, but on a different scale. It turns out this particular device has no actual way of detecting CO₂, so it just “calculates” it from the VOCs figure. Useful to know if you want to actually use that data.

Recently I came across a go library for interacting with Bambu Lab printers, so I’ve also hooked that into my automation. It exports metrics about the printer, so I now have a way of seeing what’s going on when I’m not physically present⁵. I even added a HTTP endpoint (exposed over Tailscale) that shows pictures from the built-in camera.

On the automation front, I’ve made it automatically turn off the printer’s light when it finishes a print, and also added push notifications via PushOver whenever the state changes. No longer will I be sat in another room blissfully unaware it ran out of filament seconds after I walked out the door!

Was it worth it?

These days you could probably just buy some kind of ‘smart home’ hub that works well enough, and do most of what I’ve done with a lot less effort and no coding. I’m still of the opinion that for something so essentially local, it should itself be managed entirely locally. I don’t trust companies like Google or Amazon not to kill their products, or change or remove an API I rely on.

Obviously the “Not Invented Here” approach of coding everything yourself doesn’t suit everyone, but as someone who enjoys coding and enjoys having things work just so it works very well for me.

It turns out the chipset is important. I initially used a similar Sonoff dongle that used a CC2531 chipset, and its performance wasn’t great. It often failed to pair with new devices, and dropped links to existing ones. The zigbee2mqtt docs do explicitly advise against the CC2531 chips for that reason. At the time they recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d go with whatever their latest recommendation was. ↩︎
MQTT is basically a network protocol for publishing and subscribing to arbitrary messages. A broker is the thing that sits in the middle and routes the messages. I use Mosquitto but any will do. ↩︎
They originally also re-opened at sunrise but that was a terrible mistake. Who knew the sun rose so early?! ↩︎
“open” meaning “there’s a gap so it doesn’t work” not “open for business”. This confusion in terminology also extends to drawbridges. ↩︎
I use the printer in LAN mode, which means there’s no way to monitor it from a phone, even if you’re connected to the same network still. ↩︎

A year of boardgames

2024-07-26T00:00:00Z

I realised the other day that it’s a little over a year since I started building out a collection of board games¹, and thought it’d be fun to go through the collection and what I think about the games in hindsight.

The beginning

Ticket to Ride: Europe

I don’t remember how I discovered the concept of “modern” boardgames, but just over a year ago I purchased Ticket to Ride: Europe as my first. Prior to that I thought board games were like Scrabble, Monopoly, Trivial Pursuit, etc: they might provide a distraction occasionally, but didn’t amount to actual fun in the same way a video game might. I’d characterise those “classic” games as generally being extremely dependent on luck, not presenting many interesting choices, and generally being punishing if you’re behind.²

Ticket to Ride came as a breath of fresh air. You get to decide where to route your lines, whether you want to take on more cards, how aggressive to be in blocking the other players, and more. There’s still some luck involved, but it’s managed by being able to draw from a pool of visible cards, or being able to select which route cards you keep.

Shortly after I also picked up Carcassonne, and was similarly enamoured. I’ve not played either of them recently, though: they both fall into a weird little gap for me. While I enjoy the games, I don’t like playing them with certain groups³, but I don’t love them enough to prioritise them over other games when playing with other groups.

Pandemic

The next “gateway” game I picked up was Pandemic, which was my first co-operative game. I loved everything about it, and still do. The feeling when you managed to claw back a victory after setback after setback is great, but so is the shock when everything goes even worse than you thought possible!

Pandemic was also the first game where one of the mechanics caught my attention. The way epidemics work is that you take the bottom card from the infection deck, which becomes the center of the new outbreak, then shuffle that and all the discards back onto the top of the deck. It sounds simple, but it has big implications: everywhere that was previously infected can now be re-infected, and you’re guaranteed some tense rounds while you work back through those cards. It still blows my mind a little how shuffling and re-adding some card can mix up the game so much.

I still play Pandemic when I can, and it generally has a spot in my board game bag when I’m going away. I’ve since picked up the On the Brink expansion which adds some fun new twists⁴ to the base game, and Pandemic: Hotzone – Europe which plays similarly but is a bit quicker and smaller. I’ve also recently ordered Iberia, but it hasn’t materialised yet.

More gateway games

The Quacks of Quedlinburg

Over the next few months I picked up The Quacks of Quedlinburg, Azul (the mini variant), and Splendor. Quacks is probably my least favourite out of all the games so far: it’s not terrible, but it straddles deck-building and push-your-luck in a way that I don’t find particularly satisfying. It’s fun to play the combos you’ve invested in, and it’s fun to try and push your luck, but it’s not really fun when failing at the luck element means you don’t get to use the stuff you were building.

Quacks does bear the distinction of being the first game I customised, though: it got a little plastic organiser to hold all the ingredients instead of having to sort through a massive plastic bag with them all mixed in. My appreciation for game boxes with good built-in storage has gradually increased over the year, as have the tools I have for fixing those that don’t. But we’ll get back to that…

Azul is fun when you’re in the right mood; I’m not terribly into abstract strategy games, but Azul is light enough that it’s enjoyable. Splendor, on the other hand, I really like. So much so that I wrote a digital re-implementation of it. Both Azul and Splendor allow you to mess with your opponent a little, but not too much, which keeps it interesting but also means it still plays well in groups that don’t like too much head-on confrontation.⁵

I think the thing that really caught my attention in Splendor, though, is the quasi-engine building you do. It’s not actually classed as an engine builder on BoardGameGeek, but as you collect cards you get discounts on other cards, which lets you get better cards and further discounts, and so on. I love that kind of positive feedback loop in video games, and I apparently love it in board games as well.

Terraforming Mars

A recent game of Terraforming Mars

I’d heard lots of good things about Terraforming Mars, but I was worried it sounded a bit too complicated so put off buying it for a while. I mean, its weight on BoardGameGeek is orange. That’s scary, right?! Well… no. Terraforming Mars is easily my favourite board game, and I’ve played it pretty consistently since buying it. It has such a huge stock of cards that no two games feel the same: one game you might have an arsenal of cards that generate and use microbes; another you might have a plethora of science tags that let you play some of the most powerful project cards; next you might be spamming forests on Mars, or drowning in credits because you have too much income. It has the engine building element that I love, but I also really enjoy that it’s not always obvious who’s doing well and who’s not until the final score is tallied; that keeps everyone engaged and working on things for the entire game.

I’ve added in the Prelude expansion, which I wouldn’t play the game without now, and the Venus Next expansion, which adds some interesting bits but is nowhere close to being a hard requirement. I’ll probably pick up Colonies next. Prelude adds a new type of card that you take at the start of the game, which gives you a fairly hefty boost; these help you get off to a faster start, and may also affect what strategies you try to run with for the rest of the game. Venus Next adds another resource track, but you don’t have to complete it to end the game; it also adds a bunch of project cards that have “floater” resources, but they don’t feel that much different to the base game microbes⁶. One interesting thing that Venus Next adds, though, is a “solar phase” where one of the global parameters is increased at the end of each generation. The active player decides which parameter, but gets none of the benefits for it, so you get to decide entirely tactically.

There are two drawbacks to Terraforming Mars: it’s a little lengthy, and some of the components are a bit rubbish. Timing-wise, I find it takes a good hour and a half for a two player game, while adding a third player can almost double that. Obviously it depends who you’re playing with and what’s happening in the game, but that’s enough of a commitment that you more-or-less have to plan when to play instead of just spontaneously bringing it to the table. Component-wise, the game comes with basically no storage for anything other than little plastic bags that slosh around in the box; horrible plastic cubes for resource tracking that somehow jump out of your hand constantly, and flimsy printed sheets for putting those cubes on.

My custom player boards

The reason I got a 3D printer was in part so that I could make my own upgraded player boards that actually held the pieces in place instead of letting them fly everywhere if the table was slightly knocked. You can buy third party ones but I figured it was a teach-a-man-to-fish kind of situation, and dropping £200 on a 3D printer would save me money in the long run vs buying £50 of player boards someone else had made.⁷ About seven versions later, I have some boards which I think look and feel great. It’s 3D printed in two halves, then has a laminated 2D print sandwiched in the middle for all the markings and artwork.

As well as the player mats, I 3D printed some boxes, card holders, and so on, but have since replaced them with an insert from Folded Space. These are made of foam, and are perfectly sized to the game box and all the components in it. I could have eventually got to that point with 3D prints, but it would have taken a lot of time to design and print everything. Terraforming Mars was also the game that I first sleeved, as trying to shuffle the huge decks was an exercise in frustration. With all of the cards sleeved, not only are they more protected against wear and tear, you can now shuffle the deck just by gently mashing stacks of cards together. It makes shuffling so easy I now sleeve basically all my games as I hate having to riffle shuffle unsleeved cards.

Marrakech and HeroQuest

Later in the year I bought Marrakech after playing a giant version of it at a games expo. It sits somewhere between Azul and Splendor for me: it’s fun, and good when you want a short game, but it’s not high enough on my list to want to take it away with me. Playing with felt rugs does make a nice change from flipping cards or dealing with plastic components, though.

I also picked up HeroQuest at one point when it was on offer. It’s a good concept, and I felt like I would like it, but it was a bit of a let-down. The biggest problem for me is that you either need one player to basically be the DM, and manage the board, or you use an app to do that and have to deal with a bunch of admin and duplicating things because you’re half playing on the board and half in the app. I also don’t like relying on an app because I’ve been around long enough to know how easily computer software can become defunct, get removed from stores, get “updated” to remove features you need, and so on. I feel like the game would work well with five people: one to DM, and four to control the four heroes. But if you’ve got five people who want to play a dungeon crawler why aren’t you just playing D&D or a similar TTRPG? I’ll probably sell this at some point.

Party games

Codenames

These are all smaller games, and with most of them I’d played them before and decided I wanted my own copy. Codenames is my favourite party game, and I also picked up Codenames: Duet and they now live in the same box with a nice 3D printed insert. Codenames seems to just naturally produce fun moments: when an obscure clue is understood completely it feels amazing, but when it’s totally misunderstood it’s also generally hilarious. There’s also a great tension as the spymaster when you give a clue, then immediately realise it can be misinterpreted in a way that points straight at the assassin.

Poetry for Neanderthals is fun in the same party-word-game sort of way, but basically has no replayability. There are few enough cards that if you play a few times you can remember the “harder” word and just immediately guess it once you get the “easier” one being hinted (you can also sometimes just do this outright, which feels great but it kind of sucks from a gameplay point of view). The best bits are the inflatable “no stick” you get to bop people with, and how everyone speaks like a neanderthal for a little bit out of habit even after the game has ended. I seem to be weirdly good at this game, and I’m not sure what that says about me.

Love Letter has to take the spot of best game per cubic centimeter. It’s basically just a small set of cards, but it enables some really fun emergent gameplay. I really love how small the game is, and how easy it is to set up and play. It’s the perfect time filler. I’ve also played Lovecraft Letter, which adds a lot more complexity and was great fun, but is sadly out of print and I haven’t been able to find a copy at a sensible price.

Fluxx

Star Trek mega-Fluxx

My arbitrary 12 month period ends not long after I went to a tabletop event and played, amongst other things, Cthulu Fluxx, Doctor Who Fluxx, and Star Trek: The Next Generation Fluxx. Each game has the same core Fluxx elements, but adds slight twists along with the unique themes. I ordered a copy of Star Fluxx, but then found out that you can combine the four Star Trek variants (TOS, TNG, DS9 and VOY) together with some expansions to make a mega game so ordered all of those as well. While putting together my mega-Fluxx order I noticed they had a Firefly Fluxx that was out of print so I snapped up one of the last copies of that, too.

I ordered most of the games direct from Looney Labs, as the out of print ones weren’t readily available anywhere else. Unfortunately, UK Border Force managed to misread the import label⁸ and demanded £300 of VAT and import duties on them! They refunded it eventually, but this small set of tiny games were for a little while the most expensive things in my collection.

Fluxx is fun enough, but the mega-deck version is brilliant. It gets much more chaotic, with multiple captains, teleporters, and other doodads in play to steal cards back and forth. I was worried it wouldn’t play well with non-Trekkies, but the cards from each series use a different font and the weird people that can’t identify Star Trek series based on uniform variations seem happy enough relying on that.

Future

A few more months have passed since then, and my collection is still growing. Board game stores have summer sales, it turns out! If my physical wallet was in any way involved in the purchasing flow I’m sure it’d be crying.

I’ve recently started logging my plays with BG Stats (which synchronises the information up to Board Game Geek). I’m interested to see if the hard numbers actually align with my conceptions of how often I play certain games. If I do another roundup in a year’s time, expect it to have a lot more stats in…

Other words that may be appropriate here: “hoarding”; “compulsively obtaining”; “spending all my money on”, etc ↩︎
Think about a game of Monopoly: what do you actually get to decide? It basically comes down to “buy or not” if you land on an available property, then later on “buy more houses or not”. ↩︎
e.g. a family member doesn’t like the farmer mechanic in Carcassonne, and I don’t find the game interesting enough without them. Nothing wrong with the game or the people, we’re just not going to both enjoy it the way we like when playing together. ↩︎
Which is definitely why I bought it. Not because it comes with amazing little petri dishes to keep the disease cubes in. Certainly not. ↩︎
There’s a different edge to taking a card that someone else wanted in Splendor — there’s twelve others they can pick from, or they can take a face down one, etc; it might be an inconvenience but you haven’t totally upended their plans — versus blocking a really long train line they were trying to build in Ticket to Ride. ↩︎
Floaters are meant to be big pieces of infrastructure that float in the atmosphere of Venus. The fact that they feel exactly the same way as tiny little microbes when played seems wrong. ↩︎
Having since bought a more expensive printer, I think I’m past the point of convincing myself it’s “cheaper”. It’s a lot more fun, though, and if you design parts yourself you end up with exactly what you want! ↩︎
Turns out the “weight (lbs)” column is not the “quantity” column. Who knew? ↩︎