Chameth.com - posts like break-everything-fuzz-testing, debugging-beyond-the-debugger, securing-all-the-things-with-1password, why-you-should-be-using-https but not tailscale-docker-gotcha

Securing all the things with 1Password

2025-11-26T00:00:00Z

For many years I’ve been a keen user of Bitwarden. Recently I’ve had a lot of small paper-cut problems. The browser extension was redesigned and just doesn’t quite work how I expect any more. The prompt to save new login info misfired more than it worked. The mobile app stopped background refreshing properly. No one issue was enough to make me want to leave Bitwarden, but it definitely wasn’t sparking joy. Then there were rumblings about a price increase. I don’t think they’ve announced anything, but the numbers I saw were a jump from $10/year to $24/year. A nearly 2.5x price increase for a tool that was actively getting worse over time did not seem like a good deal to me¹.

I spent a while looking at alternatives, but there were very few that met my needs. Nordpass is highly praised but doesn’t support saving TOTP codes on personal plans². All the nice open-source/self-hosted solutions fall down when you want reliable synchronisation to an iOS device (which is fair enough). Proton Pass looked like it had all the features, but the pricing was asinine. Different pages showed different prices, the prices were introductory rates that jumped up on renewal, but that was hidden away in small print, etc. Yuck. The prices are also pretty high. More or less the same as 1Password, which I’d been discounting due to the cost, until now…

After eliminating basically every other option, I took a serious look at 1Password. It seemed to tick all the right boxes, I’d often heard good things about it, and it wouldn’t actually be much more expensive than Bitwarden if they went ahead with the rumoured price increase. I started a trial to give it a test.

Initial impressions

1Password is — as you’d expect for something that originated as a Mac app — very pretty. But it’s also highly functional. The looks don’t get in the way, they’re nice extras on top. When it’s locked, you get a large version of the 1Password logo, which is a front-on view of the barrel of a lock, next to the password input. When you authenticate, the barrel turns as though you’d put a key in and rotated it, and then the background splits in two and opens like a vault door. Does a password manager need to do that? No. Does it make me a little bit happy every time I see it? Yes. Look at it, it’s amazing!

The design philosophy carries through the rest of the app. Despite using Bitwarden daily, I often clicked on the wrong thing when I wanted to do anything other than autofill; in about a month of using 1Password I don’t think I’ve ever misclicked or been confused about the UI. The browser extension just feels snappier than Bitwarden, too. I’m not sure if it’s actually faster or if the design just makes it feel that way.

Functionality-wise, 1Password does everything Bitwarden did for me but slightly better. Bitwarden kicked you to its web interface for some operations (like checking compromised passwords); 1Password has that all built into the app. Instead of editing entries in a fiddly little browser popup, 1Password just summons forth the desktop app and you edit it there. It does a good job of detecting QR codes for TOTPs and offering to save them. I think the only small complaint I have is that the browser extension doesn’t have an autofill keybind. You can rig that up via the desktop app, but it feels like 1Password leans towards you using the UI it adds on login forms. To be fair, that UI works really well, and is very useful when you have multiple accounts on a site. My muscle memory for autofilling is strong, though.

Diving deeper

At this point I’d decided to switch fully to 1Password, so I started digging into what else it could do. The “Watchtower” feature combines a bunch of reporting about passwords (weak, breached, reused, and so on), but also lists sites where 1Password knows you can use passkeys or 2FA and don’t have them set up. I’d previously disabled passkey support in Bitwarden because it just appeared one day and got in the way of me trying to use a YubiKey, but decided to give them a go with 1Password. “Passkeys” is basically just a marketing term for FIDO2’s WebAuthn with automatic discovery bolted on top. The big problem is how you synchronise or export them. It’s all well and good letting your iPhone create a passkey when the Amazon app arbitrarily prompts it to, but what happens when you log in on a different device, or swap to Android, or whatever? Having 1Password deal with them fixes that³.

The passkey flow is very smooth: 1Password pops up an account selector, with obvious options to create a new account or pass the request through to another device like a YubiKey. It’s smart enough to guess the right account most times, even when you’ve got 17 different Google accounts for $reasons. Using them is equally simple: it just shows a popup and you tap to confirm you want to use the passkey. The Watchtower functionality made it really easy to go through and add passkeys everywhere that supports them. The thing I really appreciate is that they make the data that powers it public. It’s obviously partially a marketing thing for them, but it’s also a useful resource they could have kept proprietary.

Passkeys are a bit basic, though. 1Password has many more fun offerings. There’s an entire developer section with various goodies. At first I thought “that’s nice but I won’t use it”, but now I’m using basically everything it offers. The first thing to draw me in was the CLI. It’s pretty straightforward, but like the browser extensions it can nicely integrate with the desktop app so you unlock them all at once, deal with auth prompts in the GUI, and so forth. I have a couple of command-line tools that need passwords (e.g. for dealing with encrypted backups); now instead of prompting for the passwords, they shell out to op read and I authorise it via a popup in the 1Password app. Simple, but a nice quality of life improvement.

Next up there’s a beta feature called “Environments”. This lets you define groups of environment variables within 1Password, and have them exported automatically to certain destinations. At the minute it supports exporting to the AWS SecretsManager, and local .env files. When making a local file it actually makes a named pipe, so the credentials aren’t just sitting on disk (and won’t end up accidentally committed to anything). When the 1Password vault is locked, anything trying to read from the file will cause an authentication popup, much like with the CLI tool. I used to have the mentality that if something could read arbitrary files on disk, it was basically “game over” from a security point of view. Recent events made me realise there’s some value in defence-in-depth there, though, and 1Password provides a nice solution.

Fingerprints + Linux = Sadness

One feature of 1Password had me feeling a bit left out. Instead of entering your vault password to unlock it every time, you can have it invoke the system authentication library. On an iPhone this uses Face ID, on a Mac it can invoke Touch ID, and on Linux it uses Polkit. In the default configuration, Polkit just asks for your local user password, which is not much of an improvement. A fingerprint scanner seemed like the obvious solution here: it’s both a lot more convenient than typing a long password, and provides some assurance that it’s actually me doing the action⁴. Unfortunately, fingerprint readers on Linux seem to be a bit of a sorry affair.

Don’t get me wrong: there’s support. libfprint has a big list of supported devices, but I found it almost impossible to actually find a device I could buy that had one of the listed sensors. Most of them seem to be exclusively built into laptops. A lot of the cheaper devices just use whatever sensor they can get their hands on, so it’s pot luck if you receive a version that will work with libfprint or one that won’t. There’s also not a lot of information beyond that published by libfprint. I guess it makes sense: external fingerprint readers just aren’t a thing most people care about. While that makes me feel all special, it’s somewhat prohibitive to getting an actual working solution.

I’d given up on finding an answer to this when I stumbled upon an article by Scott Laird. He uses a YubiKey Bio to do it, via pam_u2f. The Bio isn’t actually a fingerprint reader: it just has an on-board reader to secure the credentials. The OS doesn’t get involved in the process. But because it can provide a fingerprint-secured U2F key, and pam_u2f can require that key to perform system operations, you basically get the same result. Maybe it’s actually even better than using a “real” fingerprint reader, as the data is secured on the hardware device…

The YubiKey Bio was a bit more pricey than I really wanted, coming in at around £90, but it felt like a worthy investment to add a bit of security and avoid typing a password dozens of times a day. The setup was simple: the official app that you use to enrol fingerprints is packaged in the Arch User Repository, and it worked perfectly. I already had pam_u2f set up as I used a (non-Bio) YubiKey to gate access to sudo, so I just generated a new key on the Bio, updated my config to use that, and added it to the Polkit policy. Now when I need to unlock 1Password or authenticate something, it prompts me to scan my fingerprint. Perfect!

Biometric all the things \o/

One of the other things Scott mentioned in that article was SSH keys. I’d seen 1Password supports acting as an SSH agent, but didn’t really see why I’d want it to. I used a resident key on my old YubiKey, so I didn’t have much to gain from a security point of view. Scott’s article made me realise I could have some big convenience gains, though. Currently every time I need to use the key, I enter a PIN and then physically touch it. I don’t mind this for the most part, but it turns out git-lfs does something like four different requests every time you do a git operation, none of which reuse the same SSH connection for some reason. That gets painful when each request involves a PIN and a touch. I also liked the idea of using pam_rssh for sudo authentication, but didn’t want to have to do the PIN-and-touch dance every time I used it.

You can probably see where this is going: I generated an SSH key in 1Password, and then… spent literal hours disabling all the other SSH agents that were running on my system, and trying to figure out where SSH_AUTH_SOCK was being populated. I didn’t manage to figure that out in the end⁵, and ended up just overriding it to point at 1Password’s agent socket. I added the new key to GitHub, and did the usual ssh git@github.com test, and it worked perfectly. 1Password prompts to allow access to the key, showing the process requesting access, and authorising it requires a fingerprint read:

can haz ssh key?

I’m not sure why Kitty doesn’t get an icon in the prompt; other applications do. You can expand the details section to see the key fingerprint being requested, as well as the exact process name and PID requesting the key. With the proof of concept done, I generated a couple more keys: one for normal SSH, and one for gaining root access using pam_rssh. Using different keys ensures I get prompted for them separately, and means a malicious app couldn’t sneakily ask for a Git key but actually pivot to SSH into a server. Not sure that’s ever likely to be a problem, but it’s basically no additional work for a bit more defence-in-depth, so why not?

I also took a moment to configure pam_u2f for both lightdm and xfce4-screensaver, so I can now log in and unlock my PC with my fingerprint. It’s not quite as smooth as Touch ID on a Mac, but it’s pretty close.

As an aside: this is not the first time I’ve been in this situation. Both Spotify and Todoist have focused on features I don’t want or need (including, but not limited to, pretty much everything branded “AI”), and then hiked the price up afterwards. It’s not quite standard enshittification, but it’s clear that our incentives no longer align properly. ↩︎
Or didn’t, at the time I looked. It now says “All plans” on the website but the docs only exist for business accounts, and there doesn’t seem to be any announcement explaining the change. ↩︎
Having Bitwarden deal with them would also have fixed that, but it got in my way instead of helping me, so… ↩︎
or it’s not me, but I’ll be busy dealing with the presumably-undesired removal of one of my digits, so probably won’t care about whatever is happening on the computer. ↩︎
Although I’m pretty sure the answer starts and ends with “systemd, somehow”. ↩︎

10 Weeks with an Apple Watch 10

2025-09-09T00:00:00Z

My watch. Yes, I am available for wrist modelling opportunities.

Around ten weeks ago¹ I picked up an Apple Watch 10, and have been wearing it almost constantly since. It’s not my first Apple Watch — I had a Series 5 for a bit back in 2020 — but it’s the first time I’ve actually stuck with it. Ten weeks seems like an apt time to reflect on it.

Firstly, why did I even bother? Well, for a couple of years I’d been wearing a Xiaomi Smart Band 7, mainly to monitor my sleep stats and set alarms that won’t wake up everyone else nearby. Its battery life was fantastic — with notifications and other things turned off, I got about a month of use between charges — but actually using it felt like trying to order food via the medium of interpretive dance.

My biggest gripe was the screen lock. If I didn’t have the screen locked then I’d periodically trigger it during the night when I moved around and it came in contact with my chest or leg. With the lock enabled you had to deliberately swipe up from bottom to top to enable interaction, but it didn’t work reliably. When I wanted to adjust an alarm, I’d be stood swiping repeatedly trying to get it to respond. When you finally get it unlocked, the whole interface is just fiddly.

The other issue was the data quality. There were some nights when I’d been woken up, sometimes even getting up and moving around, and it just didn’t show it in the data. If it can’t even get whether I’m asleep right, can I trust anything else it says?

I spent a while researching the best devices for sleep tracking. The Oura ring came highly recommended, but it was expensive and required a subscription to do anything useful. No thanks! The Apple Watch was consistently rated pretty well, and I reasoned I could pick up a refurbished older unit. I’ve been using an iPhone as my daily driver for a while, so it’d fit right into my begrudged walled garden.

The Series 10 has a significant advantage, though: it charges much quicker than all the previous generations. On a 30-minute charge, the Series 10 can go from 0 to 60%; the 9 can only make it to 40%, and my old 5 a measly 30%. Shorter charge times means I’m far less likely to leave it on charge and wander off without it. In some ways the daily charging is more convenient than monthly: the wireless charger sits on my desk, and I plop the watch on it for a little while in the evening; I don’t need to dig out the weird pogo-pin connector that has vanished sometime in the last four weeks, then carefully arrange it so it stays attached.

How a watch maybe saved my life

One of the big features of the Apple Watch, like many other wearable devices, is health and fitness tracking. I didn’t think much about this, beyond the sleep data I wanted, at first. I’ve never had a particularly good relationship with exercise², but I do like some good statistics. I started going for walks more often to get more data and see the graphs of VO2 max and HR recovery gradually inch up. That wasn’t the most profound effect on my health, though…

The recent versions of the Apple Watch have a feature that monitors for sleep apnea, a disorder where you don’t breathe properly during your sleep. I knew I wasn’t sleeping great — that’s why I was paying attention to sleep tracking data — but was still a bit surprised to get a notification from Apple Health after wearing the watch for 30 days. It gives you a graph to print and take to your doctor. So I diligently booked an appointment and a few weeks later went to see my GP.

The appointment went about as you’d expect: talking about referral to a sleep centre for a study, and so on. Towards the end, the doctor took my blood pressure (sleep apnea can be caused by, and can cause high blood pressure, in a lovely little vicious cycle). I don’t think either of us expected anything; it was just one of those standard checks for a related problem. After taking the reading, the doctor looked at me and said, “I can’t let you leave with a BP like this”. Yikes!

Blood pressure readings are split into stages: normal is under 120 mmHg³ over 80 mmHg⁴, stage 1 is up to 140/90, stage 2 is up to 180/120, and above is simply called “crisis”. Guess where I was? Also, fun fact: depending on how exactly you count and attribute things, hypertension comes out as the single largest cause of death in humans. It doesn’t kill you outright, but contributes to strokes, coronary artery disease, heart failure, and lots of other lovely things you don’t want on your CV.

After doing a few more readings, it settled down to just under the “crisis” stage and into the “holy crap, start treatment immediately” stage instead. I won’t labour on much more about this, but things are definitely moving in the right direction now⁵.

While the Apple Watch didn’t literally save my life, it triggered the chain of events that led to this diagnosis and treatment. Who knows what would have happened had it remained undetected? Probably nothing good. Also, go check your blood pressure! It’s simple to do and simple to treat if there is an issue, but so many people are walking around with hypertension and not even realising.

Building gates in the walled garden

Even though the watch is arguably a life-saver, not everything is rosy. It’s an Apple product, so you’re firmly locked in a walled garden, jealously guarded by people wearing black turtlenecks. Fortunately, there are a few ways to make it less painful.

All the health and fitness data is stored in Apple Health. You can export data as a big messy file, but it’s a very manual process to do so and the data format is gnarly. Luckily, there’s an app for that! Health Auto Export can, well, automatically export health data. It does what it says on the tin. It can send the data to Home Assistant, over MQTT, or dump it in some cloud file services, but I just have it post it all to a REST endpoint on a service I wrote. Then I dump it all in a database and can do whatever I want with it!⁶

Another tool that’s more useful on the watch itself is Apple Shortcuts. This is their no-code “if-this-then-that”-type thing. You can make automations or shortcuts that run a number of tasks. I have a whole slew of them I access via a complication⁷ on my watch face: one which prompts for input and adds an item to my to-do list (swiping to write letters is surprisingly not horrible), one which lets me select from a bunch of pre-written ones (“charge kindle”, “take laundry out in 1 hour”, type things), one to log my weight into Apple Health, one which can open and close the blinds in my room, and so on. It’s a surprisingly robust and easy-to-use system and offers just enough freedom that I’m not constantly grating on the edge of the walled garden.

Shortcuts being able to initiate arbitrary web requests is the real killer feature for me. Anything I can’t do on the watch itself, I can just farm off to a web server and connect it up with a shortcut. No need to learn Swift or pay Apple for the privilege of being a developer! For a lot of things, like controlling the blinds or adding to-do items, I already had a HTTP endpoint available and exposed over Tailscale⁸. Adding it to the watch was just a case of entering the right things in the Shortcuts app.

Daily nitty-gritty

There are lots of other little bits and pieces that come up when using the watch daily. I don’t think I can bundle them up into a pleasing narrative arc, so instead please enjoy some disjointed paragraphs of observations.

The Apple Watch has a lot of nagging abilities. It can notify you about your fitness “rings”, prompt you to stand up every hour, count how many seconds you wash your hands for, etc. I think I’d dislike these just on general principle, but the way it does them is so condescending it’s painful. I think there’s probably a cultural divide issue here, but there is no way in British English to say “Great job! You washed your hands for 30 seconds!” without it sounding like you’re being amazingly sarcastic or like you’re talking to a young child. So I turned all of that nonsense off. It’s meant to be a tool not a wannabe life coach.

You can access Maps directly on the watch and even do navigation. It works really well. The navigation mode has some nice haptic feedback: it does a short pulse as you’re approaching a turn, and then a long pulse at the actual turn. I like it a lot more than having to dig out my phone or have the directions read out. You get one pulse, glance down and see where you need to go, then it reminds you a little later when it’s time to do it. It’s a delightful user experience.

Watchfaces aren’t quite so delightful. There’s a limited number of built-in ones, and some are “exclusive” to the Ultra Series, and you can’t use them on a peasant watch like a Series 10. Annoyingly, there’s not one that does exactly what I want: a plain analogue clock with an inset date and four complication slots around the outside. Instead, I have to use one of the slots to show the date. There are third-party watchfaces, but they have issues. Firstly, there’s no actual API for making watchfaces⁹, so what they do is bodge it horribly by using a photo background that has fake widgets on it. On top of that they’re almost universally subscription-based. Again, no thanks¹⁰.

The issue I had with the Smart Band triggering when I was sleeping is solved trivially on an Apple Watch, by virtue of it having a physical button in the crown. When you put it in sleep mode, you have to double-press the crown to unlock it before it’ll do anything else. It hasn’t misfired once while I’ve been using it.

Sleep data; spot when I swapped devices!

Finally, a special mention for the gesture controls. If you raise the watch it reliably wakes up (switching from a low refresh rate, dimmed screen to an active, brighter one); you can then double-tap your index finger and thumb together and it will scroll down or page through whatever you’re looking at. The killer feature for this is navigating recipes: you can advance to the next step in a recipe while your hands are covered in flour. It’s also handy for reading notifications: when one pops up, you can double-tap to expand it, then, when it gets to the bottom, it’ll outline the default button (often “Dismiss”) and you can double-tap again to click it.

The verdict

I normally don’t like writing an actual labelled conclusion, but it feels like one is needed here! Overall, I’m happy with the watch. The daily charging doesn’t bother me, the data gathered seems reliable, the health monitoring has obviously paid dividends already, and the walled garden isn’t too chafing. It’s a straight upgrade over my old Smart Band, and I think it was worth the cost.

The original reason for getting the watch was better sleep tracking, though, so how well did it do? I’m much happier with the data: it seems to more accurately represent when I was awake in the night, and overall the sleep phases just seem to make more sense. You can see in the graph that the old data switched frequently between phases, and they didn’t quite line up for some reason; towards the right when the Apple Watch is supplying the data instead there’s a much more consistent pattern of sleep phases that repeat over the course of the night.

I’m not going to advocate that you go out and buy one, though. I know my requirements and usage aren’t typical, and I’ve also not got experience with any recent Android Wear alternatives or the new version of the Pebble watch that’s coming soon. You should definitely get your blood pressure checked, though!

OK, it’s more like 14 now, it’s taken a while for this post to make its way from my brain into text. ↩︎
Exercise for the sake of exercising just seems so overwhelmingly tedious and boring to me. And other types of exercise generally require social interaction, coordinating with people, and so on. ↩︎
Who decided to use “millimetres of mercury displaced” as a unit? You can’t just put random chemical symbols in units! That’s not how this works! ↩︎
Blood pressure readings have two parts: systolic (the pressure when the heart is beating) and diastolic (the pressure between those beats). They’re generally presented with the systolic reading on top and the diastolic reading below, and read as “X over Y”. Now you know what some of the random numbers they shout in medical shows mean! Yay learning! ↩︎
No thanks to how much salt is in everything. I’m pretty sure I’ve had at least twice the recommended daily amount of salt in a single serving before. Don’t even get me started on the things that are “low salt” but are still full of sodium from other sources. I don’t have a problem with ionic compounds, I have a problem with sodium! ↩︎
This mainly looks like drawing graphs that are slightly different to the graphs in the Apple Health graphs, for reasons I’m not sure I can explain. Making graphs is fun, OK? ↩︎
Complications are basically just home screen widgets, but with a fancy name because they’re on a watch. ↩︎
Tailscale actually causes me some problems here: everything works fine when the watch is connected to my phone, as the phone handles the Tailscale part, but if I’m not carrying my phone the watch will try to connect over WiFi directly and doesn’t understand anything about Tailscale. It happens infrequently enough that I’ll just live with it; it’s not much worse than having no signal on a phone. ↩︎
Yay walled gardens… ↩︎
I don’t object to subscribing to things in general, but it has to be something that’s worth the ongoing cost and offers something in return for the subscription. A watchface doesn’t need enough ongoing maintenance to justify subscribing to it, it’s just a cash grab. ↩︎

Finding an awkward bug with Claude Code

2025-06-30T00:00:00Z

The many times Claude thought it was right…

I recently encountered a bug in one of my projects that I couldn’t immediately figure out. It was an issue in Centauri, my reverse proxy. After its config was updated, I noticed¹ it stopped serving responses. Looking at the logs, I could see it was obtaining new certificates from Let’s Encrypt for a couple of domains, but I’d designed it so that wouldn’t block requests (or so I thought). After a few minutes of confusion, everything went back to working.

It felt like an issue with locking, but a quick glance didn’t show me anything obvious. I decided to throw the problem at Claude Code and see how it did. With some guidance² it managed to identify the cause, but it was a bit of a journey to get there. I’m going to share the prompts and responses, as it was an interesting process.

The initial prompt

I thought the issue might be quite obvious, and I’d just overlooked it. Centauri isn’t that complex a project, after all. So I gave Claude a fairly minimal prompt to set it off:

> Hello! Please examine this codebase. I have noticed an issue that when a
  certificate is being renewed, the application seems to not serve any other
  requests. I suspect there is a misplaced lock somewhere.
 
  Can you see what's happening? Think very hard.

Claude Code has special hardcoded triggers to engage “thinking” mode, which makes it take more time to think about alternatives. The keywords are: “think”, “think hard”, “think harder”, and “ultrathink”. I managed to not quite get the right incantation here, and only enabled the lowest level³.

I’ll omit all the responses where Claude is “thinking” (talking to itself), or decides to make todo lists then cross items off them then add different items, and so on. Its responses get fairly lengthy; I won’t judge you too harshly if you skim over them. Its first response was blessedly brief:

● I found the issue! Looking at the certificate manager code, I can see the
  problem:

… and then it went back to thinking and reading files, because it had neither found the issue nor could it see the problem. Ho-hum. Some time later it came back with:

● I found the issue! The problem is in certificate/manager.go:53-54:
  
  m.store.LockCertificate(subject, altNames)
  defer m.store.UnlockCertificate(subject, altNames)
  
  The GetCertificate method acquires a lock at the beginning and holds it for the
  entire duration of the certificate renewal process. This includes:
  
  1. Contacting the ACME server
  2. Solving DNS-01 challenges (can take minutes for DNS propagation)
  3. Obtaining the certificate
  4. Getting OCSP staple
  5. Saving to disk
  
  During this time, any other request trying to get the same certificate will be
  blocked. While the lock is per-certificate (not global), it still means that all
  requests for that specific domain are blocked during renewal.
  
  Let me check if there are other places where this might cause blocking:

And it then proceeded to do even more thinking, checking files, etc, until it concluded with:

● I found the root cause! The issue is actually worse than just a
  per-certificate lock. Here's what's happening:
  
  The main blocking issue is in proxy/manager.go:133-144:
   
  func (m *Manager) CheckCertificates() {
      m.lock.RLock()  // ← Acquires read lock on the entire manager
      for i := range m.routes {
          route := m.routes[i]
          // ...
          m.updateCert(route)  // ← This can take MINUTES for ACME renewal!
      }
      m.lock.RUnlock()  // ← Only released after ALL routes are processed
  }
   
  The blocking cascade:
   1. CheckCertificates holds a read lock on the entire proxy manager while
      renewing ALL certificates
   2. Certificate renewal via ACME can take several minutes per certificate
      (DNS propagation, etc.)
   3. Every incoming request calls routeFor (proxy/manager.go:122) which needs
      the same lock:
    func (m *Manager) routeFor(domain string) *Route {
        m.lock.RLock()  // ← BLOCKED until CheckCertificates completes!
        match := m.domains[strings.ToLower(domain)]
        m.lock.RUnlock()
        // ...
    }
  
  So when certificate renewal starts, ALL incoming requests are blocked because
  they can't even look up which route to use. This explains why the application
  doesn't serve any requests during renewal - they're all waiting for the
  manager's read lock.
  
  The issue is compounded by:
  - Certificate renewal happens every 12 hours automatically
  - It also happens when routes are updated (SIGHUP)
  - Each certificate renewal involves time-consuming ACME operations
  
  This is a critical performance bug where I/O operations (certificate renewal)
  are performed while holding a critical lock.

This doesn’t actually follow on from the earlier response, it’s completely flipped onto a different lock being the issue. Amusingly, it’s nearly right here, but it’s highlighting this code for completely the wrong reason.

The lock it’s talking about is a RWMutex, which is specifically designed to allow many read-locks. Everything it’s pointing out are read locks, so they cannot block one another as it described. The response Claude has given sounds very confident, seems appropriately technical, but is in fact complete rubbish. This is one of the big issue with all LLMs at the minute, and it makes me despair at the idea of people “vibe coding” things when they don’t know enough to push back against nonsense like this.

Course correction

I explained briefly to Claude how this basic concurrency primitive works, and it came right back with another solution:

> But that's a RW Mutex. You can hold many read locks concurrently.

● You're absolutely right! I apologize for the confusion. With a RWMutex,
  multiple readers can hold the lock concurrently. Let me trace through this
  more carefully.
  
● Task(Analyze updateCert method)
  ⎿ Done (8 tool uses · 20.0k tokens · 1m 11.9s)
  
● Excellent analysis! You're right about RWMutex allowing concurrent readers.
  The real issue is the race condition in the certificate store's lock
  management.

  Looking at certificate/store.go:82-91:

  func (j *JsonStore) lockFor(subjectName string, altNames []string) *sync.Mutex {
      key := strings.Join(append([]string{subjectName}, altNames...), ";")

      if mu, ok := j.locks[key]; ok {
          return mu
      } else {
          mu = &sync.Mutex{}
          j.locks[key] = mu  // ← Concurrent map write!
          return mu
      }
  }

  This is a classic concurrent map access bug. When multiple requests hit the
  same domain during certificate operations:

  1. Multiple goroutines call lockFor() simultaneously
  2. They read/write to j.locks map without synchronization
  3. Go runtime can:
    - Panic with "concurrent map writes"
    - Corrupt the map structure
    - Block indefinitely if map internals get corrupted

  This explains the blocking behavior - it's not the RWMutex, but rather a race
  condition causing either panics or deadlocks when the map gets corrupted
  during concurrent certificate operations.

  The fix would be to either:
  1. Add a mutex to protect j.locks map access
  2. Use sync.Map instead of a regular map
  3. Pre-initialize all locks at startup

I do appreciate when me stating a basic fact is called out as “excellent analysis”. Definitely doesn’t make me feel like I’m having smoke blown up my ass.

I think the issue it’s describing here is actually a problem⁴, albeit one that’s unlikely to be hit. But it’s definitely not the one we’re looking for. Claude even describes the possible consequences of the bug, and they don’t match what happened at all.

Maybe logs will help?

While Claude was chugging away, I’d been looking into the issue as well. I had logs from when it happened, and figured Claude may be able to glean something from them too. The first thing I’d noticed is that I wasn’t logging when a process finished, so I’d already added that to the code base, hence the note so Claude doesn’t get itself confused:

> That may be an issue, but it's not the one I was looking for. I've found logs
  from the time it happened. Note that a few logs have been added to the
  codebase since (so we can't tell from the logs when ACME or OCSP processes
  *finished*, just when they started). Here are the logs:
  
  {"time":"22:40:53.187887976Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:40:53.19947532Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:40:53.199507942Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:09.048465968Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:09.048482309Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:17.419299795Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:17.419325744Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:23.238027386Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:41:29.994330646Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:29.994353329Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:43.658794811Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:43.658819548Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:57.635325513Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:57.635347515Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:10.52439779Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:10.524417107Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:23.159430249Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:23.159452171Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:36.347407326Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:36.347433295Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:52.544628775Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:52.544653993Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:25.409579107Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:25.409602842Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:43.819479448Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:43.819502041Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:54.358957563Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:54.358979355Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:44:06.180278128Z","level":"INFO","msg":"Renewing certificate", <big list of alt names>}
  {"time":"22:44:06.180302585Z","level":"INFO","msg":"Starting ACME process to obtain certificate", <big list of alt names>}
  {"time":"22:45:03.830331922Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.830528134Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.832388931Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.225854865Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488432434Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488439397Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:30.617614536Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}

I’ve cut down the logs a bit to remove the specific domains, simplify the timestamps, and remove a bunch of other cruft. The key points are there were two SIGHUPs received, and the last certificate before everything started breaking covered 6 different hostnames.

Claude then went through a phase of periodically shouting “Eureka!” then deciding it hadn’t actually found anything yet:

● Now I see it! The key insight is in the timing and the "context canceled"
  errors. Let me trace through what's happening:
  
● Ah, now I see what's happening! Looking at the logs more carefully:

  1. 22:40:53 - First SIGHUP received, routes updated
  2. 22:40:53 to 22:44:06 - Multiple certificates start renewing (14 certificates!)
  3. 22:41:23 - Another SIGHUP received while renewals are still in progress
  4. 22:45:03 onwards - "context canceled" errors start appearing

  The issue is that SetRoutes is being called AGAIN (at 22:41:23) while the
  previous CheckCertificates() goroutine is still running and renewing
  certificates. Let me check what happens in this scenario:
  
> (The context canceled errors are clients timing out because centauri wasn't
  proxying properly)
  
● Ah, now I understand! The "context canceled" errors are from clients timing
  out because Centauri wasn't proxying their requests. This confirms the
  blocking issue.

  Let me check how incoming requests actually get their certificates:
  
● Now I found the real issue! Looking at the timeline and the code:

  1. 22:40:53 - SIGHUP triggers SetRoutes() which:
    - Calls loadCertificate() for each route
    - Sets route.certificate to the existing certificate
    - Starts go m.CheckCertificates() in background
  2. 22:41:23 - Another SIGHUP while certificates are still renewing
    - SetRoutes() is called AGAIN
    - Creates NEW Route objects with route.certificate = nil initially
    - Replaces the old routes in m.routes
    - The old Route objects (with valid certificates) are replaced!

  The critical issue is in proxy/manager.go:42-65:

  func (m *Manager) SetRoutes(newRoutes []*Route, fallback *Route) error {
      // ... validation ...

      for i := range newRoutes {
          route := newRoutes[i]  // These are NEW Route objects!
          // ...
          m.loadCertificate(route)  // Try to load certificate
      }

      m.lock.Lock()
      m.routes = newRoutes  // REPLACES all routes!
      m.lock.Unlock()
  }

  When the second SIGHUP arrives:
  1. New Route objects are created
  2. For domains currently being renewed, GetExistingCertificate might fail or
     return expired certs
  3. These routes get route.certificate = nil and CertificateMissing status
  4. CertificateForClient returns nil for these domains
  5. TLS handshake fails → clients can't connect → "context canceled" errors

  The fix would be to either:
  1. Prevent concurrent CheckCertificates runs
  2. Copy certificates from existing routes before replacing them
  3. Make certificate renewal truly independent of route management

Midway through it had started searching for “context canceled” errors in the codebase, which doesn’t make much sense. That’s what happens when a client gives up waiting, so the corresponding backend request is terminated. You won’t actually find “context canceled” anywhere in Centauri, because it’s a feature of the Go standard library. I gave it a little nudge to try to stop it rabbit holing.

As far as I can tell, what it’s talking about in this response isn’t an issue. I guess it’s possible that there’s a race condition whereby a route would be replaced and end up with an old version of a certificate that was just being renewed, but it wouldn’t cause an issue in practice; it’d just get updated next time the certificates were refreshed.

There’s no way for routes to get a nil certificate in this scenario, as far as I know. But I don’t have the unfounded confidence of an LLM, so I simply said:

> I don't follow why the certificate would be nil?

Attempt the third

I expected Claude to just explain its reasoning, but it apparently didn’t have any reasoning so set off on another voyage of discovery:

● You're right to question that. Let me trace through this more carefully.

● Good point! Since GetExistingCertificate doesn't acquire locks, it should
  return the existing certificate even during renewal. Let me dig deeper:
  
● Now I see a potential race condition! Here's what could happen:

● I found it! There's a critical race condition that explains the blocking:

  The Race Condition

  1. Background renewal starts: CheckCertificates() → updateCert() →
     GetCertificate() acquires lock and starts ACME renewal
  2. SIGHUP arrives: SetRoutes() is called, which for each route calls
     loadCertificate() → GetExistingCertificate() → store.GetCertificate()
  3. The race:
    - store.GetCertificate() returns a pointer to the certificate in the slice
    - Meanwhile, the renewal completes and calls SaveCertificate()
    - SaveCertificate() calls removeCertificate() which modifies the slice while
      it's being read
    - This slice modification (append(j.certificates[:i], j.certificates[i+1:]...))
      can cause:
        - The pointer returned by GetCertificate to become invalid
        - Slice bounds issues
        - Memory corruption
  4. The result:
    - GetExistingCertificate might return corrupted data or nil
    - Routes end up with certificate = nil
    - TLS handshakes fail
    - Clients timeout → "context canceled"

  The Timeline Confirms This

  - 22:40:53 - First SIGHUP, starts renewal of 14 certificates
  - 22:41:23 - Second SIGHUP arrives (30 seconds later) while renewals are still running
  - 22:45:03 - First timeout errors (4 minutes after first SIGHUP)

  The 4-minute delay matches ACME renewal time. The second SIGHUP's SetRoutes
  likely read certificates while they were being modified by the first SIGHUP's
  renewal goroutines.

  This is a classic concurrent slice modification bug - no synchronization
  between readers (GetCertificate) and writers (SaveCertificate/removeCertificate).

Once again we have a very confident and technical answer that does not explain the symptoms we’re looking for. It’s also mostly rubbish again. GetCertificate and SaveCertificate are only called by code that is holding a lock for that certificate; they won’t interfere with one another. There may, again, be a small concurrency issue if different certificates are handled in parallel, but it’s definitely not the cause of our current problems.⁵

Working through these explanations and debunking them was annoying me at this point. I decided to give it once last chance, and used the “ultrathink” shibboleth:

> So there may be a concurrent modification issue, but that doesn't explain the
  symptoms. If a TLS handshake failed it wouldn't result in the clients timing
  out. Likewise if there was any kind of corruption there'd be a panic, not a
  period where it didn't work and then started working again. Engage ultrathink.

Fourth time lucky

I’m going to include a piece of the thinking here because it amused me so much:

✻ Thinking…

  Oh! I need to think about this differently. Let me trace through what happens
  when a request comes in:

Apparently it takes “ultrathink” to actually work through the problem logically? Amusingly it did actually get there:

● You're absolutely right. Let me think more carefully about this...

● EXACTLY! That's the issue. Looking at the timeline:

  1. 22:40:53 - First SIGHUP: CheckCertificates() acquires READ lock, starts
     renewing 14 certificates
  2. 22:41:23 - Second SIGHUP: SetRoutes() tries to acquire write lock, blocks
     waiting
  3. Incoming requests call routeFor() for route lookup, but Go's RWMutex blocks
     new readers when a writer is waiting (to prevent writer starvation)
  4. 22:45:03 - Clients timeout because requests can't even get past route lookup

  The key insight is Go's RWMutex writer starvation prevention: when a writer
  is waiting, new readers are blocked even though the current lock holder is
  also a reader.

  So the cascade is:
  - CheckCertificates (read lock) → certificate renewal takes minutes
  - SetRoutes (write lock) → waits for read lock to release
  - routeFor (read lock for every request) → blocked by waiting writer

  Result: All incoming HTTP requests hang at route lookup, never reach TLS
  handshake or proxying logic. Clients timeout waiting for any response
  → "context canceled".

  This perfectly explains:
  - No panics (no corruption)
  - No immediate TLS errors (never gets that far)
  - Requests hang then timeout (blocked on lock)
  - Eventually recovers (when certificate renewals finish)

  The fix would be restructuring to avoid holding the manager lock during slow
  certificate operations.

We’re back to the RWMutex again, but this time it’s dropping knowledge on me not the other way around: if there’s a writer waiting, then new readers are blocked. This does indeed perfectly explain the symptoms, and was something I’d overlooked. The write lock is only held briefly to update some struct members to prevent it from blocking the readers, but this behaviour coupled with the long-running read lock ends up doing that anyway.

The fix

After all this work it’s very tempting just to ask Claude to fix it, but I had almost no confidence it’d be able to maintain any of the concurrency requirements given how much it flailed around with bad explanations. But in the interests of science, I asked it to try. Here’s what it came up with:

   // CheckCertificates checks and updates the certificates required for registered routes.
   // It should be called periodically to renew certificates and obtain new OCSP staples.
   func (m *Manager) CheckCertificates() {
+    // Create a snapshot of routes to avoid holding the lock during slow certificate operations
     m.lock.RLock()
+    routes := make([]*Route, len(m.routes))
+    copy(routes, m.routes)
+    provider := m.provider
+    m.lock.RUnlock()

+    // Process certificate renewals without holding the manager lock
-    for i := range m.routes {
+    for i := range routes {
-      route := m.routes[i]
+      route := routes[i]
 
       if provider == nil {
         route.certificateStatus = CertificateNotRequired
       } else {
         m.updateCert(route)
       }
     }
-    m.lock.RUnlock()
   }

   // updateCert updates the certificate for the given route.

That’s about what I expect. It definitely fixes the exact issue described, but it’s a bit… sloppy. Routes are now being updated when the lock isn’t being held, which is a change in semantics. It also doesn’t address the underlying issue that requests can be blocked if there’s a pending writer, and pending writes can be blocked by readers. A better approach would be to allow these operations to happen without blocking, maybe by using an atomic pointer to an immutable slice. I guess I’ve got some coding to do!

As for Claude: while the journey was frustrating, I think trying to figure this out by myself would have been even more so. It’s one of those things where everything looks right to me, and I’d have to go down a fairly long chain of checking my assumptions to find where I’d gone wrong.

There’s definitely a lot of room for improvement in coding LLMs: being able to estimate their own confidence level instead of stating everything as absolute truth would go a long way. I imagine there’s also a lot of benefit of having one LLM vet the responses of another; that’s sort-of how thinking mode works, but it doesn’t seem to go far enough to stop it outputting rubbish. It’ll be interesting to see how they develop in the future, anyway.

read as “I got spammed by a bunch of monitoring notifications” ↩︎
read as “Me repeatedly telling it that it was talking rubbish” ↩︎
If only we had some kind of language model technology that could understand this kind of thing. ↩︎
I sense some fun times with Go’s race detector in my near future ↩︎
The most likely outcome from these concurrency issues is Centauri panicking, which isn’t great but would only interrupt ongoing connections before it restarts and starts serving traffic again. Much less problematic than the “block for five minutes” behaviour I’m trying to fix. ↩︎

How I Get Things Done

2024-09-01T00:00:00Z

I always quite like reading about how other people do things. What software or hardware they use, or how they manage reminders, todo lists, and so on. I’ve never actually written about how I do any of that, though. So here it is!

Productivity

In the past I’ve fallen victim to the idea of there being One True Productivity System that would solve all my problems and make me amazing at getting things done. The title of this post is a nod to Getting Things Done, which I’ve read and tried to religiously follow in the past, but it’s just not for me. One thing that did actually stick from it, though, was the idea of “open loops”:

Anything that does not belong where it is, the way it is, is an “open loop,” which will be pulling on your attention if it’s not appropriately managed.

If I want to do something and it’s not recorded in a way I trust, it weighs on me a little. Those little weights all add up, and just make life uncomfortable. I deal with those in a few ways:

Inbox Zero-ish: if an e-mail needs me to do something, it sits in my inbox until it’s done. When it’s done, it gets archived.
Notes: random things I try to remember I put into my notes, so I can search for them later.
Budget: anything monetary I just adjust in my budget or create a category for.
Todo list: any other kind of task I want/need to do either now or at some point in the future goes into Todoist.

Todoist

In the past I’ve gone through a lot of different todo systems, once again hunting for one that will fix all my problems etc¹. In the end I always ended up coming back to Todoist, and have stuck with it for a decent period now.

In Todoist, I have everything split up into projects. Work projects are one colour and personal projects are another. Anything that needs more than a handful of individual tasks gets a project, which is archived when it’s done. Any one-off things that don’t neatly fall into projects go into “Random junk” project.

As I’m writing this, I have 116 items in 22 projects. A good deal of those are things that I might want to do at some point, not things that I actively need to worry about. If I want to get one of them done, I schedule it so it shows up in the “today” view. I have the Todoist desktop app open automatically when I login to my computer, and it gets positioned on the right hand side of my ultra-wide monitor, so I can see the “today” view and add/remove things immediately at any point.

I’m not precious about tasks becoming overdue, or unscheduling things I’d previously scheduled. Priorities change, ideas that sounded good a week ago might no longer sound good, and so on.

I do a few things to help me manage tasks and keep track of things:

I have a monthly recurring task to look through everything that’s not scheduled, just so I keep a rough idea of what’s there, and can update or get rid of it as needed.
I wrote todoistager and have it running on a server to automatically apply tags to tasks based on their rough age: weeks, months or years. This lets me identify things that are lingering and reconsider them².
I automate adding tasks for things that demand my attention, like incoming GitHub pull requests. The Todoist API is nice and straight forward, and hooking it up to things is trivial.

Other desktop software

For notes I use Logseq, and I just put whatever I need to write under today’s note and rely on search or tags to find it again when I need to. I note basically anything that I might like to use again in the future. One of my most common tags is #blub, a term I came across in the excellent article “In defense of blub studies”. Blub is random ultra-specific knowledge; it might not be eminently useful to know on its own, but over time it can pay dividends.

I’ve previously tried a bunch of note taking apps, and methodologies, but like with todo lists I end up just wanting something that just works and stays out of the way. Logseq stores notes in Markdown, and I have a cronjob to commit and push my changes to a private git repository.

Like Todoist, Logseq launches when I log into my computer, and sits in the same 500px-or-so region on the right of my monitor. This allows me to switch from Todoist to take or refer to notes when needed. To do this positioning, I use devilspie2. I also use this to position Discord and my IRC client side-by-side on a small portable monitor that sits underneath my ultrawide.

More mundane software: I use IntejjiJ IDEA for almost all development and complex text editing³. The only exception is Android development where Android Studio rules the roost. For lighter text editing, Sublime Text is my choice.

I spend a fair bit of the time at the command line. I use kitty as a terminal emulator, with the zsh shell and Oh My Zsh for some nice plugins and enhancements. One thing I couldn’t live without is autojump, which allows quickly jumping to any directory you’ve previously visited using a substring of the name.

Web browsing is all Firefox, with Bitwarden for password management and two pinned Fastmail tabs, one for personal e-mail and one for work. I don’t think there’s much else to note there.

Oh, I use Arch, by the way.

Infrastructure

This is the hodge-podge of things that support my computer usage but sit outside the actual computer. In no particular order…

All my DNS queries go through NextDNS using DNS-over-HTTPS. NextDNS lets you add custom responses, block certain things from resolving, and subscribe to pre-made anti-tracking or anti-advertising lists. It gives you far more control over everything than most other services I’ve looked at.

Every device I control runs Tailscale to enable remote access. I also use Tailscale’s SSH authentication feature for my servers, allowing quick and easy access from my phone where I don’t have or want normal SSH keys.

One exciting service that combines these two is golink. It was created by Tailscale, presumably based on the go service used extensively inside Google. I run a version on my tailnet, and have NextDNS configured to resolve go to the tailscale IP of the service. That lets me type “go/whatever” on any machine I set up with NextDNS and Tailscale, including mobile devices. I use go links for all sorts of things:

Shortcuts to tools I use commonly: go/board gets updated to my current client’s kanban board or similar tracker, whether it’s in Jira, GitHub, Asana or whatever; go/meet goes to Google Meet with the user parameter set so I’m logged in with my work account not my personal account.
In place of bookmarks: go/flex goes to the CSS-Tricks flexbox guide⁴, go/cad goes to the OnShape login page, go/hmrc goes to the HMRC online services page, and so on
A few dynamic things: go/github/chameth.com will go to the csmith/chameth.com project on GitHub; go/ref/pipico will go to the Pi Pico page on my electronics reference wiki.

For my servers I have a minimal Ansible setup that does some basic configuration and deploys SSH keys to the right places. Almost everything runs inside Docker, and I use my own software for HTTP proxying: Dotege uses tags I apply to containers to build a config for Centauri which handles reverse proxying, using Let’s Encrypt for TLS certs. At the minute I have two servers: a dedicated server that runs basically all my services (including this website), and a little VPS for monitoring and backups. Both are from Hetzner.

I wrote my own basic monitoring software called Goplum to make sure things that should be running on my servers actually are. If it fails checks it sends a notification to my phone using Pushover, and a message to a channel on my private IRC server (running Ergo).

Hardware

For the last two years I’ve used a laptop as my main computer⁵. It’s a Dell G15 Special Edition. After some upgrades its specs are:

Intel i7-12700H processor
NVIDIA GeForce RTX 3060 graphics card
2x32GB DDR5 RAM
2TB Gen4 SSD

At home it sits behind my Alienware AW3423DWF monitor. This is a Quantum Dot OLED monitor, and I bought it almost immediately upon seeing someone else use it at a LAN event. As did three other people that were sat near us. It just looks so good. I wasn’t sure I’d get on with a widescreen monitor, and made sure it supported picture-by-picture before buying it, but have never actually used that. I generally run whatever I’m working on at nearly-fullscreen, then have Todoist or Logseq on the far right. Games go fullscreen over everything, of course.

Above the widescreen I have a curved lightbar that I find helps reduce eyestrain and makes it easier to focus on the monitor. And on top of that I have a Logitech Brio webcam which gives a great picture with lots of control over pan/zoom/etc for meetings. I have a little set of systemd scripts that detect when the external webcam is connected and disable the laptop’s built-in webcam, and vice-versa when it’s disconnected; that means whenever I jump into a video call in any app there’s only one input and it’s always the right one, which is nice.

Below the main monitor and its friends, I have a generic 1080p portable monitor connected to the laptop via USB-C. This comes with me when I travel, and serves as a Discord/IRC monitor when at home. It’s handy for throwing other things on as well, like if I need to reference a design while writing code, and don’t want to keep alt+tabbing.

I have two Durgod K320 keyboards: one with Cherry MX Silent Red switches for when I’m at home and don’t want to annoy people with the sounds, and one with Cherry MX Blues for when I’m travelling and can make a bit more noise in exchange for a much nicer typing experience. They’re both tenkeyless so they can handily fit in a backpack.

I use a Zowie EC1-C wired mouse, which does the job well. I got fed up with mice from more mainstream manufacturers failing: my previous mouse was a Razer Viper that lasted 7 months before it started losing clicks. All my audio goes through a HyperX Cloud Alpha Wireless, which has the most insanely good battery life of any wireless headset I’ve ever owned or seen. Their claim of 300 hours is not just marketing!

In non-computer hardware, I have a Bambu P1S 3D printer that I’ve mentioned in my blog before. For printing in fewer dimensions I have an Epson EcoTank ET-1810, which is surprisingly reasonable for an inkjet printer, and the ink costs aren’t as astronomical as normal cartridges. Plus, you can’t decide to DRM ink when it’s loaded raw into the printer…

I have a small Zigbee network set up mainly to control the lights and the old 3D printer I no longer use (the P1S has a sensible standby mode so you don’t have to yank its power away to make the fans stop spinning!). That’s run through a Raspberry Pi running Zigbee2Mqtt and a custom app I wrote to monitor events on MQTT and generate the appropriate responses (or just log the data).

My daily-driver phone is an iPhone 15 Pro, but I have a bunch of Android devices kicking around for work purposes. I read nearly every day on a Kindle Paperwhite Signature Edition.

Money and work

I use You Need A Budget for figuring out how much money I can spend on what, and making sure I have enough put aside for bills and taxes and so on. I use it for both business and personal accounts together, but I mark all the business categories and accounts with a 💼 emoji, and have some scripts that use the API to make sure the money split makes sense⁶.

Every recurring monthly and annual payment has a category, as do all upcoming events I plan to attend, and all the wonderful taxes I have to pay. I also have categories for hobbies and other expenditure. I think all of that is pretty typical for a normal YNAB setup.

I also use YNAB for tracking anything I might like to buy at some point: it gets a category in a “Wishlist” group, and I’ll occasionally budget to buy things from there. I try (and often fail) to use that as a way to put off impulse purchasing. As my income is often very inconsistent, I try to set aside money so that I can “spend” a bit every week on things in the wishlist group. That feels a lot nicer to me than just budgeting everything in one go as soon as I get paid, and gives me more time to consider whether I really want those things⁷.

For work, I do all the more technical accountancy and tax filing and so on through FreeAgent. For my fairly straight-forward tax situation, I can basically manage everything myself using FreeAgent without paying for an accountant. That does mean occasionally spending a bunch of time reading tax manuals, but I’d rather fully understand the rules than just follow advice anyway.

The problem with implementing productivity systems or switching todo apps or anything of that ilk is that it can feel like you’re being productive by doing so. It takes a while to realise that you’re not, and actually getting on with things is better. ↩︎
Or just accept that I’ll maybe do them at some point. I kept the labels vague to try and not make it feel judgmental. ↩︎
Including writing this blog post right now. ↩︎
One day I will get all the justify- and align- options right without looking them up, but until then… ↩︎
This is very normal for developers, but much less common for computer gamers. I’ll probably go back to a desktop next time I upgrade, as I’m travelling a lot less. ↩︎
I can’t just arbitrarily move money from the business to myself, it has to be given as salary or a divided etc, which has tax implications. If I budget “business money” for personal things, then it actually has to be properly transferred and accounted for before being spent (as I’m in effect budgeting a future paycheck). It would be cleaner to keep the accounts completely separate, but I find this way works well for me as I can be a bit more flexible. ↩︎
It feels a little like I’m giving myself pocket money, which is a bit weird, but it’s a lot better than having no “income” for several months when client invoices fall weirdly or I’m between contracts. ↩︎

How to break everything by fuzz testing

2020-04-26T00:00:00Z

Fuzz testing is a bit like the infinite monkey theorem, but instead of Shakespeare you get crashes.

Fuzz testing, if you’re not aware, is a form of testing that uses procedurally generated random inputs to see how a program behaves. For instance, if you were fuzz testing a web page renderer you might generate a bunch of HTML - some valid, and some not - and make sure the rendering process didn’t unexpectedly crash.

Fuzz testing doesn’t readily lend itself to all types of software, but it particularly shines in cases where some kind of complex user input is accepted and processed in some way - like the aforementioned web page renderer. I was recently adding a library to parse EXIF data to images to an Internet-facing service and realised it was a perfect opportunity to do some fuzz testing. Even if I didn’t find any issues, I’d improve my confidence that the library was safe enough to expose to the Internet.

Breaking my EXIF library

I wrote a quick harness to run go-fuzz on the library, and gave it some pre-existing demo files as sample input. The way go-fuzz works is that it instruments your code and then mutates the inputs to try to improve the coverage. For example, if I had some sample data that had an EXIF tag with a value of 1 then go-fuzz might change it to a 2 and see if the code follows a different path. In most cases it won’t but when it does, they tend to be very interesting cases.

One of the first issues that go-fuzz found was that some values in a maker note field would cause the library to panic (i.e., crash). This happened because there was a check to see if the first six characters were “Nikon” and a null byte, without first checking to see if there were actually six characters available. This is a kind of bug that doesn’t happen much with “real” data - as the field is either not present or completed correctly - but could easily be exploited once this code is exposed to the Internet.

Another interesting bug that go-fuzz found was that if a tag had a particularly large count, the library would try to allocate an obscene amount of memory and die. There was already a check in the code that was meant to avoid this exact scenario, but go-fuzz managed to find a way around it. Each tag has a size (for example an integer tag takes a fixed number of bytes) and a count; the existing check multiplied the two together and made sure that the result wasn’t too large. For most cases this was fine but go-fuzz found a case where the count was so large that when multiplied by the size of the tag it overflowed the integer and became negative, thus passing the sanity check but then subsequently failing when it came around to actually allocating the memory.

The final bug of note that go-fuzz found was the most interesting. EXIF data is stored in IFDs (“Image File Directories”), and each IFD provides what is effectively a pointer (a byte offset) to the next one. The EXIF library already had a check to make sure that these didn’t loop, but it only checked the immediately preceding IFD - so if IFD 1 linked to IFD 2, it would catch IFD 2 linking back to IFD 1 and break the loop with an error. Go-fuzz found that having three interlinked IFDs had the same issue, though, and the guard code wasn’t triggered. This created an infinite loop, maxing out a CPU core until the process was eventually killed - one of the worst kind of bugs you could have in an Internet-facing service which doesn’t deal with private data! The fix for this was fairly straightforward - I just made the library keep a record of the previously visited IFDs and bail out if it found a loop.

Breaking my IDE

When go-fuzz detects an issue it outputs not only the details of the problem (the stack trace, error message, and so forth) but also the input that generated the problem. This is useful for reproducing and making sure the issue is fixed, but it also makes it really easy to write a test to ensure that the behaviour never regresses in the future.

As I was working through fixing the bugs that go-fuzz found, I dutifully added new tests where needed. After adding the sample input with looping IFDs to the project, I switched to IDEA to write a test to use it. I clicked on the input file to copy the file name, and then the entire IDE hung and had to be restarted. Uh oh! When I restarted IDEA, it immediately began indexing the project and again hung. It turns out IDEA parses EXIF data (presumably, even if it does nothing else with the data, to get the rotation property for images), and the library they use - an independent one written on Java - had the same bug as the Go library I was using.

In order to stop IDEA from indexing the file and becoming unusable I renamed it from a ‘.tif’ extension to ‘.dat’, and everything went back to normal. I thought I’d best report the bug to JetBrains, though, so they could put a proper fix in.

Breaking YouTrack

JetBrains use their own issue tracker called YouTrack for reporting bugs in IDEA. I dutifully went over and described the problem, attaching the log files from the IDE, a description of how the file was malformed, and carefully selected the .dat version of the file to upload so that it wouldn’t cause anyone else the same immediate problem.

After trying to upload the file I got a strange error back. Uh oh! I submitted the IDEA issue as it stood, unable to see if the attachments had even uploaded, and went and wrote up an issue for YouTrack itself about the error message. While I was doing that, YouTrack seemed to slow down and become really annoying to use. I had a sinking feeling the exact same thing was probably happening as with IDEA and my library - but this time YouTrack had content-sniffed the file instead of relying on the file extension. In hindsight, I should’ve put the file in a passworded archive to ensure no automated tools got hold of it. I marked the issue as a security problem as in a service like YouTrack it presents a denial-of-service opportunity¹ (remember when I said it was one of the worst kinds of bugs you could have in an Internet-facing service?…)

Shortly after I raised my YouTrack ticket, a notice appeared at the top of the page saying they were investigating the current performance issues. Uh oh! I was holding out hope that this was unrelated to me uploading the buggy dat file, but the timing all seemed a bit suspect. I shot support an e-mail saying I think I might be the root cause for their performance issues and linked to the ticket. In the time it took me to e-mail them, the entire site had been put into maintenance mode. I got an e-mail back a few hours later confirming the outage was in fact all my fault, as I’d feared. Within the space of days the JetBrains security team had fixed the issue in YouTrack, which was a pretty nice turnaround.

So if you were trying to access YouTrack at the start of March and couldn’t - I’m sorry, I didn’t mean to! Also, if you’re building an Internet-facing service that takes user input you should really consider running a fuzz tester against it!

“We have a problem”. “Remember, there are no such things as problems, only opportunities”. “Well then we have a DDoS opportunity.” – @J4vv4d ↩︎