Chameth.com - posts like break-everything-fuzz-testing, building-a-new-computer, debugging-beyond-the-debugger but not docker-automatic-nginx-proxy, modern-css-is-fun, why-you-should-be-using-https

Fixing a loud PSU fan without dying

2025-07-30T00:00:00Z

Three months after I built my new computer, it started annoying me. There would occasionally be a noise that sounded like a fan was catching on a cable, but there weren’t any loose cables to be a problem. Over the course of a few weeks, the sound got progressively worse to the extent that I didn’t want to use the computer without headphones on. I measured the sound at 63 dB, which is about the sound of someone talking. That may not sound terrible, but it’s a constant, nasty noise coming from something that sits about 40cm from my head.

After some investigating¹, I identified the PSU fan as the culprit. I have a Cooler Master V750 SFX, which is not super high-end, but wasn’t cheap, either. It shouldn’t be developing issues after three months. Thankfully, it comes with a ten-year warranty, so it should be easy to get sorted, right?

Warranty woes

I looked at Cooler Master’s warranty, and for issues within the first two years you have to deal with the retailer. That would be Amazon in my case. So I looked at Amazon’s information on warranty issues. Their policy is that if it’s more than 30 days since purchase, you have to send it off to a third-party repair center and wait for them to diagnose and try to repair it. Here’s the kicker:

Usually repairs take up to 20 business days (including delivery time), but could take slightly longer

I use the computer for work, have upcoming LAN parties to go to, and generally can’t do without it for an entire month. That’s assuming they reproduce the issue: the PSU fan only turns on when it reaches a certain temperature, so if you just plug it in briefly it won’t exhibit any symptoms.

If I was going to do a warranty return, I’d need to buy a new PSU first to use while this one was away. If I’m spending £100+ on a PSU, then I’d be getting a better one that doesn’t have known issues with the fans. So when the Cooler Master eventually got returned I’d have no use for it. What a waste.

So the warranty is about as much use as a chocolate teapot. Good to know, I guess. If I wanted decent service, I should have probably used a real retailer, not Amazon. Lesson learnt.

Doing some research

If the warranty is useless to me, maybe I should just void it and try to fix it myself. I did some research online. Most of the results went like this:

if you’re not a trained professional you should not be opening up your PSU at all. There’s a lot of current that goes through it which will seriously hurt/main/kill you

it’s time consuming and potentially dangerous

or some other variety of doom and gloom. There were some actually useful results sprinkled in, though. A post on the smallformfactor.net forums details someone swapping out the fan on the 850W version of the same power supply because it was too noisy. It looks like the fan connector in the PSU is a two-pin connector, and they just used an off-the-shelf adapter to convert the standard four-pin case fan connector. It sounded a bit annoying to make it all fit, though.

A similar post on reddit showed basically the same upgrade, but identified the connector as a 2.54mm JST-XH. They crimped their own connector on, which feels a lot better than using a bulky connector. I do have a crimping set for JST connectors, but I absolutely hate doing them because they’re so fiddly and annoying. I’m not sure if I trust one of my crimps inside a power supply, either.

Looking at the pictures in those posts, I identified the fan as an Ong Hua HA9215VH12FD-F00. I searched to see if I could see any specifications for it. I couldn’t. But what I did get back as my first result was another Reddit post, this one titled “Getting rid of a HA9215SH12FD-F00 fan from your PSU”. It says something about the quality of a component when the first result is a post about getting rid of it because it’s annoying².

All of these mods seemed easy enough, and I’m reasonably confident in my ability to both discharge and avoid touching capacitors, so I probably wouldn’t die. I ordered a Noctua NF-A9x14 fan and some pre-made JST-XH connectors and waited for them to arrive.

Upgrade time

The replacement parts arrived one evening, so when I turned the computer off for the night I unplugged it from the mains, held the power button down for a while, and left it overnight for good measure. The next morning I popped open the case, disconnected and removed the PSU, gleefully tore the “warranty void if removed” sticker off, and opened it up.

Removing the fan was simple: unplug the JST connector and undo the screws holding it to the outer panel. I confirmed that my JST connectors were the right size, then started surgery on the Noctua fan. I carefully removed the heatshrink, cut off the normal fan connector, and separated out the four wires. A quick check of their FAQ confirmed the wiring colours: black was ground, yellow +12V, and blue and green used for speed control (which we’re not using in this instance). I clipped the blue and green wires off near the hub so they wouldn’t be in the way, then cut the yellow and black wires down to length, and attached the JST connector with a solder seal:

Old fan, new fan, burnt mouse mat

In the process I discovered my mouse mat doesn’t like being hit with a heat gun. Whoops. The solder seals are great: the solder in the center melts to form an electrical connection, the parts at the end grip the cable to provide mechanical strength, and the whole thing acts like heatshrink and keeps the connection insulated. A lot less fiddly than crimping JST connectors, and a lot smaller and cheaper than an adapter.

I installed the fan and reassembled the PSU. Then undid it all and tried again, as apparently the face the fan mounts to isn’t quite symmetrical and I’d installed the fan upside down. Connected everything back up, plugged the computer in, booted it up, and… nothing. The PSU doesn’t actually power the fan until it hits a certain temperature. I figured putting it under load would probably heat it up, so ran a graphics benchmark a few times. Eventually the fan spun up. Huzzah.

Soon the entire computer will be beige and brown

Peace at last

I measured the sound of the new fan: 43 dB. That’s a decrease of 20 dB. Due to the weird logarithmic properties of the dB scale that no one understands, that means it sounds about 4 times quieter to human ears. Big improvement!

It’s worth pointing out this isn’t a straight one-to-one upgrade. I don’t know the specs of the old fan in terms of airflow, and I’m just assuming that the Noctua will be good enough by virtue of being a decent fan. They also behave slightly differently as the power ramps up: the stock fan starts spinning at a lower voltage than the Noctua, which needs around 7V to get started properly³. I don’t think this will be an issue, as it’s effectively just putting a small step at the lower end of the fan curve. It might shorten the overall lifespan of the PSU, but from my point of view it failed after four months, so any extra time is a bonus.

I recorded some before/after audio, so you can hear the difference. I think there’s some amount of volume normalisation going on, so it’s not quite as stark a difference as in person, but it’s still noticeable.

Audio sample from the old fan

Audio sample from the new fan

And if you couldn’t hear it, here’s a visual representation:

Side-by-side comparison of the audio clips

The whole upgrade took about an hour (less time than writing this post about it!), and I even managed to avoid killing myself.

Opening the case and sticking my ear near to various components. High-tech stuff. ↩︎
On closer inspection it’s actually a slightly different model — there’s an S not a V in the middle there — but I’m going to assume that doesn’t change much on the quality/noise front. ↩︎
It then keeps spinning down to about 5V or so, it just needs enough power to initially get started. ↩︎

Finding an awkward bug with Claude Code

2025-06-30T00:00:00Z

The many times Claude thought it was right…

I recently encountered a bug in one of my projects that I couldn’t immediately figure out. It was an issue in Centauri, my reverse proxy. After its config was updated, I noticed¹ it stopped serving responses. Looking at the logs, I could see it was obtaining new certificates from Let’s Encrypt for a couple of domains, but I’d designed it so that wouldn’t block requests (or so I thought). After a few minutes of confusion, everything went back to working.

It felt like an issue with locking, but a quick glance didn’t show me anything obvious. I decided to throw the problem at Claude Code and see how it did. With some guidance² it managed to identify the cause, but it was a bit of a journey to get there. I’m going to share the prompts and responses, as it was an interesting process.

The initial prompt

I thought the issue might be quite obvious, and I’d just overlooked it. Centauri isn’t that complex a project, after all. So I gave Claude a fairly minimal prompt to set it off:

> Hello! Please examine this codebase. I have noticed an issue that when a
  certificate is being renewed, the application seems to not serve any other
  requests. I suspect there is a misplaced lock somewhere.
 
  Can you see what's happening? Think very hard.

Claude Code has special hardcoded triggers to engage “thinking” mode, which makes it take more time to think about alternatives. The keywords are: “think”, “think hard”, “think harder”, and “ultrathink”. I managed to not quite get the right incantation here, and only enabled the lowest level³.

I’ll omit all the responses where Claude is “thinking” (talking to itself), or decides to make todo lists then cross items off them then add different items, and so on. Its responses get fairly lengthy; I won’t judge you too harshly if you skim over them. Its first response was blessedly brief:

● I found the issue! Looking at the certificate manager code, I can see the
  problem:

… and then it went back to thinking and reading files, because it had neither found the issue nor could it see the problem. Ho-hum. Some time later it came back with:

● I found the issue! The problem is in certificate/manager.go:53-54:
  
  m.store.LockCertificate(subject, altNames)
  defer m.store.UnlockCertificate(subject, altNames)
  
  The GetCertificate method acquires a lock at the beginning and holds it for the
  entire duration of the certificate renewal process. This includes:
  
  1. Contacting the ACME server
  2. Solving DNS-01 challenges (can take minutes for DNS propagation)
  3. Obtaining the certificate
  4. Getting OCSP staple
  5. Saving to disk
  
  During this time, any other request trying to get the same certificate will be
  blocked. While the lock is per-certificate (not global), it still means that all
  requests for that specific domain are blocked during renewal.
  
  Let me check if there are other places where this might cause blocking:

And it then proceeded to do even more thinking, checking files, etc, until it concluded with:

● I found the root cause! The issue is actually worse than just a
  per-certificate lock. Here's what's happening:
  
  The main blocking issue is in proxy/manager.go:133-144:
   
  func (m *Manager) CheckCertificates() {
      m.lock.RLock()  // ← Acquires read lock on the entire manager
      for i := range m.routes {
          route := m.routes[i]
          // ...
          m.updateCert(route)  // ← This can take MINUTES for ACME renewal!
      }
      m.lock.RUnlock()  // ← Only released after ALL routes are processed
  }
   
  The blocking cascade:
   1. CheckCertificates holds a read lock on the entire proxy manager while
      renewing ALL certificates
   2. Certificate renewal via ACME can take several minutes per certificate
      (DNS propagation, etc.)
   3. Every incoming request calls routeFor (proxy/manager.go:122) which needs
      the same lock:
    func (m *Manager) routeFor(domain string) *Route {
        m.lock.RLock()  // ← BLOCKED until CheckCertificates completes!
        match := m.domains[strings.ToLower(domain)]
        m.lock.RUnlock()
        // ...
    }
  
  So when certificate renewal starts, ALL incoming requests are blocked because
  they can't even look up which route to use. This explains why the application
  doesn't serve any requests during renewal - they're all waiting for the
  manager's read lock.
  
  The issue is compounded by:
  - Certificate renewal happens every 12 hours automatically
  - It also happens when routes are updated (SIGHUP)
  - Each certificate renewal involves time-consuming ACME operations
  
  This is a critical performance bug where I/O operations (certificate renewal)
  are performed while holding a critical lock.

This doesn’t actually follow on from the earlier response, it’s completely flipped onto a different lock being the issue. Amusingly, it’s nearly right here, but it’s highlighting this code for completely the wrong reason.

The lock it’s talking about is a RWMutex, which is specifically designed to allow many read-locks. Everything it’s pointing out are read locks, so they cannot block one another as it described. The response Claude has given sounds very confident, seems appropriately technical, but is in fact complete rubbish. This is one of the big issue with all LLMs at the minute, and it makes me despair at the idea of people “vibe coding” things when they don’t know enough to push back against nonsense like this.

Course correction

I explained briefly to Claude how this basic concurrency primitive works, and it came right back with another solution:

> But that's a RW Mutex. You can hold many read locks concurrently.

● You're absolutely right! I apologize for the confusion. With a RWMutex,
  multiple readers can hold the lock concurrently. Let me trace through this
  more carefully.
  
● Task(Analyze updateCert method)
  ⎿ Done (8 tool uses · 20.0k tokens · 1m 11.9s)
  
● Excellent analysis! You're right about RWMutex allowing concurrent readers.
  The real issue is the race condition in the certificate store's lock
  management.

  Looking at certificate/store.go:82-91:

  func (j *JsonStore) lockFor(subjectName string, altNames []string) *sync.Mutex {
      key := strings.Join(append([]string{subjectName}, altNames...), ";")

      if mu, ok := j.locks[key]; ok {
          return mu
      } else {
          mu = &sync.Mutex{}
          j.locks[key] = mu  // ← Concurrent map write!
          return mu
      }
  }

  This is a classic concurrent map access bug. When multiple requests hit the
  same domain during certificate operations:

  1. Multiple goroutines call lockFor() simultaneously
  2. They read/write to j.locks map without synchronization
  3. Go runtime can:
    - Panic with "concurrent map writes"
    - Corrupt the map structure
    - Block indefinitely if map internals get corrupted

  This explains the blocking behavior - it's not the RWMutex, but rather a race
  condition causing either panics or deadlocks when the map gets corrupted
  during concurrent certificate operations.

  The fix would be to either:
  1. Add a mutex to protect j.locks map access
  2. Use sync.Map instead of a regular map
  3. Pre-initialize all locks at startup

I do appreciate when me stating a basic fact is called out as “excellent analysis”. Definitely doesn’t make me feel like I’m having smoke blown up my ass.

I think the issue it’s describing here is actually a problem⁴, albeit one that’s unlikely to be hit. But it’s definitely not the one we’re looking for. Claude even describes the possible consequences of the bug, and they don’t match what happened at all.

Maybe logs will help?

While Claude was chugging away, I’d been looking into the issue as well. I had logs from when it happened, and figured Claude may be able to glean something from them too. The first thing I’d noticed is that I wasn’t logging when a process finished, so I’d already added that to the code base, hence the note so Claude doesn’t get itself confused:

> That may be an issue, but it's not the one I was looking for. I've found logs
  from the time it happened. Note that a few logs have been added to the
  codebase since (so we can't tell from the logs when ACME or OCSP processes
  *finished*, just when they started). Here are the logs:
  
  {"time":"22:40:53.187887976Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:40:53.19947532Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:40:53.199507942Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:09.048465968Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:09.048482309Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:17.419299795Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:17.419325744Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:23.238027386Z","level":"INFO","msg":"Received signal, updating routes...","signal":1}
  {"time":"22:41:29.994330646Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:29.994353329Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:43.658794811Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:43.658819548Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:41:57.635325513Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:41:57.635347515Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:10.52439779Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:10.524417107Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:23.159430249Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:23.159452171Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:36.347407326Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:36.347433295Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:42:52.544628775Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:42:52.544653993Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:25.409579107Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:25.409602842Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:43.819479448Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:43.819502041Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:43:54.358957563Z","level":"INFO","msg":"Renewing certificate", ...}
  {"time":"22:43:54.358979355Z","level":"INFO","msg":"Starting ACME process to obtain certificate", ...}
  {"time":"22:44:06.180278128Z","level":"INFO","msg":"Renewing certificate", <big list of alt names>}
  {"time":"22:44:06.180302585Z","level":"INFO","msg":"Starting ACME process to obtain certificate", <big list of alt names>}
  {"time":"22:45:03.830331922Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.830528134Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:45:03.832388931Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.225854865Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488432434Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:10.488439397Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}
  {"time":"22:46:30.617614536Z","level":"WARN","msg":"Failed to connect to upstream","error":"context canceled", ...}

I’ve cut down the logs a bit to remove the specific domains, simplify the timestamps, and remove a bunch of other cruft. The key points are there were two SIGHUPs received, and the last certificate before everything started breaking covered 6 different hostnames.

Claude then went through a phase of periodically shouting “Eureka!” then deciding it hadn’t actually found anything yet:

● Now I see it! The key insight is in the timing and the "context canceled"
  errors. Let me trace through what's happening:
  
● Ah, now I see what's happening! Looking at the logs more carefully:

  1. 22:40:53 - First SIGHUP received, routes updated
  2. 22:40:53 to 22:44:06 - Multiple certificates start renewing (14 certificates!)
  3. 22:41:23 - Another SIGHUP received while renewals are still in progress
  4. 22:45:03 onwards - "context canceled" errors start appearing

  The issue is that SetRoutes is being called AGAIN (at 22:41:23) while the
  previous CheckCertificates() goroutine is still running and renewing
  certificates. Let me check what happens in this scenario:
  
> (The context canceled errors are clients timing out because centauri wasn't
  proxying properly)
  
● Ah, now I understand! The "context canceled" errors are from clients timing
  out because Centauri wasn't proxying their requests. This confirms the
  blocking issue.

  Let me check how incoming requests actually get their certificates:
  
● Now I found the real issue! Looking at the timeline and the code:

  1. 22:40:53 - SIGHUP triggers SetRoutes() which:
    - Calls loadCertificate() for each route
    - Sets route.certificate to the existing certificate
    - Starts go m.CheckCertificates() in background
  2. 22:41:23 - Another SIGHUP while certificates are still renewing
    - SetRoutes() is called AGAIN
    - Creates NEW Route objects with route.certificate = nil initially
    - Replaces the old routes in m.routes
    - The old Route objects (with valid certificates) are replaced!

  The critical issue is in proxy/manager.go:42-65:

  func (m *Manager) SetRoutes(newRoutes []*Route, fallback *Route) error {
      // ... validation ...

      for i := range newRoutes {
          route := newRoutes[i]  // These are NEW Route objects!
          // ...
          m.loadCertificate(route)  // Try to load certificate
      }

      m.lock.Lock()
      m.routes = newRoutes  // REPLACES all routes!
      m.lock.Unlock()
  }

  When the second SIGHUP arrives:
  1. New Route objects are created
  2. For domains currently being renewed, GetExistingCertificate might fail or
     return expired certs
  3. These routes get route.certificate = nil and CertificateMissing status
  4. CertificateForClient returns nil for these domains
  5. TLS handshake fails → clients can't connect → "context canceled" errors

  The fix would be to either:
  1. Prevent concurrent CheckCertificates runs
  2. Copy certificates from existing routes before replacing them
  3. Make certificate renewal truly independent of route management

Midway through it had started searching for “context canceled” errors in the codebase, which doesn’t make much sense. That’s what happens when a client gives up waiting, so the corresponding backend request is terminated. You won’t actually find “context canceled” anywhere in Centauri, because it’s a feature of the Go standard library. I gave it a little nudge to try to stop it rabbit holing.

As far as I can tell, what it’s talking about in this response isn’t an issue. I guess it’s possible that there’s a race condition whereby a route would be replaced and end up with an old version of a certificate that was just being renewed, but it wouldn’t cause an issue in practice; it’d just get updated next time the certificates were refreshed.

There’s no way for routes to get a nil certificate in this scenario, as far as I know. But I don’t have the unfounded confidence of an LLM, so I simply said:

> I don't follow why the certificate would be nil?

Attempt the third

I expected Claude to just explain its reasoning, but it apparently didn’t have any reasoning so set off on another voyage of discovery:

● You're right to question that. Let me trace through this more carefully.

● Good point! Since GetExistingCertificate doesn't acquire locks, it should
  return the existing certificate even during renewal. Let me dig deeper:
  
● Now I see a potential race condition! Here's what could happen:

● I found it! There's a critical race condition that explains the blocking:

  The Race Condition

  1. Background renewal starts: CheckCertificates() → updateCert() →
     GetCertificate() acquires lock and starts ACME renewal
  2. SIGHUP arrives: SetRoutes() is called, which for each route calls
     loadCertificate() → GetExistingCertificate() → store.GetCertificate()
  3. The race:
    - store.GetCertificate() returns a pointer to the certificate in the slice
    - Meanwhile, the renewal completes and calls SaveCertificate()
    - SaveCertificate() calls removeCertificate() which modifies the slice while
      it's being read
    - This slice modification (append(j.certificates[:i], j.certificates[i+1:]...))
      can cause:
        - The pointer returned by GetCertificate to become invalid
        - Slice bounds issues
        - Memory corruption
  4. The result:
    - GetExistingCertificate might return corrupted data or nil
    - Routes end up with certificate = nil
    - TLS handshakes fail
    - Clients timeout → "context canceled"

  The Timeline Confirms This

  - 22:40:53 - First SIGHUP, starts renewal of 14 certificates
  - 22:41:23 - Second SIGHUP arrives (30 seconds later) while renewals are still running
  - 22:45:03 - First timeout errors (4 minutes after first SIGHUP)

  The 4-minute delay matches ACME renewal time. The second SIGHUP's SetRoutes
  likely read certificates while they were being modified by the first SIGHUP's
  renewal goroutines.

  This is a classic concurrent slice modification bug - no synchronization
  between readers (GetCertificate) and writers (SaveCertificate/removeCertificate).

Once again we have a very confident and technical answer that does not explain the symptoms we’re looking for. It’s also mostly rubbish again. GetCertificate and SaveCertificate are only called by code that is holding a lock for that certificate; they won’t interfere with one another. There may, again, be a small concurrency issue if different certificates are handled in parallel, but it’s definitely not the cause of our current problems.⁵

Working through these explanations and debunking them was annoying me at this point. I decided to give it once last chance, and used the “ultrathink” shibboleth:

> So there may be a concurrent modification issue, but that doesn't explain the
  symptoms. If a TLS handshake failed it wouldn't result in the clients timing
  out. Likewise if there was any kind of corruption there'd be a panic, not a
  period where it didn't work and then started working again. Engage ultrathink.

Fourth time lucky

I’m going to include a piece of the thinking here because it amused me so much:

✻ Thinking…

  Oh! I need to think about this differently. Let me trace through what happens
  when a request comes in:

Apparently it takes “ultrathink” to actually work through the problem logically? Amusingly it did actually get there:

● You're absolutely right. Let me think more carefully about this...

● EXACTLY! That's the issue. Looking at the timeline:

  1. 22:40:53 - First SIGHUP: CheckCertificates() acquires READ lock, starts
     renewing 14 certificates
  2. 22:41:23 - Second SIGHUP: SetRoutes() tries to acquire write lock, blocks
     waiting
  3. Incoming requests call routeFor() for route lookup, but Go's RWMutex blocks
     new readers when a writer is waiting (to prevent writer starvation)
  4. 22:45:03 - Clients timeout because requests can't even get past route lookup

  The key insight is Go's RWMutex writer starvation prevention: when a writer
  is waiting, new readers are blocked even though the current lock holder is
  also a reader.

  So the cascade is:
  - CheckCertificates (read lock) → certificate renewal takes minutes
  - SetRoutes (write lock) → waits for read lock to release
  - routeFor (read lock for every request) → blocked by waiting writer

  Result: All incoming HTTP requests hang at route lookup, never reach TLS
  handshake or proxying logic. Clients timeout waiting for any response
  → "context canceled".

  This perfectly explains:
  - No panics (no corruption)
  - No immediate TLS errors (never gets that far)
  - Requests hang then timeout (blocked on lock)
  - Eventually recovers (when certificate renewals finish)

  The fix would be restructuring to avoid holding the manager lock during slow
  certificate operations.

We’re back to the RWMutex again, but this time it’s dropping knowledge on me not the other way around: if there’s a writer waiting, then new readers are blocked. This does indeed perfectly explain the symptoms, and was something I’d overlooked. The write lock is only held briefly to update some struct members to prevent it from blocking the readers, but this behaviour coupled with the long-running read lock ends up doing that anyway.

The fix

After all this work it’s very tempting just to ask Claude to fix it, but I had almost no confidence it’d be able to maintain any of the concurrency requirements given how much it flailed around with bad explanations. But in the interests of science, I asked it to try. Here’s what it came up with:

   // CheckCertificates checks and updates the certificates required for registered routes.
   // It should be called periodically to renew certificates and obtain new OCSP staples.
   func (m *Manager) CheckCertificates() {
+    // Create a snapshot of routes to avoid holding the lock during slow certificate operations
     m.lock.RLock()
+    routes := make([]*Route, len(m.routes))
+    copy(routes, m.routes)
+    provider := m.provider
+    m.lock.RUnlock()

+    // Process certificate renewals without holding the manager lock
-    for i := range m.routes {
+    for i := range routes {
-      route := m.routes[i]
+      route := routes[i]
 
       if provider == nil {
         route.certificateStatus = CertificateNotRequired
       } else {
         m.updateCert(route)
       }
     }
-    m.lock.RUnlock()
   }

   // updateCert updates the certificate for the given route.

That’s about what I expect. It definitely fixes the exact issue described, but it’s a bit… sloppy. Routes are now being updated when the lock isn’t being held, which is a change in semantics. It also doesn’t address the underlying issue that requests can be blocked if there’s a pending writer, and pending writes can be blocked by readers. A better approach would be to allow these operations to happen without blocking, maybe by using an atomic pointer to an immutable slice. I guess I’ve got some coding to do!

As for Claude: while the journey was frustrating, I think trying to figure this out by myself would have been even more so. It’s one of those things where everything looks right to me, and I’d have to go down a fairly long chain of checking my assumptions to find where I’d gone wrong.

There’s definitely a lot of room for improvement in coding LLMs: being able to estimate their own confidence level instead of stating everything as absolute truth would go a long way. I imagine there’s also a lot of benefit of having one LLM vet the responses of another; that’s sort-of how thinking mode works, but it doesn’t seem to go far enough to stop it outputting rubbish. It’ll be interesting to see how they develop in the future, anyway.

read as “I got spammed by a bunch of monitoring notifications” ↩︎
read as “Me repeatedly telling it that it was talking rubbish” ↩︎
If only we had some kind of language model technology that could understand this kind of thing. ↩︎
I sense some fun times with Go’s race detector in my near future ↩︎
The most likely outcome from these concurrency issues is Centauri panicking, which isn’t great but would only interrupt ongoing connections before it restarts and starts serving traffic again. Much less problematic than the “block for five minutes” behaviour I’m trying to fix. ↩︎

Building a new Computer

2025-06-03T00:00:00Z

The finished PC. Cat pen for scale.

I recently¹ built a new computer, after exclusively using a laptop for three years. It’s also the first time I’ve departed from the usual combo of an Intel CPU and Nvidia GPU.

While the form factor of a laptop did make it amazingly handy for travelling and attending LAN events, it was starting to show its age and there was basically no sane upgrade path. The main problem was its 3060 mobile graphics card, which was okay for the first few years and then slowly descended into painful. At the time, the (fairly disappointing) 50-series had just been released, but hadn’t yet made it into laptops. Nvidia had stopped production on the 40-series beforehand, so there also weren’t any compelling options there, either.

That’s not to say there were no laptops at all that I could have upgraded to. There were. Just not really any that ticked all of my perfectly normal boxes like “run Linux”, “have at least 64GB of ram”, and “run modern games well”. Having to upgrade the entire system just because the graphics card was showing its age was a bit of a drag, too. So back to the land of desktop computers I went!

To try to preserve some of that convenience, I opted for a small form factor (SFF) case. The computer and all the accessories can fit inside a hand-luggage-sized flight case. More on that later, though.

Components, Choices, and Cramming

So what, exactly, is in this computer? The case is a Fractal Design Terra, which is a lovely little 10.4L case². My last desktop PC was a normal-sized Fractal Design case, which I liked a lot, and it seems like they’ve upped their game since then. The Terra is both pleasant to look at (just look at that wooden panel!), and a breeze to work in. The sides and top are fully and easily removable, giving you great access to everything inside³. I expected an SFF build to be fiddly, but the case made it feel about the same as a normal full-sized build⁴.

The case with sides and top off, and wires everywhere in desperate need of management.

Another nice feature of the case is the movable spine. I think this is relatively common in SFF cases, but it was new to me. You mount the graphics card on one side of a metal divider, and everything else on the other. The divider (or spine) can be moved side-to-side depending on what exactly you’re putting in the case.

As I mentioned earlier, this was my first departure from the Intel + Nvidia bandwagon. Nvidia’s 50 series was underwhelming, and Intel managed to put out a generation of chips that literally fried themselves. It seemed like a good time to try out AMD’s offering! I went for the newly released Radeon RX 9070XT graphics card, and a Ryzen 7 9800X3D processor⁵. The processor has a Thermalright AXP90 X47 Full heatsink on top of it, with a Noctua NF-A9 fan on top of that.

The motherboard is a Gigabyte B850I Aorus Pro, and it hosts 2 32GB Corsair Vengeance RAM modules, and a 2TB Crucial T705 NVMe drive. Powering it all is a Cooler Master V750 SFX. The PSU is modular to reduce the number of wires that need to be fitted in, but they’re still pretty big and bulky. I’m tempted by custom cables but they’re rather pricey, and you can’t actually see inside with the sides closed…

With the diminutive size of the case and all the components crammed in, there’s actually no room left over for case fans. The PSU, GPU and CPU all have fans attached, but that’s it. There are stories online of people having heat issues, but thankfully everything seems pretty happy in my setup. The components all draw in air from the sides, and the hot air passively vents out the top. If I keep the CPU under maximum load for a while then it starts to thermal throttle slightly, but the only time I’ve actually managed that is when compiling a kernel. Why was I compiling a kernel, you ask? Well, we’ll come to that.

But does it Linux?

I didn’t go into this build completely blind. I did a bit of reading and chatting, and it sounded like AMD drivers were pretty good on Linux these days. I didn’t check specifically if any of the components were supported.

The 9070XT was released on the 6th of March. I built my PC on the 26th of March. The first version of Mesa (the 3D graphics lib) that was generally stable with 9070XT cards landed in Arch on the 20th of March. If I did the build just one week earlier, I would’ve had a much less fun experience.

It still wasn’t quite perfectly smooth, though. When playing games I’d occasionally have issues. In some cases the screen just went blank and came back after switching windows, but in others the graphics drivers couldn’t recover and the PC needed rebooting. Fortunately the drivers are open source, and they have a public issue tracker hosted by the Freedesktop project. Searching through the issues, I found they’d already been reported (by another Arch user, of course), and there were various suggestions for work-arounds, as well as lots of work trying to reproduce the issue.

Within a week, an AMD developer had posted a set of patches that might address the issues, and asked people to test them. That’s fortunately pretty easy on Arch: you just clone the repository containing the PKGBUILD file, drop in the patches, and update the checksums for them. Then you start compiling, realise you didn’t tell it to use more than one CPU core, abort it, and start again but this time running on all cylinders. Soon enough I was running a patched 6.14 kernel and the problems stopped completely. Others reported similar outcomes, and the patches worked their way into the kernel proper.

I spent a few weeks having to compile a custom kernel every time I accidentally updated it (I should’ve just stopped pacman from touching the kernel for a bit…), then one day the patches didn’t apply because they’d been merged upstream. Since then everything has been perfect.

A friend commented that “this wouldn’t happen with Nvidia”. They meant it negatively, but I wholeheartedly agree! Maybe Nvidia have fewer launch bugs than AMD, but you absolutely wouldn’t be able to talk to their driver engineers on a public bug tracker, and get access to test patches that fix your issue. I’ll happily take a few bugs in exchange for a company that operates openly in a way consistent with the rest of the Linux community.

But does it travel?

Travel mode, engage!

So the computer works fine at home, but how do I get it to other places? I just shoved a laptop in a backpack⁶, but you can’t quite do that with a PC even one this size. I’d seen a few people at LANs who packed everything in flight cases, but they always seemed a bit unwieldy. Doing some research, though, I found that I’d just be able to squeeze everything into a flight case sized to be taken onto a plane as cabin baggage.

Peli (or Pelican in the US) are the most well known maker of flight cases, but I chose to go with Nanuk instead. Spec-wise they’re fairly similar. Nanuk are a bit cheaper. The killer difference for me, though, is that Nanuk publish STEP files of their cases. As someone with a 3D printer, giving me CAD models of things so that I can easily mod them is definitely a way to win my favour.

After lots of measuring and thinking and modelling I came up with an arrangement where everything could fit snugly into the case. The case has cutouts for the wheels and handles, which constrain what can go where. I ended up wanting the computer at the handle-end of the case to make best use of space, but that means it needs something to take its weight when the case is vertical. The solution was, of course, 3D printing.

Using the STEP files from Nanuk I designed an insert that spans across the case, with screw holes that line up with the ones in the flight case. I wasn’t sure it would be able to take the weight like that, so I also added in some supports in the other direction (running down to the bottom of the case, if it’s stood up). I ended up printing it in three parts, and it came out perfectly first time. I can’t emphasise how nice it is having CAD models instead of having to measure and guess at things.

With the support in place, I focused on cushioning the computer. I opted for 10mm thick EVA foam as it’s easy to obtain and work with, and doesn’t take up too much room. I used contact adhesive to glue the foam directly to the case and the 3D printed divider. There’s a piece that goes between the computer and the keyboard, so I printed some little 3D supports to hold that more firmly in place, and stuck those in with contact adhesive as well.

So how does it all hold up? It’s been to one LAN event so far, and it’s great. I think the flight case might actually be easier to carry than hefting a heavy backpack: the handles are sturdy and comfortable, and it has chunky wheels for pulling it around. It’s also actually easier to get everything set up when you can see it all laid out in the flight case and pull out what you need, and then the case can be tucked away out the way for the duration of the event.

Recently-ish. It may or may not have taken me a few months to get around to writing this post. ↩︎
Using litres to measure computer cases always feels wrong. Those are the units we use for water. Don’t put them near the computers! ↩︎
The case also has holes to put security screws in to stop all of these nice easily removable parts from being removable. Very handy when you’re leaving it unattended at a LAN. ↩︎
Which for me means it was good fun for about an hour, and then I got fed up and just wanted it to be done already. ↩︎
Three months in and I still can’t remember these random strings of numbers. Someone even explained the versioning scheme to me. ↩︎
Along with a portable monitor, mouse, keyboard, headset, mouse mat, kensington lock, power supply, cables, and controller. It was a heavy bag. ↩︎

Adventures in 3D printing

2023-09-17T00:00:00Z

The Sovol SV06.

I’d been idly considering getting a 3D printer for a while, but have only recently taken the plunge. I picked up a Sovol SV06 from Amazon for £199.99¹, which is a model commonly recommended for beginners. About three weeks later, I think I’ve finally finished fixing all the problems the printer has, and thought I’d document them.

Setup and out of the box performance

The setup of the printer itself was straight forward. Most parts are assembled, you just have to bolt the frame together, bolt the various parts to the frame, and connect some wires. The hex bits in the iFixit Mako Bit Set were a godsend for this, as the included hex keys were a bit flimsy.

Once everything is bolted and connected, you turn it on and immediately start looking around for the jet engine you can hear. The fans are loud. The instructions walk you through the initial calibration, which consists of:

Levelling the X-Axis gantry (where the print head moves). The printer does this by (gently) ramming the gantry up as far as it will go.
Adjusting the Z-offset of the print head. The printer has an inductive proximity sensor attached to the extruder, but it doesn’t know how high it is relative to the nozzle. The calibration process involves manually dialing in this offset while moving a piece of paper back and forth under the nozzle.
Levelling the bed. This is entirely automatic: the printer just probes a bunch of points on the bed with the proximity sensor and builds up a mesh it can use to compensate for the bed not being level.

After that’s done, the included SD card has the standard “Benchy” model on it, pre-sliced and ready to print. It came out looking good:

A nice, problem free Benchy. No dramatic irony here.

The noise. Oh god, the noise.

I set about printing a few more things, but the noise of the machine quickly got to me. It sits approximately 80cm from my ear, and between the fans and the noises it makes when moving it was driving me mad. The fan in the PSU seems to sit at 100% power regardless of temperature or load, and sounds about three times louder than my laptop’s fans do when they’re running at full tilt. Fortunately, someone had designed a printable PSU fan silencer that redirects the airflow (and most of the noise) from the fan. It doesn’t feel like it should help that much, but it does significantly reduce the noise as it claims. This was the first of many times I’d use the 3D printer to print parts for itself.

The other noise was a lot less straightforward. The linear rods that all three axes run on pass through several bearings. Sovol apparently don’t bother to apply any lubrication to these, so it sounds like an elephant is moving through a gravel pit when the printer is running. Some people online suggest rubbing lube on the rods and pushing the movable parts back and forth, but this just results in a load of lube caught on the dust filters of the bearings².

The solution to this, then, was to dismantle most of the printer to gain access to each of the ten bearings, and manually lubricate them. This was tedious and messy, but not actually that difficult. It doesn’t seem like the kind of thing you should have to do on a brand-new machine, but I guess you get what you pay for.

Speeeeeeed

3D printers are slow. The SV06 is not one of the faster ones. One simple way to increase the print speed is to change out the nozzle. The SV06 comes with a 0.4mm nozzle, which is pretty standard, but if you upgrade to a 0.6mm nozzle you can extrude more than twice the volume of plastic in the same amount of time³. The consensus among anonymous internet people is that if you’re using a modern slicer there isn’t much difference in print quality when doing this.

I dutifully ordered a new nozzle, and after it turned up set about installing it. This is mostly straight forward, aside from the complication that you’re meant to do it when the hot-end is hot. That makes sense: you want it be under the same thermal expansion it will be under during use, but the tool they provide to change the nozzle is all metal. So you have a relatively delicate operation that you have to do in a bit of a rush before the heat creeps all the way up to your hand.

With the new nozzle in place, the printer was noticeably faster. I didn’t notice any difference in print quality, either.

SD cards are not fun

Up until this point, my process for printing anything was to slice it on the computer, export the gcode file to an SD card, turn around and walk over to the printer, insert the SD card, and start the print from the LCD. I don’t particularly mind those steps, but the SD card slot on the Sovol is terrible. It doesn’t line up well to the hole in the case, doesn’t guide the card in well, and is all round a pain to use.⁴

To remove the sneaker-net element of this process, I ordered a Raspberry Pi and installed OctoPrint on it. OctoPrint provides a web interface to the printer (which it controls over USB), and most slicers can automatically export the gcode to an OctoPrint instance.

While OctoPrint itself is great, there was one small problem: if you connect anything to the USB port on the printer, it tries to draw 5V from it and “backpower” itself. It’s not meant to draw power from the USB port, it’s just not designed well. A lot of the cheaper 3D printers share this flaw, it seems. The fix here is to get a power blocker such as the one made by PortaPow that simply leaves the +5V line unconnected between the two ends. Presumably Sovol could fix this by adding a diode to the USB power line, but maybe that would push their costs too high?

After getting OctoPrint working, I printed a nice case for it to mount on the side of the printer. I slotted the Pi in, and when I powered it on again it didn’t boot. When I went to pick it up to debug the problem, I burnt my finger on the SD card⁵. Apparently if you break SD cards in a certain way, they can short out and get very hot. Fun! Maybe I shouldn’t have been using the no-brand card that came with the printer…

What do we do with a drunken bed?

I continued printing bits and bobs, but was starting to notice a new issue: if I was printing multiple copies of the same object, those on the right hand side of the bed often failed. OctoPrint has a nifty plugin that lets you cancel certain regions from the build, which saves it ruining everything, but only being able to print in certain places was annoying.

I ran through all the troubleshooting steps I could think of, including a foray into custom firmware with different bed levelling abilities, but nothing seemed to help. Another OctoPrint plugin later and I could see the problem:

This should probably be straight.

The orientation of the graphic is a bit confusing, but you can see how the right hand side of the bed is significantly higher than the left. There’s a total variance of just over 0.5mm, which is apparently enough to cause problems even with the auto bed levelling. I spoke to Sovol support, and eventually they suggested disassembling the print bed and measuring the aluminium spacers that hold the bed. Sure enough, they were all different heights:

You had one job…

There’s a 0.13mm variance in their heights, which doesn’t account for the 0.5mm seen in the bed mesh, but it’s a good starting point. Rather than go back and forth with Sovol support over the course of many days, I just ordered some nylock nuts and reinstalled the bed without spacers. The nuts hold the bed in place, while still allowing you to tighten or loosen the screws to adjust the height.

After five or six rounds of totally and utterly uneventful adjusting and re-leveling, I got to this point:

This is much more straight.

There’s only so much you can do when you only have five screws to adjust, but I’m pretty happy the variance is now under 0.15mm. That’s definitely in the “good enough” range that the bed leveling algorithm can compensate for.

Whoops

So, about that “totally and utterly uneventful adjusting” I did. To get at the screws that hold the bed in place, you have to remove the thin sheet that acts as the build surface. This attaches magnetically to the bed, so it’s easy to do, but I am lazy and decided to leave it off while I fiddled with the bed.

There’s a slight problem here, though: when it’s working out the bed mesh the printer uses the inductive proximity sensor. The inductive proximity sensor detects metal. The metal is in the removable sheet. The first time I tried to run the bed mesh, the printer rammed the extruder assembly into the bed at a decent speed and ended up at a rather nasty angle.

I checked over the extruder and found the heat break had bent:

This should also probably be straight.

I tried printing with it as-is, but the filament came out too thin and at an angle, which made the print go about as well as you’d expect. I ordered a replacement, installed it, and… nothing worked still.

At this point I was getting thoroughly fed up. There seemed to be one problem after another, and no good way to diagnose problems with the print not sticking (because there are a dozen or so different potential causes). After a lot of frustrated googling, I found a post on the Prusa forums that suggested cleaning the build sheet with dish soap. I’d cleaned it with IPA, but several people were saying that dish soap magically fixed issues that IPA couldn’t. I took the plate to the kitchen, gave it a wash, and the next print went down perfectly.

Growing a backbone

For a blissful period of about four days the printer was chugging along perfectly with no issues. Then it started making a weird noise. In a display of amazing restraint, I didn’t immediately throw the whole thing out the window. Looking at the printer, I could see that the cable running from the extruder assembly to the main board was sagging and rubbing on top of the print as the print head was moving around. I’m not sure what property of the cable was keeping it aloft before, but it apparently ran out after about three weeks.

Like most of the other problems with the SV06, this design flaw has been noted by others and worked around. I printed a cable chain and installed it⁶. Amusingly, while I was printing the chain, the cable dragged on the print and dislodged one of the pieces entirely before it was finished. Fortunately, it didn’t affect the neighbouring pieces, so I just had to run off a replacement for that one.

The difference is rather pronounced:

It feels like I own a 3D printer to print parts for my 3D printer…

If you can’t see it: in the first image the cable is sagging and actually ends up below the print bed. The individual links of the cable chain limit its freedom of movement, stopping it sagging.

The end?

Once again the printer is printing properly again. I’m sure there will be more problems, but hopefully I get at least a week or so before anything else goes wrong. Maybe I’ll print an “X days since the last SV06 problem” counter…

I think, if I had to do this all again, but with the knowledge I have now, I’d have stumped up the extra cash to get a Prusa MK3S+⁷. This is the original machine on which the SV06 (and a lot of other cheap imitations) is based. Some obvious benefits it has over the SV06:

The extruder cable runs on the outside of the machine, so won’t droop onto prints
The heated bed has copper embedded in it for use in the induction sensor, so it won’t crash into the bed if the build sheet is removed
They actually put lubricant in the bearings instead of shipping them bone dry
The bed is held in place with screws instead of poorly sized aluminium spacers, so you can adjust the height without having to take the whole thing apart
They don’t use the cheapest possible power supply and main board, so it doesn’t backpower off of USB devices or have the loudest fans known to mankind

Prusa also have a much better attitude towards 3D printing: they open source the hardware designs (which is why there are so many cheap clones), and they actually use the printers themselves. The plastic parts of an SV06 are injection moulded, but those on a Prusa printer are actually printed on a Prusa printer. Not only is it cool that they use their own products, it means they have to run them en-masse so they’re obviously invested in making them reliable. They also have a reputation for exceptional support.

Despite all the problems, I do enjoy having the printer. I’ll probably talk more about the prints I’ve done that weren’t for the 3D printer itself in the future.

The RRP is £279.99, Amazon discounted it down to £269.99 then offered a £70 voucher. The voucher says it expires at midnight, but it’s said that every day for the past three weeks… ↩︎
Which is exactly what another set of people said would happen. It was worth a try, though. ↩︎
This feels strange, but if you work out the areas of the nozzles it makes sense: a 0.4mm diameter nozzle has an area of 0.04πmm² (the radius is half the diameter, and the area of a circle is πr²), and a 0.6mm diameter nozzle has an area of 0.09πmm². Maths! ↩︎
At least it doesn’t eject the SD card across the room like this Ender 3 V2! ↩︎
My only 3D printer related injury so far. It’s impressive I can be around a 200 degree hot-end and somehow burn myself on a storage device. ↩︎
Well actually I printed 5 copies of their calibration piece until it was perfect, then printed the actual cable chain, then had to sand each of the individual links because they still weren’t good enough. God, I hate sanding. ↩︎
Or maybe I’d cave and get the newer, shinier, much more expensive MK4. That wouldn’t be out of character. No, I didn’t just check my bank balance. You’re imagining things. ↩︎

How to break everything by fuzz testing

2020-04-26T00:00:00Z

Fuzz testing is a bit like the infinite monkey theorem, but instead of Shakespeare you get crashes.

Fuzz testing, if you’re not aware, is a form of testing that uses procedurally generated random inputs to see how a program behaves. For instance, if you were fuzz testing a web page renderer you might generate a bunch of HTML - some valid, and some not - and make sure the rendering process didn’t unexpectedly crash.

Fuzz testing doesn’t readily lend itself to all types of software, but it particularly shines in cases where some kind of complex user input is accepted and processed in some way - like the aforementioned web page renderer. I was recently adding a library to parse EXIF data to images to an Internet-facing service and realised it was a perfect opportunity to do some fuzz testing. Even if I didn’t find any issues, I’d improve my confidence that the library was safe enough to expose to the Internet.

Breaking my EXIF library

I wrote a quick harness to run go-fuzz on the library, and gave it some pre-existing demo files as sample input. The way go-fuzz works is that it instruments your code and then mutates the inputs to try to improve the coverage. For example, if I had some sample data that had an EXIF tag with a value of 1 then go-fuzz might change it to a 2 and see if the code follows a different path. In most cases it won’t but when it does, they tend to be very interesting cases.

One of the first issues that go-fuzz found was that some values in a maker note field would cause the library to panic (i.e., crash). This happened because there was a check to see if the first six characters were “Nikon” and a null byte, without first checking to see if there were actually six characters available. This is a kind of bug that doesn’t happen much with “real” data - as the field is either not present or completed correctly - but could easily be exploited once this code is exposed to the Internet.

Another interesting bug that go-fuzz found was that if a tag had a particularly large count, the library would try to allocate an obscene amount of memory and die. There was already a check in the code that was meant to avoid this exact scenario, but go-fuzz managed to find a way around it. Each tag has a size (for example an integer tag takes a fixed number of bytes) and a count; the existing check multiplied the two together and made sure that the result wasn’t too large. For most cases this was fine but go-fuzz found a case where the count was so large that when multiplied by the size of the tag it overflowed the integer and became negative, thus passing the sanity check but then subsequently failing when it came around to actually allocating the memory.

The final bug of note that go-fuzz found was the most interesting. EXIF data is stored in IFDs (“Image File Directories”), and each IFD provides what is effectively a pointer (a byte offset) to the next one. The EXIF library already had a check to make sure that these didn’t loop, but it only checked the immediately preceding IFD - so if IFD 1 linked to IFD 2, it would catch IFD 2 linking back to IFD 1 and break the loop with an error. Go-fuzz found that having three interlinked IFDs had the same issue, though, and the guard code wasn’t triggered. This created an infinite loop, maxing out a CPU core until the process was eventually killed - one of the worst kind of bugs you could have in an Internet-facing service which doesn’t deal with private data! The fix for this was fairly straightforward - I just made the library keep a record of the previously visited IFDs and bail out if it found a loop.

Breaking my IDE

When go-fuzz detects an issue it outputs not only the details of the problem (the stack trace, error message, and so forth) but also the input that generated the problem. This is useful for reproducing and making sure the issue is fixed, but it also makes it really easy to write a test to ensure that the behaviour never regresses in the future.

As I was working through fixing the bugs that go-fuzz found, I dutifully added new tests where needed. After adding the sample input with looping IFDs to the project, I switched to IDEA to write a test to use it. I clicked on the input file to copy the file name, and then the entire IDE hung and had to be restarted. Uh oh! When I restarted IDEA, it immediately began indexing the project and again hung. It turns out IDEA parses EXIF data (presumably, even if it does nothing else with the data, to get the rotation property for images), and the library they use - an independent one written on Java - had the same bug as the Go library I was using.

In order to stop IDEA from indexing the file and becoming unusable I renamed it from a ‘.tif’ extension to ‘.dat’, and everything went back to normal. I thought I’d best report the bug to JetBrains, though, so they could put a proper fix in.

Breaking YouTrack

JetBrains use their own issue tracker called YouTrack for reporting bugs in IDEA. I dutifully went over and described the problem, attaching the log files from the IDE, a description of how the file was malformed, and carefully selected the .dat version of the file to upload so that it wouldn’t cause anyone else the same immediate problem.

After trying to upload the file I got a strange error back. Uh oh! I submitted the IDEA issue as it stood, unable to see if the attachments had even uploaded, and went and wrote up an issue for YouTrack itself about the error message. While I was doing that, YouTrack seemed to slow down and become really annoying to use. I had a sinking feeling the exact same thing was probably happening as with IDEA and my library - but this time YouTrack had content-sniffed the file instead of relying on the file extension. In hindsight, I should’ve put the file in a passworded archive to ensure no automated tools got hold of it. I marked the issue as a security problem as in a service like YouTrack it presents a denial-of-service opportunity¹ (remember when I said it was one of the worst kinds of bugs you could have in an Internet-facing service?…)

Shortly after I raised my YouTrack ticket, a notice appeared at the top of the page saying they were investigating the current performance issues. Uh oh! I was holding out hope that this was unrelated to me uploading the buggy dat file, but the timing all seemed a bit suspect. I shot support an e-mail saying I think I might be the root cause for their performance issues and linked to the ticket. In the time it took me to e-mail them, the entire site had been put into maintenance mode. I got an e-mail back a few hours later confirming the outage was in fact all my fault, as I’d feared. Within the space of days the JetBrains security team had fixed the issue in YouTrack, which was a pretty nice turnaround.

So if you were trying to access YouTrack at the start of March and couldn’t - I’m sorry, I didn’t mean to! Also, if you’re building an Internet-facing service that takes user input you should really consider running a fuzz tester against it!

“We have a problem”. “Remember, there are no such things as problems, only opportunities”. “Well then we have a DDoS opportunity.” – @J4vv4d ↩︎