Chameth.com - posts like artisanal-docker-images, building-a-new-computer, docker-automatic-nginx-proxy, sense-api, why-you-should-be-using-https

Migrating from GitHub to Forgejo

2026-04-30T00:00:00Z

When Microsoft bought GitHub in 2018 my kneejerk reaction — like so many others — was to start looking for alternatives. For a while I self hosted a Gitea instance but I never totally bought into it: some repositories I still pushed to GitHub, some I pushed to Gitea and they got mirrored, and I ended up causing myself problems when I got the two confused. Part of the problem was that the GitHub UI was faster and cleaner than Gitea’s at the time; using Gitea felt like a chore compared to GitHub. I ended up not maintaining it and eventually binning it and just going back to GitHub.

An all-too familiar unicorn

Fast forward eight years, and GitHub is about what we all imagined when Microsoft bought it. If you take the most pessimistic way of counting, they have zero nines of reliability¹. If you take the most generous, they have a single nine. That’s around 30 minutes of downtime every day. It feels like it must be more than that, given how many times you see the damned unicorn.

Uptime aside, they’ve crowbarred LLMs in all over the place; they don’t have the time to maintain official GitHub actions; the ones they do maintain have Copilot running roughshod all over them²; actions themselves are slow to run³, often have weird transient failures, and there are so many footguns that I’m pretty sure GitHub actions should be an entire category in the CWE top 10.

Speaking of security, while I was drafting this post a remote code execution was reported and the details are… impressive. The actual security issue was a pretty stupid oversight. Those happen. But they end up being able to execute code as a globally shared git user with access to all the other repositories on the node. What? I can’t quite get my head around it. No sandboxing, no containers, it just runs as a git user?

So, yeah, I’m not a fan of GitHub in its current state. A few months back I took the plunge and set up Forgejo, a fork of Gitea where they did radical things like add tests and ensure that it’s community operated not twisted for commercial use. Forgejo also potentially has some fun security problems⁴, but the way I host it mitigates more-or-less all problems. It’s also slightly less shocking when an open source project has lapses in basic security versus a $2 trillion corporation.

It’s taken a bit of doing, but I’m very happy with the setup I have.

The basic setup

I run all my server software using Docker, and Forgejo is no exception. They have decent documentation on how to get it running, and publish rootless images which is nice (and diminishes my urge to make an artisanal version myself). You can set configuration options using environment variables, which is perfect for containers, although the names can end up a bit unwieldy. Like this thing: FORGEJO__repository.signing__SIGNING_NAME=Chris Smith. It gets the job done, but the environment-to-ini mapping is pretty ugly.

I didn’t want to expose Forgejo publicly, as I wanted to avoid having to try and secure it⁵, or dealing with bots scraping it, or users signing up, and so on. I wanted somewhere to store my git repositories, handle CI for me, and mirror them somewhere public for other people to see and interact with. If you’ve read some of my other posts you’ll probably guess where this is heading: Tailscale. Alongside the Forgejo container, I run a Tailscale instance with a serve.json that covers both the web frontend and the SSH listener used for git operations:

{
    "TCP": {
        "22": {
            "TCPForward": "forgejo:2222"
        },
        "443": {
            "HTTPS": true
        }
    },
    "Web": {
        "${TS_CERT_DOMAIN}:443": {
            "Handlers": {
                "/": {
                    "Proxy": "http://forgejo:3000"
                }
            }
        }
    },
    "AllowFunnel": {
        "${TS_CERT_DOMAIN}:443": false
    }
}

This allows me to interact normally with Forgejo from any of my devices or servers (which all run Tailscale), while not exposing it to the Internet at all. It also handles the TLS certificates automatically, which saves me setting up something else to do that.

For CI, I run the Forgejo runner image on the same server, and gave it a dind container to use to actually run the workloads. dind is the cutesy name for Docker-in-Docker, basically a Docker daemon running inside a Docker container. That keeps the CI workload separated nicely from the “production” workload running on the server, which is nice from both a security and a monitoring point of view.

The uncanny valley of actions

Forgejo actions are basically like GitHub actions: they run a series of steps using one or more container images. You define those steps in the exact same clunky YAML format as with GitHub. The big difference between them is that GitHub defaults to using an absolutely massive base image filled with out-of-date preinstalled software, and Forgejo leaves that for the administrator to configure. You could, of course, just use the same image as GitHub, but lugging around a multi-gigabyte image that doesn’t even have recent versions of the software I want isn’t really my style. Instead, I set about using a plain debian image as the base, and then did a face-palm when I tried to run a “normal” action and remembered they’re all written in JavaScript, so expect node to exist.

Accommodating JavaScript isn’t really my style, either, so I did the very sensible thing of writing my own suite of actions in Go. Being written in Go means they can be compiled statically, published as a container, and manage their own very limited dependencies. For example, the checkout action is built into an alpine container with git added, while the dockerbuild action uses the buildah image as a base. Each image has the tools it needs, pinned to a recent version, and nothing more. This is very much not a sensible approach to take for most people, but it made me happy. With six basic actions, I could replace almost all the ad-hoc workflows I had in GitHub⁶.

Other than the base images and my self-inflicted JavaScript machinations, everything works the same as GitHub. But faster. So much faster. Most of my CI workflows finish on Forgejo before GitHub would even have allocated a runner to the job⁷. And I can actually view the logs without them glitching out.

Step aside Dependabot, here comes Renovate

One of the boons and/or banes of hosting code on GitHub is access to Dependabot⁸. It’s a tool that monitors your dependencies, and automatically submits PRs when new versions are available. It also does security alerts, but they’re comically bad for Go at least⁹. I do like having dependencies automatically handled, especially with a cooldown, which is how I had Dependabot configured before moving.

To replace it, I’ve configured Renovate, an open source alternative. I’d seen it used a bunch before: both by projects on GitHub who prefer it over Dependabot, and by people using GitLab and other platforms where Dependabot isn’t. It mostly does the same things, but Renovate has a few very nice extras.

Firstly, it supports transcluding config from another repo. So in each of my many, many projects, I just have this stub of a renovate config:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "local>meta/renovate"
  ]
}

Then the actual config lives in a default.json in the meta/renovate repository. I mirror that repository to GitHub, if you want to have a look. This is an amazing feature. When I enabled cooldowns in my Dependabot config it was a slog to go through all my active repositories and make the same change over and over again. With Renovate I can configure that centrally. As far as I know Dependabot can’t do anything like that, even for enterprises.

Within that central config I use some of the other nice features. It has the ability to auto merge changes. I use this to automatically accept changes from trusted projects, or my own libraries:

    {
      "matchManagers": ["gomod"],
      "matchPackagePatterns": [
        "^github\\.com\\/csmith\\/",
        "^chameth\\.com\\/"
      ],
      "matchUpdateTypes": ["minor", "patch"],
      "automerge": true,
      "minimumReleaseAge": "0 days"
    },
    {
      "matchManagers": ["gomod"],
      "matchPackagePatterns": [
        "^github\\.com\\/csmith\\/",
        "^chameth\\.com\\/"
      ],
      "matchUpdateTypes": ["major"],
      "minimumReleaseAge": "0 days"
    },

This takes any minor or patch release of my libraries, and automerges it with no cooldown. For major version changes it overrides the cooldown but doesn’t automerge. To automerge Dependabot PRs you have to write an action, implement that logic yourself, and hope you’ve successfully avoided all the footguns inherent in that.

Speaking of major version updates, Renovate also has an option to automatically update the imports in Go packages when it’s offering a major update. In the config I have:

"postUpdateOptions": ["gomodTidy", "gomodUpdateImportPaths"],

Which means that every time it does an update it’ll run go mod tidy, and it’ll update the import paths. In Go, v2 of a module has a different import path to v1 (so they can coexist if transient dependencies need different versions), so usually you have to go through and find and replace those paths. Renovate handles that for me now! I still have to address whatever breaking changes are in the new version, but there’s a lot less grunt work.

It can also run custom commands¹⁰, so for example in my legotapas project I have this extra snippet in the renovate.json:

  "postUpgradeTasks": {
    "commands": [
      "sh -c 'rm plate_*.go'",
      "go run ./cmd/generate"
    ],
    "executionMode": "branch",
    "fileFilters": ["*.go"]
  }

I won’t go into the gory details, but this handles automatically regenerating a bunch of files that change every time a certain dependency is updated. Previously I had a separate action to do that on GitHub; just being able to drop commands in the config file is much nicer.

To actually run Renovate, I simply have a workflow in Forgejo on a schedule that uses their official Docker image. It also triggers if I update the config, which is nice when I add a new rule, as it’ll immediately update the open PRs to comply. Overall, Renovate has felt like a pretty big quality-of-life upgrade.

Odd problems

Not everything has been smooth sailing. Sometimes when Renovate automerged a change, it would just… disappear. The PR was closed as merged, but the commit never landed on master. It wasn’t a huge issue as the next Renovate run would just recreate it, but it would then disable the automerge and I’d get confused as to why. It turns out there was a major problem with how I had everything set up. Renovate was telling Forgejo to automerge when the required checks pass, but I had no branch protection rules set up in Forgejo, so as soon as the PR was created it was eligible to merge. As the CI was so fast I hadn’t noticed that the automerges were happening before the CI results were posted.

The disappearing commits were because the repository was getting into a bad state when Renovate tried to automerge multiple changes instantly in the same repo. Forgejo has a bunch of maintenance commands that helped to fix that problem, and I then wrote a little script to enable branch protection rules so the CI would actually run. The automerging has been smooth since.

Another issue I had early on was that merging would fail because the GPG keyring was locked. I had Forgejo configured to sign all the automatic commits, so you can verify what commits came from my infrastructure. Forgejo is the only process using the keyring, so I wasn’t sure why it was becoming locked. It turns out that GnuPG uses the hostname as part of the lockfile, and because Forgejo was running in Docker it was getting random hostnames based on the container ID. Explicitly setting a hostname made GnuPG and by extension Forgejo much happier.

Webhooks and updating

Forgejo has this amazing feature: you can add a webhook that’s called for all your repos. GitHub, for some reason, supports this for organisations but not people. Even though the two are treated the same in a lot of places. It doesn’t seem like it would be hard, and it’s such an obvious thing to want, but… no. I wrote a whole tool to add webhooks for me to get around this. Now I don’t need it, Forgejo does the right thing!

One particularly interesting use of these webhooks has been making my Docker containers auto-update when a new image is pushed. I previously had a webhook receiver for a few projects that would update them automatically, but GitHub package notifications were extremely laggy. Every other webhook fired within a minute or so of the event happening, but package webhooks would sometimes take 30-60+ minutes to fire. I opened a support case with GitHub and it got escalated to engineering and fixed about 10 weeks later¹¹. It was better for a while, and then got worse again. Trying to do continuous deployment with a 30 minute random lag was frustrating.

So now I had reliable webhooks, but the update process was a bit gross. I have a hangup about internet-facing services not having access to the Docker socket. It just feels like too big a security risk. So I had a process that received hooks, and then another that figured out if containers needed to be updated and restarted them. Adding a new container to the system meant recompiling the second process, which was a massive faff.

Fortunately, I was saved from this mess by my friend Greg. He wrote adze, a tool that receives webhooks and uses Docker Compose to update matching containers. He wrote a blog post about it. It’s all one process, but that’s no longer a blocker for me because the webhooks can be delivered privately from my Forgejo instance without exposing adze to the internet. It finds matching containers automatically based on the images they’re using¹², and combined with Forgejo not forcing me to add a webhook to every single repository makes my life so much easier. I write some code, push it to Forgejo, and within a minute or so it’s running on my dev instance.

I have IRC notifications set up for when adze does updates, and I still have some old ones that announce GitHub package webhooks. I keep seeing things like this and chuckling:

15:44:42 <@ircjag> [ADZE] chameth.com: pending updating git.yak-wall.ts.net/public/chameth.com
15:44:45 <@ircjag> [ADZE] chameth.com: success updating git.yak-wall.ts.net/public/chameth.com
17:11:54 <@ircjag> [GHCR] Container pushed to csmith/chameth.com:dev.

I pushed a change at 15:43, Forgejo built it and pushed the image (to both its own registry and to GitHub), then at 15:44 adze updated the container to the new version. An hour and a half later GitHub gets around to delivering its own webhook.

Mirroring and PRs

The final piece I had to figure out was how to actually interact with people. Most of my projects are small and don’t get a massive amount of public contributions, but the option should still be there. And ideally not involve anyone having to figure out how to e-mail patches¹³.

Mirroring outwards is simple: Forgejo supports it natively. Whenever I push to Forgejo, it pushes out to the corresponding repo on GitHub. I’ve also started mirroring some projects to Codeberg, to avoid having GitHub as my single “public” presence. But what happens when I get a pull request? If I merge it in the GitHub web UI, Forgejo will push its own branch over the top and get rid of it¹⁴. I could pull the changes, merge them, and push the result to Forgejo but then I don’t get the benefit of any CI.

I ended up writing a small private tool that deals with this for me. I give it a GitHub PR, and it effectively imports it into Forgejo for me. I created a prs organisation, and it’ll create a fork of the project in that org if it doesn’t exist. Then it pushes the PR contents to a branch, and creates a PR from the prs repo into the main one. It runs as a user which can write to the PRs org, but only submit pull requests to public repositories outside of it.

When the PR is imported, I can review it in the Forgejo UI, and then I can approve the CI workflows for that particular PR. This gives me some extra defence-in-depth in case anyone is trying to abuse one of those action footguns. Once the CI passes, I can merge it as normal in the Forgejo UI, it gets mirrored out to GitHub, and GitHub marks the PR as merged automatically. It’s not the most elegant solution, but the actual workflow is pretty smooth, especially for the low volumes of PRs I get.

With that, I have Forgejo set up to do everything I need from a git host, and I’m no longer hamstrung every time GitHub goes down.

This way of counting isn’t particularly fair, but it’s somewhat realistic. GitHub break out their uptime for things like git operations, pull requests, actions, webhooks and packages. Realistically if any of those independent things are down then my overall workflow doesn’t, well, work. ↩︎
To be clear, I’m not objecting to the use of Copilot here. It’s a tool. But if you use a tool to do something fairly simple and it spews out 35 commits, including at least 5 reverts, then either it’s a terrible tool or you’re using it very wrong. I didn’t hunt down this example, I came across it while trying to debug something and wasted a bunch of time trying to figure out what on earth was going on. ↩︎
You could self-host runners, which may help, except they’ve already threatened to charge you for the privilege of bringing your own runners. ↩︎
If people could stop finding RCEs in git services for a few days so I can stop redrafting this intro, that’d be great. ↩︎
I’m feeling pretty smug about this decision after reading about the security issues. ↩︎
There’s one holdout: my dockerfiles repo which is a very special snowflake with a very complex workflow and I’m still deciding how to handle it. ↩︎
Which is to be expected, given I’ve got a server dedicated to one person, and they’re trying to serve however-many million users at once. That doesn’t make it any less nice, though! ↩︎
I’d link to it but there’s nowhere really good to link to. It’s just been subsumed into the ‘security’ functions of GitHub, along with a bunch of copilot nonsense. ↩︎
The go project maintain a vulnerability database that includes details about which exact symbols are affected, and the govulncheck tool uses this to only report security issues in code that’s actually used. Dependabot doesn’t do that: it constantly alerts about code that’s not imported, so won’t ever be shipped. ↩︎
In the default config it doesn’t allow anything of the sort, of course. You can whitelist individual commands, but because this is a private server I can merrily whitelist .* and do whatever I like. ↩︎
The turnaround time isn’t particularly impressive, but the fact that they kept the support case open, and the (human!) support agent updated me several times is. I don’t have a lot of good things to say about GitHub, but I don’t think I’ve ever had a support experience that good from a company of a similar size. Especially as I wasn’t a paying customer at the time. ↩︎
Why didn’t I do that in the first place? It seems so obvious now… ↩︎
I really like the idea of an e-mail based workflow, but I absolutely hate actually doing it. ↩︎
This is what happened to me a lot when I was half using Gitea. ↩︎

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

10 Weeks with an Apple Watch 10

2025-09-09T00:00:00Z

My watch. Yes, I am available for wrist modelling opportunities.

Around ten weeks ago¹ I picked up an Apple Watch 10, and have been wearing it almost constantly since. It’s not my first Apple Watch — I had a Series 5 for a bit back in 2020 — but it’s the first time I’ve actually stuck with it. Ten weeks seems like an apt time to reflect on it.

Firstly, why did I even bother? Well, for a couple of years I’d been wearing a Xiaomi Smart Band 7, mainly to monitor my sleep stats and set alarms that won’t wake up everyone else nearby. Its battery life was fantastic — with notifications and other things turned off, I got about a month of use between charges — but actually using it felt like trying to order food via the medium of interpretive dance.

My biggest gripe was the screen lock. If I didn’t have the screen locked then I’d periodically trigger it during the night when I moved around and it came in contact with my chest or leg. With the lock enabled you had to deliberately swipe up from bottom to top to enable interaction, but it didn’t work reliably. When I wanted to adjust an alarm, I’d be stood swiping repeatedly trying to get it to respond. When you finally get it unlocked, the whole interface is just fiddly.

The other issue was the data quality. There were some nights when I’d been woken up, sometimes even getting up and moving around, and it just didn’t show it in the data. If it can’t even get whether I’m asleep right, can I trust anything else it says?

I spent a while researching the best devices for sleep tracking. The Oura ring came highly recommended, but it was expensive and required a subscription to do anything useful. No thanks! The Apple Watch was consistently rated pretty well, and I reasoned I could pick up a refurbished older unit. I’ve been using an iPhone as my daily driver for a while, so it’d fit right into my begrudged walled garden.

The Series 10 has a significant advantage, though: it charges much quicker than all the previous generations. On a 30-minute charge, the Series 10 can go from 0 to 60%; the 9 can only make it to 40%, and my old 5 a measly 30%. Shorter charge times means I’m far less likely to leave it on charge and wander off without it. In some ways the daily charging is more convenient than monthly: the wireless charger sits on my desk, and I plop the watch on it for a little while in the evening; I don’t need to dig out the weird pogo-pin connector that has vanished sometime in the last four weeks, then carefully arrange it so it stays attached.

How a watch maybe saved my life

One of the big features of the Apple Watch, like many other wearable devices, is health and fitness tracking. I didn’t think much about this, beyond the sleep data I wanted, at first. I’ve never had a particularly good relationship with exercise², but I do like some good statistics. I started going for walks more often to get more data and see the graphs of VO2 max and HR recovery gradually inch up. That wasn’t the most profound effect on my health, though…

The recent versions of the Apple Watch have a feature that monitors for sleep apnea, a disorder where you don’t breathe properly during your sleep. I knew I wasn’t sleeping great — that’s why I was paying attention to sleep tracking data — but was still a bit surprised to get a notification from Apple Health after wearing the watch for 30 days. It gives you a graph to print and take to your doctor. So I diligently booked an appointment and a few weeks later went to see my GP.

The appointment went about as you’d expect: talking about referral to a sleep centre for a study, and so on. Towards the end, the doctor took my blood pressure (sleep apnea can be caused by, and can cause high blood pressure, in a lovely little vicious cycle). I don’t think either of us expected anything; it was just one of those standard checks for a related problem. After taking the reading, the doctor looked at me and said, “I can’t let you leave with a BP like this”. Yikes!

Blood pressure readings are split into stages: normal is under 120 mmHg³ over 80 mmHg⁴, stage 1 is up to 140/90, stage 2 is up to 180/120, and above is simply called “crisis”. Guess where I was? Also, fun fact: depending on how exactly you count and attribute things, hypertension comes out as the single largest cause of death in humans. It doesn’t kill you outright, but contributes to strokes, coronary artery disease, heart failure, and lots of other lovely things you don’t want on your CV.

After doing a few more readings, it settled down to just under the “crisis” stage and into the “holy crap, start treatment immediately” stage instead. I won’t labour on much more about this, but things are definitely moving in the right direction now⁵.

While the Apple Watch didn’t literally save my life, it triggered the chain of events that led to this diagnosis and treatment. Who knows what would have happened had it remained undetected? Probably nothing good. Also, go check your blood pressure! It’s simple to do and simple to treat if there is an issue, but so many people are walking around with hypertension and not even realising.

Building gates in the walled garden

Even though the watch is arguably a life-saver, not everything is rosy. It’s an Apple product, so you’re firmly locked in a walled garden, jealously guarded by people wearing black turtlenecks. Fortunately, there are a few ways to make it less painful.

All the health and fitness data is stored in Apple Health. You can export data as a big messy file, but it’s a very manual process to do so and the data format is gnarly. Luckily, there’s an app for that! Health Auto Export can, well, automatically export health data. It does what it says on the tin. It can send the data to Home Assistant, over MQTT, or dump it in some cloud file services, but I just have it post it all to a REST endpoint on a service I wrote. Then I dump it all in a database and can do whatever I want with it!⁶

Another tool that’s more useful on the watch itself is Apple Shortcuts. This is their no-code “if-this-then-that”-type thing. You can make automations or shortcuts that run a number of tasks. I have a whole slew of them I access via a complication⁷ on my watch face: one which prompts for input and adds an item to my to-do list (swiping to write letters is surprisingly not horrible), one which lets me select from a bunch of pre-written ones (“charge kindle”, “take laundry out in 1 hour”, type things), one to log my weight into Apple Health, one which can open and close the blinds in my room, and so on. It’s a surprisingly robust and easy-to-use system and offers just enough freedom that I’m not constantly grating on the edge of the walled garden.

Shortcuts being able to initiate arbitrary web requests is the real killer feature for me. Anything I can’t do on the watch itself, I can just farm off to a web server and connect it up with a shortcut. No need to learn Swift or pay Apple for the privilege of being a developer! For a lot of things, like controlling the blinds or adding to-do items, I already had a HTTP endpoint available and exposed over Tailscale⁸. Adding it to the watch was just a case of entering the right things in the Shortcuts app.

Daily nitty-gritty

There are lots of other little bits and pieces that come up when using the watch daily. I don’t think I can bundle them up into a pleasing narrative arc, so instead please enjoy some disjointed paragraphs of observations.

The Apple Watch has a lot of nagging abilities. It can notify you about your fitness “rings”, prompt you to stand up every hour, count how many seconds you wash your hands for, etc. I think I’d dislike these just on general principle, but the way it does them is so condescending it’s painful. I think there’s probably a cultural divide issue here, but there is no way in British English to say “Great job! You washed your hands for 30 seconds!” without it sounding like you’re being amazingly sarcastic or like you’re talking to a young child. So I turned all of that nonsense off. It’s meant to be a tool not a wannabe life coach.

You can access Maps directly on the watch and even do navigation. It works really well. The navigation mode has some nice haptic feedback: it does a short pulse as you’re approaching a turn, and then a long pulse at the actual turn. I like it a lot more than having to dig out my phone or have the directions read out. You get one pulse, glance down and see where you need to go, then it reminds you a little later when it’s time to do it. It’s a delightful user experience.

Watchfaces aren’t quite so delightful. There’s a limited number of built-in ones, and some are “exclusive” to the Ultra Series, and you can’t use them on a peasant watch like a Series 10. Annoyingly, there’s not one that does exactly what I want: a plain analogue clock with an inset date and four complication slots around the outside. Instead, I have to use one of the slots to show the date. There are third-party watchfaces, but they have issues. Firstly, there’s no actual API for making watchfaces⁹, so what they do is bodge it horribly by using a photo background that has fake widgets on it. On top of that they’re almost universally subscription-based. Again, no thanks¹⁰.

The issue I had with the Smart Band triggering when I was sleeping is solved trivially on an Apple Watch, by virtue of it having a physical button in the crown. When you put it in sleep mode, you have to double-press the crown to unlock it before it’ll do anything else. It hasn’t misfired once while I’ve been using it.

Sleep data; spot when I swapped devices!

Finally, a special mention for the gesture controls. If you raise the watch it reliably wakes up (switching from a low refresh rate, dimmed screen to an active, brighter one); you can then double-tap your index finger and thumb together and it will scroll down or page through whatever you’re looking at. The killer feature for this is navigating recipes: you can advance to the next step in a recipe while your hands are covered in flour. It’s also handy for reading notifications: when one pops up, you can double-tap to expand it, then, when it gets to the bottom, it’ll outline the default button (often “Dismiss”) and you can double-tap again to click it.

The verdict

I normally don’t like writing an actual labelled conclusion, but it feels like one is needed here! Overall, I’m happy with the watch. The daily charging doesn’t bother me, the data gathered seems reliable, the health monitoring has obviously paid dividends already, and the walled garden isn’t too chafing. It’s a straight upgrade over my old Smart Band, and I think it was worth the cost.

The original reason for getting the watch was better sleep tracking, though, so how well did it do? I’m much happier with the data: it seems to more accurately represent when I was awake in the night, and overall the sleep phases just seem to make more sense. You can see in the graph that the old data switched frequently between phases, and they didn’t quite line up for some reason; towards the right when the Apple Watch is supplying the data instead there’s a much more consistent pattern of sleep phases that repeat over the course of the night.

I’m not going to advocate that you go out and buy one, though. I know my requirements and usage aren’t typical, and I’ve also not got experience with any recent Android Wear alternatives or the new version of the Pebble watch that’s coming soon. You should definitely get your blood pressure checked, though!

OK, it’s more like 14 now, it’s taken a while for this post to make its way from my brain into text. ↩︎
Exercise for the sake of exercising just seems so overwhelmingly tedious and boring to me. And other types of exercise generally require social interaction, coordinating with people, and so on. ↩︎
Who decided to use “millimetres of mercury displaced” as a unit? You can’t just put random chemical symbols in units! That’s not how this works! ↩︎
Blood pressure readings have two parts: systolic (the pressure when the heart is beating) and diastolic (the pressure between those beats). They’re generally presented with the systolic reading on top and the diastolic reading below, and read as “X over Y”. Now you know what some of the random numbers they shout in medical shows mean! Yay learning! ↩︎
No thanks to how much salt is in everything. I’m pretty sure I’ve had at least twice the recommended daily amount of salt in a single serving before. Don’t even get me started on the things that are “low salt” but are still full of sodium from other sources. I don’t have a problem with ionic compounds, I have a problem with sodium! ↩︎
This mainly looks like drawing graphs that are slightly different to the graphs in the Apple Health graphs, for reasons I’m not sure I can explain. Making graphs is fun, OK? ↩︎
Complications are basically just home screen widgets, but with a fancy name because they’re on a watch. ↩︎
Tailscale actually causes me some problems here: everything works fine when the watch is connected to my phone, as the phone handles the Tailscale part, but if I’m not carrying my phone the watch will try to connect over WiFi directly and doesn’t understand anything about Tailscale. It happens infrequently enough that I’ll just live with it; it’s not much worse than having no signal on a phone. ↩︎
Yay walled gardens… ↩︎
I don’t object to subscribing to things in general, but it has to be something that’s worth the ongoing cost and offers something in return for the subscription. A watchface doesn’t need enough ongoing maintenance to justify subscribing to it, it’s just a cash grab. ↩︎

Fixing a loud PSU fan without dying

2025-07-30T00:00:00Z

Three months after I built my new computer, it started annoying me. There would occasionally be a noise that sounded like a fan was catching on a cable, but there weren’t any loose cables to be a problem. Over the course of a few weeks, the sound got progressively worse to the extent that I didn’t want to use the computer without headphones on. I measured the sound at 63 dB, which is about the sound of someone talking. That may not sound terrible, but it’s a constant, nasty noise coming from something that sits about 40cm from my head.

After some investigating¹, I identified the PSU fan as the culprit. I have a Cooler Master V750 SFX, which is not super high-end, but wasn’t cheap, either. It shouldn’t be developing issues after three months. Thankfully, it comes with a ten-year warranty, so it should be easy to get sorted, right?

Warranty woes

I looked at Cooler Master’s warranty, and for issues within the first two years you have to deal with the retailer. That would be Amazon in my case. So I looked at Amazon’s information on warranty issues. Their policy is that if it’s more than 30 days since purchase, you have to send it off to a third-party repair center and wait for them to diagnose and try to repair it. Here’s the kicker:

Usually repairs take up to 20 business days (including delivery time), but could take slightly longer

I use the computer for work, have upcoming LAN parties to go to, and generally can’t do without it for an entire month. That’s assuming they reproduce the issue: the PSU fan only turns on when it reaches a certain temperature, so if you just plug it in briefly it won’t exhibit any symptoms.

If I was going to do a warranty return, I’d need to buy a new PSU first to use while this one was away. If I’m spending £100+ on a PSU, then I’d be getting a better one that doesn’t have known issues with the fans. So when the Cooler Master eventually got returned I’d have no use for it. What a waste.

So the warranty is about as much use as a chocolate teapot. Good to know, I guess. If I wanted decent service, I should have probably used a real retailer, not Amazon. Lesson learnt.

Doing some research

If the warranty is useless to me, maybe I should just void it and try to fix it myself. I did some research online. Most of the results went like this:

if you’re not a trained professional you should not be opening up your PSU at all. There’s a lot of current that goes through it which will seriously hurt/main/kill you

it’s time consuming and potentially dangerous

or some other variety of doom and gloom. There were some actually useful results sprinkled in, though. A post on the smallformfactor.net forums details someone swapping out the fan on the 850W version of the same power supply because it was too noisy. It looks like the fan connector in the PSU is a two-pin connector, and they just used an off-the-shelf adapter to convert the standard four-pin case fan connector. It sounded a bit annoying to make it all fit, though.

A similar post on reddit showed basically the same upgrade, but identified the connector as a 2.54mm JST-XH. They crimped their own connector on, which feels a lot better than using a bulky connector. I do have a crimping set for JST connectors, but I absolutely hate doing them because they’re so fiddly and annoying. I’m not sure if I trust one of my crimps inside a power supply, either.

Looking at the pictures in those posts, I identified the fan as an Ong Hua HA9215VH12FD-F00. I searched to see if I could see any specifications for it. I couldn’t. But what I did get back as my first result was another Reddit post, this one titled “Getting rid of a HA9215SH12FD-F00 fan from your PSU”. It says something about the quality of a component when the first result is a post about getting rid of it because it’s annoying².

All of these mods seemed easy enough, and I’m reasonably confident in my ability to both discharge and avoid touching capacitors, so I probably wouldn’t die. I ordered a Noctua NF-A9x14 fan and some pre-made JST-XH connectors and waited for them to arrive.

Upgrade time

The replacement parts arrived one evening, so when I turned the computer off for the night I unplugged it from the mains, held the power button down for a while, and left it overnight for good measure. The next morning I popped open the case, disconnected and removed the PSU, gleefully tore the “warranty void if removed” sticker off, and opened it up.

Removing the fan was simple: unplug the JST connector and undo the screws holding it to the outer panel. I confirmed that my JST connectors were the right size, then started surgery on the Noctua fan. I carefully removed the heatshrink, cut off the normal fan connector, and separated out the four wires. A quick check of their FAQ confirmed the wiring colours: black was ground, yellow +12V, and blue and green used for speed control (which we’re not using in this instance). I clipped the blue and green wires off near the hub so they wouldn’t be in the way, then cut the yellow and black wires down to length, and attached the JST connector with a solder seal:

Old fan, new fan, burnt mouse mat

In the process I discovered my mouse mat doesn’t like being hit with a heat gun. Whoops. The solder seals are great: the solder in the center melts to form an electrical connection, the parts at the end grip the cable to provide mechanical strength, and the whole thing acts like heatshrink and keeps the connection insulated. A lot less fiddly than crimping JST connectors, and a lot smaller and cheaper than an adapter.

I installed the fan and reassembled the PSU. Then undid it all and tried again, as apparently the face the fan mounts to isn’t quite symmetrical and I’d installed the fan upside down. Connected everything back up, plugged the computer in, booted it up, and… nothing. The PSU doesn’t actually power the fan until it hits a certain temperature. I figured putting it under load would probably heat it up, so ran a graphics benchmark a few times. Eventually the fan spun up. Huzzah.

Soon the entire computer will be beige and brown

Peace at last

I measured the sound of the new fan: 43 dB. That’s a decrease of 20 dB. Due to the weird logarithmic properties of the dB scale that no one understands, that means it sounds about 4 times quieter to human ears. Big improvement!

It’s worth pointing out this isn’t a straight one-to-one upgrade. I don’t know the specs of the old fan in terms of airflow, and I’m just assuming that the Noctua will be good enough by virtue of being a decent fan. They also behave slightly differently as the power ramps up: the stock fan starts spinning at a lower voltage than the Noctua, which needs around 7V to get started properly³. I don’t think this will be an issue, as it’s effectively just putting a small step at the lower end of the fan curve. It might shorten the overall lifespan of the PSU, but from my point of view it failed after four months, so any extra time is a bonus.

I recorded some before/after audio, so you can hear the difference. I think there’s some amount of volume normalisation going on, so it’s not quite as stark a difference as in person, but it’s still noticeable.

Audio sample from the old fan

Audio sample from the new fan

And if you couldn’t hear it, here’s a visual representation:

Side-by-side comparison of the audio clips

The whole upgrade took about an hour (less time than writing this post about it!), and I even managed to avoid killing myself.

Opening the case and sticking my ear near to various components. High-tech stuff. ↩︎
On closer inspection it’s actually a slightly different model — there’s an S not a V in the middle there — but I’m going to assume that doesn’t change much on the quality/noise front. ↩︎
It then keeps spinning down to about 5V or so, it just needs enough power to initially get started. ↩︎

Building a new Computer

2025-06-03T00:00:00Z

The finished PC. Cat pen for scale.

I recently¹ built a new computer, after exclusively using a laptop for three years. It’s also the first time I’ve departed from the usual combo of an Intel CPU and Nvidia GPU.

While the form factor of a laptop did make it amazingly handy for travelling and attending LAN events, it was starting to show its age and there was basically no sane upgrade path. The main problem was its 3060 mobile graphics card, which was okay for the first few years and then slowly descended into painful. At the time, the (fairly disappointing) 50-series had just been released, but hadn’t yet made it into laptops. Nvidia had stopped production on the 40-series beforehand, so there also weren’t any compelling options there, either.

That’s not to say there were no laptops at all that I could have upgraded to. There were. Just not really any that ticked all of my perfectly normal boxes like “run Linux”, “have at least 64GB of ram”, and “run modern games well”. Having to upgrade the entire system just because the graphics card was showing its age was a bit of a drag, too. So back to the land of desktop computers I went!

To try to preserve some of that convenience, I opted for a small form factor (SFF) case. The computer and all the accessories can fit inside a hand-luggage-sized flight case. More on that later, though.

Components, Choices, and Cramming

So what, exactly, is in this computer? The case is a Fractal Design Terra, which is a lovely little 10.4L case². My last desktop PC was a normal-sized Fractal Design case, which I liked a lot, and it seems like they’ve upped their game since then. The Terra is both pleasant to look at (just look at that wooden panel!), and a breeze to work in. The sides and top are fully and easily removable, giving you great access to everything inside³. I expected an SFF build to be fiddly, but the case made it feel about the same as a normal full-sized build⁴.

The case with sides and top off, and wires everywhere in desperate need of management.

Another nice feature of the case is the movable spine. I think this is relatively common in SFF cases, but it was new to me. You mount the graphics card on one side of a metal divider, and everything else on the other. The divider (or spine) can be moved side-to-side depending on what exactly you’re putting in the case.

As I mentioned earlier, this was my first departure from the Intel + Nvidia bandwagon. Nvidia’s 50 series was underwhelming, and Intel managed to put out a generation of chips that literally fried themselves. It seemed like a good time to try out AMD’s offering! I went for the newly released Radeon RX 9070XT graphics card, and a Ryzen 7 9800X3D processor⁵. The processor has a Thermalright AXP90 X47 Full heatsink on top of it, with a Noctua NF-A9 fan on top of that.

The motherboard is a Gigabyte B850I Aorus Pro, and it hosts 2 32GB Corsair Vengeance RAM modules, and a 2TB Crucial T705 NVMe drive. Powering it all is a Cooler Master V750 SFX. The PSU is modular to reduce the number of wires that need to be fitted in, but they’re still pretty big and bulky. I’m tempted by custom cables but they’re rather pricey, and you can’t actually see inside with the sides closed…

With the diminutive size of the case and all the components crammed in, there’s actually no room left over for case fans. The PSU, GPU and CPU all have fans attached, but that’s it. There are stories online of people having heat issues, but thankfully everything seems pretty happy in my setup. The components all draw in air from the sides, and the hot air passively vents out the top. If I keep the CPU under maximum load for a while then it starts to thermal throttle slightly, but the only time I’ve actually managed that is when compiling a kernel. Why was I compiling a kernel, you ask? Well, we’ll come to that.

But does it Linux?

I didn’t go into this build completely blind. I did a bit of reading and chatting, and it sounded like AMD drivers were pretty good on Linux these days. I didn’t check specifically if any of the components were supported.

The 9070XT was released on the 6th of March. I built my PC on the 26th of March. The first version of Mesa (the 3D graphics lib) that was generally stable with 9070XT cards landed in Arch on the 20th of March. If I did the build just one week earlier, I would’ve had a much less fun experience.

It still wasn’t quite perfectly smooth, though. When playing games I’d occasionally have issues. In some cases the screen just went blank and came back after switching windows, but in others the graphics drivers couldn’t recover and the PC needed rebooting. Fortunately the drivers are open source, and they have a public issue tracker hosted by the Freedesktop project. Searching through the issues, I found they’d already been reported (by another Arch user, of course), and there were various suggestions for work-arounds, as well as lots of work trying to reproduce the issue.

Within a week, an AMD developer had posted a set of patches that might address the issues, and asked people to test them. That’s fortunately pretty easy on Arch: you just clone the repository containing the PKGBUILD file, drop in the patches, and update the checksums for them. Then you start compiling, realise you didn’t tell it to use more than one CPU core, abort it, and start again but this time running on all cylinders. Soon enough I was running a patched 6.14 kernel and the problems stopped completely. Others reported similar outcomes, and the patches worked their way into the kernel proper.

I spent a few weeks having to compile a custom kernel every time I accidentally updated it (I should’ve just stopped pacman from touching the kernel for a bit…), then one day the patches didn’t apply because they’d been merged upstream. Since then everything has been perfect.

A friend commented that “this wouldn’t happen with Nvidia”. They meant it negatively, but I wholeheartedly agree! Maybe Nvidia have fewer launch bugs than AMD, but you absolutely wouldn’t be able to talk to their driver engineers on a public bug tracker, and get access to test patches that fix your issue. I’ll happily take a few bugs in exchange for a company that operates openly in a way consistent with the rest of the Linux community.

But does it travel?

Travel mode, engage!

So the computer works fine at home, but how do I get it to other places? I just shoved a laptop in a backpack⁶, but you can’t quite do that with a PC even one this size. I’d seen a few people at LANs who packed everything in flight cases, but they always seemed a bit unwieldy. Doing some research, though, I found that I’d just be able to squeeze everything into a flight case sized to be taken onto a plane as cabin baggage.

Peli (or Pelican in the US) are the most well known maker of flight cases, but I chose to go with Nanuk instead. Spec-wise they’re fairly similar. Nanuk are a bit cheaper. The killer difference for me, though, is that Nanuk publish STEP files of their cases. As someone with a 3D printer, giving me CAD models of things so that I can easily mod them is definitely a way to win my favour.

After lots of measuring and thinking and modelling I came up with an arrangement where everything could fit snugly into the case. The case has cutouts for the wheels and handles, which constrain what can go where. I ended up wanting the computer at the handle-end of the case to make best use of space, but that means it needs something to take its weight when the case is vertical. The solution was, of course, 3D printing.

Using the STEP files from Nanuk I designed an insert that spans across the case, with screw holes that line up with the ones in the flight case. I wasn’t sure it would be able to take the weight like that, so I also added in some supports in the other direction (running down to the bottom of the case, if it’s stood up). I ended up printing it in three parts, and it came out perfectly first time. I can’t emphasise how nice it is having CAD models instead of having to measure and guess at things.

With the support in place, I focused on cushioning the computer. I opted for 10mm thick EVA foam as it’s easy to obtain and work with, and doesn’t take up too much room. I used contact adhesive to glue the foam directly to the case and the 3D printed divider. There’s a piece that goes between the computer and the keyboard, so I printed some little 3D supports to hold that more firmly in place, and stuck those in with contact adhesive as well.

So how does it all hold up? It’s been to one LAN event so far, and it’s great. I think the flight case might actually be easier to carry than hefting a heavy backpack: the handles are sturdy and comfortable, and it has chunky wheels for pulling it around. It’s also actually easier to get everything set up when you can see it all laid out in the flight case and pull out what you need, and then the case can be tucked away out the way for the duration of the event.

Recently-ish. It may or may not have taken me a few months to get around to writing this post. ↩︎
Using litres to measure computer cases always feels wrong. Those are the units we use for water. Don’t put them near the computers! ↩︎
The case also has holes to put security screws in to stop all of these nice easily removable parts from being removable. Very handy when you’re leaving it unattended at a LAN. ↩︎
Which for me means it was good fun for about an hour, and then I got fed up and just wanted it to be done already. ↩︎
Three months in and I still can’t remember these random strings of numbers. Someone even explained the versioning scheme to me. ↩︎
Along with a portable monitor, mouse, keyboard, headset, mouse mat, kensington lock, power supply, cables, and controller. It was a heavy bag. ↩︎