Chameth.com - posts like building-a-new-computer, docker-automatic-nginx-proxy, why-you-should-be-using-https but not android-espresso-spoon, debugging-beyond-the-debugger, migrating-from-github-to-forgejo

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Fixing a loud PSU fan without dying

2025-07-30T00:00:00Z

Three months after I built my new computer, it started annoying me. There would occasionally be a noise that sounded like a fan was catching on a cable, but there weren’t any loose cables to be a problem. Over the course of a few weeks, the sound got progressively worse to the extent that I didn’t want to use the computer without headphones on. I measured the sound at 63 dB, which is about the sound of someone talking. That may not sound terrible, but it’s a constant, nasty noise coming from something that sits about 40cm from my head.

After some investigating¹, I identified the PSU fan as the culprit. I have a Cooler Master V750 SFX, which is not super high-end, but wasn’t cheap, either. It shouldn’t be developing issues after three months. Thankfully, it comes with a ten-year warranty, so it should be easy to get sorted, right?

Warranty woes

I looked at Cooler Master’s warranty, and for issues within the first two years you have to deal with the retailer. That would be Amazon in my case. So I looked at Amazon’s information on warranty issues. Their policy is that if it’s more than 30 days since purchase, you have to send it off to a third-party repair center and wait for them to diagnose and try to repair it. Here’s the kicker:

Usually repairs take up to 20 business days (including delivery time), but could take slightly longer

I use the computer for work, have upcoming LAN parties to go to, and generally can’t do without it for an entire month. That’s assuming they reproduce the issue: the PSU fan only turns on when it reaches a certain temperature, so if you just plug it in briefly it won’t exhibit any symptoms.

If I was going to do a warranty return, I’d need to buy a new PSU first to use while this one was away. If I’m spending £100+ on a PSU, then I’d be getting a better one that doesn’t have known issues with the fans. So when the Cooler Master eventually got returned I’d have no use for it. What a waste.

So the warranty is about as much use as a chocolate teapot. Good to know, I guess. If I wanted decent service, I should have probably used a real retailer, not Amazon. Lesson learnt.

Doing some research

If the warranty is useless to me, maybe I should just void it and try to fix it myself. I did some research online. Most of the results went like this:

if you’re not a trained professional you should not be opening up your PSU at all. There’s a lot of current that goes through it which will seriously hurt/main/kill you

it’s time consuming and potentially dangerous

or some other variety of doom and gloom. There were some actually useful results sprinkled in, though. A post on the smallformfactor.net forums details someone swapping out the fan on the 850W version of the same power supply because it was too noisy. It looks like the fan connector in the PSU is a two-pin connector, and they just used an off-the-shelf adapter to convert the standard four-pin case fan connector. It sounded a bit annoying to make it all fit, though.

A similar post on reddit showed basically the same upgrade, but identified the connector as a 2.54mm JST-XH. They crimped their own connector on, which feels a lot better than using a bulky connector. I do have a crimping set for JST connectors, but I absolutely hate doing them because they’re so fiddly and annoying. I’m not sure if I trust one of my crimps inside a power supply, either.

Looking at the pictures in those posts, I identified the fan as an Ong Hua HA9215VH12FD-F00. I searched to see if I could see any specifications for it. I couldn’t. But what I did get back as my first result was another Reddit post, this one titled “Getting rid of a HA9215SH12FD-F00 fan from your PSU”. It says something about the quality of a component when the first result is a post about getting rid of it because it’s annoying².

All of these mods seemed easy enough, and I’m reasonably confident in my ability to both discharge and avoid touching capacitors, so I probably wouldn’t die. I ordered a Noctua NF-A9x14 fan and some pre-made JST-XH connectors and waited for them to arrive.

Upgrade time

The replacement parts arrived one evening, so when I turned the computer off for the night I unplugged it from the mains, held the power button down for a while, and left it overnight for good measure. The next morning I popped open the case, disconnected and removed the PSU, gleefully tore the “warranty void if removed” sticker off, and opened it up.

Removing the fan was simple: unplug the JST connector and undo the screws holding it to the outer panel. I confirmed that my JST connectors were the right size, then started surgery on the Noctua fan. I carefully removed the heatshrink, cut off the normal fan connector, and separated out the four wires. A quick check of their FAQ confirmed the wiring colours: black was ground, yellow +12V, and blue and green used for speed control (which we’re not using in this instance). I clipped the blue and green wires off near the hub so they wouldn’t be in the way, then cut the yellow and black wires down to length, and attached the JST connector with a solder seal:

Old fan, new fan, burnt mouse mat

In the process I discovered my mouse mat doesn’t like being hit with a heat gun. Whoops. The solder seals are great: the solder in the center melts to form an electrical connection, the parts at the end grip the cable to provide mechanical strength, and the whole thing acts like heatshrink and keeps the connection insulated. A lot less fiddly than crimping JST connectors, and a lot smaller and cheaper than an adapter.

I installed the fan and reassembled the PSU. Then undid it all and tried again, as apparently the face the fan mounts to isn’t quite symmetrical and I’d installed the fan upside down. Connected everything back up, plugged the computer in, booted it up, and… nothing. The PSU doesn’t actually power the fan until it hits a certain temperature. I figured putting it under load would probably heat it up, so ran a graphics benchmark a few times. Eventually the fan spun up. Huzzah.

Soon the entire computer will be beige and brown

Peace at last

I measured the sound of the new fan: 43 dB. That’s a decrease of 20 dB. Due to the weird logarithmic properties of the dB scale that no one understands, that means it sounds about 4 times quieter to human ears. Big improvement!

It’s worth pointing out this isn’t a straight one-to-one upgrade. I don’t know the specs of the old fan in terms of airflow, and I’m just assuming that the Noctua will be good enough by virtue of being a decent fan. They also behave slightly differently as the power ramps up: the stock fan starts spinning at a lower voltage than the Noctua, which needs around 7V to get started properly³. I don’t think this will be an issue, as it’s effectively just putting a small step at the lower end of the fan curve. It might shorten the overall lifespan of the PSU, but from my point of view it failed after four months, so any extra time is a bonus.

I recorded some before/after audio, so you can hear the difference. I think there’s some amount of volume normalisation going on, so it’s not quite as stark a difference as in person, but it’s still noticeable.

Audio sample from the old fan

Audio sample from the new fan

And if you couldn’t hear it, here’s a visual representation:

Side-by-side comparison of the audio clips

The whole upgrade took about an hour (less time than writing this post about it!), and I even managed to avoid killing myself.

Opening the case and sticking my ear near to various components. High-tech stuff. ↩︎
On closer inspection it’s actually a slightly different model — there’s an S not a V in the middle there — but I’m going to assume that doesn’t change much on the quality/noise front. ↩︎
It then keeps spinning down to about 5V or so, it just needs enough power to initially get started. ↩︎

Building a new Computer

2025-06-03T00:00:00Z

The finished PC. Cat pen for scale.

I recently¹ built a new computer, after exclusively using a laptop for three years. It’s also the first time I’ve departed from the usual combo of an Intel CPU and Nvidia GPU.

While the form factor of a laptop did make it amazingly handy for travelling and attending LAN events, it was starting to show its age and there was basically no sane upgrade path. The main problem was its 3060 mobile graphics card, which was okay for the first few years and then slowly descended into painful. At the time, the (fairly disappointing) 50-series had just been released, but hadn’t yet made it into laptops. Nvidia had stopped production on the 40-series beforehand, so there also weren’t any compelling options there, either.

That’s not to say there were no laptops at all that I could have upgraded to. There were. Just not really any that ticked all of my perfectly normal boxes like “run Linux”, “have at least 64GB of ram”, and “run modern games well”. Having to upgrade the entire system just because the graphics card was showing its age was a bit of a drag, too. So back to the land of desktop computers I went!

To try to preserve some of that convenience, I opted for a small form factor (SFF) case. The computer and all the accessories can fit inside a hand-luggage-sized flight case. More on that later, though.

Components, Choices, and Cramming

So what, exactly, is in this computer? The case is a Fractal Design Terra, which is a lovely little 10.4L case². My last desktop PC was a normal-sized Fractal Design case, which I liked a lot, and it seems like they’ve upped their game since then. The Terra is both pleasant to look at (just look at that wooden panel!), and a breeze to work in. The sides and top are fully and easily removable, giving you great access to everything inside³. I expected an SFF build to be fiddly, but the case made it feel about the same as a normal full-sized build⁴.

The case with sides and top off, and wires everywhere in desperate need of management.

Another nice feature of the case is the movable spine. I think this is relatively common in SFF cases, but it was new to me. You mount the graphics card on one side of a metal divider, and everything else on the other. The divider (or spine) can be moved side-to-side depending on what exactly you’re putting in the case.

As I mentioned earlier, this was my first departure from the Intel + Nvidia bandwagon. Nvidia’s 50 series was underwhelming, and Intel managed to put out a generation of chips that literally fried themselves. It seemed like a good time to try out AMD’s offering! I went for the newly released Radeon RX 9070XT graphics card, and a Ryzen 7 9800X3D processor⁵. The processor has a Thermalright AXP90 X47 Full heatsink on top of it, with a Noctua NF-A9 fan on top of that.

The motherboard is a Gigabyte B850I Aorus Pro, and it hosts 2 32GB Corsair Vengeance RAM modules, and a 2TB Crucial T705 NVMe drive. Powering it all is a Cooler Master V750 SFX. The PSU is modular to reduce the number of wires that need to be fitted in, but they’re still pretty big and bulky. I’m tempted by custom cables but they’re rather pricey, and you can’t actually see inside with the sides closed…

With the diminutive size of the case and all the components crammed in, there’s actually no room left over for case fans. The PSU, GPU and CPU all have fans attached, but that’s it. There are stories online of people having heat issues, but thankfully everything seems pretty happy in my setup. The components all draw in air from the sides, and the hot air passively vents out the top. If I keep the CPU under maximum load for a while then it starts to thermal throttle slightly, but the only time I’ve actually managed that is when compiling a kernel. Why was I compiling a kernel, you ask? Well, we’ll come to that.

But does it Linux?

I didn’t go into this build completely blind. I did a bit of reading and chatting, and it sounded like AMD drivers were pretty good on Linux these days. I didn’t check specifically if any of the components were supported.

The 9070XT was released on the 6th of March. I built my PC on the 26th of March. The first version of Mesa (the 3D graphics lib) that was generally stable with 9070XT cards landed in Arch on the 20th of March. If I did the build just one week earlier, I would’ve had a much less fun experience.

It still wasn’t quite perfectly smooth, though. When playing games I’d occasionally have issues. In some cases the screen just went blank and came back after switching windows, but in others the graphics drivers couldn’t recover and the PC needed rebooting. Fortunately the drivers are open source, and they have a public issue tracker hosted by the Freedesktop project. Searching through the issues, I found they’d already been reported (by another Arch user, of course), and there were various suggestions for work-arounds, as well as lots of work trying to reproduce the issue.

Within a week, an AMD developer had posted a set of patches that might address the issues, and asked people to test them. That’s fortunately pretty easy on Arch: you just clone the repository containing the PKGBUILD file, drop in the patches, and update the checksums for them. Then you start compiling, realise you didn’t tell it to use more than one CPU core, abort it, and start again but this time running on all cylinders. Soon enough I was running a patched 6.14 kernel and the problems stopped completely. Others reported similar outcomes, and the patches worked their way into the kernel proper.

I spent a few weeks having to compile a custom kernel every time I accidentally updated it (I should’ve just stopped pacman from touching the kernel for a bit…), then one day the patches didn’t apply because they’d been merged upstream. Since then everything has been perfect.

A friend commented that “this wouldn’t happen with Nvidia”. They meant it negatively, but I wholeheartedly agree! Maybe Nvidia have fewer launch bugs than AMD, but you absolutely wouldn’t be able to talk to their driver engineers on a public bug tracker, and get access to test patches that fix your issue. I’ll happily take a few bugs in exchange for a company that operates openly in a way consistent with the rest of the Linux community.

But does it travel?

Travel mode, engage!

So the computer works fine at home, but how do I get it to other places? I just shoved a laptop in a backpack⁶, but you can’t quite do that with a PC even one this size. I’d seen a few people at LANs who packed everything in flight cases, but they always seemed a bit unwieldy. Doing some research, though, I found that I’d just be able to squeeze everything into a flight case sized to be taken onto a plane as cabin baggage.

Peli (or Pelican in the US) are the most well known maker of flight cases, but I chose to go with Nanuk instead. Spec-wise they’re fairly similar. Nanuk are a bit cheaper. The killer difference for me, though, is that Nanuk publish STEP files of their cases. As someone with a 3D printer, giving me CAD models of things so that I can easily mod them is definitely a way to win my favour.

After lots of measuring and thinking and modelling I came up with an arrangement where everything could fit snugly into the case. The case has cutouts for the wheels and handles, which constrain what can go where. I ended up wanting the computer at the handle-end of the case to make best use of space, but that means it needs something to take its weight when the case is vertical. The solution was, of course, 3D printing.

Using the STEP files from Nanuk I designed an insert that spans across the case, with screw holes that line up with the ones in the flight case. I wasn’t sure it would be able to take the weight like that, so I also added in some supports in the other direction (running down to the bottom of the case, if it’s stood up). I ended up printing it in three parts, and it came out perfectly first time. I can’t emphasise how nice it is having CAD models instead of having to measure and guess at things.

With the support in place, I focused on cushioning the computer. I opted for 10mm thick EVA foam as it’s easy to obtain and work with, and doesn’t take up too much room. I used contact adhesive to glue the foam directly to the case and the 3D printed divider. There’s a piece that goes between the computer and the keyboard, so I printed some little 3D supports to hold that more firmly in place, and stuck those in with contact adhesive as well.

So how does it all hold up? It’s been to one LAN event so far, and it’s great. I think the flight case might actually be easier to carry than hefting a heavy backpack: the handles are sturdy and comfortable, and it has chunky wheels for pulling it around. It’s also actually easier to get everything set up when you can see it all laid out in the flight case and pull out what you need, and then the case can be tucked away out the way for the duration of the event.

Recently-ish. It may or may not have taken me a few months to get around to writing this post. ↩︎
Using litres to measure computer cases always feels wrong. Those are the units we use for water. Don’t put them near the computers! ↩︎
The case also has holes to put security screws in to stop all of these nice easily removable parts from being removable. Very handy when you’re leaving it unattended at a LAN. ↩︎
Which for me means it was good fun for about an hour, and then I got fed up and just wanted it to be done already. ↩︎
Three months in and I still can’t remember these random strings of numbers. Someone even explained the versioning scheme to me. ↩︎
Along with a portable monitor, mouse, keyboard, headset, mouse mat, kensington lock, power supply, cables, and controller. It was a heavy bag. ↩︎

Adventures in 3D printing

2023-09-17T00:00:00Z

The Sovol SV06.

I’d been idly considering getting a 3D printer for a while, but have only recently taken the plunge. I picked up a Sovol SV06 from Amazon for £199.99¹, which is a model commonly recommended for beginners. About three weeks later, I think I’ve finally finished fixing all the problems the printer has, and thought I’d document them.

Setup and out of the box performance

The setup of the printer itself was straight forward. Most parts are assembled, you just have to bolt the frame together, bolt the various parts to the frame, and connect some wires. The hex bits in the iFixit Mako Bit Set were a godsend for this, as the included hex keys were a bit flimsy.

Once everything is bolted and connected, you turn it on and immediately start looking around for the jet engine you can hear. The fans are loud. The instructions walk you through the initial calibration, which consists of:

Levelling the X-Axis gantry (where the print head moves). The printer does this by (gently) ramming the gantry up as far as it will go.
Adjusting the Z-offset of the print head. The printer has an inductive proximity sensor attached to the extruder, but it doesn’t know how high it is relative to the nozzle. The calibration process involves manually dialing in this offset while moving a piece of paper back and forth under the nozzle.
Levelling the bed. This is entirely automatic: the printer just probes a bunch of points on the bed with the proximity sensor and builds up a mesh it can use to compensate for the bed not being level.

After that’s done, the included SD card has the standard “Benchy” model on it, pre-sliced and ready to print. It came out looking good:

A nice, problem free Benchy. No dramatic irony here.

The noise. Oh god, the noise.

I set about printing a few more things, but the noise of the machine quickly got to me. It sits approximately 80cm from my ear, and between the fans and the noises it makes when moving it was driving me mad. The fan in the PSU seems to sit at 100% power regardless of temperature or load, and sounds about three times louder than my laptop’s fans do when they’re running at full tilt. Fortunately, someone had designed a printable PSU fan silencer that redirects the airflow (and most of the noise) from the fan. It doesn’t feel like it should help that much, but it does significantly reduce the noise as it claims. This was the first of many times I’d use the 3D printer to print parts for itself.

The other noise was a lot less straightforward. The linear rods that all three axes run on pass through several bearings. Sovol apparently don’t bother to apply any lubrication to these, so it sounds like an elephant is moving through a gravel pit when the printer is running. Some people online suggest rubbing lube on the rods and pushing the movable parts back and forth, but this just results in a load of lube caught on the dust filters of the bearings².

The solution to this, then, was to dismantle most of the printer to gain access to each of the ten bearings, and manually lubricate them. This was tedious and messy, but not actually that difficult. It doesn’t seem like the kind of thing you should have to do on a brand-new machine, but I guess you get what you pay for.

Speeeeeeed

3D printers are slow. The SV06 is not one of the faster ones. One simple way to increase the print speed is to change out the nozzle. The SV06 comes with a 0.4mm nozzle, which is pretty standard, but if you upgrade to a 0.6mm nozzle you can extrude more than twice the volume of plastic in the same amount of time³. The consensus among anonymous internet people is that if you’re using a modern slicer there isn’t much difference in print quality when doing this.

I dutifully ordered a new nozzle, and after it turned up set about installing it. This is mostly straight forward, aside from the complication that you’re meant to do it when the hot-end is hot. That makes sense: you want it be under the same thermal expansion it will be under during use, but the tool they provide to change the nozzle is all metal. So you have a relatively delicate operation that you have to do in a bit of a rush before the heat creeps all the way up to your hand.

With the new nozzle in place, the printer was noticeably faster. I didn’t notice any difference in print quality, either.

SD cards are not fun

Up until this point, my process for printing anything was to slice it on the computer, export the gcode file to an SD card, turn around and walk over to the printer, insert the SD card, and start the print from the LCD. I don’t particularly mind those steps, but the SD card slot on the Sovol is terrible. It doesn’t line up well to the hole in the case, doesn’t guide the card in well, and is all round a pain to use.⁴

To remove the sneaker-net element of this process, I ordered a Raspberry Pi and installed OctoPrint on it. OctoPrint provides a web interface to the printer (which it controls over USB), and most slicers can automatically export the gcode to an OctoPrint instance.

While OctoPrint itself is great, there was one small problem: if you connect anything to the USB port on the printer, it tries to draw 5V from it and “backpower” itself. It’s not meant to draw power from the USB port, it’s just not designed well. A lot of the cheaper 3D printers share this flaw, it seems. The fix here is to get a power blocker such as the one made by PortaPow that simply leaves the +5V line unconnected between the two ends. Presumably Sovol could fix this by adding a diode to the USB power line, but maybe that would push their costs too high?

After getting OctoPrint working, I printed a nice case for it to mount on the side of the printer. I slotted the Pi in, and when I powered it on again it didn’t boot. When I went to pick it up to debug the problem, I burnt my finger on the SD card⁵. Apparently if you break SD cards in a certain way, they can short out and get very hot. Fun! Maybe I shouldn’t have been using the no-brand card that came with the printer…

What do we do with a drunken bed?

I continued printing bits and bobs, but was starting to notice a new issue: if I was printing multiple copies of the same object, those on the right hand side of the bed often failed. OctoPrint has a nifty plugin that lets you cancel certain regions from the build, which saves it ruining everything, but only being able to print in certain places was annoying.

I ran through all the troubleshooting steps I could think of, including a foray into custom firmware with different bed levelling abilities, but nothing seemed to help. Another OctoPrint plugin later and I could see the problem:

This should probably be straight.

The orientation of the graphic is a bit confusing, but you can see how the right hand side of the bed is significantly higher than the left. There’s a total variance of just over 0.5mm, which is apparently enough to cause problems even with the auto bed levelling. I spoke to Sovol support, and eventually they suggested disassembling the print bed and measuring the aluminium spacers that hold the bed. Sure enough, they were all different heights:

You had one job…

There’s a 0.13mm variance in their heights, which doesn’t account for the 0.5mm seen in the bed mesh, but it’s a good starting point. Rather than go back and forth with Sovol support over the course of many days, I just ordered some nylock nuts and reinstalled the bed without spacers. The nuts hold the bed in place, while still allowing you to tighten or loosen the screws to adjust the height.

After five or six rounds of totally and utterly uneventful adjusting and re-leveling, I got to this point:

This is much more straight.

There’s only so much you can do when you only have five screws to adjust, but I’m pretty happy the variance is now under 0.15mm. That’s definitely in the “good enough” range that the bed leveling algorithm can compensate for.

Whoops

So, about that “totally and utterly uneventful adjusting” I did. To get at the screws that hold the bed in place, you have to remove the thin sheet that acts as the build surface. This attaches magnetically to the bed, so it’s easy to do, but I am lazy and decided to leave it off while I fiddled with the bed.

There’s a slight problem here, though: when it’s working out the bed mesh the printer uses the inductive proximity sensor. The inductive proximity sensor detects metal. The metal is in the removable sheet. The first time I tried to run the bed mesh, the printer rammed the extruder assembly into the bed at a decent speed and ended up at a rather nasty angle.

I checked over the extruder and found the heat break had bent:

This should also probably be straight.

I tried printing with it as-is, but the filament came out too thin and at an angle, which made the print go about as well as you’d expect. I ordered a replacement, installed it, and… nothing worked still.

At this point I was getting thoroughly fed up. There seemed to be one problem after another, and no good way to diagnose problems with the print not sticking (because there are a dozen or so different potential causes). After a lot of frustrated googling, I found a post on the Prusa forums that suggested cleaning the build sheet with dish soap. I’d cleaned it with IPA, but several people were saying that dish soap magically fixed issues that IPA couldn’t. I took the plate to the kitchen, gave it a wash, and the next print went down perfectly.

Growing a backbone

For a blissful period of about four days the printer was chugging along perfectly with no issues. Then it started making a weird noise. In a display of amazing restraint, I didn’t immediately throw the whole thing out the window. Looking at the printer, I could see that the cable running from the extruder assembly to the main board was sagging and rubbing on top of the print as the print head was moving around. I’m not sure what property of the cable was keeping it aloft before, but it apparently ran out after about three weeks.

Like most of the other problems with the SV06, this design flaw has been noted by others and worked around. I printed a cable chain and installed it⁶. Amusingly, while I was printing the chain, the cable dragged on the print and dislodged one of the pieces entirely before it was finished. Fortunately, it didn’t affect the neighbouring pieces, so I just had to run off a replacement for that one.

The difference is rather pronounced:

It feels like I own a 3D printer to print parts for my 3D printer…

If you can’t see it: in the first image the cable is sagging and actually ends up below the print bed. The individual links of the cable chain limit its freedom of movement, stopping it sagging.

The end?

Once again the printer is printing properly again. I’m sure there will be more problems, but hopefully I get at least a week or so before anything else goes wrong. Maybe I’ll print an “X days since the last SV06 problem” counter…

I think, if I had to do this all again, but with the knowledge I have now, I’d have stumped up the extra cash to get a Prusa MK3S+⁷. This is the original machine on which the SV06 (and a lot of other cheap imitations) is based. Some obvious benefits it has over the SV06:

The extruder cable runs on the outside of the machine, so won’t droop onto prints
The heated bed has copper embedded in it for use in the induction sensor, so it won’t crash into the bed if the build sheet is removed
They actually put lubricant in the bearings instead of shipping them bone dry
The bed is held in place with screws instead of poorly sized aluminium spacers, so you can adjust the height without having to take the whole thing apart
They don’t use the cheapest possible power supply and main board, so it doesn’t backpower off of USB devices or have the loudest fans known to mankind

Prusa also have a much better attitude towards 3D printing: they open source the hardware designs (which is why there are so many cheap clones), and they actually use the printers themselves. The plastic parts of an SV06 are injection moulded, but those on a Prusa printer are actually printed on a Prusa printer. Not only is it cool that they use their own products, it means they have to run them en-masse so they’re obviously invested in making them reliable. They also have a reputation for exceptional support.

Despite all the problems, I do enjoy having the printer. I’ll probably talk more about the prints I’ve done that weren’t for the 3D printer itself in the future.

The RRP is £279.99, Amazon discounted it down to £269.99 then offered a £70 voucher. The voucher says it expires at midnight, but it’s said that every day for the past three weeks… ↩︎
Which is exactly what another set of people said would happen. It was worth a try, though. ↩︎
This feels strange, but if you work out the areas of the nozzles it makes sense: a 0.4mm diameter nozzle has an area of 0.04πmm² (the radius is half the diameter, and the area of a circle is πr²), and a 0.6mm diameter nozzle has an area of 0.09πmm². Maths! ↩︎
At least it doesn’t eject the SD card across the room like this Ender 3 V2! ↩︎
My only 3D printer related injury so far. It’s impressive I can be around a 200 degree hot-end and somehow burn myself on a storage device. ↩︎
Well actually I printed 5 copies of their calibration piece until it was perfect, then printed the actual cable chain, then had to sand each of the individual links because they still weren’t good enough. God, I hate sanding. ↩︎
Or maybe I’d cave and get the newer, shinier, much more expensive MK4. That wouldn’t be out of character. No, I didn’t just check my bank balance. You’re imagining things. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.