Chameth.com - posts like docker-automatic-nginx-proxy, just-a-nod, why-you-should-be-using-https but not debugging-beyond-the-debugger, g15-ram-upgrade, migrating-from-github-to-forgejo

Monthly Meanderings: February 2026

2026-03-01T00:00:00Z

It doesn’t feel like a whole month has gone past since I wrote the last instalment of Monthly Meanderings, even allowing for how short a month February is. For more context on this series, you can check out the introduction to the first edition.

Website updates

I only wrote one new blog post this month: Just a nod, which is about the “nod” button I added to the bottom of most pages. I did a lot of other little bits and pieces, though:

My blog posts are now automatically posted to a dedicated Bluesky account, and posts now include syndication links at the bottom.
The site now tracks how long each request takes to generate along with how many SQL queries were used. You can see the details for the current request by hovering/tapping over the pi symbol at the bottom right of a page. To understand why it’s a pi symbol, you should watch The Net!
Speaking of films, I redid the star rating rendering so instead of unicode stars it uses some hand-drawn pixel art stickers. They’re somewhat randomised, so they don’t look repetitive. This is what it now looks like: .
I spent some time tweaking various styles and layouts so everything works nicely on smaller screens. I try to make sure these things work when I make them, but it’s easy to miss a particular interaction, or make an innocent-seeming change that breaks tiny screens.
I added a new page that shows stats about my walks, based on my Apple Health data. I had a private view of this for a while, but decided there’s no reason it couldn’t be more public.
The films page has gained a search box, to directly find my review of a particular film. I used CSS anchor positioning for the results panel. Modern CSS is fun!
I fixed the 3D prints page being utterly broken in numerous ways. It showed up on my shiny new “how many SQL queries does each page do?” dashboard, as it was doing multiple queries for each individual print instead of just getting all the data in one go. Then I noticed a badly written redirect rule had broken all the images and download links. Whoops!
I did a lot of under-the-hood cleaning, most of which won’t be visible. Besides just making me happy, this means the contact form and nod buttons now work with Javascript disabled.
Finally, I added a page that shows my board game plays. The data comes from exports from bgstats. I have a bunch of ideas for other things to do with that data, but haven’t got around to it yet.

Phew. After writing all that out it feels like a really long month, not a short one!

Other projects

Not much going on with my other projects. The only big thing was goplum, my self-hosted monitoring/alerting tool, getting a 1.0 release. It adds a reminder feature so you can get alerted repeatedly if a service is down. It also reworks how plugins work so they’re compile-time instead of trying to be clever and dynamic, as that never really gave any benefits.

I’m still continuing my migration of projects to a private Forgejo instance. All of my personal repos are now sourced there, and mirrored out to GitHub, except one: my Dockerfiles. I’m not sure how I want to approach that yet, as the action that does the updating is pretty involved. I also have some repositories split off into their own GitHub organisations which I haven’t got around to mirroring yet.

Entertainment

Unlike the past few months, I didn’t touch many video games in February. The ones I’d been dabbling with didn’t really hook me enough to make me keep going back. I made up for it with lots of other media forms though. First up, a whole bunch of films:

I won’t write too much about them here — you can click through to read my reviews if you want — other than to call out the three excellent horror films I hadn’t seen before: Black Swan, The Cabin in the Woods, and Get Out. Not so long ago I would’ve said that I didn’t enjoy horror films, now they’re some my favourites.

I don’t have any TV show tracking on my website (yet?), as I haven’t really watched much recently. This month I started watching The Pitt though, and it rapidly become one of my favourite shows. By “started watching” I mean “watched every aired episode of both season 1 and season 2”. I highly recommend it. The only problem is that it’s hard to find anything to watch afterwards, as nothing else can quite measure up.

I’ve also been binging a book series: Beneath the Dragoneye Moons. It’s a LitRPG series: basically a fantasy setting that borrows systems from an RPG, like classes and levelling. Something about the genre really works for me, and Beneath the Dragoneye Moons is one of the more interesting series I’ve come across in the genre. I’ve been going through the series at the rate of about a book a day.

Finally, board games! Nothing particularly notable, but now I can embed a pretty display of what I actually played:

6 plays
4 plays
4 plays
4 plays
2 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play

All of these were online via Board Game Arena. I haven’t really had time for in-person games with all the TV, films and reading!

Around the web

Start all of your commands with a comma

An old post that was doing the rounds again on Hacker News. It’s an interesting idea. I recently aliased cc to launch Claude Code with some custom parameters, and it horribly backfired when I tried to compile a C program and remembered that cc was part of GCC. I haven’t transitioned all my personal commands to use a prefix yet, but I might.

Tactical tornado is the new default

An interesting post about software developers who act as “tactical tornadoes”, and how that’s basically the role that all LLMs fill. Whenever I use coding agents, I often find myself following behind them in my IDE cleaning up bits and pieces. It’s often faster than prompting them to do it. I definitely get the “cleaning up after a tornado” metaphor.

Amusingly, the agents will sometimes notice that you’re tidying up after them and ascribe it to an automated process, outputting something like “the automated linter hasn’t run yet” if you fall behind. There’s probably a deeper philosophical discussion about what it means when the agent thinks that the human is the automated part of the system…

My smart sleep mask broadcasts users’ brainwaves to an open MQTT broker

Another one for the “the ’s’ in ‘IoT’ stands for security” books. I find it hard to actually comprehend how these things get to market. Does nobody actually think about it, or do they know full well how bad it is and just launch it anyway?

The IndieWeb and Small web

Ask three people to define ‘IndieWeb’ and ‘Small web’ and you’ll get about 20 different answers. This post by Christiano almost exactly nails my own views on the terms, and explains them well. I think most personal sites (my own included) straddle both terms to some extent, but having the labels is useful in discussions.

Just a nod

2026-02-11T00:00:00Z

Last month I added a new feature to the bottom of almost all pages on this site: a “nod” button. It’s a bit like the upvote buttons you see on bear blog posts, or the ubiquitous like button you get on social media, with one key difference: it doesn’t track or count anything. Before I get into the whys and hows, this is what it currently looks like:

The nod button in all its glory. If you want a clickable version you’ll need to scroll a bit more.

So why bother? I like hearing from people who have looked at my website. I have a contact form that’s also embedded at the bottom of all my blog posts; it’s always a delight when people reach out. Actually writing a message is a pretty high bar that most visitors won’t clear, though. A like/upvote/whatever-you-want-to-call-it is a much less demanding interaction.

And why no counting? I’d been thinking about some kind of upvote button for a while, and counting was always the thing that put me off. Having a number attached to a post like that feels like I’d end up invoking Goodhart’s law and start aiming for a certain number of upvotes or likes or whatever. I don’t want to be anxiously watching numbers, or feel pressured to make a post “good enough” (I already self-filter too much…).

As for not tracking people, that feels like table stakes to me: I’m an individual concerned about privacy, not a giant mega-corporation who wants to slurp up all the data to serve ads or engagement bait feeds. I don’t really need to know someone’s identity to receive a message from them. In the olden days of the internet, you just knew everyone by screen names which could change on a whim and didn’t reveal anything about the real you. On the internet, nobody knows you’re a dog, as Peter Steiner said.

All these constraints felt insurmountable for a while. I considered just making a button that sent a contact form submission with a pre-filled message, but those end up in my e-mail inbox. I don’t really want to have to process them, especially if there’s a big traffic spike like when Hacker News notices one of my posts. I could store the button presses in a database, but then how would I actually know they happened, especially if I didn’t want to watch a number go up?

An Amazon Dash button

So I did what I usually do with these types of problems: ignored it for a bit. The idea rattles around at the back of my head, and sometimes I’ll come up with an answer. I’ve seen this called diffuse thinking. Eventually the idea bounced to the forefront of my head while I was looking at my IRC client, and the solution suddenly clicked. If you’ve not heard of it, IRC is a chat protocol that’s almost as old as I am. Daniel Stenberg has compiled a nice history of IRC that I won’t repeat here. I use IRC to stay in touch with friends, hang around the channels for some open source projects, and I also run a private server with just me and some notification bots.

The notifications I have on IRC are all “transient”: they’re handy to see in the moment, but it doesn’t matter if I miss them. I have some high-volume news feeds, log messages from some self-hosted services, details about git commits, and so on. Basically just background noise of things that are happening that don’t need action. Perfect for the kind of interaction I want!

Now that I’d finally figured out that the transient-notification-shape goes in the transient-notification-hole, I just needed to actually implement it. As part of that private IRC setup I already have a bot that accepts messages over HTTP and sends them to a channel, so the backend was pretty simple. For the frontend I was imagining something like the ill-fated Amazon Dash buttons. My first attempt was… underwhelming:

Flat UIs are in right now, right?… Right?

I guess I shouldn’t give up the day job to become a graphic designer. Fortunately, I came across this codepen from Thierry Michel with some very fun skeuomorphic arcade buttons. I spent some time and LLM tokens on converting the SCSS to CSS, changing the colour, and making a few other tweaks. I really love how this looks and feels.

This is what it looks like in my IRC client when someone nods (and when containers get pushed to the GitHub Container Registry for my projects…):

Receiving some nods on IRC via a bot

When I added the button I wasn’t sure if I’d keep it, but I’ve very much enjoyed seeing the “nods” come in every now and then. I’ve been surprised by how many come in for the various snippets and other non-blog pages on the site. It’s a nice reminder that people are out there browsing the site, without all the baggage of monitoring analytics or tracking likes.

Monthly Meanderings: January 2026

2026-02-01T00:00:00Z

Welcome to the second edition of my monthly meanderings. For a bit of context, you can check out the introduction to the first edition.

Website updates

It’s been a pretty busy month for chameth.com. Three blog posts: The Meaning of Life — an entry into the IndieWeb carnival where I mostly review a book on Stoicism — Surge Protectors: Marketing vs Reality which is a dump of a research rabbit hole I fell into, and An Interesting Tailscale + Docker Gotcha which documents a fun problem I encountered.

I’ve also made a new films section, which is basically my Letterboxd data but free range instead of cooped up in a silo. I imported my 200-odd reviews, and for the last few I wrote the review on my site and then copied it over to Letterboxd. That’s what the IndieWeb folks call POSSE: publish on your own site, syndicate elsewhere. I like the idea that my website is the canonical source of truth for all the public things I generate. I want to expand this to some other areas in the future: music, video games, board games, etc.

I made most of the film components into “shortcodes”, so I can drop them into arbitrary pages or posts, like so:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Finally, I added a new widget to the bottom of most pages: a “nod” button. It’s effectively a “like” or “kudos” button but deliberately does not track how many times it’s been pressed, has no social network connotations, and won’t flood my inbox or hit rate limits if it gets spammed. If you click the button, it simply sends a message to a private IRC channel that says “Someone nodded at $URL”. I’ll probably write a bit more about it in a blog post in the future.

Other projects

I only have one small non-chameth.com update: a new project called pigeonhole which provides a really basic web service that lets you leave messages in a named cubby, and retrieve them at a later date. I’m using it to automate adding things to my Obsidian notes from my Apple Watch. There’s lots of reasons why that isn’t as simple as it sounds. I went through a bunch of iterations to try and make it work before just writing code to fix it.

I wanted to do this because I’m trying to replace Todoist with just having tasks in Obsidian. I already used Obsidian for longer form notes, but I’ve now also started using it for drafting blog posts (I’m typing this in Obsidian right now!), and trying to keep more useful day-to-day notes. I’ll probably write a blog post with more details once I’ve figured everything out.

Finally, I’ve been moving a bunch of projects to a private Forgejo instance, with the ultimate goal of having all of my projects live there. I’m going to mirror the contents out to GitHub and maybe other places, but Forgejo will be the single source of truth, and where all the workflows/dependency wrangling/etc happens. I’ve got Renovate configured to do dependency bumps, and I’m very much liking the whole setup so far. I’m saying it a lot but again: I’ll blog about it in more detail when it’s a bit further along.

Entertainment

I mentioned last month that I was playing Hogwarts: Legacy. I finished that off, and have been struggling to find something to scratch the same itch. I’ve tried a whole bunch of open world games, and basically none of them work for me. They have cumbersome control schemes (yay console ports), really awkward or sluggish movement, and so on. I think I’m just too fussy, but if I’m going to sink dozens of hours into a game it shouldn’t feel annoying or like a chore!

I did play through Thank Goodness You’re Here, which is delightfully funny but sadly only took 3.6 hours to finish with 100% of the achievements. Still highly recommend picking it up, especially when it’s on sale for under ‘10 bob’. I’ve also started playing through Control, and I’m enjoying the ambience. There’s an insanely sharp difficulty spike for the first boss, though, which nearly made me put it down.

In other entertainment, I’ve been playing a load of turn-based games on Board Game Arena , and I’ve watched a bunch of films after barely watching any last month. Here’s another of those shortcodes to show them off:

Around the web

Remotely unlocking an encrypted hard disk

A fun post documenting using Tailscale in a very weird way to solve a relatable problem. Very much my cup of tea.

I charged $18,000 for a static HTML page

This one’s actually from 2019, but it’s the first time I’ve come across it, and it’s a fun story. As a contractor, deciding what to bill for and what not to can be a bit messy. Downtime between meetings? The ten minutes in the shower you spent thinking through a bug? It’s a lot more clear cut if the client wants you to sit in an office, twiddling your thumbs, though.

Oxide RFD: Using LLMs at Oxide

A wonderfully well thought out piece about using LLMs in a respectful, sensible manner. I’d been thinking about writing something similar, but it wouldn’t have been anywhere near as well crafted as this. While I basically agree with it all, the part about LLM-generated text really resonates:

LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion.

Your app subscription is now my weekend project

Roberto points out that monthly SaaS subscriptions start looking even less appealing when you can just prompt an alternative into existence. I talked a bit about using LLMs for one-person software before, and it becomes even more appealing when the alternative is paying monthly for a service that keeps shoe-horning in features you don’t want, making you agree to share data with 329 “trusted partners”, and/or repeatedly ratcheting up the price.

Ekul’s Internet Space

I love the aesthetics of this personal site, and the data tracking that goes along with it. I’m also slightly aghast at the amount of Monster consumed! Makes me want to track more things and expose them on my own site.

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.