Chameth.com - posts like artisanal-docker-images, just-a-nod but not debugging-beyond-the-debugger, migrating-from-github-to-forgejo, why-you-should-be-using-https

Monthly Meanderings: February 2026

2026-03-01T00:00:00Z

It doesn’t feel like a whole month has gone past since I wrote the last instalment of Monthly Meanderings, even allowing for how short a month February is. For more context on this series, you can check out the introduction to the first edition.

Website updates

I only wrote one new blog post this month: Just a nod, which is about the “nod” button I added to the bottom of most pages. I did a lot of other little bits and pieces, though:

My blog posts are now automatically posted to a dedicated Bluesky account, and posts now include syndication links at the bottom.
The site now tracks how long each request takes to generate along with how many SQL queries were used. You can see the details for the current request by hovering/tapping over the pi symbol at the bottom right of a page. To understand why it’s a pi symbol, you should watch The Net!
Speaking of films, I redid the star rating rendering so instead of unicode stars it uses some hand-drawn pixel art stickers. They’re somewhat randomised, so they don’t look repetitive. This is what it now looks like: .
I spent some time tweaking various styles and layouts so everything works nicely on smaller screens. I try to make sure these things work when I make them, but it’s easy to miss a particular interaction, or make an innocent-seeming change that breaks tiny screens.
I added a new page that shows stats about my walks, based on my Apple Health data. I had a private view of this for a while, but decided there’s no reason it couldn’t be more public.
The films page has gained a search box, to directly find my review of a particular film. I used CSS anchor positioning for the results panel. Modern CSS is fun!
I fixed the 3D prints page being utterly broken in numerous ways. It showed up on my shiny new “how many SQL queries does each page do?” dashboard, as it was doing multiple queries for each individual print instead of just getting all the data in one go. Then I noticed a badly written redirect rule had broken all the images and download links. Whoops!
I did a lot of under-the-hood cleaning, most of which won’t be visible. Besides just making me happy, this means the contact form and nod buttons now work with Javascript disabled.
Finally, I added a page that shows my board game plays. The data comes from exports from bgstats. I have a bunch of ideas for other things to do with that data, but haven’t got around to it yet.

Phew. After writing all that out it feels like a really long month, not a short one!

Other projects

Not much going on with my other projects. The only big thing was goplum, my self-hosted monitoring/alerting tool, getting a 1.0 release. It adds a reminder feature so you can get alerted repeatedly if a service is down. It also reworks how plugins work so they’re compile-time instead of trying to be clever and dynamic, as that never really gave any benefits.

I’m still continuing my migration of projects to a private Forgejo instance. All of my personal repos are now sourced there, and mirrored out to GitHub, except one: my Dockerfiles. I’m not sure how I want to approach that yet, as the action that does the updating is pretty involved. I also have some repositories split off into their own GitHub organisations which I haven’t got around to mirroring yet.

Entertainment

Unlike the past few months, I didn’t touch many video games in February. The ones I’d been dabbling with didn’t really hook me enough to make me keep going back. I made up for it with lots of other media forms though. First up, a whole bunch of films:

I won’t write too much about them here — you can click through to read my reviews if you want — other than to call out the three excellent horror films I hadn’t seen before: Black Swan, The Cabin in the Woods, and Get Out. Not so long ago I would’ve said that I didn’t enjoy horror films, now they’re some my favourites.

I don’t have any TV show tracking on my website (yet?), as I haven’t really watched much recently. This month I started watching The Pitt though, and it rapidly become one of my favourite shows. By “started watching” I mean “watched every aired episode of both season 1 and season 2”. I highly recommend it. The only problem is that it’s hard to find anything to watch afterwards, as nothing else can quite measure up.

I’ve also been binging a book series: Beneath the Dragoneye Moons. It’s a LitRPG series: basically a fantasy setting that borrows systems from an RPG, like classes and levelling. Something about the genre really works for me, and Beneath the Dragoneye Moons is one of the more interesting series I’ve come across in the genre. I’ve been going through the series at the rate of about a book a day.

Finally, board games! Nothing particularly notable, but now I can embed a pretty display of what I actually played:

6 plays
4 plays
4 plays
4 plays
2 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play

All of these were online via Board Game Arena. I haven’t really had time for in-person games with all the TV, films and reading!

Around the web

Start all of your commands with a comma

An old post that was doing the rounds again on Hacker News. It’s an interesting idea. I recently aliased cc to launch Claude Code with some custom parameters, and it horribly backfired when I tried to compile a C program and remembered that cc was part of GCC. I haven’t transitioned all my personal commands to use a prefix yet, but I might.

Tactical tornado is the new default

An interesting post about software developers who act as “tactical tornadoes”, and how that’s basically the role that all LLMs fill. Whenever I use coding agents, I often find myself following behind them in my IDE cleaning up bits and pieces. It’s often faster than prompting them to do it. I definitely get the “cleaning up after a tornado” metaphor.

Amusingly, the agents will sometimes notice that you’re tidying up after them and ascribe it to an automated process, outputting something like “the automated linter hasn’t run yet” if you fall behind. There’s probably a deeper philosophical discussion about what it means when the agent thinks that the human is the automated part of the system…

My smart sleep mask broadcasts users’ brainwaves to an open MQTT broker

Another one for the “the ’s’ in ‘IoT’ stands for security” books. I find it hard to actually comprehend how these things get to market. Does nobody actually think about it, or do they know full well how bad it is and just launch it anyway?

The IndieWeb and Small web

Ask three people to define ‘IndieWeb’ and ‘Small web’ and you’ll get about 20 different answers. This post by Christiano almost exactly nails my own views on the terms, and explains them well. I think most personal sites (my own included) straddle both terms to some extent, but having the labels is useful in discussions.

Just a nod

2026-02-11T00:00:00Z

Last month I added a new feature to the bottom of almost all pages on this site: a “nod” button. It’s a bit like the upvote buttons you see on bear blog posts, or the ubiquitous like button you get on social media, with one key difference: it doesn’t track or count anything. Before I get into the whys and hows, this is what it currently looks like:

The nod button in all its glory. If you want a clickable version you’ll need to scroll a bit more.

So why bother? I like hearing from people who have looked at my website. I have a contact form that’s also embedded at the bottom of all my blog posts; it’s always a delight when people reach out. Actually writing a message is a pretty high bar that most visitors won’t clear, though. A like/upvote/whatever-you-want-to-call-it is a much less demanding interaction.

And why no counting? I’d been thinking about some kind of upvote button for a while, and counting was always the thing that put me off. Having a number attached to a post like that feels like I’d end up invoking Goodhart’s law and start aiming for a certain number of upvotes or likes or whatever. I don’t want to be anxiously watching numbers, or feel pressured to make a post “good enough” (I already self-filter too much…).

As for not tracking people, that feels like table stakes to me: I’m an individual concerned about privacy, not a giant mega-corporation who wants to slurp up all the data to serve ads or engagement bait feeds. I don’t really need to know someone’s identity to receive a message from them. In the olden days of the internet, you just knew everyone by screen names which could change on a whim and didn’t reveal anything about the real you. On the internet, nobody knows you’re a dog, as Peter Steiner said.

All these constraints felt insurmountable for a while. I considered just making a button that sent a contact form submission with a pre-filled message, but those end up in my e-mail inbox. I don’t really want to have to process them, especially if there’s a big traffic spike like when Hacker News notices one of my posts. I could store the button presses in a database, but then how would I actually know they happened, especially if I didn’t want to watch a number go up?

An Amazon Dash button

So I did what I usually do with these types of problems: ignored it for a bit. The idea rattles around at the back of my head, and sometimes I’ll come up with an answer. I’ve seen this called diffuse thinking. Eventually the idea bounced to the forefront of my head while I was looking at my IRC client, and the solution suddenly clicked. If you’ve not heard of it, IRC is a chat protocol that’s almost as old as I am. Daniel Stenberg has compiled a nice history of IRC that I won’t repeat here. I use IRC to stay in touch with friends, hang around the channels for some open source projects, and I also run a private server with just me and some notification bots.

The notifications I have on IRC are all “transient”: they’re handy to see in the moment, but it doesn’t matter if I miss them. I have some high-volume news feeds, log messages from some self-hosted services, details about git commits, and so on. Basically just background noise of things that are happening that don’t need action. Perfect for the kind of interaction I want!

Now that I’d finally figured out that the transient-notification-shape goes in the transient-notification-hole, I just needed to actually implement it. As part of that private IRC setup I already have a bot that accepts messages over HTTP and sends them to a channel, so the backend was pretty simple. For the frontend I was imagining something like the ill-fated Amazon Dash buttons. My first attempt was… underwhelming:

Flat UIs are in right now, right?… Right?

I guess I shouldn’t give up the day job to become a graphic designer. Fortunately, I came across this codepen from Thierry Michel with some very fun skeuomorphic arcade buttons. I spent some time and LLM tokens on converting the SCSS to CSS, changing the colour, and making a few other tweaks. I really love how this looks and feels.

This is what it looks like in my IRC client when someone nods (and when containers get pushed to the GitHub Container Registry for my projects…):

Receiving some nods on IRC via a bot

When I added the button I wasn’t sure if I’d keep it, but I’ve very much enjoyed seeing the “nods” come in every now and then. I’ve been surprised by how many come in for the various snippets and other non-blog pages on the site. It’s a nice reminder that people are out there browsing the site, without all the baggage of monitoring analytics or tracking likes.

Monthly Meanderings: January 2026

2026-02-01T00:00:00Z

Welcome to the second edition of my monthly meanderings. For a bit of context, you can check out the introduction to the first edition.

Website updates

It’s been a pretty busy month for chameth.com. Three blog posts: The Meaning of Life — an entry into the IndieWeb carnival where I mostly review a book on Stoicism — Surge Protectors: Marketing vs Reality which is a dump of a research rabbit hole I fell into, and An Interesting Tailscale + Docker Gotcha which documents a fun problem I encountered.

I’ve also made a new films section, which is basically my Letterboxd data but free range instead of cooped up in a silo. I imported my 200-odd reviews, and for the last few I wrote the review on my site and then copied it over to Letterboxd. That’s what the IndieWeb folks call POSSE: publish on your own site, syndicate elsewhere. I like the idea that my website is the canonical source of truth for all the public things I generate. I want to expand this to some other areas in the future: music, video games, board games, etc.

I made most of the film components into “shortcodes”, so I can drop them into arbitrary pages or posts, like so:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Finally, I added a new widget to the bottom of most pages: a “nod” button. It’s effectively a “like” or “kudos” button but deliberately does not track how many times it’s been pressed, has no social network connotations, and won’t flood my inbox or hit rate limits if it gets spammed. If you click the button, it simply sends a message to a private IRC channel that says “Someone nodded at $URL”. I’ll probably write a bit more about it in a blog post in the future.

Other projects

I only have one small non-chameth.com update: a new project called pigeonhole which provides a really basic web service that lets you leave messages in a named cubby, and retrieve them at a later date. I’m using it to automate adding things to my Obsidian notes from my Apple Watch. There’s lots of reasons why that isn’t as simple as it sounds. I went through a bunch of iterations to try and make it work before just writing code to fix it.

I wanted to do this because I’m trying to replace Todoist with just having tasks in Obsidian. I already used Obsidian for longer form notes, but I’ve now also started using it for drafting blog posts (I’m typing this in Obsidian right now!), and trying to keep more useful day-to-day notes. I’ll probably write a blog post with more details once I’ve figured everything out.

Finally, I’ve been moving a bunch of projects to a private Forgejo instance, with the ultimate goal of having all of my projects live there. I’m going to mirror the contents out to GitHub and maybe other places, but Forgejo will be the single source of truth, and where all the workflows/dependency wrangling/etc happens. I’ve got Renovate configured to do dependency bumps, and I’m very much liking the whole setup so far. I’m saying it a lot but again: I’ll blog about it in more detail when it’s a bit further along.

Entertainment

I mentioned last month that I was playing Hogwarts: Legacy. I finished that off, and have been struggling to find something to scratch the same itch. I’ve tried a whole bunch of open world games, and basically none of them work for me. They have cumbersome control schemes (yay console ports), really awkward or sluggish movement, and so on. I think I’m just too fussy, but if I’m going to sink dozens of hours into a game it shouldn’t feel annoying or like a chore!

I did play through Thank Goodness You’re Here, which is delightfully funny but sadly only took 3.6 hours to finish with 100% of the achievements. Still highly recommend picking it up, especially when it’s on sale for under ‘10 bob’. I’ve also started playing through Control, and I’m enjoying the ambience. There’s an insanely sharp difficulty spike for the first boss, though, which nearly made me put it down.

In other entertainment, I’ve been playing a load of turn-based games on Board Game Arena , and I’ve watched a bunch of films after barely watching any last month. Here’s another of those shortcodes to show them off:

Around the web

Remotely unlocking an encrypted hard disk

A fun post documenting using Tailscale in a very weird way to solve a relatable problem. Very much my cup of tea.

I charged $18,000 for a static HTML page

This one’s actually from 2019, but it’s the first time I’ve come across it, and it’s a fun story. As a contractor, deciding what to bill for and what not to can be a bit messy. Downtime between meetings? The ten minutes in the shower you spent thinking through a bug? It’s a lot more clear cut if the client wants you to sit in an office, twiddling your thumbs, though.

Oxide RFD: Using LLMs at Oxide

A wonderfully well thought out piece about using LLMs in a respectful, sensible manner. I’d been thinking about writing something similar, but it wouldn’t have been anywhere near as well crafted as this. While I basically agree with it all, the part about LLM-generated text really resonates:

LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion.

Your app subscription is now my weekend project

Roberto points out that monthly SaaS subscriptions start looking even less appealing when you can just prompt an alternative into existence. I talked a bit about using LLMs for one-person software before, and it becomes even more appealing when the alternative is paying monthly for a service that keeps shoe-horning in features you don’t want, making you agree to share data with 329 “trusted partners”, and/or repeatedly ratcheting up the price.

Ekul’s Internet Space

I love the aesthetics of this personal site, and the data tracking that goes along with it. I’m also slightly aghast at the amount of Monster consumed! Makes me want to track more things and expose them on my own site.

Reproducible Builds and Docker Images

2022-02-18T00:00:00Z

XKCD 2347: Dependency

Reproducible builds are builds which you are able to reproduce byte-for-byte, given the same source input. Your initial reaction to that statement might be “Aren’t nearly all builds ‘reproducible builds’, then? If I give my compiler a source file it will always give me the same binary, won’t it?” It sounds simple, like it’s something that should just be fundamentally true unless we go out of our way to break it, but in reality it’s actually quite a challenge. A group of Debian developers have been working on reproducible packages for the best part of a decade and while they’ve made fantastic progress, Debian still isn’t reproducible. Before we talk about why it’s a hard problem, let’s take a minute to ponder why it’s worth that much effort.

On supply chain attacks

Suppose you want to run some open-source software. One of the many benefits of open-source software is that anyone can look at the source and, in theory, spot bugs or malicious code. Some projects even have sponsored audits or penetration tests to affirm that the software is safe. But how do you actually deploy that software? You’re probably not building from source - more likely you’re using a package manager to install a pre-built version, or downloading a binary archive, or running a docker image. How do you know whoever prepared those binary artifacts did so from an un-doctored copy of the source? How do you know a middle-man hasn’t decided to add malware to the binaries to make money?

Even worse: if the software you’re trying to use includes any dependencies, you have the same issue of trust with them. Maybe your supplier isn’t compromising the software, but that doesn’t mean their supplier isn’t. The beauty-cum-horror of a supply chain attack is that it can target the weakest link anywhere along the supply chain. Even if there aren’t any binary files involved, dependencies can still be attacked: what if npmjs.com or proxy.golang.org or github.com return a different version of a dependency-of-a-dependency when the request comes from your IP address? It doesn’t even need to be a modified dependency, it could be a perfectly un-tampered, properly signed copy of the source, just from an older version with a known vulnerability.

Enter stage left: reproducible builds, here to save the day! If the build process is reproducible then you - or anyone else on the internet - can perform the same build on the same source and validate the output has the same checksum or hash. If Debian publish a binary package and an independent re-builder comes up with the exact same build artifact, there’s a reasonably good chance that the build is good. An attacker would have to compromise both the build machine and the re-build machine to do anything nefarious. The more re-builders there are, the less feasible a supply chain attack is.

So why isn’t software just reproducible?

Compilers

As a bit of an experiment, I asked some friends to run the following for me and report the answer:

echo -e "#include <stdio.h>\nint main() { printf(\"Hello\"); return 0; }" | \
  gcc -x c -o hello.out - && \
  sha256sum hello.out

This compiles a super-simple hello world program and then prints the SHA-256 hash of the resulting binary. Here are the results:

Hash	System	GCC
1f62feab5a06861dc575201d807781926d1ae49fb113da018fde8b670a1346f7	Arch	11.2.0
b8e6f2c7082be69f65ffa5e7a3d749eb47866a1b2e1ec19efb63cc59a8b160cd	Debian	8.3.0
cbad2e47a22c234b5e7fa55e029a8db4d64ac7a962e2176bd2e1373d78954088	Debian	8.3.0
e0f6bbc13b29fea8cfa2a975ba4661e781323298aec166c8311d342e6f93c4a6	Alpine	10.3.1
e379156895e06c7a0bf18ac4d648860edcb2655576b0ab9fab172bd6c8b92075	Debian	10.2.1
7ffdaee4eb64e016b89dc5e54d2c8eebab3cebafe2c7aa97de627b5972ecea46	Debian	11.2.0
8ae52cc166743b6ae1eb3e14179ef33de5061a04237f8f97088c896c41a2f698	Arch	11.1.0
8ae52cc166743b6ae1eb3e14179ef33de5061a04237f8f97088c896c41a2f698	Arch	11.1.0

As you can see, there are barely any duplicates. Even the same version of GCC on the same OS sometimes produces different results. And this is the most basic program I could write! Differences arise from the compiler version, the build flags, the libraries installed, and a whole host of other factors. If you compile a Go application instead of a C one, then by default the compiler will include debug information in the binary. This includes the full path to the source file on disk, so building a project in /home/chris/ will produce a different binary to building the same source in /tmp. Future versions of Go are also going to stamp in other meta-data such as VCS info, so building inside and outside a Git repository will produce different binaries.

Iteration order

So we’ve pinned our build environment, we’re manipulating timestamps when adding files to archives, now what? Imagine part of the build process involves looping through all the files in a directory and doing something. What order do these files get iterated in? Well, that very much depends on the filesystem and perhaps when the files themselves were created. To ensure this is reproducible we need to explicitly sort any such operation so that it’s always consistent. This iteration could be happening in a tool that’s called by another tool that’s called by a build script, so the fix isn’t necessarily straight-forward.

Side note: a bug war story

I’ve personally been victim to this kind of non-determinism. I was working on an Android app, and committed a new test that worked fine on my machine, and worked fine on the CI server. But it failed consistently for a colleague.

We both did fresh checkouts of the source, and ran the tests. Mine passed, his failed. He sent me an archive of his checkout in case there was something weird going on there, and the tests passed on my machine. We compared hashes of our checkouts, and they were the same. It was obviously environmental somehow, but everything else worked fine, and the build system went to great pains to ensure things were the same.

After a lot of debugging, I worked out that his test was running with a different version of a library to me, despite the libraries being defined in the build files and the build files being identical. After even more debugging it turned out there were two versions of the library on the classpath, and the ordering of them was different between my machine and his.

The actual issue turned out to be that the build tool generated the classpath by iterating over the library files, and that iteration was done in order of file creation time. The two libraries were added at different points in the project history, so the creation time in your local cache depended on which versions of the app you’d built in the past. With no cache everything worked as expected but there was a slim range of commits where only one library was in use, and if you had run the tests during that period your cache was effectively poisoned.

We fixed the issue by excluding the older version of the library (which was being pulled in as a transient dependency), and filed a bug against the build tool to make the classpath properly deterministic. I think that stands as the most difficult to diagnose bug I’ve ever dealt with.

Interestingly, if you iterate over a map in Go, the iteration is deliberately non-deterministic. That’s an attempt to defeat Hyrum’s Law and prevent developers from relying on whatever the current behaviour happens to be. This actually makes it easier to make things reproducible as the problem is loud and in-your-face, rather than subtle and hard to spot.

Other sources

There’s an awful lot of other places that non-determinism can come from. If the app pulls in dependencies, their versions have to be pinned, otherwise your build changes depending on the latest release of that dependency. If the build process pulls any information from a website, it’s liable to change. Hopefully the website is under your control so that you can version the resource and pin that version. Obviously, anything to do with dates or the current user will probably cause problems. Timezones and locales can cause subtle differences.

What about Docker?

Docker comes with some good and some bad points for reproducibility. The biggest advantage is that it inherently completely describes the build environment; it should work exactly the same from one system to another, even across different OS families. The biggest drawback is it sprays timestamps around like no-one’s business. Each layer in a container image is a .tar.gz file, meaning each file within it is timestamped as discussed above. Making an image involves a lot of copying of files around, so these timestamps invariably end up causing reproducibility issues.

Even worse than timestamps in the filesystem, the image format also contains some meta-data that includes the timestamp at which each layer was built. That means even if you go out of your way to set the timestamp of every single file in your image, the image itself will be different every time you rebuild it. There is no way to deal with this in Docker, which is a very sad state of affairs. Fortunately, Buildah provides a --timestamp flag for its build commands; this not only sets the layer timestamp but also the creation timestamp of any file within the layer.

The other major issue that affects Docker images is the pinning of packages pulled in by package managers. An awful lot of images are based on Alpine or Debian derivatives, and use apk or apt to install dependencies. These need to have a version specified as otherwise the package manager will just pull in the latest at the time of the build. But this isn’t quite enough: you also need to pin the version of any packages that they depend on, recursively. This means flattening the entire package hierarchy and installing all the packages explicitly and with pinned versions.

One more wrinkle in the package management space is that Alpine don’t keep old packages in their main repositories. If you have a Docker image with pinned alpine packages in, it will stop building if the package is updated. This isn’t necessarily fatal to making a reproducible build — as long as it’s reproducible for its useful lifetime, I don’t really see an issue.

Honestly, though, the biggest issue with making Docker images reproducible is getting people to care. Dockerfiles are a relatively new way of packaging software, and there’s no centralised organisation like you find with Linux distributions. There are enough challenges that most casual packagers aren’t going to bother, and no real incentive for them to. That won’t stop me trying, though!

Artisanal Docker images

2022-02-05T00:00:00Z

Artisanal containers…

I run a fair number of services as docker containers. Recently, I’ve been moving away from pre-built images pulled from Docker Hub in favour of those I’ve hand-crafted myself. If you’re thinking “that sounds like a lot of effort”, you’re right. It also comes with a number of advantages, though, and has been a fairly fun journey.

The problems with Docker Hub and its images

Rate limits

For the last few years, I’ve been getting increasingly unhappy with Docker Hub itself. Docker-the-technology is wonderful, but Docker-the-company has been making some rather large missteps. The biggest and most impactful of these has been introducing “pull rate” limits. At the time of writing, if you want to just pull a public image without logging in then you are limited to 100 pulls every 6 hours. If you log in then you’re limited to 200 pulls per 6 hours, but it’s account wide. This might seem like a big enough number, but I repeatedly hit it and there is no way to actually audit what is causing it. I have various containers that may all pull images at arbitrary times (e.g. continuous integration build agents), and the only information you get back from Docker Hub is the number of pulls remaining.

Obviously, I could start paying Docker Hub for a “Pro” plan. That gets you 5,000 pulls per day for $7/month. The downside is that every docker client would have to be authenticated, which presents a fair annoyance in terms of credential management. I also don’t really like how they positioned the service as a public utility with special treatment in the docker software, and then start tightening the ratchet to make money.

“Bad” images

I’m fairly opinionated about what a container image should look like: most importantly it should run just a single process, and only include the bare minimum dependencies required for that. Other people think differently, and it’s very hard to tell at a glance whether an image on Docker Hub contains just the application you want, or whether it also bundles MySQL, Redis, Elasticsearch, and a partridge in a pear tree. Some people want that kind of thing, but I really don’t. It’s also very hard to tell whether an image is officially endorsed by the upstream project, and where the source Dockerfile is. This used to be better because most projects used Docker Hub’s automatic builds, but they’re now a “pro” feature.

I quite often found that I’d be looking for an image for X, and there would be 5-10 images from different users. None of them looked official, some of them were out-of-date, some bundled the kitchen sink. Even when one looked good, it’s a bit of a gamble whether the author is going to keep it updated or not.

Doijanky

The rate limits and other problems were annoying, but they weren’t really annoying enough to force me to do anything about it. The straw that broke the camel’s back came later: I was looking at the official golang images, and noticed that all the tags were pushed by a random user account called “doijanky”:

An ‘official’ Docker Hub image pushed by user ‘doijanky’

I, perhaps naively, assumed that official images were built on Docker Hub’s own infrastructure. Why would all the Golang images be attributed to this user? Checking out their profile, they’re simply identified as a “Community User” like everyone else, with no repositories of their own. The only thing in the profile is their homepage, which is a link to a Jenkins dashboard: https://doi-janky.infosiftr.net/. It appears legitimate: “Infosiftr” are a container consultancy and the dashboard is linked to from the README in the official images git repository, but I find it baffling that they’re using third-party infrastructure and a normal user account (with a dubious name) to push these images. There doesn’t seem to be a good way to verify what you pull corresponds to the Dockerfile it came from; if infosiftr wanted to inject something into the build they could happily do so, and who knows how good their infosec posture is? If someone got access to the “doijanky” account, how long could they upload malicious images before someone noticed?

This little roller-coaster ride from “are all the official images compromised?!” to “oh, no, they’re not, it’s all just awful” finally convinced me to look at building my own images from scratch.

The implementation: templating with contempt

One of the big issues I needed to tackle was how to deal with updates. I didn’t want to have to go and edit a file every time some minor release was made of some software, or every time there was a security vulnerability in a common library. The official images use a shell-scripting based system to check for updates and generate Dockerfiles, I decided to do something similar but with Go templates. The result is a tool called contempt. It takes a template like:

FROM {{image "golang"}} AS build

ARG TAG="{{github_tag "example/project"}}"

RUN apk add --no-cache \
        {{range $key, $value := alpine_packages "git" -}}
        {{$key}}={{$value}}\
        {{end}}; \
# ...

Contempt has support for getting information from a variety of sources. In this case, it’s getting the latest digest of another Docker image, the latest tag from a Git repository, and the latest version of an alpine package and all its dependencies. The resulting Dockerfile looks something like this:

# Generated from https://github.com/csmith/dockerfiles/blob/master/miniflux/Dockerfile.gotpl
# BOM: {"apk:brotli-libs":"1.0.9-r5","apk:busybox":"1.34.1-r4", <snip> }

FROM reg.c5h.io/golang@sha256:ac8fa5f4078b0a697796b5d741... AS build

ARG TAG="2.0.35"

RUN apk add --no-cache \
        brotli-libs=1.0.9-r5\
        busybox=1.34.1-r4\
        # <snip>
        pcre2=10.39-r0\
        zlib=1.2.11-r3\
        ; \
# ...

I’ve cut out the longer parts for readability. You can see that it pins all versions of the alpine packages in use, as well as the base image. This ensures that if you build from the same Dockerfile at a later time it will build the same image (or will fail entirely, as Alpine doesn’t keep their old packages around indefinitely). It also produces a “bill of materials” as a really long JSON-encoded comment. If you let contempt commit the Dockerfile it uses the BOM to generate useful commit messages like: [project] apk:busybox: 1.34.1-r3->1.34.1-r4, so you can see exactly what changed.

Contempt also has support for building and pushing images whenever it changes the Dockerfile. I use it in a GitHub action that runs daily to check all my images are up-to-date and push those that aren’t. It understands the dependencies between images (by pre-analysing the templates) so it will always check and build base images before ones that require them. This means an update to, say, the “alpine” base image will cause anything that depends on it to get updated at the same time, ensuring security updates are rolled out promptly.

The result

You can see my collection of lovingly hand-crafted Dockerfiles in my dockerfiles repository.

There are a number of advantages to handwriting all the images I use. The obvious one is that they’re all built how I want: there are no extraneous dependencies, they’re all based on the same small set of base images (rather than pulling around 10 different versions of debian), nothing tries to also run a DBMS in its container, etc.

This level of customisation goes further, though. Because I’m packaging the software myself, I can tweak how it’s built to fit my needs. A couple of things I run need their own TLS certificates separate from my normal HTTPS setup, so I bake my certwrapper tool in to manage those; I can even set the build flags on certwrapper to only enable the particular DNS provider I personally need (thus avoiding dragging in clients for AWS, GCP, etc). Some software like Hashicorp Vault has an optional web interface that I don’t need, so I simply don’t enable it in the build. These changes save build time, reduce image sizes, in some cases improve runtime performance, and generally reduce the attack surface of what’s running in the container.

It’s also been a great way to learn more about how software is distributed. Writing a Dockerfile is not that distant from writing a PKGBUILD file for an Arch Linux package, or the equivalent for other distributions. In a couple of instances I’ve googled how to solve a particular issue, and found an Arch or Void linux maintainer asking the upstream project about the exact same issue.

Finally, all the images I build I push to my own registry so there are obviously no rate limiting issues. Standing up a service (assuming the Dockerfile has been written!) is amazingly quick because the base layers are all shared and cached, and the registry is a lot physically closer than Docker Hub. Bootstrapping this whole thing becomes an interesting problem because the image for the registry is stored on the registry, but I’ll leave that discussion for another post…

Chameth.com - posts like artisanal-docker-images, just-a-nod but not debugging-beyond-the-debugger, migrating-from-github-to-forgejo, why-you-should-be-using-https

Monthly Meanderings: February 2026

Website updates

Other projects

Entertainment

Around the web

Just a nod

Monthly Meanderings: January 2026

Website updates

Watched films ranked

Other projects

Entertainment

Around the web

Reproducible Builds and Docker Images

On supply chain attacks

So why isn’t software just reproducible?

Compilers

Archives

Iteration order

Other sources

What about Docker?

Artisanal Docker images

The problems with Docker Hub and its images

Rate limits

“Bad” images

Doijanky

The implementation: templating with contempt

The result