Chameth.com - posts like docker-automatic-nginx-proxy, making-a-font-of-my-handwriting but not why-you-should-be-using-https

Monthly Meanderings: January 2026

2026-02-01T00:00:00Z

Welcome to the second edition of my monthly meanderings. For a bit of context, you can check out the introduction to the first edition.

Website updates

It’s been a pretty busy month for chameth.com. Three blog posts: The Meaning of Life — an entry into the IndieWeb carnival where I mostly review a book on Stoicism — Surge Protectors: Marketing vs Reality which is a dump of a research rabbit hole I fell into, and An Interesting Tailscale + Docker Gotcha which documents a fun problem I encountered.

I’ve also made a new films section, which is basically my Letterboxd data but free range instead of cooped up in a silo. I imported my 200-odd reviews, and for the last few I wrote the review on my site and then copied it over to Letterboxd. That’s what the IndieWeb folks call POSSE: publish on your own site, syndicate elsewhere. I like the idea that my website is the canonical source of truth for all the public things I generate. I want to expand this to some other areas in the future: music, video games, board games, etc.

I made most of the film components into “shortcodes”, so I can drop them into arbitrary pages or posts, like so:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Finally, I added a new widget to the bottom of most pages: a “nod” button. It’s effectively a “like” or “kudos” button but deliberately does not track how many times it’s been pressed, has no social network connotations, and won’t flood my inbox or hit rate limits if it gets spammed. If you click the button, it simply sends a message to a private IRC channel that says “Someone nodded at $URL”. I’ll probably write a bit more about it in a blog post in the future.

Other projects

I only have one small non-chameth.com update: a new project called pigeonhole which provides a really basic web service that lets you leave messages in a named cubby, and retrieve them at a later date. I’m using it to automate adding things to my Obsidian notes from my Apple Watch. There’s lots of reasons why that isn’t as simple as it sounds. I went through a bunch of iterations to try and make it work before just writing code to fix it.

I wanted to do this because I’m trying to replace Todoist with just having tasks in Obsidian. I already used Obsidian for longer form notes, but I’ve now also started using it for drafting blog posts (I’m typing this in Obsidian right now!), and trying to keep more useful day-to-day notes. I’ll probably write a blog post with more details once I’ve figured everything out.

Finally, I’ve been moving a bunch of projects to a private Forgejo instance, with the ultimate goal of having all of my projects live there. I’m going to mirror the contents out to GitHub and maybe other places, but Forgejo will be the single source of truth, and where all the workflows/dependency wrangling/etc happens. I’ve got Renovate configured to do dependency bumps, and I’m very much liking the whole setup so far. I’m saying it a lot but again: I’ll blog about it in more detail when it’s a bit further along.

Entertainment

I mentioned last month that I was playing Hogwarts: Legacy. I finished that off, and have been struggling to find something to scratch the same itch. I’ve tried a whole bunch of open world games, and basically none of them work for me. They have cumbersome control schemes (yay console ports), really awkward or sluggish movement, and so on. I think I’m just too fussy, but if I’m going to sink dozens of hours into a game it shouldn’t feel annoying or like a chore!

I did play through Thank Goodness You’re Here, which is delightfully funny but sadly only took 3.6 hours to finish with 100% of the achievements. Still highly recommend picking it up, especially when it’s on sale for under ‘10 bob’. I’ve also started playing through Control, and I’m enjoying the ambience. There’s an insanely sharp difficulty spike for the first boss, though, which nearly made me put it down.

In other entertainment, I’ve been playing a load of turn-based games on Board Game Arena , and I’ve watched a bunch of films after barely watching any last month. Here’s another of those shortcodes to show them off:

Around the web

Remotely unlocking an encrypted hard disk

A fun post documenting using Tailscale in a very weird way to solve a relatable problem. Very much my cup of tea.

I charged $18,000 for a static HTML page

This one’s actually from 2019, but it’s the first time I’ve come across it, and it’s a fun story. As a contractor, deciding what to bill for and what not to can be a bit messy. Downtime between meetings? The ten minutes in the shower you spent thinking through a bug? It’s a lot more clear cut if the client wants you to sit in an office, twiddling your thumbs, though.

Oxide RFD: Using LLMs at Oxide

A wonderfully well thought out piece about using LLMs in a respectful, sensible manner. I’d been thinking about writing something similar, but it wouldn’t have been anywhere near as well crafted as this. While I basically agree with it all, the part about LLM-generated text really resonates:

LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion.

Your app subscription is now my weekend project

Roberto points out that monthly SaaS subscriptions start looking even less appealing when you can just prompt an alternative into existence. I talked a bit about using LLMs for one-person software before, and it becomes even more appealing when the alternative is paying monthly for a service that keeps shoe-horning in features you don’t want, making you agree to share data with 329 “trusted partners”, and/or repeatedly ratcheting up the price.

Ekul’s Internet Space

I love the aesthetics of this personal site, and the data tracking that goes along with it. I’m also slightly aghast at the amount of Monster consumed! Makes me want to track more things and expose them on my own site.

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Making a font of my handwriting

2025-08-08T00:00:00Z

Recently I’ve been on a small campaign to try to make my personal website more… personal. Little ways to make it obvious it’s mine and personal, not just another piece of the boring corporate dystopia that is most of the web these days. I don’t quite want to fully regress to the Geocities era and fill the screen with animated under construction GIFs, but I do want to capture some of that vibe.

I’d added some bits and pieces along those lines: floating images in articles now look like they’re stuck to the page with sellotape, related post links have a wavy border that animates when you hover over them, and so on. Next, I wanted to change the heading fonts from a monospace font to something cursive, to resemble handwriting. Less terminal output, more handwritten letter. I couldn’t find one I liked, though. So why not make my own? It can’t be that hard, right?

Failing to do it myself

I set out to try to make the font myself using open source tools. After doing a bit of research, it seemed like the general approach was to create vectors of each character and then import them into a font editor. That seems to mean either Adobe Illustrator and FontLab (if you have too much money) or Inkscape and FontForge (if you like open source). I fall firmly into the latter category, so I grabbed my graphics tablet and opened Inkscape.

I wrote out my first three letters: capital A, B and C. Saved them in Inkscape, and attempted to import them into FontForge. Then I remembered one crucial thing that had slipped my mind: I absolutely loathe using FontForge. It’s a bit like when you open an old version of GIMP and get a bunch of weird looking windows floating all over the place; it feels like you’re fighting against the tool to do even the most basic operations. The difference is I have cause to edit images a lot more than I edit fonts, and GIMP has actually significantly improved their UI over the years.

Here are the rough steps I went through with FontForge:

Launch Font Forge. It shows a weird bit of art in one window, and an open file dialog in another.
I don’t want to open a file, so I close that dialog. The program exits.
Relaunch Font Forge, and realise that within the “Open Font” dialog is a “New” button. Click it.
Get to the standard font-editing UI. Right-click on the “A” looking for a way to import an SVG. Don’t see one.
Click around a bit, exploring the menus. Everything feels a bit off. You can’t open one menu then hover over the next to see its content, like basically every UI toolkit in existence. I think FontForge has eschewed QT and GTK in favour of doing things itself.
Find the “Import” option in the File menu. Hope it’s for a single glyph not the whole font.
A file picker opens. Again it’s all a bit off from normal desktop conventions. Try to resize it, and just get blank grey space at the bottom.
Type the absolute path I want to go to in the text field.
Get a dialog saying “Not a bdf file /home/chris/etc”. Press OK.
Get a dialog saying “Could not find a bitmap font in”. Press OK.
Press Ctrl+L to see if that lets me enter a path. Click everything in the dialog to try to find a way to enter a path. Get annoyed. Give up. Click through folder-by-folder to get to where I want to be.
Get to the folder and don’t see any files. Change the format to “SVG”. Double-click the newly-visible SVG file.
Get a dialog saying “You must select a glyph before you can import an image into it”. Press OK.
The import dialog goes away, having not imported.
Select the glyph in the main tool area, then repeat the File→Import dance.
It’s actually there now! Open the glyph in the editor and see it’s a complete mess of Bézier curves. I can’t click what I want without accidentally moving a handle for an adjacent curve.
Rage-quit.

I’m sure FontForge is less anger inducing once you’re used to it. And you definitely could use it to build a font like this if you had much more patience than me. I’d had enough of death-by-a-thousand-paper-cuts though.

I briefly tried Inkscape’s built-in support for making an SVG font. It annoyed me a lot less, but it’s fiddly: it seemed like each glyph had to be a single path, so you had to convert the glyphs to paths, then merge them correctly. If you merge them incorrectly then the wrong bits of your letters end up filled (like the inside of the ‘B’). Path manipulation is getting towards the limit of my knowledge of vector editing, and it took a bit of trial and error for each letter that had more than a single stroke. I didn’t fancy doing that for every letter.

I’m usually a big advocate of open source, but this was one of those painful times when it feels like it just falls short. Clunky, painful UI and processes where commercial tools just let you get on with your work.

You can exchange money for goods and services

When I’d been looking for open source tutorials, I found many mentions of a closed source, hosted tool: Calligraphr. It has a free version with limitations (no ligatures, no variations, 75 glyphs per font), and a pro version for £8/month. I’d normally balk at the idea of a subscription for this, but they have the perfect answer: you can make a one-time payment, and your account automatically downgrades back to free after a month. It’s not a hidden option, either, it’s the most prominent button on the upgrade page. That made me happy to give them £8 to play around with the service for a month.

Calligraphr works by having you print templates, write out the letters, then scan them in. It does some magical processing to extract the glyphs, provides tools to tidy them up, align them, etc, and then produces a TTF file for you. You can see some of my completed templates here:

Most of the templates I used for the font

Calligraphr has a nice UI to generate the templates, allowing you to select which glyphs to include. I added the “minimal English”, “basic punctuation” and “Ligatures” sets. That gave me four pages to fill out, and I did them all twice. That let me filter out versions that didn’t work well, and have variants for some letters so the font wasn’t too repetitive. Later on, I went back and added some custom ligatures based on blog post titles that didn’t look quite right: “Re”, “To”, “ers”, “ey”, “hy”, “ra”, “re” and “ty”. Ligatures like this help it look more natural: when we write we don’t just stamp out identical letters regardless of their surroundings, instead they will connect to their neighbours, or overlap slightly, or even share a stroke.

I filled these templates in with a Sharpie, as I wanted a fairly informal, scrap-booky look, and it would also give good solid shapes that should be easy to pick out of the template. I scanned them with the “Scan Document” function on my iPhone, and uploaded the PDFs to Calligraphr.

Iterating and tweaking

The Calligraphr UI allows you to preview the font, but I found it a lot more useful to just download a copy and use it on a local copy of my website. That let me test it with real text, and see how it’d look at the different font sizes I use on the site.

The first version was not great. Despite the guidelines on the template, I apparently wasn’t good at sticking to them. Some letters were floating way off the baseline, and some were sunken below. When those opposites met it looked terrible. Fortunately Calligraphr has a pretty easy tool to slide each letter up and down, and scale it up or down if needed, and you can see it next to other letters as you do it. It took a little bit of time to go through all the variants of all the letters, but the next version looked a lot better.

Another tweak I ended up doing was reducing the spacing between letters. The defaults Calligraphr uses are probably good for a blocky font, but I wanted to put the letters close together to give it more of a joined-up look. Again, this is an easy tool to use, you just drag the sides in or out as desired. While these tweaking steps were probably as fiddly as some of the Inkscape steps I refused to do earlier, they’re a lot more rewarding as you see things improving with each one. It’s a lot easier for me to commit time and effort to improving something that’s already working reasonably, than put that time and energy into an unknown.

Later, I noticed that occasionally there would be a huge gap in a title. Not “the kerning is slightly off” but “there’s enough room to park a bus”. It took me a while to figure out what was happening: a couple of glyphs hadn’t been isolated perfectly and had picked up a few pixels from the template lines at the edge of their boxes. That meant the glyph had a width that covered the actual written glyph, a big gap, and then the rogue marks. At first, I fixed this by just adjusting the width, but that left the little pixels floating awkwardly down-sentence. The proper fix was to use the editing tool and simply delete them, and then Calligraphr snapped the width back to what it should be.

These iterations took a while to do, but I just dipped in and out occasionally over the course of a week, so it didn’t actually feel like too much work. I quite enjoy the process of refining things, too.

Result and a surprise

If you’re viewing this post on my website¹, you can see the font in the headers, captions, and a few other places. Here’s how it compares to my actual handwriting:

My handwriting vs my handwriting font

It’s not close enough to forge documents, but I think it definitely gets across my style, and that’s exactly what I wanted. It’s surprisingly legible even at smaller font sizes — I think the weight of the Sharpie helps here — and at £8 and a bit of manual work was a lot more economical than spending days wresting with open source tools.

A few weeks after I put the finishing touches on the font, I got an e-mail from Calligraphr. As my account had lapsed back to the free version, I was no longer eligible for the “server-side backup” feature. So what did they do? They e-mailed me an exported copy! It’s a JSON file with the properties of each glyph and a base64 encoded image. Not only can I re-upload this to Calligraphr if I resubscribe, I can probably hook something up to edit it should I ever need to. I’m blown away by how pro-user Calligraphr’s business practices are. They’re up-front about pricing, don’t try to get you stuck on an auto-renewing subscription, and automatically export your data. It’s like a breath of fresh air compared to the barrage of dark patterns that other websites foist on us. If you want to make this kind of font, I’d definitely recommend them just because of how nice they are.

And I haven’t changed everything since writing this post… ↩︎

An app can be a ready meal

2025-06-11T00:00:00Z

It’s not a home-cooked meal, but it does the job sometimes.

Three years ago I read “an app can be a home-cooked meal” by Robin Sloan. It’s a great article about how Robin cooked up an app for his family to replace a commercial one that died. It’s been stuck in my head ever since. It’s only recently that I’ve actually done anything like Robin described, though. Part of the reason was my brain got too hung up on the family aspect: in my head, a home-cooked meal is one where your family or friends all gather around to eat it with you (in much the same way as Robin’s app is used in the article). It took me an embarrassingly long time to realise that you can apply all the same arguments to an app built just for you. And it doesn’t even have to be difficult. In fact, it can be more like a ready meal¹ than a family dinner.

Why not open source?

I love open source software. Almost everything I use day-to-day is open source, and most things I write for myself I release as open source. I believe that should be the default stance for most software. So why would you want to make something and keep it just for yourself?

Even if an open source project garners no users whatsoever, there’s still some pressure for it to meet certain standards²: if it needs configuring, then there needs to be a mechanism for people to do that; the code needs to be of a reasonable quality — my open source projects are linked from my CV, potential clients might be looking at them — and there needs to be at least some attempt at documentation or showing other people how to use it. There’s “building in public” and then there’s “inviting the whole world to poke around your drawer of shame”.

The way I usually work on things that are mostly for me is I hack them together, and then I gradually force myself to fix them up into a state where I consider them acceptable for release. That stage isn’t fun, and sometimes it adds more complexity than the whole original project. I’m not saying open source isn’t worth it — far from it! — but there is definitely a balance to be struck between the utility of making something open source and the amount of effort it takes.

For example, I mentioned previously that I wrote a home automation system. Hidden within that is a ~40 line function that determines whether a fan should be turned on to keep a room cool. The logic is entirely unique to the devices on my network and my requirements. To try to put it into a form that would be useful to anyone but me would take exponentially more effort. I’d basically be re-inventing Home Assistant, and I explicitly started that project because I didn’t need that kind of complexity. For a while I had this nagging feeling that I should find a way to open source it. Then I had my³ “lightbulb” moment: I could just not.

Writing something entirely for yourself, without planning on open sourcing it is surprisingly liberating: you don’t need to worry about documenting things; you can change anything and everything at will without having to worry about migration paths or whether it’ll break anyone’s workflow; you don’t have to configure things, you can just code them how you want. Hell, you can hard-code API keys right in the source code. Who cares!

The microwave revolution

XKCD 1205

One of the other problems with making software just for yourself is that often the time investment just isn’t worth it. If I spend a day automating something that normally takes me five minutes, it’s going to take an awful long time to “break even” on that time spent. If you’re open sourcing something then there are ancillary benefits that may tip those scales: you might help other people, have something to show off, etc. But if it’s just for you, then XKCD 1205 is a harsh mistress.

Recently, though, a new tool has emerged that tips the balance: LLM assistants. I’ve talked about my experience before. I’m definitely not comfortable letting an LLM run roughshod over my published code, even if it can help write some of it, but for private code? Why not! For a trivial example: I have a USB key with an Arch Linux ISO on it, in case I need to troubleshoot or reinstall my PC. Every so often I’ll update the ISO to the latest version. It probably takes 5 minutes to do, so it’s probably only worth 25-50 minutes to optimise it. Can I write, test, and debug a script to do it myself in that time? Probably not. Can I get an LLM to generate some code to do it for me, and then spend 10 minutes reviewing it⁴? Easily.

To continue with the strained metaphor: LLMs are like the microwave that nukes your ready meal. Pop in a prompt, let it use a load of power for a while, and out pops your app. You can even do it on your phone while sat on a sofa⁵. That changes the time equation even more: you can just squeeze in a prompt whenever something comes to mind.

This also goes back to the open source issue: I don’t feel comfortable publishing something created by an LLM with minimal intervention on my part. If I can prompt an LLM to do something, so can anyone else. We don’t need more slop out there, and I don’t want there to be any confusion about what I’ve written and what I’ve just prompted into existence. But for personal projects that will never see the light of day it’s perfect.

What I’ve made

Besides the home automation controller, I’ve built a constellation of smaller tools: a script to arrange windows on my monitors how I want them⁶, one to help me verify that my backups are working and can be restored, one to create the files/folders for a new blog post, and some other odds and ends. They’re all pretty small, and pretty specific to my setup.

My biggest just-for-me project is a web app that I started to help me aggregate film recommendations. It’s since morphed into a general personal data aggregation service: it deals with data from GitHub, Todoist, Letterboxd, TMDB, Healthkit, and others. It also lets me make re-orderable lists, store recipes, and more. Parts of this could definitely be open sourced, and I might carve them out at some point, but it’s mostly a glorious hodge-podge of things specific to me. Having all these services in one place lets me make quick and dirty automations, for example: when I create a Todoist note on my phone or watch, I often forget to set the due date, so it doesn’t show up in the “Today” view. It was literally a few lines of code to plumb things together so any inbox task without a due date gets set to today automatically.

If I was trying to write these things in a way that would be useful to other people, or — for some of the features — without the aid of an LLM, I just wouldn’t be bothered. It’d take too much time for questionable benefit. But there’s a kind of joy in just being able to hack things together that work just well enough for you. A ready meal will never compete with a home-cooked meal, but sometimes it perfectly hits the spot.

Image credits	Creator	Licence	Source
Spaghetti carbonara	Wikimedia user Geni	CC BY-SA 4.0	Wikimedia
XKCD 1205	Randall Munroe	CC BY-NC 2.5	XKCD

or a “TV dinner”, if you’re North-American-ly inclined. ↩︎
Self-imposed pressure, for sure, but brains are going to do brain things. ↩︎
very obvious in retrospect ↩︎
Reviewing LLM-generated code that shells out to dd to write to the root of a storage drive is perhaps the most intensely I’ve ever reviewed any code to date. I really didn’t want to write a blog post about how an LLM blatted my hard drive! ↩︎
Phones are terrible input devices for code, but they’re just about passable for typing English. ↩︎
I basically want a tiling window manager, but without all the effort and weirdness of a tiling window manager. ↩︎

Generating infinite avatars

2022-12-30T00:00:00Z

An example of one of the unique avatars

I recently added a new ‘about’ section to the top of my website. Like most about pages, it has a picture. Instead of a normal photograph, however, you’ll see an AI-generated avatar. This is admittedly fairly trendy at the minute — apps like Lensa offer to make you profile pictures if you give them a set of photos and some cash — but I’ve done something a bit different.

You see, there is not just one image that has been carefully curated, edited, and uploaded. No, the image you see quite possibly has never been seen before and will never be seen again. It’s unique. Just for you.

Background: Stable Diffusion, DreamBooth, et al

You’ve probably heard of Stable Diffusion, the open text-to-image model developed by LMU Munich. Given a text prompt it starts with a random array of static and repeatedly transforms it, each step moving away from pure entropy and towards a real image that befits the prompt¹. It stands in contrast to competitors like DALL-E and Midjourney in both the code and the model being freely and publicly available.

One interesting side effect of that is that you can take the pre-trained Stable Diffusion model, and run your own further training on top. It takes hundreds of thousands of GPU hours to train such a model from scratch, but only a few to add some specific tweaks on top. Earlier this year researchers from Boston University and Google Research published a paper titled DreamBooth which presents a model for doing exactly that.

This research has spawned a slew of startups that do the training and/or generation for you in return for cold, hard cash. The most popular of these at present is Lensa, a mobile app that will generate 200 avatars for you in pre-set styles for £9.99. You can’t change the styles or regenerate any you don’t like, but from what I hear 200 is just about enough that you’ll find one or two that you like.

The lovely ASCII art shown while training is underway

While £9.99 isn’t much money, if you’re technically inclined then it’s not very difficult to do the work yourself for free. That also lets you come up with unique prompts, creating pictures in different styles, with different backgrounds, and so on. I used the wonderful notebooks from TheLastBen that run in Google Colab. The generous free tier offered by Colab is plenty enough to run a DreamBooth training session, and the notebook walks you through pretty much everything².

If you train a custom model then you end up with a weighty file called a “checkpoint”, which you can provide to most Stable Diffusion tools to use when generating images. I use AUTOMATIC1111’s stable-diffusion-webui which not only offers a simple web UI, but also a REST API for accessing it programmatically. I installed this on my laptop and spent a happy hour or two generating weird and wonderful pictures of me.

Automating it

When I was training the model, I was planning on finding a single nice avatar to use. After playing around with it for a while, though, I wanted to expose all the wacky and unique pictures that it was generating. I came up with the rough idea of batch generating a number of avatars, then having a custom webserver that served you one and deleted it.

My first attempt at this was to try and run the Stable Diffusion process entirely on CPU on a server. I’d previously run an SD generator on my laptop without CUDA support, and while it was deathly slow it still worked. With this custom model, though, it took longer to initialise than I was prepared to wait — and I hate to think how long the subsequent image generation would have taken!³

I obviously needed something with a GPU, but I didn’t want to use my laptop as it may be unavailable or doing other more important things with its GPU like playing games. So I turned to AWS, and found they have GPU-enabled instances that can be obtained for reasonable amounts of money. As I didn’t really care when the batch processing ran, I could use “spot” instances which offer a decent discount in exchange for only being able to run when there aren’t reserved instances that need the resources.

After much fiddling in the AWS console, I got a spot reservation set up for a GPU-enabled instance. After waiting a while and not seeing any instances appear, I checked the logs and found it was erroring because I was trying to exceed my vCPU limit. Odd. A bit of googling⁴ later and I discover there’s a separate limit for that type of machine and the default limit is 0. There’s a whole mini application in AWS for requesting limit increases, so I requested a modest increase to 8 vCPUs (the minimum configuration for the “accelerated computing” images is 4 or 8 vCPUs depending on the exact type). After a brief wait, Amazon declined my request:

I am sorry but at this time we are unable to approve your service quota increase request.

Service quotas are put in place to help you gradually ramp up activity and decrease the likelihood of large bills due to sudden, unexpected spikes.

I’m not entirely sure how you’re meant to ramp up without being able to run a single instance. There are lots of theories online about account age requirements, minimum spends, etc, but I wasn’t willing to jump through inscrutable hoops in order to try to give Jeff Bezos more money. Instead, I looked at Paperspace, a service I’d come across previously when trying to run a GPU-enabled Windows box. They have a variety of GPUs on offer, and a lovely API to remotely manage machines. [If you want to try Paperspace you can use this referral link to get $10 off. In doing so you’ll give me enough credit to generate around 10,000 avatars. If that’s not a worthy cause, I don’t know what is.]

I went a bit overboard investigating the different GPU offerings and their relative bang for the buck:

A slightly over-the-top analysis of the GPUs offered by paperspace

The A4000 comes out on top: it’s built on a modern architecture with a large number of CUDA cores, and is really competitively priced. Paperspace only give you access to the M4000 and P4000 initially and make you request access to the higher tier units. I dutifully filled out the very brief form, and a day later it was approved. At least someone is willing to accept my money!

Writing some code

After setting up the machine on Paperspace and copying over my custom model, I set about writing code to handle the generating and the serving. I eventually settled on having two buckets of images: ones that will be shown to only one person and deleted on use, and a fallback bucket that will be used multiple times. The fallback bucket is so that I can limit how many avatars I need to generate (and thus how much money I pay for GPU time⁵). I set a global limit of one avatar used every 10 minutes, as well as a per-IP limit of one unique avatar per 24 hours.

In order to distinguish whether you’re seeing a unique avatar, the server adds a border around it and a “UNIQUE LIMITED EDITION” label at the bottom. If you see that text, you’re looking at an image that has never been seen before and that has already been deleted. The server sends some aggressive caching headers, so in normal day-to-day operations you should see a different unique avatar every day you visit the site.

The generating side is a bit more interesting. It monitors the contents of the two avatar buckets, and springs into action if they fall below a configured minimum. It starts the process by calling Paperspace and requesting the machine is started up, then repeatedly polls the status endpoint until it’s ready. It then generates images individually using the REST API until it hits the bucket’s configured maximum. Once it’s done, it asks Paperspace to shut the machine back down.

Initially I just hardcoded a set of prompts for the generator to use, but they resulted in a lot of fairly similar images. To make things more interesting, I started dynamically generating the prompt using a combination of:

A prefix such as “A painting of”, “A sketch of”, “A photograph of”
In 80% of prompts, an artist reference such as “in the style of Andy Warhol”
In 30% of prompts, a film reference such as “from the film The Matrix”
1-10 random suffixes such as “bokeh”, “8K”, “trending in Artstation”⁶

Initially I had the film and artist prompts independent, but the occasions where neither appeared in the prompt lead to pretty bad images. Instead, there’s now a 20% chance of a film reference, a 70% chance of an artist reference, and a 10% chance of both⁷. There’s a list of around 10 prefixes, 70 artists, 20 films and 20 suffixes which gives a large pool of random prompts.

End results

Everything I’ve described is now live on chameth.com – if you visit you might get a unique, never-been-seen before version of me. The code for the generating and serving is available on GitHub if you’re interested or want to replicate this for yourself.

To finish off, I ran off a batch of 200 avatars and have selected the most interesting ones:

A selection of avatars produced by the generator

As you can see there was one output that appears to be a cat with a ball of yarn, rather than a picture of me. That seems to happen occasionally when the various parts of the prompt don’t gel well, but I’m happy with 0.5% or so of the images being somewhat random! The batch of 200 avatars took just shy of 17 minutes to generate, which will result in a bill of $0.22 from Paperspace.

The famous quote from Arthur C Clark comes to mind when I think too much about how this works: “Any sufficiently advanced technology is indistinguishable from magic”. ↩︎
The only thing it doesn’t help with is finding enough pictures of yourself to use for the training data. That’s presumably easier if you’re more of a “selfie person” than I am. ↩︎
There were a lot of differences that could account for the extra slowness: my custom model was based on the larger 2.1 SD model rather than 1.5; I was using different software; and my laptop CPU is far more modern than the server’s. ↩︎
In the genericised sense: I used Duck Duck Go. ↩︎
One of my biggest concerns here was to avoid putting a “make Chris pay money” button on the Internet. That felt like a bad idea. ↩︎
AKA the random detritus that gets appended to prompts to make images better in mysterious ways. The English pedant in me hates this nonsense, but the results when you spam rubbish modifiers are inarguably better than when using straight forward prose. ↩︎
Imagining the Venn Diagrams is left as an exercise for the reader. ↩︎