Chameth.com - posts like debugging-beyond-the-debugger, docker-automatic-nginx-proxy, securing-all-the-things-with-1password but not migrating-from-github-to-forgejo, sense-api, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Securing all the things with 1Password

2025-11-26T00:00:00Z

For many years I’ve been a keen user of Bitwarden. Recently I’ve had a lot of small paper-cut problems. The browser extension was redesigned and just doesn’t quite work how I expect any more. The prompt to save new login info misfired more than it worked. The mobile app stopped background refreshing properly. No one issue was enough to make me want to leave Bitwarden, but it definitely wasn’t sparking joy. Then there were rumblings about a price increase. I don’t think they’ve announced anything, but the numbers I saw were a jump from $10/year to $24/year. A nearly 2.5x price increase for a tool that was actively getting worse over time did not seem like a good deal to me¹.

I spent a while looking at alternatives, but there were very few that met my needs. Nordpass is highly praised but doesn’t support saving TOTP codes on personal plans². All the nice open-source/self-hosted solutions fall down when you want reliable synchronisation to an iOS device (which is fair enough). Proton Pass looked like it had all the features, but the pricing was asinine. Different pages showed different prices, the prices were introductory rates that jumped up on renewal, but that was hidden away in small print, etc. Yuck. The prices are also pretty high. More or less the same as 1Password, which I’d been discounting due to the cost, until now…

After eliminating basically every other option, I took a serious look at 1Password. It seemed to tick all the right boxes, I’d often heard good things about it, and it wouldn’t actually be much more expensive than Bitwarden if they went ahead with the rumoured price increase. I started a trial to give it a test.

Initial impressions

1Password is — as you’d expect for something that originated as a Mac app — very pretty. But it’s also highly functional. The looks don’t get in the way, they’re nice extras on top. When it’s locked, you get a large version of the 1Password logo, which is a front-on view of the barrel of a lock, next to the password input. When you authenticate, the barrel turns as though you’d put a key in and rotated it, and then the background splits in two and opens like a vault door. Does a password manager need to do that? No. Does it make me a little bit happy every time I see it? Yes. Look at it, it’s amazing!

The design philosophy carries through the rest of the app. Despite using Bitwarden daily, I often clicked on the wrong thing when I wanted to do anything other than autofill; in about a month of using 1Password I don’t think I’ve ever misclicked or been confused about the UI. The browser extension just feels snappier than Bitwarden, too. I’m not sure if it’s actually faster or if the design just makes it feel that way.

Functionality-wise, 1Password does everything Bitwarden did for me but slightly better. Bitwarden kicked you to its web interface for some operations (like checking compromised passwords); 1Password has that all built into the app. Instead of editing entries in a fiddly little browser popup, 1Password just summons forth the desktop app and you edit it there. It does a good job of detecting QR codes for TOTPs and offering to save them. I think the only small complaint I have is that the browser extension doesn’t have an autofill keybind. You can rig that up via the desktop app, but it feels like 1Password leans towards you using the UI it adds on login forms. To be fair, that UI works really well, and is very useful when you have multiple accounts on a site. My muscle memory for autofilling is strong, though.

Diving deeper

At this point I’d decided to switch fully to 1Password, so I started digging into what else it could do. The “Watchtower” feature combines a bunch of reporting about passwords (weak, breached, reused, and so on), but also lists sites where 1Password knows you can use passkeys or 2FA and don’t have them set up. I’d previously disabled passkey support in Bitwarden because it just appeared one day and got in the way of me trying to use a YubiKey, but decided to give them a go with 1Password. “Passkeys” is basically just a marketing term for FIDO2’s WebAuthn with automatic discovery bolted on top. The big problem is how you synchronise or export them. It’s all well and good letting your iPhone create a passkey when the Amazon app arbitrarily prompts it to, but what happens when you log in on a different device, or swap to Android, or whatever? Having 1Password deal with them fixes that³.

The passkey flow is very smooth: 1Password pops up an account selector, with obvious options to create a new account or pass the request through to another device like a YubiKey. It’s smart enough to guess the right account most times, even when you’ve got 17 different Google accounts for $reasons. Using them is equally simple: it just shows a popup and you tap to confirm you want to use the passkey. The Watchtower functionality made it really easy to go through and add passkeys everywhere that supports them. The thing I really appreciate is that they make the data that powers it public. It’s obviously partially a marketing thing for them, but it’s also a useful resource they could have kept proprietary.

Passkeys are a bit basic, though. 1Password has many more fun offerings. There’s an entire developer section with various goodies. At first I thought “that’s nice but I won’t use it”, but now I’m using basically everything it offers. The first thing to draw me in was the CLI. It’s pretty straightforward, but like the browser extensions it can nicely integrate with the desktop app so you unlock them all at once, deal with auth prompts in the GUI, and so forth. I have a couple of command-line tools that need passwords (e.g. for dealing with encrypted backups); now instead of prompting for the passwords, they shell out to op read and I authorise it via a popup in the 1Password app. Simple, but a nice quality of life improvement.

Next up there’s a beta feature called “Environments”. This lets you define groups of environment variables within 1Password, and have them exported automatically to certain destinations. At the minute it supports exporting to the AWS SecretsManager, and local .env files. When making a local file it actually makes a named pipe, so the credentials aren’t just sitting on disk (and won’t end up accidentally committed to anything). When the 1Password vault is locked, anything trying to read from the file will cause an authentication popup, much like with the CLI tool. I used to have the mentality that if something could read arbitrary files on disk, it was basically “game over” from a security point of view. Recent events made me realise there’s some value in defence-in-depth there, though, and 1Password provides a nice solution.

Fingerprints + Linux = Sadness

One feature of 1Password had me feeling a bit left out. Instead of entering your vault password to unlock it every time, you can have it invoke the system authentication library. On an iPhone this uses Face ID, on a Mac it can invoke Touch ID, and on Linux it uses Polkit. In the default configuration, Polkit just asks for your local user password, which is not much of an improvement. A fingerprint scanner seemed like the obvious solution here: it’s both a lot more convenient than typing a long password, and provides some assurance that it’s actually me doing the action⁴. Unfortunately, fingerprint readers on Linux seem to be a bit of a sorry affair.

Don’t get me wrong: there’s support. libfprint has a big list of supported devices, but I found it almost impossible to actually find a device I could buy that had one of the listed sensors. Most of them seem to be exclusively built into laptops. A lot of the cheaper devices just use whatever sensor they can get their hands on, so it’s pot luck if you receive a version that will work with libfprint or one that won’t. There’s also not a lot of information beyond that published by libfprint. I guess it makes sense: external fingerprint readers just aren’t a thing most people care about. While that makes me feel all special, it’s somewhat prohibitive to getting an actual working solution.

I’d given up on finding an answer to this when I stumbled upon an article by Scott Laird. He uses a YubiKey Bio to do it, via pam_u2f. The Bio isn’t actually a fingerprint reader: it just has an on-board reader to secure the credentials. The OS doesn’t get involved in the process. But because it can provide a fingerprint-secured U2F key, and pam_u2f can require that key to perform system operations, you basically get the same result. Maybe it’s actually even better than using a “real” fingerprint reader, as the data is secured on the hardware device…

The YubiKey Bio was a bit more pricey than I really wanted, coming in at around £90, but it felt like a worthy investment to add a bit of security and avoid typing a password dozens of times a day. The setup was simple: the official app that you use to enrol fingerprints is packaged in the Arch User Repository, and it worked perfectly. I already had pam_u2f set up as I used a (non-Bio) YubiKey to gate access to sudo, so I just generated a new key on the Bio, updated my config to use that, and added it to the Polkit policy. Now when I need to unlock 1Password or authenticate something, it prompts me to scan my fingerprint. Perfect!

Biometric all the things \o/

One of the other things Scott mentioned in that article was SSH keys. I’d seen 1Password supports acting as an SSH agent, but didn’t really see why I’d want it to. I used a resident key on my old YubiKey, so I didn’t have much to gain from a security point of view. Scott’s article made me realise I could have some big convenience gains, though. Currently every time I need to use the key, I enter a PIN and then physically touch it. I don’t mind this for the most part, but it turns out git-lfs does something like four different requests every time you do a git operation, none of which reuse the same SSH connection for some reason. That gets painful when each request involves a PIN and a touch. I also liked the idea of using pam_rssh for sudo authentication, but didn’t want to have to do the PIN-and-touch dance every time I used it.

You can probably see where this is going: I generated an SSH key in 1Password, and then… spent literal hours disabling all the other SSH agents that were running on my system, and trying to figure out where SSH_AUTH_SOCK was being populated. I didn’t manage to figure that out in the end⁵, and ended up just overriding it to point at 1Password’s agent socket. I added the new key to GitHub, and did the usual ssh git@github.com test, and it worked perfectly. 1Password prompts to allow access to the key, showing the process requesting access, and authorising it requires a fingerprint read:

can haz ssh key?

I’m not sure why Kitty doesn’t get an icon in the prompt; other applications do. You can expand the details section to see the key fingerprint being requested, as well as the exact process name and PID requesting the key. With the proof of concept done, I generated a couple more keys: one for normal SSH, and one for gaining root access using pam_rssh. Using different keys ensures I get prompted for them separately, and means a malicious app couldn’t sneakily ask for a Git key but actually pivot to SSH into a server. Not sure that’s ever likely to be a problem, but it’s basically no additional work for a bit more defence-in-depth, so why not?

I also took a moment to configure pam_u2f for both lightdm and xfce4-screensaver, so I can now log in and unlock my PC with my fingerprint. It’s not quite as smooth as Touch ID on a Mac, but it’s pretty close.

As an aside: this is not the first time I’ve been in this situation. Both Spotify and Todoist have focused on features I don’t want or need (including, but not limited to, pretty much everything branded “AI”), and then hiked the price up afterwards. It’s not quite standard enshittification, but it’s clear that our incentives no longer align properly. ↩︎
Or didn’t, at the time I looked. It now says “All plans” on the website but the docs only exist for business accounts, and there doesn’t seem to be any announcement explaining the change. ↩︎
Having Bitwarden deal with them would also have fixed that, but it got in my way instead of helping me, so… ↩︎
or it’s not me, but I’ll be busy dealing with the presumably-undesired removal of one of my digits, so probably won’t care about whatever is happening on the computer. ↩︎
Although I’m pretty sure the answer starts and ends with “systemd, somehow”. ↩︎

How I Get Things Done

2024-09-01T00:00:00Z

I always quite like reading about how other people do things. What software or hardware they use, or how they manage reminders, todo lists, and so on. I’ve never actually written about how I do any of that, though. So here it is!

Productivity

In the past I’ve fallen victim to the idea of there being One True Productivity System that would solve all my problems and make me amazing at getting things done. The title of this post is a nod to Getting Things Done, which I’ve read and tried to religiously follow in the past, but it’s just not for me. One thing that did actually stick from it, though, was the idea of “open loops”:

Anything that does not belong where it is, the way it is, is an “open loop,” which will be pulling on your attention if it’s not appropriately managed.

If I want to do something and it’s not recorded in a way I trust, it weighs on me a little. Those little weights all add up, and just make life uncomfortable. I deal with those in a few ways:

Inbox Zero-ish: if an e-mail needs me to do something, it sits in my inbox until it’s done. When it’s done, it gets archived.
Notes: random things I try to remember I put into my notes, so I can search for them later.
Budget: anything monetary I just adjust in my budget or create a category for.
Todo list: any other kind of task I want/need to do either now or at some point in the future goes into Todoist.

Todoist

In the past I’ve gone through a lot of different todo systems, once again hunting for one that will fix all my problems etc¹. In the end I always ended up coming back to Todoist, and have stuck with it for a decent period now.

In Todoist, I have everything split up into projects. Work projects are one colour and personal projects are another. Anything that needs more than a handful of individual tasks gets a project, which is archived when it’s done. Any one-off things that don’t neatly fall into projects go into “Random junk” project.

As I’m writing this, I have 116 items in 22 projects. A good deal of those are things that I might want to do at some point, not things that I actively need to worry about. If I want to get one of them done, I schedule it so it shows up in the “today” view. I have the Todoist desktop app open automatically when I login to my computer, and it gets positioned on the right hand side of my ultra-wide monitor, so I can see the “today” view and add/remove things immediately at any point.

I’m not precious about tasks becoming overdue, or unscheduling things I’d previously scheduled. Priorities change, ideas that sounded good a week ago might no longer sound good, and so on.

I do a few things to help me manage tasks and keep track of things:

I have a monthly recurring task to look through everything that’s not scheduled, just so I keep a rough idea of what’s there, and can update or get rid of it as needed.
I wrote todoistager and have it running on a server to automatically apply tags to tasks based on their rough age: weeks, months or years. This lets me identify things that are lingering and reconsider them².
I automate adding tasks for things that demand my attention, like incoming GitHub pull requests. The Todoist API is nice and straight forward, and hooking it up to things is trivial.

Other desktop software

For notes I use Logseq, and I just put whatever I need to write under today’s note and rely on search or tags to find it again when I need to. I note basically anything that I might like to use again in the future. One of my most common tags is #blub, a term I came across in the excellent article “In defense of blub studies”. Blub is random ultra-specific knowledge; it might not be eminently useful to know on its own, but over time it can pay dividends.

I’ve previously tried a bunch of note taking apps, and methodologies, but like with todo lists I end up just wanting something that just works and stays out of the way. Logseq stores notes in Markdown, and I have a cronjob to commit and push my changes to a private git repository.

Like Todoist, Logseq launches when I log into my computer, and sits in the same 500px-or-so region on the right of my monitor. This allows me to switch from Todoist to take or refer to notes when needed. To do this positioning, I use devilspie2. I also use this to position Discord and my IRC client side-by-side on a small portable monitor that sits underneath my ultrawide.

More mundane software: I use IntejjiJ IDEA for almost all development and complex text editing³. The only exception is Android development where Android Studio rules the roost. For lighter text editing, Sublime Text is my choice.

I spend a fair bit of the time at the command line. I use kitty as a terminal emulator, with the zsh shell and Oh My Zsh for some nice plugins and enhancements. One thing I couldn’t live without is autojump, which allows quickly jumping to any directory you’ve previously visited using a substring of the name.

Web browsing is all Firefox, with Bitwarden for password management and two pinned Fastmail tabs, one for personal e-mail and one for work. I don’t think there’s much else to note there.

Oh, I use Arch, by the way.

Infrastructure

This is the hodge-podge of things that support my computer usage but sit outside the actual computer. In no particular order…

All my DNS queries go through NextDNS using DNS-over-HTTPS. NextDNS lets you add custom responses, block certain things from resolving, and subscribe to pre-made anti-tracking or anti-advertising lists. It gives you far more control over everything than most other services I’ve looked at.

Every device I control runs Tailscale to enable remote access. I also use Tailscale’s SSH authentication feature for my servers, allowing quick and easy access from my phone where I don’t have or want normal SSH keys.

One exciting service that combines these two is golink. It was created by Tailscale, presumably based on the go service used extensively inside Google. I run a version on my tailnet, and have NextDNS configured to resolve go to the tailscale IP of the service. That lets me type “go/whatever” on any machine I set up with NextDNS and Tailscale, including mobile devices. I use go links for all sorts of things:

Shortcuts to tools I use commonly: go/board gets updated to my current client’s kanban board or similar tracker, whether it’s in Jira, GitHub, Asana or whatever; go/meet goes to Google Meet with the user parameter set so I’m logged in with my work account not my personal account.
In place of bookmarks: go/flex goes to the CSS-Tricks flexbox guide⁴, go/cad goes to the OnShape login page, go/hmrc goes to the HMRC online services page, and so on
A few dynamic things: go/github/chameth.com will go to the csmith/chameth.com project on GitHub; go/ref/pipico will go to the Pi Pico page on my electronics reference wiki.

For my servers I have a minimal Ansible setup that does some basic configuration and deploys SSH keys to the right places. Almost everything runs inside Docker, and I use my own software for HTTP proxying: Dotege uses tags I apply to containers to build a config for Centauri which handles reverse proxying, using Let’s Encrypt for TLS certs. At the minute I have two servers: a dedicated server that runs basically all my services (including this website), and a little VPS for monitoring and backups. Both are from Hetzner.

I wrote my own basic monitoring software called Goplum to make sure things that should be running on my servers actually are. If it fails checks it sends a notification to my phone using Pushover, and a message to a channel on my private IRC server (running Ergo).

Hardware

For the last two years I’ve used a laptop as my main computer⁵. It’s a Dell G15 Special Edition. After some upgrades its specs are:

Intel i7-12700H processor
NVIDIA GeForce RTX 3060 graphics card
2x32GB DDR5 RAM
2TB Gen4 SSD

At home it sits behind my Alienware AW3423DWF monitor. This is a Quantum Dot OLED monitor, and I bought it almost immediately upon seeing someone else use it at a LAN event. As did three other people that were sat near us. It just looks so good. I wasn’t sure I’d get on with a widescreen monitor, and made sure it supported picture-by-picture before buying it, but have never actually used that. I generally run whatever I’m working on at nearly-fullscreen, then have Todoist or Logseq on the far right. Games go fullscreen over everything, of course.

Above the widescreen I have a curved lightbar that I find helps reduce eyestrain and makes it easier to focus on the monitor. And on top of that I have a Logitech Brio webcam which gives a great picture with lots of control over pan/zoom/etc for meetings. I have a little set of systemd scripts that detect when the external webcam is connected and disable the laptop’s built-in webcam, and vice-versa when it’s disconnected; that means whenever I jump into a video call in any app there’s only one input and it’s always the right one, which is nice.

Below the main monitor and its friends, I have a generic 1080p portable monitor connected to the laptop via USB-C. This comes with me when I travel, and serves as a Discord/IRC monitor when at home. It’s handy for throwing other things on as well, like if I need to reference a design while writing code, and don’t want to keep alt+tabbing.

I have two Durgod K320 keyboards: one with Cherry MX Silent Red switches for when I’m at home and don’t want to annoy people with the sounds, and one with Cherry MX Blues for when I’m travelling and can make a bit more noise in exchange for a much nicer typing experience. They’re both tenkeyless so they can handily fit in a backpack.

I use a Zowie EC1-C wired mouse, which does the job well. I got fed up with mice from more mainstream manufacturers failing: my previous mouse was a Razer Viper that lasted 7 months before it started losing clicks. All my audio goes through a HyperX Cloud Alpha Wireless, which has the most insanely good battery life of any wireless headset I’ve ever owned or seen. Their claim of 300 hours is not just marketing!

In non-computer hardware, I have a Bambu P1S 3D printer that I’ve mentioned in my blog before. For printing in fewer dimensions I have an Epson EcoTank ET-1810, which is surprisingly reasonable for an inkjet printer, and the ink costs aren’t as astronomical as normal cartridges. Plus, you can’t decide to DRM ink when it’s loaded raw into the printer…

I have a small Zigbee network set up mainly to control the lights and the old 3D printer I no longer use (the P1S has a sensible standby mode so you don’t have to yank its power away to make the fans stop spinning!). That’s run through a Raspberry Pi running Zigbee2Mqtt and a custom app I wrote to monitor events on MQTT and generate the appropriate responses (or just log the data).

My daily-driver phone is an iPhone 15 Pro, but I have a bunch of Android devices kicking around for work purposes. I read nearly every day on a Kindle Paperwhite Signature Edition.

Money and work

I use You Need A Budget for figuring out how much money I can spend on what, and making sure I have enough put aside for bills and taxes and so on. I use it for both business and personal accounts together, but I mark all the business categories and accounts with a 💼 emoji, and have some scripts that use the API to make sure the money split makes sense⁶.

Every recurring monthly and annual payment has a category, as do all upcoming events I plan to attend, and all the wonderful taxes I have to pay. I also have categories for hobbies and other expenditure. I think all of that is pretty typical for a normal YNAB setup.

I also use YNAB for tracking anything I might like to buy at some point: it gets a category in a “Wishlist” group, and I’ll occasionally budget to buy things from there. I try (and often fail) to use that as a way to put off impulse purchasing. As my income is often very inconsistent, I try to set aside money so that I can “spend” a bit every week on things in the wishlist group. That feels a lot nicer to me than just budgeting everything in one go as soon as I get paid, and gives me more time to consider whether I really want those things⁷.

For work, I do all the more technical accountancy and tax filing and so on through FreeAgent. For my fairly straight-forward tax situation, I can basically manage everything myself using FreeAgent without paying for an accountant. That does mean occasionally spending a bunch of time reading tax manuals, but I’d rather fully understand the rules than just follow advice anyway.

The problem with implementing productivity systems or switching todo apps or anything of that ilk is that it can feel like you’re being productive by doing so. It takes a while to realise that you’re not, and actually getting on with things is better. ↩︎
Or just accept that I’ll maybe do them at some point. I kept the labels vague to try and not make it feel judgmental. ↩︎
Including writing this blog post right now. ↩︎
One day I will get all the justify- and align- options right without looking them up, but until then… ↩︎
This is very normal for developers, but much less common for computer gamers. I’ll probably go back to a desktop next time I upgrade, as I’m travelling a lot less. ↩︎
I can’t just arbitrarily move money from the business to myself, it has to be given as salary or a divided etc, which has tax implications. If I budget “business money” for personal things, then it actually has to be properly transferred and accounted for before being spent (as I’m in effect budgeting a future paycheck). It would be cleaner to keep the accounts completely separate, but I find this way works well for me as I can be a bit more flexible. ↩︎
It feels a little like I’m giving myself pocket money, which is a bit weird, but it’s a lot better than having no “income” for several months when client invoices fall weirdly or I’m between contracts. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.

An introduction to containers

2020-03-01T00:00:00Z

So. Many. Containers.

I’m a huge fan of (software) containers. Most people I know fall in to one of two camps: either they also use, and are fans of, containers, or they haven’t yet really figured them out and view them as some kind of voodoo that they don’t really want or need.

I’m writing this short guide to explain a little how containers work - and how running something in a container isn’t really that much different to running it normally - to hopefully enable more people in that second group to give them a try. It’s aimed at people who have a fairly good grasp of how Linux works.

Containers are often mentioned in the same breath as VMs, which is not a helpful comparison or analogy. Think of containers as standard units of software, much like how Intermodal containers are standard units of freight transport across the world. When a company internationally ships goods in volume there isn’t a question about how they’re packaged - they go in an intermodal container. The same container can be deployed on a freight train, a lorry, or a ship. The haulage company doesn’t need to care what’s in the container because they’re completely standardised. Likewise, with software containers you don’t really need to care about what’s inside: the software you’re deploying could be written in Go, Python2, Python3, Bash, PHP, LOLCODE, or anything¹.

What does a running container look like?

When you run a container, you’re just running a process. In a lot of ways it’s not very different to what happens if you were to start the same process on the host computer.

For example I have a container that just runs cat(1). With no arguments, cat will read from stdin until it receives an EOF, so it’s handy to test with. If I run ps a on my computer, I can see the cat process in amongst everything else I’m currently running:

    PID TTY      STAT   TIME COMMAND
   7199 pts/1    Ss     0:01 /usr/bin/zsh
 323806 pts/0    Ss+    0:00 /bin/cat
 324120 pts/4    R+     0:00 ps a

The /bin/cat process is in a container, and the ps a underneath it is just running like normal on my desktop. They look very similar, right? If I look under /proc/323806 I can see all the usual attributes, the same as any other process running.

If I run ps in a container, though, it’s a different story:

    PID TTY      STAT   TIME COMMAND
      1 pts/0    Rs+    0:00 ps a

So inside the container it looks like there’s only one process running. It can’t see anything running “outside” on my desktop. The secret here is that this isn’t a special container trick: this is just a feature of the Linux kernel called namespacing.

If we go back to procfs and look at the ns/pid node we can see the process in the container is in a separate PID (process ID) namespace to the one on my desktop:

# readlink /proc/323806/ns/pid  
pid:[4026534564]
# readlink /proc/7199/ns/pid
pid:[4026531836]

Almost all² processes running ’normally’ on my desktop have the same PID namespace, whereas each container gets their own by default. PID namespaces are hierarchical: a new process is assigned a PID in its own namespace, and the parent namespace, and the grandparent namespace, and so on. That’s why I can see the process running in the container from my normal shell - the container’s namespace is a child of the main namespace all of my desktop software is running in.

Linux supports - and container software makes use of - a bunch of other namespaces too: mount points, network, UTS³, cgroups, and more. These all play a part in isolating a container from the system it is running on.

You can manually run a process with unshare(1) to “unshare” some namespaces from the parent process. For example if I run unshare -fp --mount-proc ps a, it looks very similar to running ps instead the container:

    PID TTY      STAT   TIME COMMAND
      1 pts/4    R+     0:00 ps a

So: a process running inside a container is just a heavily namespaced process running otherwise normally in the operating system. No voodoo magic here!

What about the filesystem? What are ‘images’?

Containers run in their own mount namespace meaning mount points can be different inside the container to those on the host. This means the container can have a different / mounted to the host, effectively giving it its own filesystem.

The root filesystem of the container is defined in the container’s image. If I use Docker to run a container using the Ubuntu image (docker run ubuntu), the root filesystem inside that container will resemble a minimal ubuntu install. Note that this is just the filesystem: the container doesn’t have its own kernel.

You might be thinking that sounds pretty inefficient. Downloading Ubuntu is definitely not instant, and doing it for every application you run would be insane! Quite. Containers solve this by using filesystem layers. These are stacked on top of one another to create the final filesystem. Each layer can be retrieved and cached independently of all others.

Say (for simplicity) that the Ubuntu image is a single layer. If I run one container with that image, then the layer will be downloaded and cached once. If I run three hundred containers with that image then the layer will be downloaded and cached once. Even better, if I use another image that’s based on Ubuntu but adds some software on top, only the “on top” layer will be downloaded if I already have the relevant “Ubuntu” layer cached.

If all the layers are cached, what happens when you change a file? This is dealt with using the copy-on-write technique: when you modify a file it is copied from the source layer and the changes are only made in a new layer. This is handled by the OverlayFS filesystem which is part of the mainline kernel.

When a container is running, changes made to its filesystem are temporary, and do not persist across container restarts. To persist data - or introduce new data to a container - you can mount volumes. How this works varies depending on how you’re running your container, but at the basic level it is pretty much the same as bind-mounting (mount -o bind).

You may be familiar with using chroots to change the apparent root directory of processes, perhaps with full-blown “jails” built on top. Containers offer much better isolation thanks to the use of namespaces. Instead of being constrained to a portion of the host’s filesystem, they don’t even have it mounted! Containers also get to specify their environment - if they expect in certain places, for example - instead of the sysadmin having to manually set up the chroot. Finally, containers offer much more fine-grained control over what processes can do (if you want it), and allow much more advanced use-cases such as inter-container networking.

Images and filesystems employ a little magic to ensure that layers are reusable and cacheable, but again there’s nothing terribly special about them: a container has a filesystem that appears to it to work the same way as a filesystem on the host, and it’s using a standard filesystem shipped with the kernel.

How about networking?

Again, networking is namespaced, so a container has its own network stack, its own virtual network interface, its own IP address and so on. How that network interacts with your real network depends on how you’re running the container. Docker, for example, can add iptables rules to NAT traffic between containers' networks and the outside world.

Containers can generally be connected into networks, and can communicate amongst themselves without the traffic actually leaving the host machine. This allows you to, for example, run a SQL database and connect it to a web application without ever exposing the database to the outside world. Moreover, as well as being isolated from the outside world, it’s isolated from other containers in other networks. If one of your applications has a crazy bug or is compromised, this significantly limits the damage it can do.

You have to explicitly opt in to “publishing” ports from a container, which exposes them to the outside world (either directly, or via a load balancer or some other middle-man, depending on how you’re running the container). This means you can pick and chose how the outside world sees the app you’re deploying: if it’s a web service that listens on both port 443 and port 80, you can chose to only expose the encrypted port.

If you run some containers and create some networks, you can see the interfaces and bridges on the host:

$ ip l
...
6: br-2405a8cc0445: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:3e:fa:23:62 brd ff:ff:ff:ff:ff:ff
8: veth8ed0735@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2405a8cc0445 state UP mode DEFAULT group default 
    link/ether b2:c1:5d:55:26:9b brd ff:ff:ff:ff:ff:ff link-netnsid 2
10: veth541d84b@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-9d7bc4024c1a state UP mode DEFAULT group default 
    link/ether 86:aa:5f:ee:da:1a brd ff:ff:ff:ff:ff:ff link-netnsid 1

This shows a bridge and two virtual NICs, just the same as if you’d manually created them (albeit with seemingly random names). So: as before, nothing special here.

Docker? Compose? K8s? Floccinaucinihilipilification?

(OK, Floccinaucinihilipilification isn’t actually a container technology, as far as I’m aware.)

Docker is the most popular container engine - that is, the bit of software that actually runs containers. It’s responsible for setting up all those namespaces we found, downloading and caching image layers, and actually starting and stopping the processes. Docker runs as a system-wide daemon - when you run a command like docker run ubuntu it actually just instructs the daemon to do the work.

There are several alternatives to Docker for running containers; one interesting one is Podman which runs containers without a daemon. Container engines have all standardised around the same image format looked after by the Open Container Initiative, so an image you build in Docker can be used in Podman, or pretty much any other engine.

Docker compose is a tool for defining and running multiple-container applications. I mentioned earlier running a database alongside a webapp - in practice to do this you’re going to have to configure a network for them, configure a mount point for the database to persist its data on, pass credentials in to both the database and the application, and so on. Doing all that by hand is tedious and error prone.

Docker compose lets you write “compose files”, which are simple yaml descriptions of the containers you wish to run, their properties, and details about any volumes or networks you may want. Out of the box, docker-compose will create a default network for each compose file you run so the containers within it can communicate.

Kubernetes is a container orchestrator, designed to automate deployment and management of large numbers of containers. It works with Docker under the hood, but provides a huge amount of tooling on top to allow you to deploy applications and manage their dependencies. It runs across multiple physical (or virtual) machines (while still allowing containers to communicate privately), and can support massive workloads by scaling out services (running multiple copies of a container on different hosts) and load balancing. Kubernetes is sometimes shorted to k8s (as in K - 8 elided letters - s) because computer people don’t like long words.

OK, they make sense now. But why bother?

Hopefully if you’ve read this far you’ve already picked up on some of the potential benefits, but this is my personal list:

Isolation. If I run some software in a container, there is very little it can do to upset me. It’s not going to leave bits of itself all over my filesystem, it can’t steal all of the secrets in my home directory, I can even limit its CPU and memory resources if I want. If I decide to stop running it, I just delete the container and it is completely gone: no trace remains.

Ease of use. If you give me a container image I have a very good idea of how to run it already. I might need to do some minor configuration to expose ports or mount volumes, but there’s no question about how to run it, how to make it automatically start, and there’s no “installation” procedure. If I want to then swap it with an alternative (say, move from MySQL to MariaDB), it’s potentially just a case of changing the name of the image I pull.

Dependencies included. Containers just run. Python 2 software includes Python 2 and just work. Python 3 software includes Python 3 and just work. I don’t have a massive headache trying to run both at the same time, because they take care of their own messes. Similarly I’m not going to have to install npm or cargo or composer to pull in dependencies for an application: that’s going to have been done in the build process.

Reproducibility. As a fallout from having dependencies included and being isolated from everything else, containers give you amazing reproducibility. If it “works on your machine” in a container, it’ll almost certainly work in production because it’s the exact same environment.

Standardisation. At the start of this article I called containers standard units of software. One of my favourite advantages of containers is that you basically get an API to list all the software you’re running. Most container engines let you supply labels attached to containers as well, so you can add your own annotations. I use this to annotate services which expose HTTP endpoints, and I have a tool that automatically generates SSL certificates for them and configures haproxy to route traffic to them. I can’t imagine how I’d do this without containers - I imagine it’d involve a lot of manual work.

Hopefully this has helped demystify containers a little. If you feel like I’ve missed something important out, or I’ve left you more confused than when you started, feel free to drop me a note using the feedback form below.

OK, maybe you should care if you’re deploying something written in crazy languages like PHP. ↩︎
Some multi-process apps, such as web browsers, are starting to use namespaces to enhance security, as do certain package systems like Flatpak ↩︎
“Unix timesharing system”; in practice this means having a separate hostname ↩︎