Chameth.com - posts not debugging-beyond-the-debugger, monthly-meanderings-2026-04, why-you-should-be-using-https

Tracking PBs without the dark patterns

2026-07-23T00:00:00Z

I’ve recently been bitten by some kind of strange exercise bug. I’ve gone from being mostly sedentary to struggling to not go out for a cheeky bike ride on a day that’s meant to be a rest day. Maybe this is what a midlife crisis feels like?

Anyway, one of the things I’ve enjoyed about this development is seeing my personal best efforts gradually go up. There’s something incredibly rewarding about doing a run, and then seeing a nice little gold medal because as part of that you ran your fastest 1km ever. Unfortunately the only software I’ve found that does that is Strava, and boy do I not get on with it.

The many shortcomings of Strava

Strava is basically the app for running and cycling. Everyone talks about it. It sounds like they like it. But I really, really don’t. It’s like someone went through a catalogue of user-hostile behaviours and said “we’ll take them all!”.

The problems started in the setup wizard: it touts a load of privacy features, but doesn’t offer any way to enable them. All your activities are public by default, and you have to spelunk around in the settings to turn that off. I wasn’t actually sure if I wanted my activities to be public or not, and ended up flip-flopping around a bit. There is a clunky workflow to bulk change past activities, but it’s arbitrarily time-gated so you can’t use it too often…

The bulk operations are extremely limited, too: you can set the activity visibility, and your heart rate visibility, and that’s it. Want to bulk delete things? You’re out of luck. Bulk rename? No chance. You either do them one-by-one or use some janky third-party script. I think I pressed around 300 little ‘delete’ links in the course of trying to get my activity history imported in a way I liked.

Those problems give a bit of a signal about Strava’s priorities, but I could live with them. Once I’ve stopped faffing with the initial import I’m not really going to want to change the privacy settings or bulk edit things. Alas, the problems didn’t end there.

Look at me! Look at me! GIVE ME ENGAGEMENT!

The most obvious one is that everything is an upsell. Vast swathes of the app are premium-only, and they don’t ever get out of your way: they stick around acting as adverts. We’re all used to freemium apps that try to hawk upgrades, but the sheer amount of screen real estate taken up by these is egregious. I started a premium trial, expecting things to get a lot better, and… they didn’t really. All the premium features are still branded as such, so now instead of “look at what you could have if you paid us” it’s “look at what you’re getting for paying us (don’t even think about cancelling)”. Subscribing also makes your avatar get a premium outline, and there’s no way to disable it. I really hate that. Now I’m an advert myself. I don’t want to be part of their weird dark patterns, thank you very much.

And while we’re talking about dark patterns: there’s so much engagement bait that I physically cringe at it. You get a notification for logging an activity with a vague call to action to check it out. You get a separate notification when you break or come close to one of your PBs, but it doesn’t tell you which; there’s plenty of space in the notification for it, but they want you to hit the app instead. It all just feels so slimy to me. I don’t know whether it doesn’t bother other people as much, or if they’re desensitised to it, or just don’t realise how pleasant things could actually be.

Making a replacement

I realised pretty early on that if I only want automatic PB tracking, I could just write something myself to do it. I already use Health Auto Export to exfiltrate data from Apple Health, including workouts. I can just make a service that receives that data, crunches some numbers, and activates the dopamine receivers in my brain when I set a new record. But I didn’t really want to bother doing that if I could avoid it. I tried a whole slew of different apps and none of them did the thing, at least not without a subscription. And Strava had soured me on fitness subscriptions. I finally resigned myself to making something.

And by “myself” I mean “an LLM”. This is one of the first projects where I’ve not written a single line of code, and I’ve barely even reviewed anything produced by the LLM. I gave it an overall architecture and some sample data, and had it write a plan. I critiqued the plan, had it broken down into a few different phases, and then had it work through them. I occasionally stopped it when I saw it doing something stupid, and gave it a couple of technical steers, but my feedback has mostly been functional. I’m probably not going to be releasing it publicly any time soon for that reason — I don’t want to publish it under my name when I’ve not even looked at the code. I also wouldn’t be comfortable making software this way if other people were going to rely on it, but for non-critical, personal-use-only software I’m happy with the approach.

Anyway, I called it “Pompei Band”. Project names are hard. One of my common techniques is to put a term into an anagram finder; in this case “PB dopamine” anagrams to “Pompei Band”, and it even abbreviates to PB. Weird, but I like it. This is how it shows recent activities on the homepage:

Dashboard entry for a recent run in PB

You can click through to see the standard activity details most tracking apps show: a map, heart rate, pace, gradient, and heart rate zones. You can also click on each effort to see a history of them, which gives a nice overview of progress:

1 mile run PB history

It pretty much does exactly what I want, and has none of the annoying behaviours that Strava leans into. When I first drafted this post I wrote “there’s a few more bits I’d like to add at some point, but I can happily prompt them into existence when I want them”, but in the time I spent procrastinating before taking screenshots I ended up implementing them all. My favourite is this little history widget I added to the homepage, shown here next to Strava’s:

Recent activity calendar in Pompei Band and in Strava

The Strava one just confuses me. I’m not sure what the scale of the circles means. I’m guessing it’s time, which in effect means it just highlights my bike rides which are longer than runs. The colours for the line charts don’t work for me, and I’m not sure what three zig-zag lines are meant to represent. Pompei Band’s version uses distinct, obvious colours for different activities, and scales the brightness independently for each type of activity based on length (so long cycle rides don’t affect the short runs). I also flipped the axes so it looks more like a GitHub activity timeline, because that’s how my brain works. Nothing groundbreaking, but also none of the paper-cuts the Strava version inflicted on me.

Software doesn’t have to be awful

This blog post has been rattling around in my head for a few days. I didn’t want it to just be a litany about Strava being bad, and I didn’t want to sound like an LLM hype piece, which made it tricky to figure out what I was actually trying to say. I think the core of it is: software doesn’t have to be awful.

Everyone’s heard of enshittification, and it’s so prevalent that it’s easy to accept it as a fact of life. That’s just how late-stage capitalism works, right? But we can just make better software. And as LLMs slowly become more and more capable, that “we” grows to a broader and broader group. It’s easy to be negative about that: we’re going to drown in vibe-coded products, much like we’re already drowning in written LLM slop. On the other hand, a future where you can just say “make me single-player Strava without all the dark patterns” and get something that works just how you want it is pretty exciting.

The one thing I can’t decide: if everyone can whip up their own individual software, will the existing platforms actually improve? If Strava didn’t have all the dark patterns and engagement bait I would have just have paid for it. Will they realise that? Or will they double down even harder, try to lock you into their platform, and build moats around user data? Now I’ve said it, it seems pretty obvious they’ll try to do that. But I hope they won’t.

Modern CSS is fun

2026-03-17T00:00:00Z

I’ve been doing a bunch of CSS tweaking recently, and keep being surprised by how nice modern CSS is to work with. As someone grey-haired enough to remember writing HTML without CSS, it’s amazing to think how far along web technology has come¹. I wanted to demonstrate some of the handy bits and pieces I’ve used recently.

:has

The :has selector allows you to effectively query for child elements. While a span will match a <span> within an <a>, a:has(span) will match an <a> that contains a <span>. This really shines when combined with more complex selectors, for example:

input {
    border-radius: var(--border-radius);

    &:has(+ .results:not(:empty)) {
        border-bottom-left-radius: 0;
        border-bottom-right-radius: 0;
    }
}

This is part of the styling for my film search box. When the search box is presented initially it has fully rounded corners. But when it is followed by a non-empty results element, it removes the rounding on the bottom corners so that the border continues in a straight line down into the results. You can see it in action below; just enter a few characters (like “the”) to get some results:

You could achieve the same effect by having JavaScript add a class to the <input> element, but I’ll take a CSS solution over a JavaScript solution any day.

Nested rules

You probably spotted this in the example above. It’s what finally made me switch from SCSS to plain CSS. If you have a rule for .foo and a rule for .foo .bar you can just nest them. Not only does it save repeating yourself an awful lot, it keeps everything organised nicely. For example, the CSS for my film list embeds looks like this:

a.film-list {
    display: grid;
    grid-template-areas: "header images" "description images" "meta images";
    grid-template-columns: 1fr auto;
    grid-template-rows: auto 1fr auto;
    row-gap: var(--small-space);

    @media (width <= 800px) {
        grid-template-areas: "header" "description" "images" "meta";
        grid-template-columns: 1fr;
        grid-template-rows: auto auto auto auto;
    }

    & + & {
        margin-top: var(--medium-space);
    }

    h3 {
        font-size: var(--font-size-xxlarge);
    }

    .description {
        /* ... */
    }
}

Everything is wrapped up in a single bundle, including the @media rules for changing the layout on smaller screens, the lovely little & + & rule that adds some extra margin if there are two lists in a row, etc. The & syntax refers to the parent selector, so & + & in this case is the same as a.film-list + a.film-list: it’s very handy! The best part about this is that it’s the exact same syntax as SCSS, so converting is easy². The styled element looks like this:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Media range syntax

Once again, you might have spotted this in the previous snippet. I’m not deliberately teasing things, I promise! Back in the day, you did media queries like so:

@media (min-width: 1000px) and (max-width: 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I always hated this syntax. I always got muddled up as to whether I wanted “min” or “max”, and whether they were inclusive or not. It’s easy to reason through, but it never came naturally. Fortunately you can now just use ranges:

@media (1000px <= width <= 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I find this style so much easier to write and understand.

Anchor positioning

Trying to dynamically position one element next to another used to exclusively fall within the purview of JavaScript. Fortunately anchor positioning fixes all this. I use this in the film search field I showed above:

.film-search {
    input {
        anchor-name: --film-search-box;
    }
    .results {
        position: absolute;
        position-anchor: --film-search-box;
        position-area: bottom center;
        position-visibility: always;
        width: anchor-size();
    }
}

This does lots of fun things. The .results element is anchored to the input field (via the --film-search-box name), and it’s positioned on the bottom, directly under the anchor. The position-area rule describes a 3x3 grid, with the anchor at the center, so top left would put it diagonally adjacent to the anchor. Next, position-visibility keeps the .results element visible when the anchor isn’t; otherwise when you scroll the anchor off the screen the results would immediately vanish. Finally, the special anchor-size() function makes the .results element take up the same exact width as the anchor.

All-in-all, this puts the results exactly where I want them, without having to deal with any JavaScript at all. I can see how it’d also be amazingly useful trying to do tooltips or other forms of ‘floating’ content too. You can even leave the positioning up to the browser, giving it hints about which order to try, or what property to optimise for (e.g. you can say “anchor to the left or right, whichever has more horizontal space”).

attr() function

This one blew my mind a little. In some of the elements on the site, I’ve added a little label to the top. It looks something like this:

It’s meant to just be a visual flourish, not a semantic title or anything, so I initially just added a ::before selector to each element, customising the content to have the right value. I didn’t like the duplication, though. Now, instead, I do this:

&::before {
    content: attr(data-title);
}

Each component defines its title in a data attribute: <div class="raised-box" data-title="an example">, and the attr function plucks it out and puts it in the content rule. It’s worth noting that this is only widely supported for the content rule, you can’t yet use it for colours or dimensions or other things. It still feels a bit magical, though. Like you’re making a reusable, customisable component with just HTML and CSS.

layers

I’d come across layers a few times, but I never found a need for them. It seemed like something you’d only really need with complicated design systems, or something. Then I had a problem and layers were the perfect solution!

The headings on this site have a whole bunch of CSS attached to them. Currently they look a bit like blue painter’s tape with handwriting on them. This style is applied to all headers, and then the various places that don’t want it had to manually reset everything. I got fed up with that, so decided to add a plain-header class which would “disable” the extra styling:

h2,
h3,
h4,
h5,
h6 {
    /* common styles for all headers, regardless */
    font-family: "Chris Hand", sans-serif;
    font-size: var(--font-size-xxlarge);
    color: var(--text-highlight-colour);
    margin: var(--medium-space) 0;
    line-height: 1;

    &:not(.plain-header) {
        /* do fancy stuff if it _doesn't_ have the .plain-header class */
        transform: rotate(-0.15deg);
        padding: var(--smedium-space);
        font-weight: 800;
        filter: drop-shadow(3px 3px 5px rgb(0, 0, 0));

        a {
            background: none;
            text-decoration-skip-ink: auto;
        }
    }
}

This might seem a bit backwards but I want the default to be styled, as those are far more common. I added this change, sprinkled around some “plain-header” classes, tidied up a lot of duplicate CSS, and things mostly worked. Mostly. Some things stopped working. In a few places the headers had slight tweaks, and they stopped working entirely.

It turns out in moving the rules from a plain h2 selector to a h2:not(.plain-header), I’d made them more specific. Previously a selector like h2.special would have been more specific, so its properties would override those from the less specific h2 selector. My first reaction was to try and hack around it. Changing all the overrides to body h2.special would make them more specific again, but I didn’t want to have to remember to do that forever more.

Instead, I defined some custom layers, and put the header definitions in one:

@layer reset, links, headings

@layer headings {
    h2,
    h3 {
        /* etc */
    }
}

The first @layer rule defines some layers that will be handled in a specific order: first the “reset” layer, then the “links” layer, then the “headings” layer, and then anything not in a layer after that. Rules in one layer don’t have to worry about beating the specificity of rules in another layer, because they’re handled separately.

Note that I couldn’t just put the headings in a layer and call it a day: their rules have to come after the CSS reset, and after the normal link styles, as headers have a bit of extra styling for nested links. If the CSS reset weren’t in a layer, then the * { margin: 0; padding: 0; } type reset would apply over the rules from the headings layer.

Using layers is definitely a bit fiddly, and is not necessary for a lot of sites, but it’s a much cleaner alternative to ugly specificity hacking.

Things I’m looking forward to

There are a few things that aren’t (widely) available yet, that particularly interest me:

sibling-index() and sibling-count()

These new functions are available in WebKit and Blink based browsers, but not Firefox. They give you the index of the element within its siblings, or the count of siblings. For the film list component above, I currently have this abomination for the overlapping posters:

&:nth-child(1) {
    z-index: 5;
    left: 0;
}
&:nth-child(2) {
    z-index: 4;
    left: calc(133px * var(--overlap) * 1);
}
&:nth-child(3) {
    z-index: 3;
    left: calc(133px * var(--overlap) * 2);
}
&:nth-child(4) {
    z-index: 2;
    left: calc(133px * var(--overlap) * 3);
}
&:nth-child(5) {
    z-index: 1;
    left: calc(133px * var(--overlap) * 4);
}

These could be replaced with something like:

z-index: calc(1 + sibling-count() - sibling-index());
left: calc(133px * var(--overlap) * (sibling-index() - 1));

As an added bonus, it would scale to any number of posters. index being 1-based not 0-based is a bit unfortunate as it makes using it in calculations awkward (see 1 offset in both of those rules!), but it’s leagues better than writing 5 separate rules.

random()

Currently only available in Safari. There are a few places where I’d like to have slight random variations of the style. Things like elements that are rotated slightly for aesthetics; they look a bit silly if they’re all identically positioned. I also randomise the icons and positions of my rating stars to break up the visual monotony. Here are a few examples:

Currently I do the stars by hardcoding a bunch of classes and having the backend randomly apply one when it generates the markup for stars. Being able to do it in CSS would be great, though.

Mixins

This one’s so far off it’s not even listed on caniuse.com yet. Mixins are another handy feature of SCSS, that allow you to define reusable blocks of rules, then import them when needed:

@mixin fancy-background {
    background: /*...*/;
}

.some-element {
    @include fancy-background;
}

#other-element {
    @include fancy-background;
}

This would be particularly useful if you can’t easily control the markup to add classes to everything. For example, my footnotes are rendered by a markdown plugin, and don’t easily have a way to add extra classes to them. To style it the same as another element, I currently duplicate a bunch of rules between them both. Mixins would allow me to define those rules once, and then import them in both places.

Aside from JavaScript, which seems to just endlessly reinvent new frameworks and ways to make the most sprawling dependency tree possible. ↩︎
It also means that if your website’s syntax highlighting library doesn’t seem to understand nested rules, you can just pretend your perfectly valid CSS is actually SCSS and it’ll magically work. Grumble, grumble. ↩︎

The longest way to represent a date

2026-03-14T00:00:00Z

The other day, someone on IRC posed this question: “What is the longest way to represent a date using any means possible that isn’t just repeated filler?”

Some people jumped for writing the date out in languages that had longer translations. My immediate reaction was instead to suggest an obnoxious string-based representation of a unix timestamp: “one second after one second after one second after … midnight on January 1st 1970”. It’s very repetitive, but it’s not filler: taking out any of the repetitions would change the value. Effectively it’s a successor function for unix timestamps, writ long. Some quick napkin maths suggests the full string representation would be around 30GiB¹.

An obvious approach to make it bigger is to make it more precise. If we do the same sort of successor function for milliseconds it’d be a bit over 1000x longer². It actually ends up being just over 35TiB³.

There’s no reason to stop there, though. You can step down to microseconds, nanoseconds, and so on… or… we can just skip right to the end. How about: “one oscillation of the caesium-133 hyperfine transition frequency after one oscillation of the caesium-133 hyperfine transition frequency after … midnight on January 1st 1970”? There are over 9 billion of those every second, and the phrasing is wordy. That comes out to almost 1ZiB⁴. If we pooled all the data storage on the planet together we could save a couple hundred of these timestamps.

One final step, then: why use the unix epoch when we could use the universe’s own⁵? We’re something around 4×10^17 seconds past the big bang, so… hold on, I need to look up some units… that’d be over 200,000 yobibytes⁶. There’s not actually a standard prefix for that order of magnitude, according to Wikipedia. I think it should be 200 robibytes. Does anyone have a contact at the IEC?

That is a big number. Is it impossibly big? If we wanted to track our current time, we’d need to write around 600GiB/s⁷ to disk. That’s a bit more than we can manage at the minute, but isn’t completely outrageous. What about storage? Apparently hard drives require around a million atoms to store a bit of data. There’s some research showing it’s possible to use as few as 12, but I don’t think it’s anything that can be used at scale anytime soon⁸. So just for the direct data storage, not counting all the other infrastructure you need for a disk drive to work we’d need around 10^36 atoms. If we sourced our atoms for our local paperclip maximiser we’d need around 200 trillion paperclips. For just one timestamp.

On the plus side, this timestamp format would compress wonderfully.

17 bytes of repetition * 1773528583 seconds ~= 3×10^10. ↩︎
There’d be a thousand times more repetitions, and each one would have the extra five bytes from the “milli” prefix. ↩︎
22 bytes of repetition * 1773528583000 milliseconds ~= 3×10^13. ↩︎
72 bytes of repetition * 1773528583 seconds * 9192631770 oscillations per second ~= 1×10^21. ↩︎
Other than it being impractical, not known to a decent accuracy, and the myriad other problems, of course. ↩︎
72 bytes of repetition * 4×10^17 seconds * 9192631770 oscillations per second ~= 3×10^29. ↩︎
72 bytes of repetition * 9192631770 oscillations per second ~= 6×10^11. ↩︎
Not that we could scale conventional drives up this much, either. ↩︎

Just a nod

2026-02-11T00:00:00Z

Last month I added a new feature to the bottom of almost all pages on this site: a “nod” button. It’s a bit like the upvote buttons you see on bear blog posts, or the ubiquitous like button you get on social media, with one key difference: it doesn’t track or count anything. Before I get into the whys and hows, this is what it currently looks like:

The nod button in all its glory. If you want a clickable version you’ll need to scroll a bit more.

So why bother? I like hearing from people who have looked at my website. I have a contact form that’s also embedded at the bottom of all my blog posts; it’s always a delight when people reach out. Actually writing a message is a pretty high bar that most visitors won’t clear, though. A like/upvote/whatever-you-want-to-call-it is a much less demanding interaction.

And why no counting? I’d been thinking about some kind of upvote button for a while, and counting was always the thing that put me off. Having a number attached to a post like that feels like I’d end up invoking Goodhart’s law and start aiming for a certain number of upvotes or likes or whatever. I don’t want to be anxiously watching numbers, or feel pressured to make a post “good enough” (I already self-filter too much…).

As for not tracking people, that feels like table stakes to me: I’m an individual concerned about privacy, not a giant mega-corporation who wants to slurp up all the data to serve ads or engagement bait feeds. I don’t really need to know someone’s identity to receive a message from them. In the olden days of the internet, you just knew everyone by screen names which could change on a whim and didn’t reveal anything about the real you. On the internet, nobody knows you’re a dog, as Peter Steiner said.

All these constraints felt insurmountable for a while. I considered just making a button that sent a contact form submission with a pre-filled message, but those end up in my e-mail inbox. I don’t really want to have to process them, especially if there’s a big traffic spike like when Hacker News notices one of my posts. I could store the button presses in a database, but then how would I actually know they happened, especially if I didn’t want to watch a number go up?

An Amazon Dash button

So I did what I usually do with these types of problems: ignored it for a bit. The idea rattles around at the back of my head, and sometimes I’ll come up with an answer. I’ve seen this called diffuse thinking. Eventually the idea bounced to the forefront of my head while I was looking at my IRC client, and the solution suddenly clicked. If you’ve not heard of it, IRC is a chat protocol that’s almost as old as I am. Daniel Stenberg has compiled a nice history of IRC that I won’t repeat here. I use IRC to stay in touch with friends, hang around the channels for some open source projects, and I also run a private server with just me and some notification bots.

The notifications I have on IRC are all “transient”: they’re handy to see in the moment, but it doesn’t matter if I miss them. I have some high-volume news feeds, log messages from some self-hosted services, details about git commits, and so on. Basically just background noise of things that are happening that don’t need action. Perfect for the kind of interaction I want!

Now that I’d finally figured out that the transient-notification-shape goes in the transient-notification-hole, I just needed to actually implement it. As part of that private IRC setup I already have a bot that accepts messages over HTTP and sends them to a channel, so the backend was pretty simple. For the frontend I was imagining something like the ill-fated Amazon Dash buttons. My first attempt was… underwhelming:

Flat UIs are in right now, right?… Right?

I guess I shouldn’t give up the day job to become a graphic designer. Fortunately, I came across this codepen from Thierry Michel with some very fun skeuomorphic arcade buttons. I spent some time and LLM tokens on converting the SCSS to CSS, changing the colour, and making a few other tweaks. I really love how this looks and feels.

This is what it looks like in my IRC client when someone nods (and when containers get pushed to the GitHub Container Registry for my projects…):

Receiving some nods on IRC via a bot

When I added the button I wasn’t sure if I’d keep it, but I’ve very much enjoyed seeing the “nods” come in every now and then. I’ve been surprised by how many come in for the various snippets and other non-blog pages on the site. It’s a nice reminder that people are out there browsing the site, without all the baggage of monitoring analytics or tracking likes.

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎