Debugging beyond the debugger

Published on May 8, 2019

Collection of tools hanging on a wall
Real-life debugging tools

Most programming – and sysadmin – problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.



tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Understanding Docker volume mounts

Published on Apr 1, 2019

The Docker project logo
The Docker project logo

One thing that always confuses me with Docker is how exactly mounting volumes behaves. At a basic level it’s fairly straight forward: you declare a volume in a Dockerfile, and then either explicitly mount something there or docker automatically creates an anonymous volume for you. Done. But it turns out there’s quite a few edge cases…

Changing ownership of the folder

Perhaps the most common operation done on a Docker volume other than simply mounting it is trying to change the ownership of the directory. If your Docker process runs as a certain user you probably want the directory to be writable by that user.

Over-the-top optimisations with Nim

Published on Dec 9, 2018

Christmas Tree from Advent of Code 2005
Christmas Tree from Advent of Code 2005

For the past few years I’ve been taking part in Eric Wastl’s Advent of Code, a coding challenge that provides a 2-part problem each day from the 1st of December through to Christmas Day. The puzzles are always interesting — especially as they get progressively harder — and there’s an awesome community of folks that share their solutions in a huge variety of languages.

To up the ante somewhat, Shane and I usually have a little informal competition to see who can write the most performant code. This year, though, Shane went massively overboard and wrote an entire benchmarking suite and webapp to measure our performance, which I took as an invitation and personal challenge to try to beat him every single day.

For the past three years I’d used Python exclusively, as its vast standard library and awesome syntax lead to quick and elegant solutions. Unfortunately it stands no chance, at least on the earlier puzzles, of beating the speed of Shane’s preferred language of PHP. For a while I consoled myself with the notion that once the challenges get more complicated I’d be in with a shot, but after the third or fourth time that Shane’s solution finished before the Python interpreter even started I decided I’d have to jump ship. I started using Nim.

DNS-over-TLS on the EdgeRouter Lite

Published on Dec 17, 2017

An EdgeRouter Lite
An EdgeRouter Lite

DNS-over-TLS is a fairly recent specificiation described in RFC7858, which enables DNS clients to communicate with servers over a TLS (encrypted) connection instead of requests and responses being sent in plain text. I won’t ramble on about why it’s a good thing that your ISP, government, or neighbour can’t see your DNS requests…

I use an EdgeRouter Lite from Ubiquiti Networks at home, and recently configured it to use DNS-over-TLS for all DNS queries. Here’s how I did it.

Installing unbound

Out of the box, the ERL uses dnsmasq to service DNS requests from local clients. To get DNS-over-TLS support I switched to using Unbound, an open source DNS resolver with support for many modern features such as DNSSEC and DNS-over-TLS.

I was thinking about switching DNS providers recently, and found myself whoising random domains and looking at their nameservers. One thing lead to another and I ended up doing a survey of the nameservers of the top 100,000 sites according to Alexa.

The top providers by a large margin were, unsurprisingly, Cloudflare and AWS Route 53. Between them they accounted for around 30% of the top 100k sites.