Chameth.com - posts like android-espresso-spoon, docker-automatic-nginx-proxy but not debugging-beyond-the-debugger, g15-ram-upgrade, utility-user-stories, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.

An introduction to containers

2020-03-01T00:00:00Z

So. Many. Containers.

I’m a huge fan of (software) containers. Most people I know fall in to one of two camps: either they also use, and are fans of, containers, or they haven’t yet really figured them out and view them as some kind of voodoo that they don’t really want or need.

I’m writing this short guide to explain a little how containers work - and how running something in a container isn’t really that much different to running it normally - to hopefully enable more people in that second group to give them a try. It’s aimed at people who have a fairly good grasp of how Linux works.

Containers are often mentioned in the same breath as VMs, which is not a helpful comparison or analogy. Think of containers as standard units of software, much like how Intermodal containers are standard units of freight transport across the world. When a company internationally ships goods in volume there isn’t a question about how they’re packaged - they go in an intermodal container. The same container can be deployed on a freight train, a lorry, or a ship. The haulage company doesn’t need to care what’s in the container because they’re completely standardised. Likewise, with software containers you don’t really need to care about what’s inside: the software you’re deploying could be written in Go, Python2, Python3, Bash, PHP, LOLCODE, or anything¹.

What does a running container look like?

When you run a container, you’re just running a process. In a lot of ways it’s not very different to what happens if you were to start the same process on the host computer.

For example I have a container that just runs cat(1). With no arguments, cat will read from stdin until it receives an EOF, so it’s handy to test with. If I run ps a on my computer, I can see the cat process in amongst everything else I’m currently running:

    PID TTY      STAT   TIME COMMAND
   7199 pts/1    Ss     0:01 /usr/bin/zsh
 323806 pts/0    Ss+    0:00 /bin/cat
 324120 pts/4    R+     0:00 ps a

The /bin/cat process is in a container, and the ps a underneath it is just running like normal on my desktop. They look very similar, right? If I look under /proc/323806 I can see all the usual attributes, the same as any other process running.

If I run ps in a container, though, it’s a different story:

    PID TTY      STAT   TIME COMMAND
      1 pts/0    Rs+    0:00 ps a

So inside the container it looks like there’s only one process running. It can’t see anything running “outside” on my desktop. The secret here is that this isn’t a special container trick: this is just a feature of the Linux kernel called namespacing.

If we go back to procfs and look at the ns/pid node we can see the process in the container is in a separate PID (process ID) namespace to the one on my desktop:

# readlink /proc/323806/ns/pid  
pid:[4026534564]
# readlink /proc/7199/ns/pid
pid:[4026531836]

Almost all² processes running ’normally’ on my desktop have the same PID namespace, whereas each container gets their own by default. PID namespaces are hierarchical: a new process is assigned a PID in its own namespace, and the parent namespace, and the grandparent namespace, and so on. That’s why I can see the process running in the container from my normal shell - the container’s namespace is a child of the main namespace all of my desktop software is running in.

Linux supports - and container software makes use of - a bunch of other namespaces too: mount points, network, UTS³, cgroups, and more. These all play a part in isolating a container from the system it is running on.

You can manually run a process with unshare(1) to “unshare” some namespaces from the parent process. For example if I run unshare -fp --mount-proc ps a, it looks very similar to running ps instead the container:

    PID TTY      STAT   TIME COMMAND
      1 pts/4    R+     0:00 ps a

So: a process running inside a container is just a heavily namespaced process running otherwise normally in the operating system. No voodoo magic here!

What about the filesystem? What are ‘images’?

Containers run in their own mount namespace meaning mount points can be different inside the container to those on the host. This means the container can have a different / mounted to the host, effectively giving it its own filesystem.

The root filesystem of the container is defined in the container’s image. If I use Docker to run a container using the Ubuntu image (docker run ubuntu), the root filesystem inside that container will resemble a minimal ubuntu install. Note that this is just the filesystem: the container doesn’t have its own kernel.

You might be thinking that sounds pretty inefficient. Downloading Ubuntu is definitely not instant, and doing it for every application you run would be insane! Quite. Containers solve this by using filesystem layers. These are stacked on top of one another to create the final filesystem. Each layer can be retrieved and cached independently of all others.

Say (for simplicity) that the Ubuntu image is a single layer. If I run one container with that image, then the layer will be downloaded and cached once. If I run three hundred containers with that image then the layer will be downloaded and cached once. Even better, if I use another image that’s based on Ubuntu but adds some software on top, only the “on top” layer will be downloaded if I already have the relevant “Ubuntu” layer cached.

If all the layers are cached, what happens when you change a file? This is dealt with using the copy-on-write technique: when you modify a file it is copied from the source layer and the changes are only made in a new layer. This is handled by the OverlayFS filesystem which is part of the mainline kernel.

When a container is running, changes made to its filesystem are temporary, and do not persist across container restarts. To persist data - or introduce new data to a container - you can mount volumes. How this works varies depending on how you’re running your container, but at the basic level it is pretty much the same as bind-mounting (mount -o bind).

You may be familiar with using chroots to change the apparent root directory of processes, perhaps with full-blown “jails” built on top. Containers offer much better isolation thanks to the use of namespaces. Instead of being constrained to a portion of the host’s filesystem, they don’t even have it mounted! Containers also get to specify their environment - if they expect in certain places, for example - instead of the sysadmin having to manually set up the chroot. Finally, containers offer much more fine-grained control over what processes can do (if you want it), and allow much more advanced use-cases such as inter-container networking.

Images and filesystems employ a little magic to ensure that layers are reusable and cacheable, but again there’s nothing terribly special about them: a container has a filesystem that appears to it to work the same way as a filesystem on the host, and it’s using a standard filesystem shipped with the kernel.

How about networking?

Again, networking is namespaced, so a container has its own network stack, its own virtual network interface, its own IP address and so on. How that network interacts with your real network depends on how you’re running the container. Docker, for example, can add iptables rules to NAT traffic between containers' networks and the outside world.

Containers can generally be connected into networks, and can communicate amongst themselves without the traffic actually leaving the host machine. This allows you to, for example, run a SQL database and connect it to a web application without ever exposing the database to the outside world. Moreover, as well as being isolated from the outside world, it’s isolated from other containers in other networks. If one of your applications has a crazy bug or is compromised, this significantly limits the damage it can do.

You have to explicitly opt in to “publishing” ports from a container, which exposes them to the outside world (either directly, or via a load balancer or some other middle-man, depending on how you’re running the container). This means you can pick and chose how the outside world sees the app you’re deploying: if it’s a web service that listens on both port 443 and port 80, you can chose to only expose the encrypted port.

If you run some containers and create some networks, you can see the interfaces and bridges on the host:

$ ip l
...
6: br-2405a8cc0445: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:3e:fa:23:62 brd ff:ff:ff:ff:ff:ff
8: veth8ed0735@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2405a8cc0445 state UP mode DEFAULT group default 
    link/ether b2:c1:5d:55:26:9b brd ff:ff:ff:ff:ff:ff link-netnsid 2
10: veth541d84b@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-9d7bc4024c1a state UP mode DEFAULT group default 
    link/ether 86:aa:5f:ee:da:1a brd ff:ff:ff:ff:ff:ff link-netnsid 1

This shows a bridge and two virtual NICs, just the same as if you’d manually created them (albeit with seemingly random names). So: as before, nothing special here.

Docker? Compose? K8s? Floccinaucinihilipilification?

(OK, Floccinaucinihilipilification isn’t actually a container technology, as far as I’m aware.)

Docker is the most popular container engine - that is, the bit of software that actually runs containers. It’s responsible for setting up all those namespaces we found, downloading and caching image layers, and actually starting and stopping the processes. Docker runs as a system-wide daemon - when you run a command like docker run ubuntu it actually just instructs the daemon to do the work.

There are several alternatives to Docker for running containers; one interesting one is Podman which runs containers without a daemon. Container engines have all standardised around the same image format looked after by the Open Container Initiative, so an image you build in Docker can be used in Podman, or pretty much any other engine.

Docker compose is a tool for defining and running multiple-container applications. I mentioned earlier running a database alongside a webapp - in practice to do this you’re going to have to configure a network for them, configure a mount point for the database to persist its data on, pass credentials in to both the database and the application, and so on. Doing all that by hand is tedious and error prone.

Docker compose lets you write “compose files”, which are simple yaml descriptions of the containers you wish to run, their properties, and details about any volumes or networks you may want. Out of the box, docker-compose will create a default network for each compose file you run so the containers within it can communicate.

Kubernetes is a container orchestrator, designed to automate deployment and management of large numbers of containers. It works with Docker under the hood, but provides a huge amount of tooling on top to allow you to deploy applications and manage their dependencies. It runs across multiple physical (or virtual) machines (while still allowing containers to communicate privately), and can support massive workloads by scaling out services (running multiple copies of a container on different hosts) and load balancing. Kubernetes is sometimes shorted to k8s (as in K - 8 elided letters - s) because computer people don’t like long words.

OK, they make sense now. But why bother?

Hopefully if you’ve read this far you’ve already picked up on some of the potential benefits, but this is my personal list:

Isolation. If I run some software in a container, there is very little it can do to upset me. It’s not going to leave bits of itself all over my filesystem, it can’t steal all of the secrets in my home directory, I can even limit its CPU and memory resources if I want. If I decide to stop running it, I just delete the container and it is completely gone: no trace remains.

Ease of use. If you give me a container image I have a very good idea of how to run it already. I might need to do some minor configuration to expose ports or mount volumes, but there’s no question about how to run it, how to make it automatically start, and there’s no “installation” procedure. If I want to then swap it with an alternative (say, move from MySQL to MariaDB), it’s potentially just a case of changing the name of the image I pull.

Dependencies included. Containers just run. Python 2 software includes Python 2 and just work. Python 3 software includes Python 3 and just work. I don’t have a massive headache trying to run both at the same time, because they take care of their own messes. Similarly I’m not going to have to install npm or cargo or composer to pull in dependencies for an application: that’s going to have been done in the build process.

Reproducibility. As a fallout from having dependencies included and being isolated from everything else, containers give you amazing reproducibility. If it “works on your machine” in a container, it’ll almost certainly work in production because it’s the exact same environment.

Standardisation. At the start of this article I called containers standard units of software. One of my favourite advantages of containers is that you basically get an API to list all the software you’re running. Most container engines let you supply labels attached to containers as well, so you can add your own annotations. I use this to annotate services which expose HTTP endpoints, and I have a tool that automatically generates SSL certificates for them and configures haproxy to route traffic to them. I can’t imagine how I’d do this without containers - I imagine it’d involve a lot of manual work.

Hopefully this has helped demystify containers a little. If you feel like I’ve missed something important out, or I’ve left you more confused than when you started, feel free to drop me a note using the feedback form below.

OK, maybe you should care if you’re deploying something written in crazy languages like PHP. ↩︎
Some multi-process apps, such as web browsers, are starting to use namespaces to enhance security, as do certain package systems like Flatpak ↩︎
“Unix timesharing system”; in practice this means having a separate hostname ↩︎

Android testing with Espresso and Spoon

2017-05-16T00:00:00Z

I’ve been spending some time recently setting up automated testing for our collection of Android apps and libraries at work. We have a mixture of unit tests, integration tests, and UI tests for most projects, and getting them all to run reliably and automatically has posed some interesting challenges.

Running tests on multiple devices using Spoon

Spoon is a tool developed by Square that handles distributing instrumentation tests to multiple connected devices, aggregating the results, and making reports.

As part of our continuous integration we build both application and test APKs, and these are pushed to the build server as build artefacts. A separate build job then pulls these artefacts down to a Mac Mini we have in the office, and executes Spoon with a few arguments:

java -jar spoon-runner.jar \
    --apk application.apk \
    --test-apk applicationTests.apk \
    --fail-on-failure \
    --fail-if-no-device-connected

Spoon finds all devices, deploys both APKs on them, and then begins the instrumentation tests. We use two physical devices and an emulator to cover the form factors and API versions that are important to us; if any test fails on any of those devices, Spoon will return an error code and the build will fail.

For library projects, you only have a single APK containing both the tests and the library itself. The current version of Spoon requires both --apk and --test-apk to be specified, so we simply pass in the same APK to both. It looks like future versions of Spoon will be more flexible in this regard.

Spoon produces HTML reports, showing the status of each test run on each device. We have the report output folder collected as a build artefact, so the reports can be seen right from the build server:

Spoon output summary, showing results of 171 tests run on 3 devices

Flake-free UI testing with Espresso

Espresso is an Android library that provides an API for interacting with and making assertions about the UI of Android applications. Espresso has a very simple interface, and does lots of clever things under the hood to ensure that your tests only execute code when the UI is idle (and hence stable). You shouldn’t ever need to make your code sleep or wait.

In order for Espresso’s magic to work, it needs to know whenever some background activity is going on that it needs to wait for. By default, it hooks in to Android’s AsyncTask executor so it can wait for those to finish. In our apps, there were a few cases where we used an explicit Thread to do some background work, which caused tests to fail intermittently (depending on whether the thread performed its UI update before or after Espresso executed the test code). Rewriting these cases to use an AsyncTask enabled Espresso to figure out what was happening and the tests started passing reliably.

Another, similar, problem occurred where we were using RxJava to load some data. There are two possible ways to deal with this… Espresso has a concept of an idling resource, which provides a way of telling Espresso when a resource is busy so it can belay interacting with or testing the UI until the resource is finished. In our case, the code in question was going to be rewritten soon, so we went for a quicker and dirtier option: force RxJava to use the same executor as AsyncTask.

To do this, we added a simple test utility class that registers an RxJavaPlugin that overrides the Schedulers used by Rx:

/**
 * Hooks in to the RxJava plugins API to force Rx work to be scheduled on the
 * AsyncTask's thread pool executor. This is a quick and dirty hack to make
 * Espresso aware of when Rx is doing work (and wait for it).
 */
public final class RxSchedulerHook {

    private static final RxJavaSchedulersHook javaHook =
            new RxJavaTestSchedulerHook();

    private RxSchedulerHook() {
        // Should not be insantiated
    }

    public static void registerHooksForTesting() {
        if (RxJavaPlugins.getInstance().getSchedulersHook() != javaHook) {
            RxJavaPlugins.getInstance().reset();
            RxJavaPlugins.getInstance().registerSchedulersHook(javaHook);
        }
    }

    private static class RxJavaTestSchedulerHook extends RxJavaSchedulersHook {
        @Override
        public Scheduler getComputationScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getIOScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getNewThreadScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }
    }
}

With the hook registered Rx does all of its work on the same thread pool as AsyncTask, which Espresso already knows about. It’s not the best long-term solution, but it means we don’t have to spend time integrating IdlingResource for code that doesn’t have long to live. With the hook in place, the tests that were flaking because of Rx started passing reliably as well.

Getting automatic screenshots of failures

Spoon provides a client library to, among other things, take a screenshot of the device. Espresso provides a hook that can be used to change how errors are handled. Putting the two together is very simple:

final FailureHandler defaultHandler =
        new DefaultFailureHandler(
                InstrumentationRegistry.getTargetContext());

Espresso.setFailureHandler(new FailureHandler() {
    @Override
    public void handle(Throwable throwable, Matcher<View> matcher) {
        try {
            Spoon.screenshot(
                    getActivity(),
                    "espresso-failure",
                    description.getClassName(),
                    description.getMethodName());
        } catch (Exception ex) {
            Log.e(TAG, "Error capturing screenshot", ex);
        }
        defaultHandler.handle(throwable, matcher);
    }
});

In our new error handler we simply ask Spoon to take a screenshot, then call Espresso’s original handler so that it can output its debugging information and fail the test. The Spoon runner automatically picks up the screenshot and adds it to the report:

Spoon output details, showing a screenshot captured of the failure

Having the screenshot, error message and logs all presented in a clean UI makes debugging failures much, much easier than searching through a huge build log to try and find the exception.

Filtering tests based on device capabilities

Some tests won’t work on every device you throw at them. We had two problems: some tests require a higher API version than some of our devices, and some of the UIs under test were designed to run only on certain resolutions.

The main cause for our dependence on newer API versions was the use of WireMock, a brilliant library for stubbing out web services. WireMock requires API 19, while our physical devices tend to run versions older than that. Stopping these tests running is simply a case of applying an annotation to the class:

@SdkSuppress(minSdkVersion=19)

Screen resolution is a bit more complicated. One of our apps is designed for a specific tablet device (and will never be used on anything else), and trying to render the UI on smaller screens results in items overlapping, important parts ending up below the fold, and other problems.

We’d still like all the other tests for that app to run on all of the devices, though, so we can test them on a variety of API versions and in other conditions. We just need to suppress the UI tests. To do this, we subclassed the Android ActivityTestRule and overrode the apply method:

@Override
public Statement apply(Statement base, final Description description) {
    if (!canRunUiTests(InstrumentationRegistry.getContext())) {
        // If we can't run UI tests, then return a statement that does nothing
        // at all.  With normal JUnit tests we'd just throw an assumption
        // failure and the test would be ignored, but that makes the Android
        // runner angry.
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                // Do nothing
            }
        };
    }

    return super.apply(base, description);
}

/**
 * Checks that the screen size is close enough to that of our tablet device.
 */
@TargetApi(Build.VERSION_CODES.HONEYCOMB_MR2)
private boolean canRunUiTests(Context context) {
    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.HONEYCOMB_MR2) {
        return false;
    }
    int screenWidth = dpToPx(context,
            context.getResources().getConfiguration().screenWidthDp);
    return screenWidth >= 600;
}

private int dpToPx(Context context, int dp) {
    DisplayMetrics displayMetrics = context.getResources().getDisplayMetrics();
    return Math.round(dp * displayMetrics.density);
}

When the rule is run, we check if the screen width meets a minimum number of pixels. If it doesn’t, an empty Statement is returned that turns the test into a no-op. These show up as passed rather than skipped in the output, but there doesn’t seem to be a nice way to signal to the Android JUnit runner that the test is being ignored programatically.

Automatic reverse proxying with Docker and nginx

2016-05-21T00:00:00Z

The Docker project logo

Over the past few weeks I’ve gradually been migrating services from running in LXC containers to Docker containers. It takes a while to get into the right mindset for Docker - thinking of containers as basically immutable - especially when you’re coming from a background of running things without containers, or in “full” VM-like containers. Once you’ve got your head around that, though, it opens up a lot of opportunities: Docker doesn’t just provide a container platform, it turns software into discrete units with a defined interface.

With all of your software suddenly having a common interface, it becomes trivial to automate a lot of things that would be tedious or complicated otherwise. You don’t need to manage port forwards because the containers just declare their ports, for example. You can also apply labels to the application containers, and then query the labels through Docker’s API.

Reverse proxying and SSL termination with Nginx and Let’s Encrypt

A fairly significant chunk of the software I run has a web interface. I don’t really want to expose and remember dozens of non-standard ports, so I configure an nginx instance as a reverse proxy. I’m of the opinion that all web traffic should be encrypted, so I also have to provide nginx with trusted certificates to use for each site it reverse proxies. Let’s Encrypt makes the process of obtaining free, trusted certificates approximately a thousand times easier than it was previously, but my workflow still ends up looking like this:

Create a new config file from a template and save it in /etc/nginx/sites-available
Temporarily disable SSL for the site as there’s no valid certificate yet
Enable the site by symlinking to it from /etc/nginx/sites-enabled
Reload nginx
Run the Let’s Encrypt client to obtain certificates
Enable SSL and for the site
Reload nginx

… And that’s not including the extra steps when I miss a semi-colon, accidentally skip a step and have to spend time figuring out why it’s not working, or any of the other human-induced problems that creep in.

I’d been toying with making a script to run through these steps manually for me, if I gave it a domain name and a reverse proxy target, but I never got around to it. Now I’m moving things to Docker, though, there’s an opportunity to automate the entire thing with no human interaction at all.

Existing solutions

It seemed like this probably wasn’t a unique idea, so I had a look around for existing solutions. The most popular by far seems to be nginx-proxy by Jason Wilder. This is based on his docker-gen project that takes a template and populates values from docker containers.

It’s a good solution, but there were a few bits I didn’t like. Firstly, templates don’t really lend themselves well to every step of the process: to request Let’s Encrypt certificates, the container uses a template to create a shell script which it then sources. Each container that generates a template also needs access to the Docker socket. Both of those cause an itch in the back of my head and make me want to say phrases like “attack surface”. I don’t think there’s actually a problem, but it doesn’t really sit well with me.

Secondly, the whole system seems slightly too tightly coupled for my liking. The Let’s Encrypt component needs to modify the nginx config in order to obtain the certificate, while the main nginx component is also making different changes to add and remove sites. It feels like if it doesn’t just work, it’s going to be difficult to debug and pry apart the different components.

Another potential solution is Rancher. This is a complete platform for managing containers, and I’m fairly sure if configured right it can grab certificates from Let’s Encrypt and do SSL termination using haproxy. I tried it for a bit but the whole platform seemed a bit overkill for my purposes, and I didn’t want to invest the time I’d need to fully understand it all.

Rolling my own

In the end I decided to roll my own solution. Here’s a high-level overview of how it all works:

Diagram showing components of a reverse proxy implementation

As you probably noticed, there are quite a few containers involved. Each one performs a small, well-defined task, and its output can easily be inspected in either a volume or a database. I think there’s some similarity to piping commands together on a command line — it’s a lot easier to reason about simpler commands like head, cut and tr than it would be one giant command that combined them. And, if it does go wrong, you can inspect the pipe at each stage to see where the problem is happening.

service-reporter and etcd

The first part of the chain is my service-reporter container. This uses the Docker API to get a list of containers, and store information about them in etcd. Etcd is a distributed key-value store (similar in some ways to redis or memcached). The container also watches for containers that are added and removed, and keeps etcd updated appropriately.

As the service metadata is stored in a database, no other part of the system needs to interact with Docker. If the Docker API changes, or the host configuration changes, then only this container has to be updated.

service-letsencrypt and letsencrypt-lexicon

The left fork of the diagram deals with obtaining SSL certificates. To keep it separate from the nginx configuration, it uses DNS-based challenge to prove that we control the domains. It does this by plumbing together two great open source projects: letsencrypt.sh, a Let’s Encrypt client implemented in bash with support for the dns-01 challenge type, and Lexicon, a python library for updating DNS records using a variety of providers.

My service-letsencrypt container connects to etcd and pulls a list of containers that have a label with the key com.chameth.vhost. It uses this to build a plain text list of certificates we require (in a format understood by letsencrypt.sh), and then monitors etcd for changes and repeats as necessary.

The letsencrypt-lexicon container runs letsencrypt.sh, using Lexicon to perform the required DNS updates, and produces certificates. The nice thing about this is that it can be used in a completely standalone fashion (you can just write a domains.txt yourself). It uses iowait to watch the domains text file for updates, and automatically reruns when there are changes. It also runs once a day to renew any certs that are coming up for expiry.

service-nginx and nginx

The right fork of the diagram is concerned with nginx. My service-nginx container again connects to etcd and pulls a list of containers. It uses a couple of labels to determine the vhost, proxy port, and proxy protocol. It then feeds these values into a template to create a server block for each site, configured with SSL certificates and a reverse proxy setup. The template covers only the very minimal settings, with the expectation that everything else will be done in the global config (things such as SSL ciphers, redirection from HTTP, etc).

This container works completely independently of the Let’s Encrypt side. You can use the Let’s Encrypt containers and mount the certificate volume, or you could just provide your own certificates. It doesn’t really make any difference.

Putting it all together

The only downside to having many small containers is that it’s a bit of a nuisance to get them all set up. Fortunately, Docker has a solution for this in the form of Docker compose. This allows you to write a YAML file defining all of the services you want to run, and bring them up or down in one go. It can handle volumes, dependencies, networking, etc. I’ll be publishing a docker-compose.yml file to get this entire stack up and running soon.