Chameth.com - posts like ipv6-docker-routing, monitoring-power-with-wemo but not debugging-beyond-the-debugger, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Home Automation Without the Megacorps

2025-05-21T00:00:00Z

I first experimented with home automation in 2016, by picking up a Samsung “SmartThings” hub. It was terrible. The UI to configure things was slow and clunky, firmware updates were applied whether you wanted them or not, and everything stopped working if their cloud services stopped. You were also locked into whatever integrations they deigned to support, of course. After that broke for the umpteenth time I scaled back and for years the closest I got to home automation was a couple of Hue bulbs.

Recently I’ve been building it out again, though. This time using off-the-shelf components that interop using Zigbee, open-source software, and some code I wrote myself. It’s great; it runs entirely locally and has had basically zero downtime. The Zigbee ecosystem lets me integrate all sorts of things without having to spend lots of money on “smart” alternatives. I think I’ve spent less on this incarnation than I did on the original SmartThings hub all those years ago (even without adjusting for inflation!).

My current setup

I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based on the CC2652P chipset¹. Interfacing with the Zigbee stack is handled by zigbee2mqtt (z2m for short), an open-source project that basically bridges your devices to an MQTT broker². When a device reports some data, it will send a new message over MQTT; when you want to make a device do something you just post a message back. It’s incredibly lightweight, but supports a huge array of devices out of the box. And as it’s just using MQTT, it’s trivial to integrate with other software or build on top of.

I know a lot of people building out this kind of home automation use Home Assistant, but I don’t get on with it terribly well. It felt incredibly sluggish, and the entire project is just a bit… much. I want a binary or package or docker image I can just run, not an entire operating system. That’s not how computers are meant to work! In contrast, z2m is simple to set up, super light weight and responsive.

Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond this now, and I’m skipping some boring bits like error handling and JSON parsing, but at first I had something like:

for {
    message, topic, err := c.ReadSlices()
    
    if string(topic) == "zigbee2mqtt/desk-button" {
        if strings.Contains(message, "single") {
            c.Publish(nil, []byte("{\"state\": \"TOGGLE\"}"), "zigbee2mqtt/room-lights/set")
        }
    }
}

I think I actually made this harder to read by trying to simplify it here, but hopefully you can follow that it was relatively straight forward to listen for a particular action to happen (in this case me “single”-pressing on the device called “desk-button”) and then make another device do something in response (toggling the state of the “room-lights” device).

So what actually are these devices? At present in z2m I have the following:

Some generic buttons
An air quality sensor (USB powered), and a separate temperature sensor (battery powered)
A light switch
Some 240V relay modules, that I use to turn non-smart devices on and off
Some USB relay modules, for the same purpose
Some “smart” plugs that I mostly use to monitor power usage
A motorised blind roller

I think all of these are either Tuya or Moes branded, and were all sourced from AliExpress. You can get the same things from Amazon or elsewhere, but they tend to be 3-4x more expensive for the same product.

One of the cool things about Zigbee devices is that the powered ones work together to create a mesh network, so you don’t have to worry about network repeaters or signal strength like you do with Wi-Fi networks. z2m even makes a map showing the connections:

The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.

Some interesting automations

A Zigbee relay spliced into the power cable for the fan

So what do I actually do with all of this? Some of it is just what I’ll call “laziness automation”: I want to be able to turn on the lights without getting up and walking to the light switch. So there’s a Zigbee button on my desk that does it. Then there are some less manual automations: my blinds are automatically closed at sunset³, for example. It feels a bit trivial, but it’s surprisingly nice just not to have to think about that at all.

Another nice quality of life automation is the light I have above my monitor. It’s USB powered, and I now have it going through a Zigbee USB relay that lets it be toggled on and off. Then I have a small agent running on my desktop that turns the light on when the computer is unlocked, and off when it locks or shuts down. I keep meaning to make a “film mode” that detects when I’m watching a film, and turning all the lights and blinds down, but I haven’t got around to it yet.

The most complex automation is probably for a window fan. It’s not smart in any way, so I cut the power cable and inserted a Zigbee relay. The whole thing is housed in a little project box to keep it secure. The relay basically acts as a switch: the live wire leading to the fan runs to the “normally open”⁴ contact, while the upstream power goes to “common”. When the relay is told to turn the fan on, the two get connected and the fan gets power. Oh, it’s also not actually a window fan, but I designed and 3D printed some adapters so that it sits right in the window.

So now we have a window fan that can be turned off and on automatically. But I don’t want to have to press a button to do that. I’m lazy, remember? Instead, I made it so that the Go code constantly monitors the temperature reported by the air quality sensor, and queries the free OpenWeatherMap API to get the rough temperature outside. It can then turn the fan on if the room is too hot, and outside is cool enough to make a difference (there’s no point in blowing hotter air in!).

A graph of room temperature, showing the effect of the fan being turned on and off

I don’t want it to do that all the time though. The room doesn’t need to be cooled if I’m not around. There are various Zigbee presence sensors you can get, but the cheaper ones seem to be of dubious quality. Instead, I wrote some code to guess whether I’m present. It uses the state of the monitor light as a proxy for “is the computer in use”, and then makes some guesses based on the last time the computer was used and the current time (if I turn the computer off at midnight it probably means I’m going to sleep; if I turn it off at 2pm it probably means I’m going out somewhere).

You can see in the graph that the fan has a decent impact on temperature. I coded it to cool the room to 19 degrees, but then not turn on again until it was above 20. This prevents it flip-flopping on and off constantly. You can clearly see the pattern in the graph, as it actively cools and then slowly the room warms back up. This arrangement is much better than being woken up at 5am because it’s painfully cold. Trust me.

Bonuses: metrics and 3D printer

One of the things my custom Go code does is collate all the various stats reported by the Zigbee devices, and inserts them into a VictoriaMetrics database. I originally hosted this on the Pi itself, and it performed fine, but I’ve since moved it onto a server so that I can use it for some other things as well.

I set up Grafana to point to VM, and can create dashboards showing power usage, what devices are turned on when, and a bunch of environmental conditions. This also makes it easy to spot how good the data coming from the devices are. For example, the air quality sensor reports a figure for the amount of Carbon Dioxide in the air, as well as the amount of Volatile Organic Compounds (VOCs). The graphs are basically identical, but on a different scale. It turns out this particular device has no actual way of detecting CO₂, so it just “calculates” it from the VOCs figure. Useful to know if you want to actually use that data.

Recently I came across a go library for interacting with Bambu Lab printers, so I’ve also hooked that into my automation. It exports metrics about the printer, so I now have a way of seeing what’s going on when I’m not physically present⁵. I even added a HTTP endpoint (exposed over Tailscale) that shows pictures from the built-in camera.

On the automation front, I’ve made it automatically turn off the printer’s light when it finishes a print, and also added push notifications via PushOver whenever the state changes. No longer will I be sat in another room blissfully unaware it ran out of filament seconds after I walked out the door!

Was it worth it?

These days you could probably just buy some kind of ‘smart home’ hub that works well enough, and do most of what I’ve done with a lot less effort and no coding. I’m still of the opinion that for something so essentially local, it should itself be managed entirely locally. I don’t trust companies like Google or Amazon not to kill their products, or change or remove an API I rely on.

Obviously the “Not Invented Here” approach of coding everything yourself doesn’t suit everyone, but as someone who enjoys coding and enjoys having things work just so it works very well for me.

It turns out the chipset is important. I initially used a similar Sonoff dongle that used a CC2531 chipset, and its performance wasn’t great. It often failed to pair with new devices, and dropped links to existing ones. The zigbee2mqtt docs do explicitly advise against the CC2531 chips for that reason. At the time they recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d go with whatever their latest recommendation was. ↩︎
MQTT is basically a network protocol for publishing and subscribing to arbitrary messages. A broker is the thing that sits in the middle and routes the messages. I use Mosquitto but any will do. ↩︎
They originally also re-opened at sunrise but that was a terrible mistake. Who knew the sun rose so early?! ↩︎
“open” meaning “there’s a gap so it doesn’t work” not “open for business”. This confusion in terminology also extends to drawbridges. ↩︎
I use the printer in LAN mode, which means there’s no way to monitor it from a phone, even if you’re connected to the same network still. ↩︎

Adventures in IPv6 routing in Docker

2022-10-24T00:00:00Z

One of the biggest flaws in Docker’s design is that it wasn’t created with IPv6 in mind. Out of the box Docker assigns each container a private IPv4 address, and they won’t be able to reach IPv6-only services. While incoming connections might work, the containers won’t know the correct remote IP address which can cause problems for some applications. This situation is obviously suboptimal in the current day and age. It’s a bit like not supporting HTTPS on a website — you might not have any issues because of it immediately, but you’re fighting against the currents of progress and are making life worse for your users.

Thankfully, it’s now relatively easy to make Docker behave a lot nicer. The docker-ipv6nat project has been around since 2016, and uses an IPv6 overlay network and some iptables magic to route traffic to and from containers in a sensible fashion. It uses NAT to emulate the behaviour Docker employs for IPv4 traffic; while using NAT with IPv6 is an anathema, I think it makes sense for containers. You could give each container a publicly routable IPv6 address, but that brings with it a lot of headaches: you’re basically going to be forced to implement service discovery and some kind of DNS management to deal with the fact that your containers will be popping up on randomly assigned IP addresses. That is completely overkill for people running a small number of services on one or two physical boxes; and if it’s not overkill for you then you’re probably already looking at more complicated orchestration solutions like Kubernetes.

More recently, similar functionality has been built into the Docker daemon itself. You can now edit the config file to enable ipv6 and each container will be assigned an address in the range specified when it uses the default bridge network. This gives more-or-less the same functionality of docker-ipv6nat — you lose a little flexibility as you can’t disable IPv6 on the default bridge, but that’s a very worthy trade for having the functionality built-in.

So far this all seems very simple. Hardly worthy of being called an “adventure”. Enter stage left: the wicked witch of destination address selection…

Destination address selection and you

When a computer program tries to connect to an address such as google.com, it first resolves it in to an IP address. That’s DNS 101, but what happens if the address resolves to multiple IP addresses? For example, google.com resolves to both 142.250.74.206 and 2a00:1450:4001:82b::200e. You might just assume there’s a simple “prefer IPv4 / prefer IPv6” toggle somewhere that decides, but it’s actually a lot more complicated. With IPv6, devices are likely to have many addresses - a link-local address, a unique local address, a normal public address, various privacy addresses, and so on. To deal with this, a number of RFCs define a series of rules that most implementations follow. These are called the destination address selection rules. RFC6724 gives the rules as:

Avoid unusable destinations
Prefer matching scope
Avoid deprecated addresses
Prefer home addresses
Prefer matching label
Prefer higher precedence
Prefer native transport
Prefer smaller scope
Use the longest matching prefix
Otherwise, leave the order unchanged

Applying these rules will re-order the list of IP addresses such that (in theory) the most likely one to work will be first. Most of them are fairly niche rules; the ones that do the heavy lifting are rules 5 and 6, which rely on a policy table to make their decisions. The policy table ‘SHOULD’ be configurable by system administrators to allow them to tweak how traffic is routed. In the absence of an admin-provided policy table, the RFC gives the following defaults:

Prefix	Precedence	Label	Notes
`::1/128`	50	0	Loopback address
`::/0`	40	1	Any IPv6 address
`::ffff:0:0/96`	35	4	IPv4 addresses mapped as v6 addresses
`2002::/16`	30	2	6-to-4 gateways
`2001::/32`	5	5	Toredo tunnels
`fc00::/7`	3	13	Unique Local Addresses (ULAs)
`::/96`	1	3	IPv4 compatible addresses (deprecated)
`fec0::/10`	1	11	Site-local addresses (deprecated)
`3ffe::/16`	1	12	6bone (deprecated)

The policy table is a bit complex, but you can see that normal IPv6 addresses are preferred (have a higher precedence) over IPv4 addresses, which are preferred over the various tunnels, local addresses, and various deprecated ranges.

We can see the result of this when I run ping google.com on a box that has a native IPv6 connection as well as a native IPv4 connection:

PING google.com(fra07s29-in-x200e.1e100.net (2a00:1450:4001:802::200e)) 56 data bytes
64 bytes from fra24s01-in-x0e.1e100.net (2a00:1450:4001:802::200e): icmp_seq=1 ttl=119 time=5.11 ms
64 bytes from fra24s01-in-x0e.1e100.net (2a00:1450:4001:802::200e): icmp_seq=2 ttl=119 time=5.28 ms
64 bytes from fra07s29-in-x200e.1e100.net (2a00:1450:4001:802::200e): icmp_seq=3 ttl=119 time=5.31 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 5.114/5.235/5.311/0.086 ms

The address selection in this instance has been a result of rule 6: the IPv6 address has a higher precedence (40) than the IPv4 address (35).

However, if I run the same command in an IPv6-enabled Ubuntu container then it seems to prefer the IPv4 address:

PING google.com (172.217.16.206) 56(84) bytes of data.
64 bytes from fra16s08-in-f206.1e100.net (172.217.16.206): icmp_seq=1 ttl=59 time=4.86 ms
64 bytes from fra16s08-in-f14.1e100.net (172.217.16.206): icmp_seq=2 ttl=59 time=4.86 ms
64 bytes from fra16s08-in-f14.1e100.net (172.217.16.206): icmp_seq=3 ttl=59 time=5.03 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.860/4.916/5.027/0.078 ms

If I force ping to use IPv6 with the -6 flag then it works the same as on the host, but when left to its own devices it prefers IPv4. What’s going on there? The key difference between the Docker container and the host is that the host’s network interfaces have public addresses, but the container has a private IPv4 address (172.19.0.7) and a private IPv6 address (fd00:dead:beef::7). My first reaction to this was to think “Ah yes, fc00::/7 has a lower precedence than native IPv4, that makes sense”, but that’s not quite right. These are the destination address selection rules; rule 6 doesn’t care about the source addresses. This is actually rule 5 at work: the label of source IPv6 address is 13, but the label of the destination address is 1; meanwhile both the source and destination IPv4 addresses are 4. This sorts the IPv4 address before the IPv6 one, and rule 6 becomes irrelevant.

Adjusting the policy table

Thankfully, the RFC says the policy table should be configurable by system administrators, so those of us who are doing unorthodox things like NAT’ing IPv6 can customise the behaviour to fit our weird environments. The configuration is done via the /etc/gai.conf file (‘gai’ standing for getaddrinfo, the function in the standard library responsible for dealing with all these rules). The gai.conf file by default (if it exists) will likely just contain comments and examples; if there are no uncommented “label” or “precedence” lines then the library will use its built-in defaults based on the RFC requirements.

To make our container happy, we need to cause fd00::/8 to have the same label as a public IPv6 address. To do this we can uncomment the default labels in gai.conf and then add a single extra line, like so:

 label ::1/128       0
 label ::/0          1
 label 2002::/16     2
 label ::/96         3
 label ::ffff:0:0/96 4
 label fec0::/10     5
 label fc00::/7      6
 label 2001:0::/32   7
+label fd00::/8      1

Because fd00::/8 has a longer prefix than fc00::/7 it will match our addresses and give them a label of 1, the same as a public IPv6 address would get. This makes rule 5 leave it alone, and the default precedence table used by the standard library will put the IPv6 address above the IPv4 address. (They don’t bother giving fc00::/7 a separate precedence as listed in the RFC because rule 5 would have already de-prioritised those addresses, as we discovered.)

Making this small change to the gai.conf file in our Ubuntu container makes it start preferring the IPv6 address for google.com:

PING google.com(fra16s65-in-x0e.1e100.net (2a00:1450:4001:806::200e)) 56 data bytes
64 bytes from fra15s29-in-x0e.1e100.net (2a00:1450:4001:806::200e): icmp_seq=1 ttl=118 time=5.00 ms
64 bytes from fra15s29-in-x0e.1e100.net (2a00:1450:4001:806::200e): icmp_seq=2 ttl=118 time=5.10 ms
64 bytes from fra16s65-in-x0e.1e100.net (2a00:1450:4001:806::200e): icmp_seq=3 ttl=118 time=5.08 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 5.002/5.058/5.098/0.040 ms

A fairly complicated problem, but a simple fix!

But wait, there’s more…

Unfortunately, this fix won’t work on containers that use alpine. And that is a lot of containers. Alpine uses musl as its standard library, rather than the much more common glibc. There is very little documentation on the subject, but if you browse the source code for musl you will see that it doesn’t implement any way at all to configure the policy tables. In network/lookup_name.c we can see the sorting logic:

	/* The following implements a subset of RFC 3484/6724 destination
	 * address selection by generating a single 31-bit sort key for
	 * each address. Rules 3, 4, and 7 are omitted for having
	 * excessive runtime and code size cost and dubious benefit.
	 * So far the label/precedence table cannot be customized. */
	for (i=0; i<cnt; i++) {
		int family = buf[i].family;
		int key = 0;
		struct sockaddr_in6 sa6 = { 0 }, da6 = {
			.sin6_family = AF_INET6,
			.sin6_scope_id = buf[i].scopeid,
			.sin6_port = 65535
		};
		struct sockaddr_in sa4 = { 0 }, da4 = {
			.sin_family = AF_INET,
			.sin_port = 65535
		};
		void *sa, *da;
		socklen_t salen, dalen;
		if (family == AF_INET6) {
			memcpy(da6.sin6_addr.s6_addr, buf[i].addr, 16);
			da = &da6; dalen = sizeof da6;
			sa = &sa6; salen = sizeof sa6;
		} else {
			memcpy(sa6.sin6_addr.s6_addr,
				"\0\0\0\0\0\0\0\0\0\0\xff\xff", 12);
			memcpy(da6.sin6_addr.s6_addr+12, buf[i].addr, 4);
			memcpy(da6.sin6_addr.s6_addr,
				"\0\0\0\0\0\0\0\0\0\0\xff\xff", 12);
			memcpy(da6.sin6_addr.s6_addr+12, buf[i].addr, 4);
			memcpy(&da4.sin_addr, buf[i].addr, 4);
			da = &da4; dalen = sizeof da4;
			sa = &sa4; salen = sizeof sa4;
		}
		const struct policy *dpolicy = policyof(&da6.sin6_addr);
		int dscope = scopeof(&da6.sin6_addr);
		int dlabel = dpolicy->label;
		int dprec = dpolicy->prec;
		int prefixlen = 0;
		int fd = socket(family, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_UDP);
		if (fd >= 0) {
			if (!connect(fd, da, dalen)) {
				key |= DAS_USABLE;
				if (!getsockname(fd, sa, &salen)) {
					if (family == AF_INET) memcpy(
						sa6.sin6_addr.s6_addr+12,
						&sa4.sin_addr, 4);
					if (dscope == scopeof(&sa6.sin6_addr))
						key |= DAS_MATCHINGSCOPE;
					if (dlabel == labelof(&sa6.sin6_addr))
						key |= DAS_MATCHINGLABEL;
					prefixlen = prefixmatch(&sa6.sin6_addr,
						&da6.sin6_addr);
				}
			}
			close(fd);
		}
		key |= dprec << DAS_PREC_SHIFT;
		key |= (15-dscope) << DAS_SCOPE_SHIFT;
		key |= prefixlen << DAS_PREFIX_SHIFT;
		key |= (MAXADDRS-i) << DAS_ORDER_SHIFT;
		buf[i].sortkey = key;
	}
	qsort(buf, cnt, sizeof *buf, addrcmp);

We can see if the address labels match, the sort key is adjusted using the DAS_MATCHINGLABEL constant. But where do the labels come from? For that we need to investigate the policyof func:

static const struct policy {
	unsigned char addr[16];
	unsigned char len, mask;
	unsigned char prec, label;
} defpolicy[] = {
	{ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 15, 0xff, 50, 0 },
	{ "\0\0\0\0\0\0\0\0\0\0\xff\xff", 11, 0xff, 35, 4 },
	{ "\x20\2", 1, 0xff, 30, 2 },
	{ "\x20\1", 3, 0xff, 5, 5 },
	{ "\xfc", 0, 0xfe, 3, 13 },
#if 0
	/* These are deprecated and/or returned to the address
	 * pool, so despite the RFC, treating them as special
	 * is probably wrong. */
	{ "", 11, 0xff, 1, 3 },
	{ "\xfe\xc0", 1, 0xc0, 1, 11 },
	{ "\x3f\xfe", 1, 0xff, 1, 12 },
#endif
	/* Last rule must match all addresses to stop loop. */
	{ "", 0, 0, 40, 1 },
};

static const struct policy *policyof(const struct in6_addr *a)
{
	int i;
	for (i=0; ; i++) {
		if (memcmp(a->s6_addr, defpolicy[i].addr, defpolicy[i].len))
			continue;
		if ((a->s6_addr[defpolicy[i].len] & defpolicy[i].mask)
		    != defpolicy[i].addr[defpolicy[i].len])
			continue;
		return defpolicy+i;
	}
}

So the policyof func simply uses the defpolicy array to find the policy that applies to an address. This defpolicy array contains a number of hardcoded entries which correspond exactly to the ones provided in the RFC. The one causing us problems is { "\xfc", 0, 0xfe, 3, 13 } which matches fc00::/7 addresses.

Unfortunately, without recompiling musl from source there’s not much we can do to address this directly. The only way we can make the containers work as desired is to give them a different address range. Ideally this would be a range that you control and that is otherwise not used, but there’s a good chance you won’t have such a range. One possible alternative is the 2001:db8::/32 range which is reserved for documentation purposes. This doesn’t feature in any of the policy tables, so is treated like a normal public IPv6 address. It doesn’t feel great to misuse a reserved range like that, but it’s probably the least of all evils, at least until musl allows configuring the policy table.

Monitoring power draw with WeMo Insight Switches

2016-05-02T00:00:00Z

A WeMo Insight Switch

I recently picked up a couple of Belkin’s WeMo Insight Switches to monitor power usage for my PC and networking equipment. WeMo is Belkin’s home automation brand, and the switches allow you to toggle power on and off with an app, and monitor power usage.

The WeMo Android app is pretty dismal. It’s slow, doesn’t look great, and crashed about a dozen times during the setup process for each of my two switches. It also doesn’t provide much information at all about power: you can see average power draw and current power draw, and that’s basically it.

Belkin has provided an option to e-mail yourself a spreadsheet with historical power data, and can even do it on a regularly scheduled basis, but that’s not really a nice solution if you want up-to-date power stats. Even if you were happy with data arriving in batch, having to get hold of an e-mail attachment and parse out a weirdly formatted spreadsheet doesn’t make for easy automation. It also relies on Belkin supporting the service indefinitely, which isn’t necessarily going to happen.

Enter the SOAP API

Fortunately, the devices themselves expose an API to get the data. Once a switch is setup with the WeMo app, it connects to a WiFi network. You can discover WeMo devices on the network using a UPnP broadcast, and then from there a list of supported services. Each service has a number of SOAP actions, with nice obvious names, and parameters documented in the XML service description.

For the insight switches, there’s a service at /upnp/control/insight1 which has a number of actions including GetPower and GetTodayKWH. Sending a SOAP request isn’t too difficult (albeit nowhere near as nice as a REST JSON API) — here’s a sample request I made using the DHC chrome extension:

POST /upnp/control/insight1 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.8
Content-Type: text/xml
Origin: chrome-extension://aejoelaoggembcahagimdiliamlcdmfm
SOAPACTION: "urn:Belkin:service:insight:1#GetPower"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.29 Safari/537.36

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
    </s:Body>
</s:Envelope>

The name of the action (in the SOAPACTION header) and the XML namespace in the body are both constructed from information in the service definition. The result that comes back is:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPowerResponse xmlns:u="urn:Belkin:service:insight:1">
            <InstantPower>92170</InstantPower>
        </u:GetPowerResponse>
    </s:Body>
</s:Envelope>

The current power draw in milliwatts is returned in the <InstantPower> argument.

Putting the data to work

So now we have a way to pull the current power draw, a nice thing to do would be to plot a graph of it to see the variation over time. This is the short of thing you’d expect to see in the Belkin app, but it’s sadly missing. After a bit of research I settled on the tried and true rrdtool to store data and generate graphs. I created a new database to store the values from the two switches:

rrdtool create power.rrd \
    --start now \
    --step 60 \
    DS:wemoComputer:GAUGE:120:U:U \
    DS:wemoNetworking:GAUGE:120:U:U \
    RRA:AVERAGE:0.5:1:1440 \
    RRA:AVERAGE:0.5:10:1008 \
    RRA:AVERAGE:0.5:30:1488 \
    RRA:AVERAGE:0.5:120:1488 \
    RRA:AVERAGE:0.5:360:1488 \
    RRA:AVERAGE:0.5:1440:36500

This creates a database file which expects values to be given every 60 seconds for two data series: ‘wemoComputer’ and ‘wemoNetworking’. These are gauge types (i.e., a value we read off a gauge, rather than a counter that keeps going up) with no minimum and maximum (‘U’).

It then defines a series of round-robin archives. Each of these stores a fixed number of entries, and the oldest is overwritten when they become full. The interesting arguments are the step count (second to last) and number of samples (last argument). The first one takes every sample and keeps 1,440 of them (i.e., a full day at 1-minute resolution); the last one has a step of 1,440 and keeps 36500 of them (i.e., 100 years at 1-day resolution). The use of RRAs ensures that rrdtool can retain enough data to create historical graphs, while providing a fixed, finite maximum file size. With sufficient RRA coverage you can graph most timescales and have data available at a reasonable resolution.

Next, I created a small python script to retrieve the power using SOAP:

#!/usr/bin/python3

import requests
from xml.etree import ElementTree

def get_power(ip_and_port):
    headers = {'Content-type': 'text/xml', 'SOAPACTION': '"urn:Belkin:service:insight:1#GetPower"'}
    payload = '''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
                             s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
                  <s:Body>
                   <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
                  </s:Body>
                 </s:Envelope>'''
    r = requests.post("http://%s/upnp/control/insight1" % ip_and_port, headers=headers, data=payload)
    et = ElementTree.fromstring(r.text)
    return et.find('.//InstantPower').text

I then have a dictionary of IP addresses to data series names, and the script polls each one in turn and then executes an rrdtool update query to add the items to the database. I have this script running every minute via cron.

After leaving the script to run for a bit and gather data, it’s time to make some graphs. I use the following to create a graph with a background gradient:

rrdtool graph desk-1d.png \
        -o -X0 -w800 -h500 \
        -u 2000 -l 20 -r \
        DEF:raw=power.rrd:wemoComputer:AVERAGE \
        CDEF:power=raw,1000,/ \
        CDEF:powerz=power,2000,LT,power,2000,IF CDEF:powerzNoUnk=power,UN,0,powerz,IF AREA:powerzNoUnk#ff0000 \
        CDEF:powery=power,1500,LT,power,1500,IF CDEF:poweryNoUnk=power,UN,0,powery,IF AREA:poweryNoUnk#ff0000 \
        CDEF:powerx=power,1000,LT,power,1000,IF CDEF:powerxNoUnk=power,UN,0,powerx,IF AREA:powerxNoUnk#ff0000 \
        CDEF:powerw=power,900,LT,power,900,IF CDEF:powerwNoUnk=power,UN,0,powerw,IF AREA:powerwNoUnk#ff0000 \
        CDEF:powerv=power,800,LT,power,800,IF CDEF:powervNoUnk=power,UN,0,powerv,IF AREA:powervNoUnk#ff1b00 \
        CDEF:poweru=power,700,LT,power,700,IF CDEF:poweruNoUnk=power,UN,0,poweru,IF AREA:poweruNoUnk#ff4100 \
        CDEF:powert=power,600,LT,power,600,IF CDEF:powertNoUnk=power,UN,0,powert,IF AREA:powertNoUnk#ff6600 \
        CDEF:powers=power,400,LT,power,400,IF CDEF:powersNoUnk=power,UN,0,powers,IF AREA:powersNoUnk#ff8e00 \
        CDEF:powerr=power,200,LT,power,200,IF CDEF:powerrNoUnk=power,UN,0,powerr,IF AREA:powerrNoUnk#ffb500 \
        CDEF:powerq=power,180,LT,power,180,IF CDEF:powerqNoUnk=power,UN,0,powerq,IF AREA:powerqNoUnk#ffdb00 \
        CDEF:powerp=power,160,LT,power,160,IF CDEF:powerpNoUnk=power,UN,0,powerp,IF AREA:powerpNoUnk#fdff00 \
        CDEF:powero=power,140,LT,power,140,IF CDEF:poweroNoUnk=power,UN,0,powero,IF AREA:poweroNoUnk#d7ff00 \
        CDEF:powern=power,120,LT,power,120,IF CDEF:powernNoUnk=power,UN,0,powern,IF AREA:powernNoUnk#b0ff00 \
        CDEF:powerm=power,100,LT,power,100,IF CDEF:powermNoUnk=power,UN,0,powerm,IF AREA:powermNoUnk#8aff00 \
        CDEF:powerl=power,90,LT,power,90,IF CDEF:powerlNoUnk=power,UN,0,powerl,IF AREA:powerlNoUnk#65ff00 \
        CDEF:powerk=power,80,LT,power,80,IF CDEF:powerkNoUnk=power,UN,0,powerk,IF AREA:powerkNoUnk#3eff00 \
        CDEF:powerj=power,70,LT,power,70,IF CDEF:powerjNoUnk=power,UN,0,powerj,IF AREA:powerjNoUnk#17ff00 \
        CDEF:poweri=power,60,LT,power,60,IF CDEF:poweriNoUnk=power,UN,0,poweri,IF AREA:poweriNoUnk#00ff10 \
        CDEF:powerh=power,50,LT,power,50,IF CDEF:powerhNoUnk=power,UN,0,powerh,IF AREA:powerhNoUnk#00ff36 \
        CDEF:powerg=power,40,LT,power,40,IF CDEF:powergNoUnk=power,UN,0,powerg,IF AREA:powergNoUnk#00ff5c \
        CDEF:powerf=power,30,LT,power,30,IF CDEF:powerfNoUnk=power,UN,0,powerf,IF AREA:powerfNoUnk#00ff83 \
        CDEF:powere=power,20,LT,power,20,IF CDEF:powereNoUnk=power,UN,0,powere,IF AREA:powereNoUnk#00ffa8 \
        CDEF:powerd=power,0,LT,power,0,IF CDEF:powerdNoUnk=power,UN,0,powerd,IF AREA:powerdNoUnk#00ffd0 \
        LINE:power#080

This seems a bit unweildy, but it’s fairly straight forward. The options tell rrdtool to create a graph with a canvas size of 800x500 pixels, a lower limit of 20W, upper limit of 2kW, and a log scale. The first CDEF divides our raw value (which was in millwatts) by 1000 to get a value in watts. rrdtool uses reverse polish notation like some calculators, so you provide the arguments before the operator.

The big block of CDEF/CDEF/AREA parameters creates a series of area plots in different colours according to the power level. They’re in descending order so the smaller areas are drawn on top of the larger layers. This results in a graph that looks like this:

Graph of power usage over a day

This graph shows the total power for all the things plugged in at my desk. You can see the idle power draw is around 60W. When I’m using the computer it jumps up to around 130W, and when the computer is under heavy load (playing games, for example) it goes up even further to the 200W mark. With a couple of small tweaks to the rrdtool command, I also have a graph showing the entire week:

Graph of power usage over a week

The two huge spikes near the start of the data are caused by a heater under my desk. They’re also one of the main reasons I chose to plot the graphs with a logarithmic scale. With a linear scale between 20W and 2kW, the graph would be completely dominated by these large spikes — in fact the difference in height between the two spikes would take almost half the graph! The relative difference between the normal values of 60-200W and the heater’s value of 1.8kW isn’t actually that interesting, and certainly not worth using about 80% of the graph to demonstrate. The log scale helps to compress this, and emphasises the difference in the smaller values more.

Reverse engineering the Sense API

2016-04-10T00:00:00Z

A Sense unit and its two pillow sensors

Sense is a little device that sits by your bedside and, in conjunction with a little ‘pill’ attached to your pillow, monitors your sleeping patterns and any environmental conditions that might hamper them. Android and iOS apps show you your sleep history, and offer suggestions for improvements.

Sense was Kickstarted in August 2014, raising over 2.4 million US dollars, and shipped to backers in mid 2015. The campaign blurb included this snippet:

Building with Sense

You’ll always have access to your data via our API. Take it, play with it, graph it, do whatever you want with it. It’s yours. That’s important to us.

We enjoy tinkering with and building on-top of other products we like. Sense will let you have that experience.

We’d love to hear your thoughts on what you might want to build with Sense, and how you could directly interact with the hardware, and the data it collects.

Sounds great! But a year after shipping, there’s no sign of an API, and some of us who enjoy tinkering are getting a bit restless…

Reverse engineering to the rescue

Data from the Sense hardware is transmitted off onto the Internet somewhere, and then pulled down by the mobile apps. If we can snoop that traffic, we can probably figure out how to grab the data from the Sense.

So the first step is to capture the raw network traffic while the Sense and the app are in use. I crafted an extremely over-the-top setup that involves my phone and the Sense being on a new, separate wireless network that gateways through a Linux server that can look in on all the traffic (because you never know when a dedicated capturing network will come in handy, right?…)

Running tshark while the app and device are active showed they were making HTTPS requests an Amazon Elastic Cloud instance. That’s not really much use, as we can’t see the content of the requests. The next step is to move up the network stack and target the HTTPS traffic specifically. Step in, mitmproxy. This automates man-in-the-middle attacks on HTTPS requests. When it receives such a request, it cooks up its own certificate and sends that to the client, and then sits in between while the client and the server communicate.

The certificates generated by mitmproxy aren’t issued by a trusted source, so (well behaved) browsers and apps will throw up lots of warnings and refuse to communicate with it for the most part. This is the mechanism that stops anyone impersonating your banking website when you access it over HTTPS, so is generally a very good thing. In this case, both the Sense itself and the Android app refused to talk with the blatant attacker.

We control the Android phone, though, so can just tell it that we trust the dodgy certificate issuer. This involves grabbing the certificate generated by mitmproxy, and adding it to Android’s key store. The process is documented nicely in the mitmproxy docs and is pretty straightforward. With Android trusting our certificate authority, the Sense app starts talking through the proxy and we can look at the plain HTTP requests.

The API

Logging in to the Android app and navigating through the various screens shows that the API is actually pretty nice. It’s RESTful, and uses an OAuth bearer token for authorisation, as you can see in the mitmproxy output of the authentication and first few requests:

POST https://api.hello.is/v1/oauth2/token
    ← 200 application/json 151B 1.17s
GET https://api.hello.is/v2/account/preferences
    ← 200 application/json 124B 109ms
GET https://api.hello.is/v1/account
    ← 200 application/json 181B 388ms
GET https://api.hello.is/v2/devices
    ← 200 application/json 235B 126ms
POST https://api.hello.is/v1/app/checkin
    ← 200 application/json 106B 119ms
GET https://api.hello.is/v2/insights
    ← 200 application/json 1.4kB 157ms
POST https://api.hello.is/v1/notifications/registration
    ← 204 [no content] 364ms
GET https://api.hello.is/v2/timeline/2016-04-09
    ← 200 application/json 160B 550ms
GET https://api.hello.is/v1/questions?date=2016-04-10
    ← 200 application/json 284B 132ms
PATCH https://api.hello.is/v1/app/stats
    ← 202 [no content] 126ms
GET https://api.hello.is/v1/app/stats/unread
    ← 200 application/json 71B 171ms

Authorisation

To get a bearer token, you send a POST request to /v1/oauth2/token, supplying the username and password, and a client ID and secret. Presumably if the API is ever opened up you’ll be able to register clients and get your own ID and secret, but for now reusing the ones from the Android app works fine. [I’ve prettified and linewrapped the content to make it a bit easier to read.]

POST /v1/oauth2/token HTTP/1.1
Host: api.hello.is
Accept: */*
Content-Length: 123
Content-Type: application/x-www-form-urlencoded

grant_type=password
 &client_id=8d3c1664-05ae-47e4-bcdb-477489590aa4
 &client_secret=4f771f6f-5c10-4104-bbc6-3333f5b11bf9
 &username=USERNAME
 &password=PASSWORD

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:25:16 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "token_type":"Bearer",
  "expires_in":31536000,
  "account_id":"1234",
  "access_token":"2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "refresh_token":"2.yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Sleep timeline

Getting the timeline for one night is just a straight up GET request. It gives a detailed log of events, including different ‘depths’ of sleep (I’ve cut out a whole bunch here for simplicity). It also shows metrics relating to the entire night, and their condition (‘warning’ or ‘ideal’).

GET /v2/timeline/2016-04-08 HTTP/1.1
Host: api.hello.is
Accept: */*
Authorization: Bearer 2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:29:00 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "score":76,
  "score_condition":"WARNING",
  "message":"You were asleep for **8.2 hours**, and sleeping soundly for 3.2 hours.",
  "date":"2016-04-08",
  "events":[
    {
      "timestamp":1460168700000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You went to bed.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"GOT_IN_BED",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460169600000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You fell asleep.",
      "sleep_depth":100,
      "sleep_state":"SOUND",
      "event_type":"FELL_ASLEEP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460177040000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You were moving around quite a bit.",
      "sleep_depth":2,
      "sleep_state":"AWAKE",
      "event_type":"GENERIC_MOTION",
      "valid_actions":["VERIFY","INCORRECT"]
    },{
      "timestamp":1460199000000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"Good morning.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"WOKE_UP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    }
  ],
  "metrics":[
    { "name":"total_sleep", "value":490, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"sound_sleep", "value":190, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"time_to_sleep", "value":15, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"times_awake", "value":2, "unit":"QUANTITY", "condition":"IDEAL" },
    { "name":"fell_asleep", "value":1460169600000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"woke_up", "value":1460199000000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"temperature", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"humidity", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"particulates", "value":null, "unit":"CONDITION", "condition":"WARNING" },
    { "name":"light", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"sound", "value":null, "unit":"CONDITION", "condition":"IDEAL"}
  ]
}

Other interesting resources

Pretty much everything displayed in the app is available as a REST resource. They clearly put some thought into API design, which makes it even more of a shame that they haven’t made it public yet. Some of the other interesting resources are:

/v1/alarms: the alarms set on the Sense.
/v1/room/current?temp_unit=c: gives the latest environmental readouts from the Sense.
/v1/room/all_sensors/hours?quantity=2&from_utc=<timestamp>: sensor history.
/v2/devices: the paired devices, including battery status of pills.
/v2/insights: JSON representation of the cards displayed on the main screen (recommendations, analysis, etc).
/v2/trends/LAST_WEEK (or MONTH or 3_MONTHS): gives graph data for sleep scores, duration and depths.

Now the API is figured out, all that’s left is to build something that uses it…