Chameth.com - posts like docker-automatic-nginx-proxy, reverse-engineering-arctis-pro-wireless-headset but not debugging-beyond-the-debugger, intro-to-containers, why-you-should-be-using-https

An interesting Tailscale + Docker gotcha

2026-01-29T00:00:00Z

As I’ve written about before, I use Tailscale for a lot of things. I thought I had it set up in a reasonably secure manner, but I recently noticed a problem.

I use Tailscale’s ACLs to limit what each node can access, based on the tags I apply to it. So an app node can’t access anything via Tailscale, while an integration or server node can access things tagged with either app or integration. This is expressed pretty simply in the Tailscale ACL JSON:

{
	"tagOwners": {
		// Servers that can be SSH'd into
		"tag:server": [],
		// Applications that are exposed on tailscale but never connect out
		"tag:app": [],
		// Things which talk to other services over tailscale (connecting to apps etc)
		"tag:integration": [],
	},
	"grants": [
		{
			// Users can access everything
			"src": ["autogroup:member"],
			"dst": ["*"],
			"ip":  ["*"],
		},
		{
			// Servers and integrations can access integrations and apps
			"src": ["tag:server", "tag:integration"],
			"dst": ["tag:integration", "tag:app"],
			"ip":  ["*"],
		},
	],
}

Over the past week I’ve been setting up a private Forgejo instance behind Tailscale, complete with an actions runner that runs things using a docker-in-docker container. I didn’t want the runner knowing anything about Tailscale, so I had it configured to speak to Forgejo direct over HTTP (forgejo:3000) instead of using full Tailscale HTTPS URL that I use when accessing it (http://git.example-net.ts.net/)¹.

Everything was going fine, until I forgot to do that translation… and it worked. My Forgejo action runners could access anything on my tailnet. I run several things on Tailscale that just have authentication turned off, on the basis that only authorised devices can access them. Things like the admin interface for this website. I definitely didn’t intend for any workflow I run on my git server to have access to edit my website!

This wasn’t just limited to Forgejo, either. Any docker container I was running could access the tailnet. In hindsight it’s fairly obvious why: the host is running Tailscale, connected as a node tagged with my server tag. That creates a tailscale0 interface, and automagically sets up iptables rules to route Tailscale traffic over the interface. Docker also automagically sets up iptables rules to bridge traffic, and apparently these two sets of rules interact in such a way that traffic from Docker containers is allowed to route via the tailscale0 interface.

I say it’s fairly obvious in hindsight — there’s no reason why Docker would special case any particular host interface after all — but it still feels pretty surprising. Because both bits of software inject their own iptables rules, I never really had a good mental model for how they interact. The host Tailscale node was a completely separate building block to Docker. It would be a pain to use either of them if they didn’t do these rules, but it’s also one of the reasons I don’t really like “magical” things².

There are lots of ways to fix this, but none of them feel particularly great. You can configure both Tailscale and Docker to not automatically fiddle with iptables and handle the rules yourself, but I really hate dealing with iptables³. As a stopgap I did hold my nose and add some iptables rules to drop traffic to the tailscale0 interface if it originated from the IP ranges that Docker was configured to use:

iptables -I DOCKER-USER -s 192.168.0.0/16 -o tailscale0 -j DROP
iptables -I DOCKER-USER -s 172.17.0.0/12 -o tailscale0 -j DROP

The DOCKER-USER chain is a nice little escape hatch; it comes before the main auto-generated DOCKER chain, and Docker leaves the rules in it alone.

So I added these rules, and felt pretty good about myself, and then… everything started breaking in weird ways. After some debugging I realised the problem was DNS⁴. The server’s DNS resolver is 100.100.100.100, a special Tailscale address. This is so that it can resolve tailnet hostnames via MagicDNS (and so I don’t have to configure my custom DNS servers manually on each device; Tailscale does it for me). My new iptables rules inadvertently dropped all the DNS packets coming from docker containers. D’oh.

Obviously the solution here is to double down and add MORE iptables rules:

iptables -I DOCKER-USER -s 192.168.0.0/16 -d 100.100.100.100 -o tailscale0 -j ACCEPT
iptables -I DOCKER-USER -s 172.17.0.0/12 -d 100.100.100.100 -o tailscale0 -j ACCEPT

This got DNS working again, but it all feels a bit gross. Not least because to actually make these persist you need to use iptables-save, and then you also get all the rules that both Docker and Tailscale have inserted. I came across a script to filter out the Docker ones, but… yuck⁵.

I realised a cleaner idea might just be to lock down what server tagged nodes could access within Tailscale itself. That way I can avoid touching iptables at all. It’s handy to be able to curl services from the host when debugging, but it’s not really necessary. So I removed that access, and… stuff broke again. I’m using my Forgejo instance as a registry for some of the docker images I run, so the docker daemon needs to be able to reach it. I ended up making a new tag for infrastructure, which can be accessed from server devices. This does still allow all the docker containers to reach Forgejo, but I already have it set up with appropriate access controls and public/private repository splits. Forgejo is a service designed to run publicly, so this seems a reasonable trade-off for convenience. I used the tests feature of Tailscale’s ACL config to make sure I’d got the rules right:

{
	"tests": [
		{
			// Servers can only access infrastructure
			"src":   "tag:server",
			"proto": "tcp",
			"allow": [
				"tag:infrastructure:8080",
			],
			"deny": [
				"100.84.16.43:8080",
				"me@example.com:8080",
				"tag:server:8080",
				"tag:app:8080",
				"tag:integration:8080",
			],
		},
	],
}

The whole situation still feels a bit messy. If I ever get around to switching to nftables I might loop back and manually craft some rules for routing traffic, instead of leaving Tailscale and Docker to do their own thing.

I discovered later on that logging into the container registry at forgejo:3000 actually issued a redirect to https://git.example-net.ts.net/ so this was all basically for naught… ↩︎
I didn’t use Tailscale’s MagicDNS for a long time just because the word “magic” put me off. Only when I eventually got around to learning how it worked, and seeing that it wasn’t really that magical under the hood, did I change my mind. ↩︎
nftables seems far better in lots of ways, but I can’t really be bothered migrating. Maybe next time I reimage the server for whatever reason… ↩︎
It’s always DNS… ↩︎
Again, nftables would almost certainly help here. It actually has (gasp) configuration files. But again, I really didn’t want to spend the time migrating. ↩︎

Docker reverse proxying, redux

2022-12-06T00:00:00Z

Six years ago, I described my system for configuring a reverse proxy for docker containers. It involved six containers including a key-value store and a webserver. Nothing in that system has persisted to this day. Don’t get me wrong – it worked – but there were a lot of rough edges and areas for improvement.

Microservices and their limitations

My goal was to follow the UNIX philosophy of “do one thing and do it well”. Unfortunately, that doesn’t really work when applied to network services that have to interact with one and other. UNIX tools are built upon a common file system and simple data passed over STDIN. Microservices don’t have that shared foundation. You could make one: companies that use microservices in anger often have a team that deals with the “developer experience” of creating and using microservices. But as a solo developer that’s not something I wanted to spend my time doing.

This became very apparent when trying to debug issues. In the UNIX world, if your series of commands piped together does something strange you can simply echo the data at various stages. Not so much when that data is flying around on a network, going into and out of things like etcd. Trying to figure out why a certificate hadn’t been acquired was a case of searching through logs from four containers, none of which had particularly good logging. There are many ways to get insight into what’s happening with microservices but, again, that’s not something I wanted to do myself.

Over time, and with experience in other projects, I came to realise that microservices only really make sense if you’re unable to deploy a monolith. For tech companies this naturally happens when different teams contribute to the same service: splitting it out into smaller services that are wholly owned by individual teams makes sense. For solo developers, that never happens. You can still gain the other benefits of microservices – such as code separation and having clearly defined APIs – by sticking to certain coding standards.

Proxy inconveniences

As well as being unhappy with the microservice nature of the solution, I wasn’t pleased with nginx. If you requested an unknown domain, nginx would use the first server block in its config to serve a response, instead of sending an “unrecognised name” alert as I wanted. It was a minor issue, but it irked me.

So from nginx I switched to haproxy. It has a strict-sni option when configuring TLS connections which makes it behave properly. It also performs a lot better for this type of workload than nginx. All was well for a while, but then I started getting alerts that requests were occasionally failing. I couldn’t reproduce the issue, but my nightly jobs to build and push containers managed to hit it nearly every night, causing them to fail.

After some investigation, I found that the haproxy developers had refactored the header parsing code, and neglected to properly reset flags when multiple requests were sent over the same connection. There was a patch, but it wasn’t released. No problem, I thought, I’ll just cherry-pick it onto the last release… Except that haproxy use Git in the most convoluted manner I’ve ever seen – they have one repository per release. This makes it harder to patch, but it also made me question whether I trusted them to ship stable software: there were no tests for the header parsing code (which is both fundamental and finicky, the perfect target for tests), the source code management was weird, and they didn’t seem in any rush to patch this bug.

Not long after that issue, Greg managed to encounter another bug where haproxy returned a 500 error whenever the upstream server replied with a particular, perfectly valid, header. The die was cast – it was time to move to something else.

Not Invented Here syndrome

Looking for a new solution, there were many more options than back in 2016. I’m still convinced, however, that anything exposed to the Internet should not have access to run docker containers. It’s the modern equivalent of running a CGI script as root. That single requirement eliminates most off-the-shelf solutions. What do you do when nothing quite meets your specific requirements? You make something yourself! My new solution has two components: Dotege and Centauri.

Dotege is a replacement for the microservices that monitored containers and obtained certificates. It’s fundamentally a templating engine - whenever the containers change, it evaluates a template and saves the result to disk. The template has access to details about the containers, their labels, ports, and so on. Dotege can also obtain certificates from Let’s Encrypt, and raise a signal against another process whenever the template or certificates change. I used this to generate the configuration and certificates used by haproxy for a while, and more recently changed the template so that it works for Centauri.

Centauri is my own reverse proxy. It’s configured using a simple text file and can also obtain certificates from an ACME provider. It doesn’t serve static content, has no knowledge about docker, and avoids the other bells and whistles that adorn most reverse proxies. It also has good test coverage to ensure that I don’t, say, accidentally break header parsing.

As a software engineer I enjoy writing software, but I also enjoy running simple, easy to understand software. That’s what I’ve achieved here: it’s very easy to identify where the problem is if anything goes wrong, both are small Go programs rather than vast sprawling C monstrosities, and their interaction is primarily through a file written to disk that can be inspected or edited as needed.

Reverse engineering an Arctis Pro Wireless Headset

2021-06-12T00:00:00Z

The Arctis Pro Wireless Headset.

For the last year and a bit, I’ve been using a SteelSeries Arctis Pro Wireless Headset for gaming and talking to friends. It’s a fine headset, but because there’s an always-on receiver there’s no way to detect if the headset is turned on or not from the desktop.

Whenever I start using the headset, I set my desktop’s sound to go to the headset, and then when I stop using the headset I set it to go back to speakers. It doesn’t take more than a second, but some days I might put the headset on a dozen times as I’m on calls, or if it’s noisy outside, etc. That means it’s probably worth at least a few hours of my time trying to automate it.

At first, I hoped I’d be able to tell from the state of the USB device whether there was a headset connected but nothing at all changed when flipping it on and off. Then I went hunting for existing open source tools that might work with it and found that while people have reverse engineered many of the older Arctis headsets, no one has done the same for the Pro Wireless. I finished off with a search to see if anyone had documented the wire protocol even if there was no nice open source software to go with it; I came up short there, too. Looks like I’d have to do it myself.

Capturing data with WireShark

The headset exposes a Human Interface Device (HID), and the wire protocols for earlier versions of the Arctis series looked to be very simple messages passed over the HID connection. It should therefore be fairly easy to use WireShark to capture the data being sent and received by the official SteelSeries application¹.

After setting up WireShark and making it record only the headset’s HID connection, it quickly became apparent that the software was sending three different requests each second, over and over again:

Wireshark, showing a packet capture of a SET_REPORT request from the host to the headset

The packets we’re interested in are the SET_REPORT Request frames sent from the host to the device. Wireshark understands the HID protocol, so it nicely shows us the raw HID data; in the screenshot this is 0x40AA (ignoring the trailing zero bytes). The other requests sent immediately after have 0x41AA and 0x42AA payloads — clearly the first byte is indicating which piece of data is being requested.

The responses to these requests come back in an URB_INTERRUPT in frame:

Wireshark, showing a packet capture of the headset’s response to the earlier request

So the answer to our first request appears to be 0x04. The second response is 0x0402 and the third is 0x04 again. The first thing I tried doing was popping the spare battery out of the charger. The response to the 0x42AA request changed from 0x04 to 0x00 - that’s certainly clear enough! After a bit of waiting around, the battery in my headset dropped from full on the display to three bars, and at that point the 0x40AA response dropped from 0x04 to 0x03. The display on the receiver shows battery state as four bars, and it appears the wire protocol directly corresponds to that particular representation.

That left the 0x41AA request as an unknown. I tried everything I could think of, but it stubbornly kept returning 0x0402. I enlisted a friend who has the same headset to run some hacky Go code and report his values, and he also got a 0x0402 response. As I explained that I couldn’t figure out what these values are, he reported back that turning his headset off made the response change to 0x0202. In all my testing, I’d forgotten to try turning the headset off! That’s the one thing I was actually trying to detect, as well. Thanks, Simon, for helping me get past that bit of stupidity!

I still don’t know what the second byte of the response is, or whether there are other values than 0x04 for on and 0x02 for off, but I’m happy enough to label it as “device status” and move on.

Exploring other features

The software allows you to tweak a bunch of different settings:

SteelSeries Good Game software, showing the advanced headset settings

I went through each one and fiddled with all the values, recording the requests in WireShark as I did so. It turns out the protocol is very simplistic - the wire protocol directly corresponds to the UI elements in the software (or on the receiver, if you navigate through its menus). For example, the software allows you to set the “Headset auto shutoff” value in increments of 10 minutes. You might usually expect this to be converted to seconds or something similar before being passed to the device, but on the wire it’s actually sent as 0x00 for off, 0x01 for 10 minutes, up through to 0x0C for the maximum of 120 minutes.

All the dropdown options seem to function this way — if you pick the 6th option then the request payload will be an 0x06 byte. The sliders have fixed positions they snap to and function similarly: for the two brightness sliders pictured they snap to 11 positions and on the wire these range from 0x00 for off to 0x0A for the maximum.

At this point I was well past my goal of being able to detect whether the headset was on or off, and I was now just trying to see if I could figure out enough of the protocol that I could reimplement the control software on Linux if I ever wanted to. Just as I was about to close the app, something caught my eye: there are integrations with games and other software that can display information on the receiver’s OLED display!

There’s an official API for this, but it involves sending JSON to a webserver which runs as part of their app². That doesn’t feel very nice, and definitely won’t work for me on Linux, so I started a new WireShark session and took some captures while I spoke to myself on Discord.

Decoding pixels

The frames sent to the device whenever Discord showed a notification had a 1060 byte payload. The display on the receiver is 140 pixels wide³ and each pixel can only seem to be on or off, so I figured each bit in the payload corresponded to one bit of the output. I exported the data to binary in the hope that I could visually see what was going on - in theory if I line wrap the data at 140 characters it should look vaguely like the final output. Unfortunately, it did not. There was roughly the right number of high bits, but no amount of fiddling with them in a text editor could give me a coherent picture.

Instead, I wrote some code to write different values to the device. Starting with a payload of all zeroes and gradually increasing a bit at a time every second. After the first few seconds, I saw a line of pixels being drawn downwards on the left of the screen; had I just got the axes the wrong way around? After the first eight pixels lit up, though, they jumped over to the next column. The actual addressing scheme looks something like this:

Diagram of how pixels are addressed on the receiver’s OLED display

So the first 140 bytes given the pixels for the first 8 rows, the next 140 bytes fill in the 8 rows below that, and so on and so forth. Armed with this information I wrote a simple program to read an image and output it to the display:

Headset receiver displaying a custom Hello World message

At this point I was thinking about trying to get Doom rendering on the screen, but I couldn’t find anything nicely hackable that would let me grab the output and pass it on to the receiver. Instead, I decided to try a GIF decoder and before very long had a nice little animated display:

Protocol reference

All the HID messages have a single byte that determines the message type, then an 0xAA byte, then any payload required by the command. These are the ones I’ve figured out:

Byte	Command	Payload
`0x09`	Save changes	None.
`0x10`	Request firmware version(?)	None.
`0x27`	Volume limiter	`0x00` for off, `0x01` for on.
`0x2E`	Equalizer preset	ID of the equalizer preset to use.
`0x39`	Sidetone level	`0x00` for lowest to `0x09` for highest.
`0x3C`	Set headset timeout	Timeout, as a number of 10 minutes. `0x00` for off to `0x0C` for 120 mins.
`0x3E`	Mic mute LED brightness	`0x00` for lowest to `0x0A` for highest.
`0x40`	Request headset battery	None.
`0x41`	Request device status	None.
`0x42`	Request receiver battery	None.
`0x51`	Surround sound mode	`0x00` for off, `0x01` for on.
`0x62`	Auto-start Bluetooth	`0x00` for off, `0x01` for on.
`0x63`	Auto-mute game audio during calls	`0x00` for off, `0x01` for on.
`0x85`	OLED brightness	`0x00` for lowest to `0x0A` for highest.
`0x83`	Equalizer	(Not yet decoded)
`0x89`	Screensaver mode	`0x00` to dim, `0x01` for off, `0x02` for screensaver.
`0xD2`	Render image	Pixel array as described above.

Some of these I’ve not dug too much into because they didn’t seem very interesting. If you have an Arctis Pro Wireless and figure anything more out, let me know, and I’ll update the list.

If you just want to check device state like I originally did, I’ve contributed support for this headset to the excellent HeadsetControl project. It should be in the next release.

Unfortunately (as you’d expect) their application only runs on Windows, so this process involved an annoying amount of rebooting to Windows, fleeing back to Linux, and then realising I hadn’t actually recorded enough to figure it out and repeating. ↩︎
You can also send lisp to the webserver, and it will execute it. No comment. ↩︎
I know this not because it’s mentioned in the technical specs (it’s not), but because I took a photo and counted them out one by one. ↩︎

Automatic reverse proxying with Docker and nginx

2016-05-21T00:00:00Z

The Docker project logo

Over the past few weeks I’ve gradually been migrating services from running in LXC containers to Docker containers. It takes a while to get into the right mindset for Docker - thinking of containers as basically immutable - especially when you’re coming from a background of running things without containers, or in “full” VM-like containers. Once you’ve got your head around that, though, it opens up a lot of opportunities: Docker doesn’t just provide a container platform, it turns software into discrete units with a defined interface.

With all of your software suddenly having a common interface, it becomes trivial to automate a lot of things that would be tedious or complicated otherwise. You don’t need to manage port forwards because the containers just declare their ports, for example. You can also apply labels to the application containers, and then query the labels through Docker’s API.

Reverse proxying and SSL termination with Nginx and Let’s Encrypt

A fairly significant chunk of the software I run has a web interface. I don’t really want to expose and remember dozens of non-standard ports, so I configure an nginx instance as a reverse proxy. I’m of the opinion that all web traffic should be encrypted, so I also have to provide nginx with trusted certificates to use for each site it reverse proxies. Let’s Encrypt makes the process of obtaining free, trusted certificates approximately a thousand times easier than it was previously, but my workflow still ends up looking like this:

Create a new config file from a template and save it in /etc/nginx/sites-available
Temporarily disable SSL for the site as there’s no valid certificate yet
Enable the site by symlinking to it from /etc/nginx/sites-enabled
Reload nginx
Run the Let’s Encrypt client to obtain certificates
Enable SSL and for the site
Reload nginx

… And that’s not including the extra steps when I miss a semi-colon, accidentally skip a step and have to spend time figuring out why it’s not working, or any of the other human-induced problems that creep in.

I’d been toying with making a script to run through these steps manually for me, if I gave it a domain name and a reverse proxy target, but I never got around to it. Now I’m moving things to Docker, though, there’s an opportunity to automate the entire thing with no human interaction at all.

Existing solutions

It seemed like this probably wasn’t a unique idea, so I had a look around for existing solutions. The most popular by far seems to be nginx-proxy by Jason Wilder. This is based on his docker-gen project that takes a template and populates values from docker containers.

It’s a good solution, but there were a few bits I didn’t like. Firstly, templates don’t really lend themselves well to every step of the process: to request Let’s Encrypt certificates, the container uses a template to create a shell script which it then sources. Each container that generates a template also needs access to the Docker socket. Both of those cause an itch in the back of my head and make me want to say phrases like “attack surface”. I don’t think there’s actually a problem, but it doesn’t really sit well with me.

Secondly, the whole system seems slightly too tightly coupled for my liking. The Let’s Encrypt component needs to modify the nginx config in order to obtain the certificate, while the main nginx component is also making different changes to add and remove sites. It feels like if it doesn’t just work, it’s going to be difficult to debug and pry apart the different components.

Another potential solution is Rancher. This is a complete platform for managing containers, and I’m fairly sure if configured right it can grab certificates from Let’s Encrypt and do SSL termination using haproxy. I tried it for a bit but the whole platform seemed a bit overkill for my purposes, and I didn’t want to invest the time I’d need to fully understand it all.

Rolling my own

In the end I decided to roll my own solution. Here’s a high-level overview of how it all works:

Diagram showing components of a reverse proxy implementation

As you probably noticed, there are quite a few containers involved. Each one performs a small, well-defined task, and its output can easily be inspected in either a volume or a database. I think there’s some similarity to piping commands together on a command line — it’s a lot easier to reason about simpler commands like head, cut and tr than it would be one giant command that combined them. And, if it does go wrong, you can inspect the pipe at each stage to see where the problem is happening.

service-reporter and etcd

The first part of the chain is my service-reporter container. This uses the Docker API to get a list of containers, and store information about them in etcd. Etcd is a distributed key-value store (similar in some ways to redis or memcached). The container also watches for containers that are added and removed, and keeps etcd updated appropriately.

As the service metadata is stored in a database, no other part of the system needs to interact with Docker. If the Docker API changes, or the host configuration changes, then only this container has to be updated.

service-letsencrypt and letsencrypt-lexicon

The left fork of the diagram deals with obtaining SSL certificates. To keep it separate from the nginx configuration, it uses DNS-based challenge to prove that we control the domains. It does this by plumbing together two great open source projects: letsencrypt.sh, a Let’s Encrypt client implemented in bash with support for the dns-01 challenge type, and Lexicon, a python library for updating DNS records using a variety of providers.

My service-letsencrypt container connects to etcd and pulls a list of containers that have a label with the key com.chameth.vhost. It uses this to build a plain text list of certificates we require (in a format understood by letsencrypt.sh), and then monitors etcd for changes and repeats as necessary.

The letsencrypt-lexicon container runs letsencrypt.sh, using Lexicon to perform the required DNS updates, and produces certificates. The nice thing about this is that it can be used in a completely standalone fashion (you can just write a domains.txt yourself). It uses iowait to watch the domains text file for updates, and automatically reruns when there are changes. It also runs once a day to renew any certs that are coming up for expiry.

service-nginx and nginx

The right fork of the diagram is concerned with nginx. My service-nginx container again connects to etcd and pulls a list of containers. It uses a couple of labels to determine the vhost, proxy port, and proxy protocol. It then feeds these values into a template to create a server block for each site, configured with SSL certificates and a reverse proxy setup. The template covers only the very minimal settings, with the expectation that everything else will be done in the global config (things such as SSL ciphers, redirection from HTTP, etc).

This container works completely independently of the Let’s Encrypt side. You can use the Let’s Encrypt containers and mount the certificate volume, or you could just provide your own certificates. It doesn’t really make any difference.

Putting it all together

The only downside to having many small containers is that it’s a bit of a nuisance to get them all set up. Fortunately, Docker has a solution for this in the form of Docker compose. This allows you to write a YAML file defining all of the services you want to run, and bring them up or down in one go. It can handle volumes, dependencies, networking, etc. I’ll be publishing a docker-compose.yml file to get this entire stack up and running soon.

Reverse engineering the Sense API

2016-04-10T00:00:00Z

A Sense unit and its two pillow sensors

Sense is a little device that sits by your bedside and, in conjunction with a little ‘pill’ attached to your pillow, monitors your sleeping patterns and any environmental conditions that might hamper them. Android and iOS apps show you your sleep history, and offer suggestions for improvements.

Sense was Kickstarted in August 2014, raising over 2.4 million US dollars, and shipped to backers in mid 2015. The campaign blurb included this snippet:

Building with Sense

You’ll always have access to your data via our API. Take it, play with it, graph it, do whatever you want with it. It’s yours. That’s important to us.

We enjoy tinkering with and building on-top of other products we like. Sense will let you have that experience.

We’d love to hear your thoughts on what you might want to build with Sense, and how you could directly interact with the hardware, and the data it collects.

Sounds great! But a year after shipping, there’s no sign of an API, and some of us who enjoy tinkering are getting a bit restless…

Reverse engineering to the rescue

Data from the Sense hardware is transmitted off onto the Internet somewhere, and then pulled down by the mobile apps. If we can snoop that traffic, we can probably figure out how to grab the data from the Sense.

So the first step is to capture the raw network traffic while the Sense and the app are in use. I crafted an extremely over-the-top setup that involves my phone and the Sense being on a new, separate wireless network that gateways through a Linux server that can look in on all the traffic (because you never know when a dedicated capturing network will come in handy, right?…)

Running tshark while the app and device are active showed they were making HTTPS requests an Amazon Elastic Cloud instance. That’s not really much use, as we can’t see the content of the requests. The next step is to move up the network stack and target the HTTPS traffic specifically. Step in, mitmproxy. This automates man-in-the-middle attacks on HTTPS requests. When it receives such a request, it cooks up its own certificate and sends that to the client, and then sits in between while the client and the server communicate.

The certificates generated by mitmproxy aren’t issued by a trusted source, so (well behaved) browsers and apps will throw up lots of warnings and refuse to communicate with it for the most part. This is the mechanism that stops anyone impersonating your banking website when you access it over HTTPS, so is generally a very good thing. In this case, both the Sense itself and the Android app refused to talk with the blatant attacker.

We control the Android phone, though, so can just tell it that we trust the dodgy certificate issuer. This involves grabbing the certificate generated by mitmproxy, and adding it to Android’s key store. The process is documented nicely in the mitmproxy docs and is pretty straightforward. With Android trusting our certificate authority, the Sense app starts talking through the proxy and we can look at the plain HTTP requests.

The API

Logging in to the Android app and navigating through the various screens shows that the API is actually pretty nice. It’s RESTful, and uses an OAuth bearer token for authorisation, as you can see in the mitmproxy output of the authentication and first few requests:

POST https://api.hello.is/v1/oauth2/token
    ← 200 application/json 151B 1.17s
GET https://api.hello.is/v2/account/preferences
    ← 200 application/json 124B 109ms
GET https://api.hello.is/v1/account
    ← 200 application/json 181B 388ms
GET https://api.hello.is/v2/devices
    ← 200 application/json 235B 126ms
POST https://api.hello.is/v1/app/checkin
    ← 200 application/json 106B 119ms
GET https://api.hello.is/v2/insights
    ← 200 application/json 1.4kB 157ms
POST https://api.hello.is/v1/notifications/registration
    ← 204 [no content] 364ms
GET https://api.hello.is/v2/timeline/2016-04-09
    ← 200 application/json 160B 550ms
GET https://api.hello.is/v1/questions?date=2016-04-10
    ← 200 application/json 284B 132ms
PATCH https://api.hello.is/v1/app/stats
    ← 202 [no content] 126ms
GET https://api.hello.is/v1/app/stats/unread
    ← 200 application/json 71B 171ms

Authorisation

To get a bearer token, you send a POST request to /v1/oauth2/token, supplying the username and password, and a client ID and secret. Presumably if the API is ever opened up you’ll be able to register clients and get your own ID and secret, but for now reusing the ones from the Android app works fine. [I’ve prettified and linewrapped the content to make it a bit easier to read.]

POST /v1/oauth2/token HTTP/1.1
Host: api.hello.is
Accept: */*
Content-Length: 123
Content-Type: application/x-www-form-urlencoded

grant_type=password
 &client_id=8d3c1664-05ae-47e4-bcdb-477489590aa4
 &client_secret=4f771f6f-5c10-4104-bbc6-3333f5b11bf9
 &username=USERNAME
 &password=PASSWORD

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:25:16 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "token_type":"Bearer",
  "expires_in":31536000,
  "account_id":"1234",
  "access_token":"2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "refresh_token":"2.yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Sleep timeline

Getting the timeline for one night is just a straight up GET request. It gives a detailed log of events, including different ‘depths’ of sleep (I’ve cut out a whole bunch here for simplicity). It also shows metrics relating to the entire night, and their condition (‘warning’ or ‘ideal’).

GET /v2/timeline/2016-04-08 HTTP/1.1
Host: api.hello.is
Accept: */*
Authorization: Bearer 2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:29:00 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "score":76,
  "score_condition":"WARNING",
  "message":"You were asleep for **8.2 hours**, and sleeping soundly for 3.2 hours.",
  "date":"2016-04-08",
  "events":[
    {
      "timestamp":1460168700000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You went to bed.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"GOT_IN_BED",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460169600000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You fell asleep.",
      "sleep_depth":100,
      "sleep_state":"SOUND",
      "event_type":"FELL_ASLEEP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460177040000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You were moving around quite a bit.",
      "sleep_depth":2,
      "sleep_state":"AWAKE",
      "event_type":"GENERIC_MOTION",
      "valid_actions":["VERIFY","INCORRECT"]
    },{
      "timestamp":1460199000000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"Good morning.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"WOKE_UP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    }
  ],
  "metrics":[
    { "name":"total_sleep", "value":490, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"sound_sleep", "value":190, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"time_to_sleep", "value":15, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"times_awake", "value":2, "unit":"QUANTITY", "condition":"IDEAL" },
    { "name":"fell_asleep", "value":1460169600000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"woke_up", "value":1460199000000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"temperature", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"humidity", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"particulates", "value":null, "unit":"CONDITION", "condition":"WARNING" },
    { "name":"light", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"sound", "value":null, "unit":"CONDITION", "condition":"IDEAL"}
  ]
}

Other interesting resources

Pretty much everything displayed in the app is available as a REST resource. They clearly put some thought into API design, which makes it even more of a shame that they haven’t made it public yet. Some of the other interesting resources are:

/v1/alarms: the alarms set on the Sense.
/v1/room/current?temp_unit=c: gives the latest environmental readouts from the Sense.
/v1/room/all_sensors/hours?quantity=2&from_utc=<timestamp>: sensor history.
/v2/devices: the paired devices, including battery status of pills.
/v2/insights: JSON representation of the cards displayed on the main screen (recommendations, analysis, etc).
/v2/trends/LAST_WEEK (or MONTH or 3_MONTHS): gives graph data for sleep scores, duration and depths.

Now the API is figured out, all that’s left is to build something that uses it…