Chameth.com - posts like reverse-engineering-arctis-pro-wireless-headset, simple-backups-restic-hetzner but not docker-automatic-nginx-proxy, migrating-from-github-to-forgejo, why-you-should-be-using-https

Thinking more about backups

2025-11-01T00:00:00Z

The Backblaze logo

Almost a year ago I wrote about how I do backups with Restic and Hetzner. That system has been ticking along well ever since, but recently I had some… thoughts. These backups are all well and good if I accidentally delete a file, or a database gets corrupted, or something, but there are two glaring issues:

Firstly, I’m backing up my Hetzner server to Hetzner cloud storage. If something happens to Hetzner — or my Hetzner account — then all my eggs go down with that basket. Obviously Hetzner are a big organisation and aren’t likely to just vanish overnight, but I’m less confident about my account. Could a false abuse report get it suspended? What if the UK passes even more dumb laws and Hetzner decide it’s easier just to not do business with people here? This is the same sort of concern I have about Google accounts: if you have half of your life in Google Drive and Google Mail, what happens if you comment on a YouTube video, get flagged by an AI moderation process, and your account gets suspended? It’s probably not very likely, but these are things my brain likes to dwell on.

Secondly, the credentials to access the backups sit on each machine that is backed up. If someone malicious gained access to the machine, they’d also have access to delete or tamper with all the backups. It feels a little silly that the same attack could take down both the originals and the backups. There’s no way to avoid that with Hetzner’s S3 implementation, as far as I can tell.

Exploring options

I toyed with the idea of making local copies of the backups, but the only way to avoid the same problems would be to keep them offline and do a manual copy every now and then. I didn’t really want to do that, and was concerned that if I did a monthly offline backup then I stood to lose up to a month of data in the worst case.

I then looked around at other S3 providers. Amazon’s glacier offering is tempting due to its very low storage costs, but you pay for that if you ever want to restore anything. There are also lots of weird pricing edge cases around moving data between storage classes, minimum file sizes, and so on. A much better option is Backblaze’s B2 product. Their pricing is much more straight-forward, and they have an interesting feature that’s particularly useful in this case: lifecycle rules. Coupled with the ability to create API keys that don’t have access to delete files (just “hide” them), this allows for what’s effectively an append-only store.

This works more-or-less out of the box with Restic. Joseph Price has a guide that goes into the setup in a bit more depth. Basically, whenever Restic would delete a file (e.g. during a “forget” or “prune” operation), it instead gets hidden and is only deleted when the B2 lifecycle rules decide it should be. I’ve kept the existing Hetzner S3 backups for now, and just added an extra step to the end of my script: a simple restic copy and a restic forget. B2 actually works out cheaper than the Hetzner storage, as they don’t bill you for a minimum of 1TB storage; my current usage is around $3/month. Not a bad price for some extra peace of mind!

Further Adventures in Music Organisation

2025-09-21T00:00:00Z

I wrote before about how I’d dropped Spotify in favour of locally stored music, but things have advanced a bit since. I had a few issues: Tauon would occasionally manage to lose its database and along with it all my carefully constructed playlists and song ratings¹, and the experience on my phone was not very fun.

I had to manually sync the music by plugging my phone in to the computer, and sometimes it just refused to mount the right partition. I don’t think there’s really a good way to debug an Apple phone not behaving properly when connected to a Linux desktop. Then I started wanting more than one playlist synced², and trying to find a way to make that work just broke me.

I spent a while looking at different ways of hosting the music centrally. Plexamp gets lots of good reviews, and I’ve used Plex a fair bit. I set about spinning up a Plex server, and just could not get it working. The server ran fine, served the web interface, but would neither associate with my account nor run standalone. The docs were contradictory and there was very little useful logging. After a lot of frustration, I stumbled across mentions that Plex block running on Hetzner. I assume that’s the cause of my issues, although I have no way to know for sure. I use Hetzner for all my servers and other hosted services, but Plex have decided I can’t run the self-hosted software that I have a lifetime subscription for there. What the actual fuck?

I could have probably worked around the arbitrary restriction, but I didn’t want to throw more time down the drain. Instead, I set up Navidrome, an open source music server. It supports the Subsonic protocol, which means you can use a whole slew of different clients with it (or even write your own). It also means there’s a nice way to get data in and out of it programmatically, which I recall being a bit of a fight with Plex.

Syncing and organising

All my music lived on my desktop, but now I wanted it on my server. At first I just used rsync to copy everything up. I’d maintain a local “master” copy, and periodically shove the changes to the server³. That quickly got old, so I set up Syncthing to keep the two folders in sync. That worked… for a while.

Concurrently, I was looking at improving the organisation of my music. It was mostly organised by some Go programs I’d thrown together to automate the importing, with no real validation of metadata or anything else. Some albums got split up into multiple folders because the tracks had different artists (and didn’t have an album artist set); some artists ended up with multiple folders with slight spelling or case variations. It was upsetting.

I’d used MusicBrainz Picard before to fix some issues, but I didn’t really like the UI, especially when trying to do bulk actions. The main alternative is beets, which describes itself as “the music geek’s media organiser”, and is entirely command line. I was intrigued.

I started playing around with beets, and quickly noticed a problem. Every time I changed a metadata tag in a music file, Syncthing had to upload the entire file. I was frequently changing tags across the whole library as I got beets set up how I wanted, and Syncthing was handling it by uploading gigabytes of files for every minor change. I was already a bit unhappy with having the content duplicated in two places, so now I was using a command line organiser I figured I could just get rid of my local copy and move everything to the server.

I set up a Docker container for beets, and then wrote some incredibly hacky shell scripts so I could run beet locally on my desktop and it would SSH to the server over Tailscale and exec into the container, passing the arguments along. Then I did one final sync of the music library, got rid of Syncthing, double checked I had a backup and deleted all of my local music.

Bears, Beets, Battlestar Galactica

Beets is amazing. It has a vast array of plugins that can do almost anything you could want with a music library, and the command line workflow works really well for me. It’s very well documented, and all the individual parts are pleasingly simple and easy to understand.

Here’s what it looks like when importing some new music:

Beets importing an album

If it doesn’t get a perfect match then it shows the closest matches, and summarises what’s different about them (missing tracks, different names, etc), and lets you decide what to do. Aside from the metadata matching, it’s doing a lot of things under the hood that aren’t necessarily apparent. It:

Looks up the genre from Last.fm
Analyses all the tracks and writes ReplayGain metadata to them
Fetches album art
Checks all the files to make sure they’re actually playable
Scrubs any existing metadata tags

It also maintains its own database, and you can query against it. The query language is both simple and quite powerful, like the rest of beets. As an example:

$ beet ls artist:"Linkin Park" year:2025 length:2:00..2:30 
Linkin Park - From Zero - Casualty

Almost all the other commands also let you use the query syntax, which makes the whole tool really powerful. Want to delete all country tracks that you added last month? No problem. Redownload all the album art for a certain band? Easy. You can even do smart playlists using the filters.

Actually playing music

With all this organisation, I’ve not actually mentioned one tiny detail: how I actually play music now. Navidrome has a web UI, which is perfectly usable, but I don’t really want my web browser involved as it makes balancing sound levels tricky, and getting media keys working is a pain. Thankfully there are loads of Subsonic clients. I’ve tried several, but eventually settled on Feishin on the desktop. It’s very pretty:

Feishin’s ’now playing’ screen

What attracted me to it, though, was its built-in support for Navidrome’s smart playlists. These aren’t even properly exposed in Navidrome’s web UI yet, but Feishin has a visual editor that lets you create and edit them. I ended up with an identical system to the playlists I had in Tauon: a “blacklist” playlist which are tracks I never want to play, a “favourites” playlist, and then an “everything” playlist which is the entire catalogue minus the blacklist.

On iOS I’ve settled on Arpeggi. It’s not actually on the App Store yet, but available via TestFlight. It’s one of those really nicely polished apps that are obviously a labour of love, not just out to do the minimum possible to get your money. It can cache songs offline, and automatically download entire playlists, and supports all of the standard Subsonic features like rating, reporting plays back to the server, and so on.

One unexpected benefit of using a central server is that it handles reporting plays to Last.fm and ListenBrainz instead of the clients. I don’t think I’ve ever bothered to configure my mobile clients to do that before, and now it just works automagically. I’m relying on those services more for recommendations and discovery, as that’s something you lose out on when self-hosting.

Making bad decisions about bitrates

With all the moving, copying, and organising, I started thinking about how large the collection was. The songs were all in different formats, depending on when and where I’d picked them up, and I couldn’t tell the difference between them. I did a test and transcoded an MP3 file down to 128kbps, and still couldn’t tell the difference. So armed with that sample size of 1, I transcoded the entire library and saved so much space.

I don’t remember which track I did that first test with, but it must have been a very unlucky pick. I quickly started to notice the distortions caused by the low bitrate. It was particularly bad in any song with a lot of treble, and started to really annoy me. I started a painful process of reimporting things in their original format. Beets came in clutch again, both with the import process (if you import a duplicate album, it asks if you want to replace the original, and shows the format and bitrate of them for comparison), and keeping track of what was left to fix (beet ls bitrate:..128000).

Having started paying more attention to the sound quality, I think I’ve been nibbled on by the audiophile bug. I’ve been trying to get new music in FLAC format where possible. I’m pretty sure I can’t tell the difference between a decent bitrate MP3 and a FLAC, but maybe if I got a better DAC and some nice headphones…? Someone please hide my credit card!

As for the library taking up a lot of space, I now realise it’s a price worth paying. I might need to swap out the server to one with more storage at some point in the future, but it’s generally worth upgrading every few years with rented dedicated servers anyway, as you often get more for the same amount of money. Next upgrade I’ll just make sure there’s an appropriately-sized disk instead of focusing entirely on RAM and CPU.

Writing more code

Naturally, this whole process spawned several side projects. When browsing albums in Navidrome, I was a bit upset that the album art wasn’t all a consistent size. I thought about scripting something to crop them consistently, but then I had a better idea, and jewelcase was born. It takes the album art, crops it down to a consistent size, and then renders it inside a jewel case. It applies some slight effects, like adjusting the colours, rounding the corners, and tweaking the edges so it looks a bit more real. I don’t know why⁴, but every time I see the effect it gives me a little spark of joy.

Album art, before and after jewelcase is applied

The other project is BASS, a tool that uses the Subsonic API to grab information about my music catalogue, and then generates a “Daily Mix” playlist. It uses a system of weights to select tracks semi-randomly, biasing towards favourite tracks, those that haven’t been played much, and a few other criteria. This gives me a nice balance between having my favourites on repeat, and exploring the full library at random.

All the weights in BASS are customisable, so I can tweak it to my heart’s desire, and anyone else can also run it and configure it entirely differently to me if they want to. I can’t imagine I would’ve been able to do that if I were using Plex!

I was running a version from git as the stable release had fun dependency issues on Arch, so it’s possible this wouldn’t be an issue in normal use. I also started backing up the database, but it’s still annoying to have to restore it. ↩︎
A “favourites” playlist for me, and a more “family friendly” playlist for when I’m playing music out loud that has a bit less screaming/swearing/etc. ↩︎
One of the reasons for starting this was being annoyed by the periodic sync to my phone, obviously it makes perfect sense to introduce a new, different periodic sync. ↩︎
Nostalgia? A little bit of OCD? Both? ↩︎

Simple backups with Restic and Hetzner Cloud

2024-12-06T00:00:00Z

Restic’s mascot, who’s dual-wielding umbrellas to save you from a rainy day.

I have a confession: for the past few years I’ve not been backing up any of my computers. Everyone knows that you should do backups, but actually getting around to doing it is another story.

Don’t get me wrong: most of my important things are “backed up” by virtue of being committed to remote git repositories, or attached to e-mails, or re-obtainable from the original source, and so on. I don’t think any machine failing completely would be a disaster for me, but it would certainly be a pain.

This week I finally got around to doing something, and it ended up being a lot more straight forward than my previous forays into backup-land.

Restic

After soliciting a few opinions, the choice of backup software came down to either Borg or Restic. I’m pretty sure either would have done what I want, but I leaned towards Restic for a few reasons: it has a more informative website, it’s written in Go rather than Python¹, and Borg seems to be transitioning between major releases at the moment².

The way Restic works is pretty simple: you initialise a ‘repository’, and can then call restic backup /some/path and it’ll get backed up to the repository. Restic handles keeping different backups separate, and only sending data that’s changed, and deduplicating, and so on. You basically point it at a thing you don’t want to lose, and it sorts it out for you. Perfect.

There are equally straight-forward commands for removing old snapshots (restic forget) and verifying backups (restic check). One of the nice things about modern backup solutions is they support a whole range of backends. I was originally going to spin up a small VPS to host my backups, but noticed that Restic supported S3-compatible stores…

Hetzner Cloud Object Storage

I host my servers with Hetzner, and was going to use them to spin up a VPS as well. Despite the “Cloud” branding on a bunch of products, they offer reasonable prices and good service. A couple of months ago, they started offering S3-compatible object storage.

The pricing isn’t totally straight forward, but for continuous use the “free quota” amounts to 1TB of storage and 1TB of egress a month. Ingress is free, as is traffic within the eu-central region (where all my servers are). That quota is only awarded when you pay the “base price”, though, which is €4.99 a month. So basically it’s €5 a month for 1TB of storage and enough egress to fully restore every single byte. That’s better value than any VPS I can find, much cheaper than Amazon S3, and about the same as Backblaze B2.

Getting them to work together

Now you’d think making the tool that supports S3-compatible object storage work with your S3-compatible object storage would be easy, right? Well not quite. The library Restic uses to deal with S3 backends has some strange logic for figuring out the bucket name given a URL. It doesn’t quite seem to work right, though…

Hetzner buckets have URLs like s3://bucketname.hel1.your-objectstorage.com³, but just passing that to Restic gives an error that the bucket is not specified. The docs mention there’s an advanced option to make it use the virtual host for the bucket name: -o s3.bucket-lookup=dns. But that… also doesn’t work. What I ended up doing was specifying the URL as s3://hel1.your-objectstorage.com/bucketname, and also passing in the dns option. The library then seems to muddle its way back to a real, working URL. I’m not sure why it works like this: maybe it’s just a weird aspect of S3 that I’m oblivious to?

The next fun part is that you can configure Restic entirely by using environment variables, except for that -o s3.bucket-lookup=dns argument. That has to go on the command line. I ended up making a little wrapper script to invoke Restic correctly:

#!/bin/sh

# The password restic will use to encrypt your data. You should generate
# something nice and secure.
export RESTIC_PASSWORD=repo-password

# The path to the repository where restic will save the backup. In our case
# this takes the form `s3:<endpoint>/<bucket>`. Your endpoint might be different
# to mine depending on the region your bucket is in.
export RESTIC_REPOSITORY=s3:hel1.your-objectstorage.com/bucket-name

# The access and secret key generated in the 'S3 credentials' section of the
# Hetzner Cloud Console
export AWS_ACCESS_KEY_ID=hetzner-access-key-id
export AWS_SECRET_ACCESS_KEY=hetzner-access-key

# Pass any arguments on to the restic command, along with the magic
# `s3.bucket-lookup` option we need to resolve the S3 URL properly.
exec restic -o s3.bucket-lookup=dns "$@"

Then in the backup script I just alias restic to use the script:

#!/bin/bash

set -eu

# This means any time we use `restic` below, we'll actually execute our special
# script which supplies all the env vars and arguments needed to find the
# repository. Make sure the path matches where you saved the script!
alias restic='~/.bin/restic'

# Actually do the backup. Each directory in the list below will be backed up
# separately.
dirs=(
	"/some/path/to/backup/"
	"/some/other/path/"
)
for i in "${dirs[@]}" 
do
	echo $i
	(cd $i && restic --verbose backup .)
done

# Prune our snapshots. You can tweak the numbers here. Run with `--dry-run` to
# see what effect any changes would have before actually committing to them. 
restic forget --keep-daily 7 --keep-weekly 10 --keep-monthly 24 --keep-yearly 10

The only other notable thing here is that the script changes into the directory to be backed up. If you give Restic an absolute path, it will create a new snapshot if the metadata of any folder in the path changes, which is not what I want.

If it wasn’t for the S3 URL issues, the whole thing would’ve probably taken me about half an hour. That’s including setting up the object storage, installing Restic, and so on. It’s painfully easy. Why didn’t I do this three years ago?!

Addendum: a step-by-step guide

If you want to do this yourself, here’s a quick step-by-step guide:

Log in to the Hetzner Cloud Console, and create a project.
On the “Object Storage” tab, create a new bucket. Note the name and the endpoint.
On the “Security” tab, go to “S3 Credentials” and generate new credentials. Note down the access key and the secret key.
Copy the first shell script above, fill in the password (you can pick!), endpoint, bucket name, access key and secret key.
Run the script with the init argument (e.g. ~/.bin/restic init). This will create a new repository, and only needs to be done once even if you backup multiple machines.
Copy the second shell script above, making sure the restic alias points at the script you saved in step 4. Change the list of directories to whatever you want to backup.
Schedule the script to be run automatically, using crontab or systemd timers or however you prefer.

To check everything is working, you can use the snapshots subcommand to see a list of saved snapshots. You might also want to try to restore a snapshot using the restore subcommand.

I’m not trying to be a language snob, but given the choice between two otherwise equal projects one in Go and one in Python, I’ll take the Go one. I know I’m not going to have weird library issues down the line, and I’m much more comfortable rummaging around the source. ↩︎
With a “Don’t use this in production!” notice on the shiny new version. ↩︎
Aside: I really hate URLs like that. What is with the trend for completely generic domains divorced from the service they’re a part of? ↩︎

Reverse engineering an Arctis Pro Wireless Headset

2021-06-12T00:00:00Z

The Arctis Pro Wireless Headset.

For the last year and a bit, I’ve been using a SteelSeries Arctis Pro Wireless Headset for gaming and talking to friends. It’s a fine headset, but because there’s an always-on receiver there’s no way to detect if the headset is turned on or not from the desktop.

Whenever I start using the headset, I set my desktop’s sound to go to the headset, and then when I stop using the headset I set it to go back to speakers. It doesn’t take more than a second, but some days I might put the headset on a dozen times as I’m on calls, or if it’s noisy outside, etc. That means it’s probably worth at least a few hours of my time trying to automate it.

At first, I hoped I’d be able to tell from the state of the USB device whether there was a headset connected but nothing at all changed when flipping it on and off. Then I went hunting for existing open source tools that might work with it and found that while people have reverse engineered many of the older Arctis headsets, no one has done the same for the Pro Wireless. I finished off with a search to see if anyone had documented the wire protocol even if there was no nice open source software to go with it; I came up short there, too. Looks like I’d have to do it myself.

Capturing data with WireShark

The headset exposes a Human Interface Device (HID), and the wire protocols for earlier versions of the Arctis series looked to be very simple messages passed over the HID connection. It should therefore be fairly easy to use WireShark to capture the data being sent and received by the official SteelSeries application¹.

After setting up WireShark and making it record only the headset’s HID connection, it quickly became apparent that the software was sending three different requests each second, over and over again:

Wireshark, showing a packet capture of a SET_REPORT request from the host to the headset

The packets we’re interested in are the SET_REPORT Request frames sent from the host to the device. Wireshark understands the HID protocol, so it nicely shows us the raw HID data; in the screenshot this is 0x40AA (ignoring the trailing zero bytes). The other requests sent immediately after have 0x41AA and 0x42AA payloads — clearly the first byte is indicating which piece of data is being requested.

The responses to these requests come back in an URB_INTERRUPT in frame:

Wireshark, showing a packet capture of the headset’s response to the earlier request

So the answer to our first request appears to be 0x04. The second response is 0x0402 and the third is 0x04 again. The first thing I tried doing was popping the spare battery out of the charger. The response to the 0x42AA request changed from 0x04 to 0x00 - that’s certainly clear enough! After a bit of waiting around, the battery in my headset dropped from full on the display to three bars, and at that point the 0x40AA response dropped from 0x04 to 0x03. The display on the receiver shows battery state as four bars, and it appears the wire protocol directly corresponds to that particular representation.

That left the 0x41AA request as an unknown. I tried everything I could think of, but it stubbornly kept returning 0x0402. I enlisted a friend who has the same headset to run some hacky Go code and report his values, and he also got a 0x0402 response. As I explained that I couldn’t figure out what these values are, he reported back that turning his headset off made the response change to 0x0202. In all my testing, I’d forgotten to try turning the headset off! That’s the one thing I was actually trying to detect, as well. Thanks, Simon, for helping me get past that bit of stupidity!

I still don’t know what the second byte of the response is, or whether there are other values than 0x04 for on and 0x02 for off, but I’m happy enough to label it as “device status” and move on.

Exploring other features

The software allows you to tweak a bunch of different settings:

SteelSeries Good Game software, showing the advanced headset settings

I went through each one and fiddled with all the values, recording the requests in WireShark as I did so. It turns out the protocol is very simplistic - the wire protocol directly corresponds to the UI elements in the software (or on the receiver, if you navigate through its menus). For example, the software allows you to set the “Headset auto shutoff” value in increments of 10 minutes. You might usually expect this to be converted to seconds or something similar before being passed to the device, but on the wire it’s actually sent as 0x00 for off, 0x01 for 10 minutes, up through to 0x0C for the maximum of 120 minutes.

All the dropdown options seem to function this way — if you pick the 6th option then the request payload will be an 0x06 byte. The sliders have fixed positions they snap to and function similarly: for the two brightness sliders pictured they snap to 11 positions and on the wire these range from 0x00 for off to 0x0A for the maximum.

At this point I was well past my goal of being able to detect whether the headset was on or off, and I was now just trying to see if I could figure out enough of the protocol that I could reimplement the control software on Linux if I ever wanted to. Just as I was about to close the app, something caught my eye: there are integrations with games and other software that can display information on the receiver’s OLED display!

There’s an official API for this, but it involves sending JSON to a webserver which runs as part of their app². That doesn’t feel very nice, and definitely won’t work for me on Linux, so I started a new WireShark session and took some captures while I spoke to myself on Discord.

Decoding pixels

The frames sent to the device whenever Discord showed a notification had a 1060 byte payload. The display on the receiver is 140 pixels wide³ and each pixel can only seem to be on or off, so I figured each bit in the payload corresponded to one bit of the output. I exported the data to binary in the hope that I could visually see what was going on - in theory if I line wrap the data at 140 characters it should look vaguely like the final output. Unfortunately, it did not. There was roughly the right number of high bits, but no amount of fiddling with them in a text editor could give me a coherent picture.

Instead, I wrote some code to write different values to the device. Starting with a payload of all zeroes and gradually increasing a bit at a time every second. After the first few seconds, I saw a line of pixels being drawn downwards on the left of the screen; had I just got the axes the wrong way around? After the first eight pixels lit up, though, they jumped over to the next column. The actual addressing scheme looks something like this:

Diagram of how pixels are addressed on the receiver’s OLED display

So the first 140 bytes given the pixels for the first 8 rows, the next 140 bytes fill in the 8 rows below that, and so on and so forth. Armed with this information I wrote a simple program to read an image and output it to the display:

Headset receiver displaying a custom Hello World message

At this point I was thinking about trying to get Doom rendering on the screen, but I couldn’t find anything nicely hackable that would let me grab the output and pass it on to the receiver. Instead, I decided to try a GIF decoder and before very long had a nice little animated display:

Protocol reference

All the HID messages have a single byte that determines the message type, then an 0xAA byte, then any payload required by the command. These are the ones I’ve figured out:

Byte	Command	Payload
`0x09`	Save changes	None.
`0x10`	Request firmware version(?)	None.
`0x27`	Volume limiter	`0x00` for off, `0x01` for on.
`0x2E`	Equalizer preset	ID of the equalizer preset to use.
`0x39`	Sidetone level	`0x00` for lowest to `0x09` for highest.
`0x3C`	Set headset timeout	Timeout, as a number of 10 minutes. `0x00` for off to `0x0C` for 120 mins.
`0x3E`	Mic mute LED brightness	`0x00` for lowest to `0x0A` for highest.
`0x40`	Request headset battery	None.
`0x41`	Request device status	None.
`0x42`	Request receiver battery	None.
`0x51`	Surround sound mode	`0x00` for off, `0x01` for on.
`0x62`	Auto-start Bluetooth	`0x00` for off, `0x01` for on.
`0x63`	Auto-mute game audio during calls	`0x00` for off, `0x01` for on.
`0x85`	OLED brightness	`0x00` for lowest to `0x0A` for highest.
`0x83`	Equalizer	(Not yet decoded)
`0x89`	Screensaver mode	`0x00` to dim, `0x01` for off, `0x02` for screensaver.
`0xD2`	Render image	Pixel array as described above.

Some of these I’ve not dug too much into because they didn’t seem very interesting. If you have an Arctis Pro Wireless and figure anything more out, let me know, and I’ll update the list.

If you just want to check device state like I originally did, I’ve contributed support for this headset to the excellent HeadsetControl project. It should be in the next release.

Unfortunately (as you’d expect) their application only runs on Windows, so this process involved an annoying amount of rebooting to Windows, fleeing back to Linux, and then realising I hadn’t actually recorded enough to figure it out and repeating. ↩︎
You can also send lisp to the webserver, and it will execute it. No comment. ↩︎
I know this not because it’s mentioned in the technical specs (it’s not), but because I took a photo and counted them out one by one. ↩︎

Reverse engineering the Sense API

2016-04-10T00:00:00Z

A Sense unit and its two pillow sensors

Sense is a little device that sits by your bedside and, in conjunction with a little ‘pill’ attached to your pillow, monitors your sleeping patterns and any environmental conditions that might hamper them. Android and iOS apps show you your sleep history, and offer suggestions for improvements.

Sense was Kickstarted in August 2014, raising over 2.4 million US dollars, and shipped to backers in mid 2015. The campaign blurb included this snippet:

Building with Sense

You’ll always have access to your data via our API. Take it, play with it, graph it, do whatever you want with it. It’s yours. That’s important to us.

We enjoy tinkering with and building on-top of other products we like. Sense will let you have that experience.

We’d love to hear your thoughts on what you might want to build with Sense, and how you could directly interact with the hardware, and the data it collects.

Sounds great! But a year after shipping, there’s no sign of an API, and some of us who enjoy tinkering are getting a bit restless…

Reverse engineering to the rescue

Data from the Sense hardware is transmitted off onto the Internet somewhere, and then pulled down by the mobile apps. If we can snoop that traffic, we can probably figure out how to grab the data from the Sense.

So the first step is to capture the raw network traffic while the Sense and the app are in use. I crafted an extremely over-the-top setup that involves my phone and the Sense being on a new, separate wireless network that gateways through a Linux server that can look in on all the traffic (because you never know when a dedicated capturing network will come in handy, right?…)

Running tshark while the app and device are active showed they were making HTTPS requests an Amazon Elastic Cloud instance. That’s not really much use, as we can’t see the content of the requests. The next step is to move up the network stack and target the HTTPS traffic specifically. Step in, mitmproxy. This automates man-in-the-middle attacks on HTTPS requests. When it receives such a request, it cooks up its own certificate and sends that to the client, and then sits in between while the client and the server communicate.

The certificates generated by mitmproxy aren’t issued by a trusted source, so (well behaved) browsers and apps will throw up lots of warnings and refuse to communicate with it for the most part. This is the mechanism that stops anyone impersonating your banking website when you access it over HTTPS, so is generally a very good thing. In this case, both the Sense itself and the Android app refused to talk with the blatant attacker.

We control the Android phone, though, so can just tell it that we trust the dodgy certificate issuer. This involves grabbing the certificate generated by mitmproxy, and adding it to Android’s key store. The process is documented nicely in the mitmproxy docs and is pretty straightforward. With Android trusting our certificate authority, the Sense app starts talking through the proxy and we can look at the plain HTTP requests.

The API

Logging in to the Android app and navigating through the various screens shows that the API is actually pretty nice. It’s RESTful, and uses an OAuth bearer token for authorisation, as you can see in the mitmproxy output of the authentication and first few requests:

POST https://api.hello.is/v1/oauth2/token
    ← 200 application/json 151B 1.17s
GET https://api.hello.is/v2/account/preferences
    ← 200 application/json 124B 109ms
GET https://api.hello.is/v1/account
    ← 200 application/json 181B 388ms
GET https://api.hello.is/v2/devices
    ← 200 application/json 235B 126ms
POST https://api.hello.is/v1/app/checkin
    ← 200 application/json 106B 119ms
GET https://api.hello.is/v2/insights
    ← 200 application/json 1.4kB 157ms
POST https://api.hello.is/v1/notifications/registration
    ← 204 [no content] 364ms
GET https://api.hello.is/v2/timeline/2016-04-09
    ← 200 application/json 160B 550ms
GET https://api.hello.is/v1/questions?date=2016-04-10
    ← 200 application/json 284B 132ms
PATCH https://api.hello.is/v1/app/stats
    ← 202 [no content] 126ms
GET https://api.hello.is/v1/app/stats/unread
    ← 200 application/json 71B 171ms

Authorisation

To get a bearer token, you send a POST request to /v1/oauth2/token, supplying the username and password, and a client ID and secret. Presumably if the API is ever opened up you’ll be able to register clients and get your own ID and secret, but for now reusing the ones from the Android app works fine. [I’ve prettified and linewrapped the content to make it a bit easier to read.]

POST /v1/oauth2/token HTTP/1.1
Host: api.hello.is
Accept: */*
Content-Length: 123
Content-Type: application/x-www-form-urlencoded

grant_type=password
 &client_id=8d3c1664-05ae-47e4-bcdb-477489590aa4
 &client_secret=4f771f6f-5c10-4104-bbc6-3333f5b11bf9
 &username=USERNAME
 &password=PASSWORD

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:25:16 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "token_type":"Bearer",
  "expires_in":31536000,
  "account_id":"1234",
  "access_token":"2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "refresh_token":"2.yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Sleep timeline

Getting the timeline for one night is just a straight up GET request. It gives a detailed log of events, including different ‘depths’ of sleep (I’ve cut out a whole bunch here for simplicity). It also shows metrics relating to the entire night, and their condition (‘warning’ or ‘ideal’).

GET /v2/timeline/2016-04-08 HTTP/1.1
Host: api.hello.is
Accept: */*
Authorization: Bearer 2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:29:00 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "score":76,
  "score_condition":"WARNING",
  "message":"You were asleep for **8.2 hours**, and sleeping soundly for 3.2 hours.",
  "date":"2016-04-08",
  "events":[
    {
      "timestamp":1460168700000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You went to bed.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"GOT_IN_BED",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460169600000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You fell asleep.",
      "sleep_depth":100,
      "sleep_state":"SOUND",
      "event_type":"FELL_ASLEEP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460177040000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You were moving around quite a bit.",
      "sleep_depth":2,
      "sleep_state":"AWAKE",
      "event_type":"GENERIC_MOTION",
      "valid_actions":["VERIFY","INCORRECT"]
    },{
      "timestamp":1460199000000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"Good morning.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"WOKE_UP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    }
  ],
  "metrics":[
    { "name":"total_sleep", "value":490, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"sound_sleep", "value":190, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"time_to_sleep", "value":15, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"times_awake", "value":2, "unit":"QUANTITY", "condition":"IDEAL" },
    { "name":"fell_asleep", "value":1460169600000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"woke_up", "value":1460199000000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"temperature", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"humidity", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"particulates", "value":null, "unit":"CONDITION", "condition":"WARNING" },
    { "name":"light", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"sound", "value":null, "unit":"CONDITION", "condition":"IDEAL"}
  ]
}

Other interesting resources

Pretty much everything displayed in the app is available as a REST resource. They clearly put some thought into API design, which makes it even more of a shame that they haven’t made it public yet. Some of the other interesting resources are:

/v1/alarms: the alarms set on the Sense.
/v1/room/current?temp_unit=c: gives the latest environmental readouts from the Sense.
/v1/room/all_sensors/hours?quantity=2&from_utc=<timestamp>: sensor history.
/v2/devices: the paired devices, including battery status of pills.
/v2/insights: JSON representation of the cards displayed on the main screen (recommendations, analysis, etc).
/v2/trends/LAST_WEEK (or MONTH or 3_MONTHS): gives graph data for sleep scores, duration and depths.

Now the API is figured out, all that’s left is to build something that uses it…