Chameth.com - posts like sense-api but not debugging-beyond-the-debugger, docker-automatic-nginx-proxy, g15-ram-upgrade, how-i-use-tailscale, migrating-from-github-to-forgejo, why-you-should-be-using-https

10 Weeks with an Apple Watch 10

2025-09-09T00:00:00Z

My watch. Yes, I am available for wrist modelling opportunities.

Around ten weeks ago¹ I picked up an Apple Watch 10, and have been wearing it almost constantly since. It’s not my first Apple Watch — I had a Series 5 for a bit back in 2020 — but it’s the first time I’ve actually stuck with it. Ten weeks seems like an apt time to reflect on it.

Firstly, why did I even bother? Well, for a couple of years I’d been wearing a Xiaomi Smart Band 7, mainly to monitor my sleep stats and set alarms that won’t wake up everyone else nearby. Its battery life was fantastic — with notifications and other things turned off, I got about a month of use between charges — but actually using it felt like trying to order food via the medium of interpretive dance.

My biggest gripe was the screen lock. If I didn’t have the screen locked then I’d periodically trigger it during the night when I moved around and it came in contact with my chest or leg. With the lock enabled you had to deliberately swipe up from bottom to top to enable interaction, but it didn’t work reliably. When I wanted to adjust an alarm, I’d be stood swiping repeatedly trying to get it to respond. When you finally get it unlocked, the whole interface is just fiddly.

The other issue was the data quality. There were some nights when I’d been woken up, sometimes even getting up and moving around, and it just didn’t show it in the data. If it can’t even get whether I’m asleep right, can I trust anything else it says?

I spent a while researching the best devices for sleep tracking. The Oura ring came highly recommended, but it was expensive and required a subscription to do anything useful. No thanks! The Apple Watch was consistently rated pretty well, and I reasoned I could pick up a refurbished older unit. I’ve been using an iPhone as my daily driver for a while, so it’d fit right into my begrudged walled garden.

The Series 10 has a significant advantage, though: it charges much quicker than all the previous generations. On a 30-minute charge, the Series 10 can go from 0 to 60%; the 9 can only make it to 40%, and my old 5 a measly 30%. Shorter charge times means I’m far less likely to leave it on charge and wander off without it. In some ways the daily charging is more convenient than monthly: the wireless charger sits on my desk, and I plop the watch on it for a little while in the evening; I don’t need to dig out the weird pogo-pin connector that has vanished sometime in the last four weeks, then carefully arrange it so it stays attached.

How a watch maybe saved my life

One of the big features of the Apple Watch, like many other wearable devices, is health and fitness tracking. I didn’t think much about this, beyond the sleep data I wanted, at first. I’ve never had a particularly good relationship with exercise², but I do like some good statistics. I started going for walks more often to get more data and see the graphs of VO2 max and HR recovery gradually inch up. That wasn’t the most profound effect on my health, though…

The recent versions of the Apple Watch have a feature that monitors for sleep apnea, a disorder where you don’t breathe properly during your sleep. I knew I wasn’t sleeping great — that’s why I was paying attention to sleep tracking data — but was still a bit surprised to get a notification from Apple Health after wearing the watch for 30 days. It gives you a graph to print and take to your doctor. So I diligently booked an appointment and a few weeks later went to see my GP.

The appointment went about as you’d expect: talking about referral to a sleep centre for a study, and so on. Towards the end, the doctor took my blood pressure (sleep apnea can be caused by, and can cause high blood pressure, in a lovely little vicious cycle). I don’t think either of us expected anything; it was just one of those standard checks for a related problem. After taking the reading, the doctor looked at me and said, “I can’t let you leave with a BP like this”. Yikes!

Blood pressure readings are split into stages: normal is under 120 mmHg³ over 80 mmHg⁴, stage 1 is up to 140/90, stage 2 is up to 180/120, and above is simply called “crisis”. Guess where I was? Also, fun fact: depending on how exactly you count and attribute things, hypertension comes out as the single largest cause of death in humans. It doesn’t kill you outright, but contributes to strokes, coronary artery disease, heart failure, and lots of other lovely things you don’t want on your CV.

After doing a few more readings, it settled down to just under the “crisis” stage and into the “holy crap, start treatment immediately” stage instead. I won’t labour on much more about this, but things are definitely moving in the right direction now⁵.

While the Apple Watch didn’t literally save my life, it triggered the chain of events that led to this diagnosis and treatment. Who knows what would have happened had it remained undetected? Probably nothing good. Also, go check your blood pressure! It’s simple to do and simple to treat if there is an issue, but so many people are walking around with hypertension and not even realising.

Building gates in the walled garden

Even though the watch is arguably a life-saver, not everything is rosy. It’s an Apple product, so you’re firmly locked in a walled garden, jealously guarded by people wearing black turtlenecks. Fortunately, there are a few ways to make it less painful.

All the health and fitness data is stored in Apple Health. You can export data as a big messy file, but it’s a very manual process to do so and the data format is gnarly. Luckily, there’s an app for that! Health Auto Export can, well, automatically export health data. It does what it says on the tin. It can send the data to Home Assistant, over MQTT, or dump it in some cloud file services, but I just have it post it all to a REST endpoint on a service I wrote. Then I dump it all in a database and can do whatever I want with it!⁶

Another tool that’s more useful on the watch itself is Apple Shortcuts. This is their no-code “if-this-then-that”-type thing. You can make automations or shortcuts that run a number of tasks. I have a whole slew of them I access via a complication⁷ on my watch face: one which prompts for input and adds an item to my to-do list (swiping to write letters is surprisingly not horrible), one which lets me select from a bunch of pre-written ones (“charge kindle”, “take laundry out in 1 hour”, type things), one to log my weight into Apple Health, one which can open and close the blinds in my room, and so on. It’s a surprisingly robust and easy-to-use system and offers just enough freedom that I’m not constantly grating on the edge of the walled garden.

Shortcuts being able to initiate arbitrary web requests is the real killer feature for me. Anything I can’t do on the watch itself, I can just farm off to a web server and connect it up with a shortcut. No need to learn Swift or pay Apple for the privilege of being a developer! For a lot of things, like controlling the blinds or adding to-do items, I already had a HTTP endpoint available and exposed over Tailscale⁸. Adding it to the watch was just a case of entering the right things in the Shortcuts app.

Daily nitty-gritty

There are lots of other little bits and pieces that come up when using the watch daily. I don’t think I can bundle them up into a pleasing narrative arc, so instead please enjoy some disjointed paragraphs of observations.

The Apple Watch has a lot of nagging abilities. It can notify you about your fitness “rings”, prompt you to stand up every hour, count how many seconds you wash your hands for, etc. I think I’d dislike these just on general principle, but the way it does them is so condescending it’s painful. I think there’s probably a cultural divide issue here, but there is no way in British English to say “Great job! You washed your hands for 30 seconds!” without it sounding like you’re being amazingly sarcastic or like you’re talking to a young child. So I turned all of that nonsense off. It’s meant to be a tool not a wannabe life coach.

You can access Maps directly on the watch and even do navigation. It works really well. The navigation mode has some nice haptic feedback: it does a short pulse as you’re approaching a turn, and then a long pulse at the actual turn. I like it a lot more than having to dig out my phone or have the directions read out. You get one pulse, glance down and see where you need to go, then it reminds you a little later when it’s time to do it. It’s a delightful user experience.

Watchfaces aren’t quite so delightful. There’s a limited number of built-in ones, and some are “exclusive” to the Ultra Series, and you can’t use them on a peasant watch like a Series 10. Annoyingly, there’s not one that does exactly what I want: a plain analogue clock with an inset date and four complication slots around the outside. Instead, I have to use one of the slots to show the date. There are third-party watchfaces, but they have issues. Firstly, there’s no actual API for making watchfaces⁹, so what they do is bodge it horribly by using a photo background that has fake widgets on it. On top of that they’re almost universally subscription-based. Again, no thanks¹⁰.

The issue I had with the Smart Band triggering when I was sleeping is solved trivially on an Apple Watch, by virtue of it having a physical button in the crown. When you put it in sleep mode, you have to double-press the crown to unlock it before it’ll do anything else. It hasn’t misfired once while I’ve been using it.

Sleep data; spot when I swapped devices!

Finally, a special mention for the gesture controls. If you raise the watch it reliably wakes up (switching from a low refresh rate, dimmed screen to an active, brighter one); you can then double-tap your index finger and thumb together and it will scroll down or page through whatever you’re looking at. The killer feature for this is navigating recipes: you can advance to the next step in a recipe while your hands are covered in flour. It’s also handy for reading notifications: when one pops up, you can double-tap to expand it, then, when it gets to the bottom, it’ll outline the default button (often “Dismiss”) and you can double-tap again to click it.

The verdict

I normally don’t like writing an actual labelled conclusion, but it feels like one is needed here! Overall, I’m happy with the watch. The daily charging doesn’t bother me, the data gathered seems reliable, the health monitoring has obviously paid dividends already, and the walled garden isn’t too chafing. It’s a straight upgrade over my old Smart Band, and I think it was worth the cost.

The original reason for getting the watch was better sleep tracking, though, so how well did it do? I’m much happier with the data: it seems to more accurately represent when I was awake in the night, and overall the sleep phases just seem to make more sense. You can see in the graph that the old data switched frequently between phases, and they didn’t quite line up for some reason; towards the right when the Apple Watch is supplying the data instead there’s a much more consistent pattern of sleep phases that repeat over the course of the night.

I’m not going to advocate that you go out and buy one, though. I know my requirements and usage aren’t typical, and I’ve also not got experience with any recent Android Wear alternatives or the new version of the Pebble watch that’s coming soon. You should definitely get your blood pressure checked, though!

OK, it’s more like 14 now, it’s taken a while for this post to make its way from my brain into text. ↩︎
Exercise for the sake of exercising just seems so overwhelmingly tedious and boring to me. And other types of exercise generally require social interaction, coordinating with people, and so on. ↩︎
Who decided to use “millimetres of mercury displaced” as a unit? You can’t just put random chemical symbols in units! That’s not how this works! ↩︎
Blood pressure readings have two parts: systolic (the pressure when the heart is beating) and diastolic (the pressure between those beats). They’re generally presented with the systolic reading on top and the diastolic reading below, and read as “X over Y”. Now you know what some of the random numbers they shout in medical shows mean! Yay learning! ↩︎
No thanks to how much salt is in everything. I’m pretty sure I’ve had at least twice the recommended daily amount of salt in a single serving before. Don’t even get me started on the things that are “low salt” but are still full of sodium from other sources. I don’t have a problem with ionic compounds, I have a problem with sodium! ↩︎
This mainly looks like drawing graphs that are slightly different to the graphs in the Apple Health graphs, for reasons I’m not sure I can explain. Making graphs is fun, OK? ↩︎
Complications are basically just home screen widgets, but with a fancy name because they’re on a watch. ↩︎
Tailscale actually causes me some problems here: everything works fine when the watch is connected to my phone, as the phone handles the Tailscale part, but if I’m not carrying my phone the watch will try to connect over WiFi directly and doesn’t understand anything about Tailscale. It happens infrequently enough that I’ll just live with it; it’s not much worse than having no signal on a phone. ↩︎
Yay walled gardens… ↩︎
I don’t object to subscribing to things in general, but it has to be something that’s worth the ongoing cost and offers something in return for the subscription. A watchface doesn’t need enough ongoing maintenance to justify subscribing to it, it’s just a cash grab. ↩︎

Home Automation Without the Megacorps

2025-05-21T00:00:00Z

I first experimented with home automation in 2016, by picking up a Samsung “SmartThings” hub. It was terrible. The UI to configure things was slow and clunky, firmware updates were applied whether you wanted them or not, and everything stopped working if their cloud services stopped. You were also locked into whatever integrations they deigned to support, of course. After that broke for the umpteenth time I scaled back and for years the closest I got to home automation was a couple of Hue bulbs.

Recently I’ve been building it out again, though. This time using off-the-shelf components that interop using Zigbee, open-source software, and some code I wrote myself. It’s great; it runs entirely locally and has had basically zero downtime. The Zigbee ecosystem lets me integrate all sorts of things without having to spend lots of money on “smart” alternatives. I think I’ve spent less on this incarnation than I did on the original SmartThings hub all those years ago (even without adjusting for inflation!).

My current setup

I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based on the CC2652P chipset¹. Interfacing with the Zigbee stack is handled by zigbee2mqtt (z2m for short), an open-source project that basically bridges your devices to an MQTT broker². When a device reports some data, it will send a new message over MQTT; when you want to make a device do something you just post a message back. It’s incredibly lightweight, but supports a huge array of devices out of the box. And as it’s just using MQTT, it’s trivial to integrate with other software or build on top of.

I know a lot of people building out this kind of home automation use Home Assistant, but I don’t get on with it terribly well. It felt incredibly sluggish, and the entire project is just a bit… much. I want a binary or package or docker image I can just run, not an entire operating system. That’s not how computers are meant to work! In contrast, z2m is simple to set up, super light weight and responsive.

Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond this now, and I’m skipping some boring bits like error handling and JSON parsing, but at first I had something like:

for {
    message, topic, err := c.ReadSlices()
    
    if string(topic) == "zigbee2mqtt/desk-button" {
        if strings.Contains(message, "single") {
            c.Publish(nil, []byte("{\"state\": \"TOGGLE\"}"), "zigbee2mqtt/room-lights/set")
        }
    }
}

I think I actually made this harder to read by trying to simplify it here, but hopefully you can follow that it was relatively straight forward to listen for a particular action to happen (in this case me “single”-pressing on the device called “desk-button”) and then make another device do something in response (toggling the state of the “room-lights” device).

So what actually are these devices? At present in z2m I have the following:

Some generic buttons
An air quality sensor (USB powered), and a separate temperature sensor (battery powered)
A light switch
Some 240V relay modules, that I use to turn non-smart devices on and off
Some USB relay modules, for the same purpose
Some “smart” plugs that I mostly use to monitor power usage
A motorised blind roller

I think all of these are either Tuya or Moes branded, and were all sourced from AliExpress. You can get the same things from Amazon or elsewhere, but they tend to be 3-4x more expensive for the same product.

One of the cool things about Zigbee devices is that the powered ones work together to create a mesh network, so you don’t have to worry about network repeaters or signal strength like you do with Wi-Fi networks. z2m even makes a map showing the connections:

The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.

Some interesting automations

A Zigbee relay spliced into the power cable for the fan

So what do I actually do with all of this? Some of it is just what I’ll call “laziness automation”: I want to be able to turn on the lights without getting up and walking to the light switch. So there’s a Zigbee button on my desk that does it. Then there are some less manual automations: my blinds are automatically closed at sunset³, for example. It feels a bit trivial, but it’s surprisingly nice just not to have to think about that at all.

Another nice quality of life automation is the light I have above my monitor. It’s USB powered, and I now have it going through a Zigbee USB relay that lets it be toggled on and off. Then I have a small agent running on my desktop that turns the light on when the computer is unlocked, and off when it locks or shuts down. I keep meaning to make a “film mode” that detects when I’m watching a film, and turning all the lights and blinds down, but I haven’t got around to it yet.

The most complex automation is probably for a window fan. It’s not smart in any way, so I cut the power cable and inserted a Zigbee relay. The whole thing is housed in a little project box to keep it secure. The relay basically acts as a switch: the live wire leading to the fan runs to the “normally open”⁴ contact, while the upstream power goes to “common”. When the relay is told to turn the fan on, the two get connected and the fan gets power. Oh, it’s also not actually a window fan, but I designed and 3D printed some adapters so that it sits right in the window.

So now we have a window fan that can be turned off and on automatically. But I don’t want to have to press a button to do that. I’m lazy, remember? Instead, I made it so that the Go code constantly monitors the temperature reported by the air quality sensor, and queries the free OpenWeatherMap API to get the rough temperature outside. It can then turn the fan on if the room is too hot, and outside is cool enough to make a difference (there’s no point in blowing hotter air in!).

A graph of room temperature, showing the effect of the fan being turned on and off

I don’t want it to do that all the time though. The room doesn’t need to be cooled if I’m not around. There are various Zigbee presence sensors you can get, but the cheaper ones seem to be of dubious quality. Instead, I wrote some code to guess whether I’m present. It uses the state of the monitor light as a proxy for “is the computer in use”, and then makes some guesses based on the last time the computer was used and the current time (if I turn the computer off at midnight it probably means I’m going to sleep; if I turn it off at 2pm it probably means I’m going out somewhere).

You can see in the graph that the fan has a decent impact on temperature. I coded it to cool the room to 19 degrees, but then not turn on again until it was above 20. This prevents it flip-flopping on and off constantly. You can clearly see the pattern in the graph, as it actively cools and then slowly the room warms back up. This arrangement is much better than being woken up at 5am because it’s painfully cold. Trust me.

Bonuses: metrics and 3D printer

One of the things my custom Go code does is collate all the various stats reported by the Zigbee devices, and inserts them into a VictoriaMetrics database. I originally hosted this on the Pi itself, and it performed fine, but I’ve since moved it onto a server so that I can use it for some other things as well.

I set up Grafana to point to VM, and can create dashboards showing power usage, what devices are turned on when, and a bunch of environmental conditions. This also makes it easy to spot how good the data coming from the devices are. For example, the air quality sensor reports a figure for the amount of Carbon Dioxide in the air, as well as the amount of Volatile Organic Compounds (VOCs). The graphs are basically identical, but on a different scale. It turns out this particular device has no actual way of detecting CO₂, so it just “calculates” it from the VOCs figure. Useful to know if you want to actually use that data.

Recently I came across a go library for interacting with Bambu Lab printers, so I’ve also hooked that into my automation. It exports metrics about the printer, so I now have a way of seeing what’s going on when I’m not physically present⁵. I even added a HTTP endpoint (exposed over Tailscale) that shows pictures from the built-in camera.

On the automation front, I’ve made it automatically turn off the printer’s light when it finishes a print, and also added push notifications via PushOver whenever the state changes. No longer will I be sat in another room blissfully unaware it ran out of filament seconds after I walked out the door!

Was it worth it?

These days you could probably just buy some kind of ‘smart home’ hub that works well enough, and do most of what I’ve done with a lot less effort and no coding. I’m still of the opinion that for something so essentially local, it should itself be managed entirely locally. I don’t trust companies like Google or Amazon not to kill their products, or change or remove an API I rely on.

Obviously the “Not Invented Here” approach of coding everything yourself doesn’t suit everyone, but as someone who enjoys coding and enjoys having things work just so it works very well for me.

It turns out the chipset is important. I initially used a similar Sonoff dongle that used a CC2531 chipset, and its performance wasn’t great. It often failed to pair with new devices, and dropped links to existing ones. The zigbee2mqtt docs do explicitly advise against the CC2531 chips for that reason. At the time they recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d go with whatever their latest recommendation was. ↩︎
MQTT is basically a network protocol for publishing and subscribing to arbitrary messages. A broker is the thing that sits in the middle and routes the messages. I use Mosquitto but any will do. ↩︎
They originally also re-opened at sunrise but that was a terrible mistake. Who knew the sun rose so early?! ↩︎
“open” meaning “there’s a gap so it doesn’t work” not “open for business”. This confusion in terminology also extends to drawbridges. ↩︎
I use the printer in LAN mode, which means there’s no way to monitor it from a phone, even if you’re connected to the same network still. ↩︎

Reverse engineering an Arctis Pro Wireless Headset

2021-06-12T00:00:00Z

The Arctis Pro Wireless Headset.

For the last year and a bit, I’ve been using a SteelSeries Arctis Pro Wireless Headset for gaming and talking to friends. It’s a fine headset, but because there’s an always-on receiver there’s no way to detect if the headset is turned on or not from the desktop.

Whenever I start using the headset, I set my desktop’s sound to go to the headset, and then when I stop using the headset I set it to go back to speakers. It doesn’t take more than a second, but some days I might put the headset on a dozen times as I’m on calls, or if it’s noisy outside, etc. That means it’s probably worth at least a few hours of my time trying to automate it.

At first, I hoped I’d be able to tell from the state of the USB device whether there was a headset connected but nothing at all changed when flipping it on and off. Then I went hunting for existing open source tools that might work with it and found that while people have reverse engineered many of the older Arctis headsets, no one has done the same for the Pro Wireless. I finished off with a search to see if anyone had documented the wire protocol even if there was no nice open source software to go with it; I came up short there, too. Looks like I’d have to do it myself.

Capturing data with WireShark

The headset exposes a Human Interface Device (HID), and the wire protocols for earlier versions of the Arctis series looked to be very simple messages passed over the HID connection. It should therefore be fairly easy to use WireShark to capture the data being sent and received by the official SteelSeries application¹.

After setting up WireShark and making it record only the headset’s HID connection, it quickly became apparent that the software was sending three different requests each second, over and over again:

Wireshark, showing a packet capture of a SET_REPORT request from the host to the headset

The packets we’re interested in are the SET_REPORT Request frames sent from the host to the device. Wireshark understands the HID protocol, so it nicely shows us the raw HID data; in the screenshot this is 0x40AA (ignoring the trailing zero bytes). The other requests sent immediately after have 0x41AA and 0x42AA payloads — clearly the first byte is indicating which piece of data is being requested.

The responses to these requests come back in an URB_INTERRUPT in frame:

Wireshark, showing a packet capture of the headset’s response to the earlier request

So the answer to our first request appears to be 0x04. The second response is 0x0402 and the third is 0x04 again. The first thing I tried doing was popping the spare battery out of the charger. The response to the 0x42AA request changed from 0x04 to 0x00 - that’s certainly clear enough! After a bit of waiting around, the battery in my headset dropped from full on the display to three bars, and at that point the 0x40AA response dropped from 0x04 to 0x03. The display on the receiver shows battery state as four bars, and it appears the wire protocol directly corresponds to that particular representation.

That left the 0x41AA request as an unknown. I tried everything I could think of, but it stubbornly kept returning 0x0402. I enlisted a friend who has the same headset to run some hacky Go code and report his values, and he also got a 0x0402 response. As I explained that I couldn’t figure out what these values are, he reported back that turning his headset off made the response change to 0x0202. In all my testing, I’d forgotten to try turning the headset off! That’s the one thing I was actually trying to detect, as well. Thanks, Simon, for helping me get past that bit of stupidity!

I still don’t know what the second byte of the response is, or whether there are other values than 0x04 for on and 0x02 for off, but I’m happy enough to label it as “device status” and move on.

Exploring other features

The software allows you to tweak a bunch of different settings:

SteelSeries Good Game software, showing the advanced headset settings

I went through each one and fiddled with all the values, recording the requests in WireShark as I did so. It turns out the protocol is very simplistic - the wire protocol directly corresponds to the UI elements in the software (or on the receiver, if you navigate through its menus). For example, the software allows you to set the “Headset auto shutoff” value in increments of 10 minutes. You might usually expect this to be converted to seconds or something similar before being passed to the device, but on the wire it’s actually sent as 0x00 for off, 0x01 for 10 minutes, up through to 0x0C for the maximum of 120 minutes.

All the dropdown options seem to function this way — if you pick the 6th option then the request payload will be an 0x06 byte. The sliders have fixed positions they snap to and function similarly: for the two brightness sliders pictured they snap to 11 positions and on the wire these range from 0x00 for off to 0x0A for the maximum.

At this point I was well past my goal of being able to detect whether the headset was on or off, and I was now just trying to see if I could figure out enough of the protocol that I could reimplement the control software on Linux if I ever wanted to. Just as I was about to close the app, something caught my eye: there are integrations with games and other software that can display information on the receiver’s OLED display!

There’s an official API for this, but it involves sending JSON to a webserver which runs as part of their app². That doesn’t feel very nice, and definitely won’t work for me on Linux, so I started a new WireShark session and took some captures while I spoke to myself on Discord.

Decoding pixels

The frames sent to the device whenever Discord showed a notification had a 1060 byte payload. The display on the receiver is 140 pixels wide³ and each pixel can only seem to be on or off, so I figured each bit in the payload corresponded to one bit of the output. I exported the data to binary in the hope that I could visually see what was going on - in theory if I line wrap the data at 140 characters it should look vaguely like the final output. Unfortunately, it did not. There was roughly the right number of high bits, but no amount of fiddling with them in a text editor could give me a coherent picture.

Instead, I wrote some code to write different values to the device. Starting with a payload of all zeroes and gradually increasing a bit at a time every second. After the first few seconds, I saw a line of pixels being drawn downwards on the left of the screen; had I just got the axes the wrong way around? After the first eight pixels lit up, though, they jumped over to the next column. The actual addressing scheme looks something like this:

Diagram of how pixels are addressed on the receiver’s OLED display

So the first 140 bytes given the pixels for the first 8 rows, the next 140 bytes fill in the 8 rows below that, and so on and so forth. Armed with this information I wrote a simple program to read an image and output it to the display:

Headset receiver displaying a custom Hello World message

At this point I was thinking about trying to get Doom rendering on the screen, but I couldn’t find anything nicely hackable that would let me grab the output and pass it on to the receiver. Instead, I decided to try a GIF decoder and before very long had a nice little animated display:

Protocol reference

All the HID messages have a single byte that determines the message type, then an 0xAA byte, then any payload required by the command. These are the ones I’ve figured out:

Byte	Command	Payload
`0x09`	Save changes	None.
`0x10`	Request firmware version(?)	None.
`0x27`	Volume limiter	`0x00` for off, `0x01` for on.
`0x2E`	Equalizer preset	ID of the equalizer preset to use.
`0x39`	Sidetone level	`0x00` for lowest to `0x09` for highest.
`0x3C`	Set headset timeout	Timeout, as a number of 10 minutes. `0x00` for off to `0x0C` for 120 mins.
`0x3E`	Mic mute LED brightness	`0x00` for lowest to `0x0A` for highest.
`0x40`	Request headset battery	None.
`0x41`	Request device status	None.
`0x42`	Request receiver battery	None.
`0x51`	Surround sound mode	`0x00` for off, `0x01` for on.
`0x62`	Auto-start Bluetooth	`0x00` for off, `0x01` for on.
`0x63`	Auto-mute game audio during calls	`0x00` for off, `0x01` for on.
`0x85`	OLED brightness	`0x00` for lowest to `0x0A` for highest.
`0x83`	Equalizer	(Not yet decoded)
`0x89`	Screensaver mode	`0x00` to dim, `0x01` for off, `0x02` for screensaver.
`0xD2`	Render image	Pixel array as described above.

Some of these I’ve not dug too much into because they didn’t seem very interesting. If you have an Arctis Pro Wireless and figure anything more out, let me know, and I’ll update the list.

If you just want to check device state like I originally did, I’ve contributed support for this headset to the excellent HeadsetControl project. It should be in the next release.

Unfortunately (as you’d expect) their application only runs on Windows, so this process involved an annoying amount of rebooting to Windows, fleeing back to Linux, and then realising I hadn’t actually recorded enough to figure it out and repeating. ↩︎
You can also send lisp to the webserver, and it will execute it. No comment. ↩︎
I know this not because it’s mentioned in the technical specs (it’s not), but because I took a photo and counted them out one by one. ↩︎

Monitoring power draw with WeMo Insight Switches

2016-05-02T00:00:00Z

A WeMo Insight Switch

I recently picked up a couple of Belkin’s WeMo Insight Switches to monitor power usage for my PC and networking equipment. WeMo is Belkin’s home automation brand, and the switches allow you to toggle power on and off with an app, and monitor power usage.

The WeMo Android app is pretty dismal. It’s slow, doesn’t look great, and crashed about a dozen times during the setup process for each of my two switches. It also doesn’t provide much information at all about power: you can see average power draw and current power draw, and that’s basically it.

Belkin has provided an option to e-mail yourself a spreadsheet with historical power data, and can even do it on a regularly scheduled basis, but that’s not really a nice solution if you want up-to-date power stats. Even if you were happy with data arriving in batch, having to get hold of an e-mail attachment and parse out a weirdly formatted spreadsheet doesn’t make for easy automation. It also relies on Belkin supporting the service indefinitely, which isn’t necessarily going to happen.

Enter the SOAP API

Fortunately, the devices themselves expose an API to get the data. Once a switch is setup with the WeMo app, it connects to a WiFi network. You can discover WeMo devices on the network using a UPnP broadcast, and then from there a list of supported services. Each service has a number of SOAP actions, with nice obvious names, and parameters documented in the XML service description.

For the insight switches, there’s a service at /upnp/control/insight1 which has a number of actions including GetPower and GetTodayKWH. Sending a SOAP request isn’t too difficult (albeit nowhere near as nice as a REST JSON API) — here’s a sample request I made using the DHC chrome extension:

POST /upnp/control/insight1 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.8
Content-Type: text/xml
Origin: chrome-extension://aejoelaoggembcahagimdiliamlcdmfm
SOAPACTION: "urn:Belkin:service:insight:1#GetPower"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.29 Safari/537.36

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
    </s:Body>
</s:Envelope>

The name of the action (in the SOAPACTION header) and the XML namespace in the body are both constructed from information in the service definition. The result that comes back is:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPowerResponse xmlns:u="urn:Belkin:service:insight:1">
            <InstantPower>92170</InstantPower>
        </u:GetPowerResponse>
    </s:Body>
</s:Envelope>

The current power draw in milliwatts is returned in the <InstantPower> argument.

Putting the data to work

So now we have a way to pull the current power draw, a nice thing to do would be to plot a graph of it to see the variation over time. This is the short of thing you’d expect to see in the Belkin app, but it’s sadly missing. After a bit of research I settled on the tried and true rrdtool to store data and generate graphs. I created a new database to store the values from the two switches:

rrdtool create power.rrd \
    --start now \
    --step 60 \
    DS:wemoComputer:GAUGE:120:U:U \
    DS:wemoNetworking:GAUGE:120:U:U \
    RRA:AVERAGE:0.5:1:1440 \
    RRA:AVERAGE:0.5:10:1008 \
    RRA:AVERAGE:0.5:30:1488 \
    RRA:AVERAGE:0.5:120:1488 \
    RRA:AVERAGE:0.5:360:1488 \
    RRA:AVERAGE:0.5:1440:36500

This creates a database file which expects values to be given every 60 seconds for two data series: ‘wemoComputer’ and ‘wemoNetworking’. These are gauge types (i.e., a value we read off a gauge, rather than a counter that keeps going up) with no minimum and maximum (‘U’).

It then defines a series of round-robin archives. Each of these stores a fixed number of entries, and the oldest is overwritten when they become full. The interesting arguments are the step count (second to last) and number of samples (last argument). The first one takes every sample and keeps 1,440 of them (i.e., a full day at 1-minute resolution); the last one has a step of 1,440 and keeps 36500 of them (i.e., 100 years at 1-day resolution). The use of RRAs ensures that rrdtool can retain enough data to create historical graphs, while providing a fixed, finite maximum file size. With sufficient RRA coverage you can graph most timescales and have data available at a reasonable resolution.

Next, I created a small python script to retrieve the power using SOAP:

#!/usr/bin/python3

import requests
from xml.etree import ElementTree

def get_power(ip_and_port):
    headers = {'Content-type': 'text/xml', 'SOAPACTION': '"urn:Belkin:service:insight:1#GetPower"'}
    payload = '''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
                             s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
                  <s:Body>
                   <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
                  </s:Body>
                 </s:Envelope>'''
    r = requests.post("http://%s/upnp/control/insight1" % ip_and_port, headers=headers, data=payload)
    et = ElementTree.fromstring(r.text)
    return et.find('.//InstantPower').text

I then have a dictionary of IP addresses to data series names, and the script polls each one in turn and then executes an rrdtool update query to add the items to the database. I have this script running every minute via cron.

After leaving the script to run for a bit and gather data, it’s time to make some graphs. I use the following to create a graph with a background gradient:

rrdtool graph desk-1d.png \
        -o -X0 -w800 -h500 \
        -u 2000 -l 20 -r \
        DEF:raw=power.rrd:wemoComputer:AVERAGE \
        CDEF:power=raw,1000,/ \
        CDEF:powerz=power,2000,LT,power,2000,IF CDEF:powerzNoUnk=power,UN,0,powerz,IF AREA:powerzNoUnk#ff0000 \
        CDEF:powery=power,1500,LT,power,1500,IF CDEF:poweryNoUnk=power,UN,0,powery,IF AREA:poweryNoUnk#ff0000 \
        CDEF:powerx=power,1000,LT,power,1000,IF CDEF:powerxNoUnk=power,UN,0,powerx,IF AREA:powerxNoUnk#ff0000 \
        CDEF:powerw=power,900,LT,power,900,IF CDEF:powerwNoUnk=power,UN,0,powerw,IF AREA:powerwNoUnk#ff0000 \
        CDEF:powerv=power,800,LT,power,800,IF CDEF:powervNoUnk=power,UN,0,powerv,IF AREA:powervNoUnk#ff1b00 \
        CDEF:poweru=power,700,LT,power,700,IF CDEF:poweruNoUnk=power,UN,0,poweru,IF AREA:poweruNoUnk#ff4100 \
        CDEF:powert=power,600,LT,power,600,IF CDEF:powertNoUnk=power,UN,0,powert,IF AREA:powertNoUnk#ff6600 \
        CDEF:powers=power,400,LT,power,400,IF CDEF:powersNoUnk=power,UN,0,powers,IF AREA:powersNoUnk#ff8e00 \
        CDEF:powerr=power,200,LT,power,200,IF CDEF:powerrNoUnk=power,UN,0,powerr,IF AREA:powerrNoUnk#ffb500 \
        CDEF:powerq=power,180,LT,power,180,IF CDEF:powerqNoUnk=power,UN,0,powerq,IF AREA:powerqNoUnk#ffdb00 \
        CDEF:powerp=power,160,LT,power,160,IF CDEF:powerpNoUnk=power,UN,0,powerp,IF AREA:powerpNoUnk#fdff00 \
        CDEF:powero=power,140,LT,power,140,IF CDEF:poweroNoUnk=power,UN,0,powero,IF AREA:poweroNoUnk#d7ff00 \
        CDEF:powern=power,120,LT,power,120,IF CDEF:powernNoUnk=power,UN,0,powern,IF AREA:powernNoUnk#b0ff00 \
        CDEF:powerm=power,100,LT,power,100,IF CDEF:powermNoUnk=power,UN,0,powerm,IF AREA:powermNoUnk#8aff00 \
        CDEF:powerl=power,90,LT,power,90,IF CDEF:powerlNoUnk=power,UN,0,powerl,IF AREA:powerlNoUnk#65ff00 \
        CDEF:powerk=power,80,LT,power,80,IF CDEF:powerkNoUnk=power,UN,0,powerk,IF AREA:powerkNoUnk#3eff00 \
        CDEF:powerj=power,70,LT,power,70,IF CDEF:powerjNoUnk=power,UN,0,powerj,IF AREA:powerjNoUnk#17ff00 \
        CDEF:poweri=power,60,LT,power,60,IF CDEF:poweriNoUnk=power,UN,0,poweri,IF AREA:poweriNoUnk#00ff10 \
        CDEF:powerh=power,50,LT,power,50,IF CDEF:powerhNoUnk=power,UN,0,powerh,IF AREA:powerhNoUnk#00ff36 \
        CDEF:powerg=power,40,LT,power,40,IF CDEF:powergNoUnk=power,UN,0,powerg,IF AREA:powergNoUnk#00ff5c \
        CDEF:powerf=power,30,LT,power,30,IF CDEF:powerfNoUnk=power,UN,0,powerf,IF AREA:powerfNoUnk#00ff83 \
        CDEF:powere=power,20,LT,power,20,IF CDEF:powereNoUnk=power,UN,0,powere,IF AREA:powereNoUnk#00ffa8 \
        CDEF:powerd=power,0,LT,power,0,IF CDEF:powerdNoUnk=power,UN,0,powerd,IF AREA:powerdNoUnk#00ffd0 \
        LINE:power#080

This seems a bit unweildy, but it’s fairly straight forward. The options tell rrdtool to create a graph with a canvas size of 800x500 pixels, a lower limit of 20W, upper limit of 2kW, and a log scale. The first CDEF divides our raw value (which was in millwatts) by 1000 to get a value in watts. rrdtool uses reverse polish notation like some calculators, so you provide the arguments before the operator.

The big block of CDEF/CDEF/AREA parameters creates a series of area plots in different colours according to the power level. They’re in descending order so the smaller areas are drawn on top of the larger layers. This results in a graph that looks like this:

Graph of power usage over a day

This graph shows the total power for all the things plugged in at my desk. You can see the idle power draw is around 60W. When I’m using the computer it jumps up to around 130W, and when the computer is under heavy load (playing games, for example) it goes up even further to the 200W mark. With a couple of small tweaks to the rrdtool command, I also have a graph showing the entire week:

Graph of power usage over a week

The two huge spikes near the start of the data are caused by a heater under my desk. They’re also one of the main reasons I chose to plot the graphs with a logarithmic scale. With a linear scale between 20W and 2kW, the graph would be completely dominated by these large spikes — in fact the difference in height between the two spikes would take almost half the graph! The relative difference between the normal values of 60-200W and the heater’s value of 1.8kW isn’t actually that interesting, and certainly not worth using about 80% of the graph to demonstrate. The log scale helps to compress this, and emphasises the difference in the smaller values more.

Reverse engineering the Sense API

2016-04-10T00:00:00Z

A Sense unit and its two pillow sensors

Sense is a little device that sits by your bedside and, in conjunction with a little ‘pill’ attached to your pillow, monitors your sleeping patterns and any environmental conditions that might hamper them. Android and iOS apps show you your sleep history, and offer suggestions for improvements.

Sense was Kickstarted in August 2014, raising over 2.4 million US dollars, and shipped to backers in mid 2015. The campaign blurb included this snippet:

Building with Sense

You’ll always have access to your data via our API. Take it, play with it, graph it, do whatever you want with it. It’s yours. That’s important to us.

We enjoy tinkering with and building on-top of other products we like. Sense will let you have that experience.

We’d love to hear your thoughts on what you might want to build with Sense, and how you could directly interact with the hardware, and the data it collects.

Sounds great! But a year after shipping, there’s no sign of an API, and some of us who enjoy tinkering are getting a bit restless…

Reverse engineering to the rescue

Data from the Sense hardware is transmitted off onto the Internet somewhere, and then pulled down by the mobile apps. If we can snoop that traffic, we can probably figure out how to grab the data from the Sense.

So the first step is to capture the raw network traffic while the Sense and the app are in use. I crafted an extremely over-the-top setup that involves my phone and the Sense being on a new, separate wireless network that gateways through a Linux server that can look in on all the traffic (because you never know when a dedicated capturing network will come in handy, right?…)

Running tshark while the app and device are active showed they were making HTTPS requests an Amazon Elastic Cloud instance. That’s not really much use, as we can’t see the content of the requests. The next step is to move up the network stack and target the HTTPS traffic specifically. Step in, mitmproxy. This automates man-in-the-middle attacks on HTTPS requests. When it receives such a request, it cooks up its own certificate and sends that to the client, and then sits in between while the client and the server communicate.

The certificates generated by mitmproxy aren’t issued by a trusted source, so (well behaved) browsers and apps will throw up lots of warnings and refuse to communicate with it for the most part. This is the mechanism that stops anyone impersonating your banking website when you access it over HTTPS, so is generally a very good thing. In this case, both the Sense itself and the Android app refused to talk with the blatant attacker.

We control the Android phone, though, so can just tell it that we trust the dodgy certificate issuer. This involves grabbing the certificate generated by mitmproxy, and adding it to Android’s key store. The process is documented nicely in the mitmproxy docs and is pretty straightforward. With Android trusting our certificate authority, the Sense app starts talking through the proxy and we can look at the plain HTTP requests.

The API

Logging in to the Android app and navigating through the various screens shows that the API is actually pretty nice. It’s RESTful, and uses an OAuth bearer token for authorisation, as you can see in the mitmproxy output of the authentication and first few requests:

POST https://api.hello.is/v1/oauth2/token
    ← 200 application/json 151B 1.17s
GET https://api.hello.is/v2/account/preferences
    ← 200 application/json 124B 109ms
GET https://api.hello.is/v1/account
    ← 200 application/json 181B 388ms
GET https://api.hello.is/v2/devices
    ← 200 application/json 235B 126ms
POST https://api.hello.is/v1/app/checkin
    ← 200 application/json 106B 119ms
GET https://api.hello.is/v2/insights
    ← 200 application/json 1.4kB 157ms
POST https://api.hello.is/v1/notifications/registration
    ← 204 [no content] 364ms
GET https://api.hello.is/v2/timeline/2016-04-09
    ← 200 application/json 160B 550ms
GET https://api.hello.is/v1/questions?date=2016-04-10
    ← 200 application/json 284B 132ms
PATCH https://api.hello.is/v1/app/stats
    ← 202 [no content] 126ms
GET https://api.hello.is/v1/app/stats/unread
    ← 200 application/json 71B 171ms

Authorisation

To get a bearer token, you send a POST request to /v1/oauth2/token, supplying the username and password, and a client ID and secret. Presumably if the API is ever opened up you’ll be able to register clients and get your own ID and secret, but for now reusing the ones from the Android app works fine. [I’ve prettified and linewrapped the content to make it a bit easier to read.]

POST /v1/oauth2/token HTTP/1.1
Host: api.hello.is
Accept: */*
Content-Length: 123
Content-Type: application/x-www-form-urlencoded

grant_type=password
 &client_id=8d3c1664-05ae-47e4-bcdb-477489590aa4
 &client_secret=4f771f6f-5c10-4104-bbc6-3333f5b11bf9
 &username=USERNAME
 &password=PASSWORD

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:25:16 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "token_type":"Bearer",
  "expires_in":31536000,
  "account_id":"1234",
  "access_token":"2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "refresh_token":"2.yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Sleep timeline

Getting the timeline for one night is just a straight up GET request. It gives a detailed log of events, including different ‘depths’ of sleep (I’ve cut out a whole bunch here for simplicity). It also shows metrics relating to the entire night, and their condition (‘warning’ or ‘ideal’).

GET /v2/timeline/2016-04-08 HTTP/1.1
Host: api.hello.is
Accept: */*
Authorization: Bearer 2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 10 Apr 2016 17:29:00 GMT
transfer-encoding: chunked
Connection: keep-alive

{
  "score":76,
  "score_condition":"WARNING",
  "message":"You were asleep for **8.2 hours**, and sleeping soundly for 3.2 hours.",
  "date":"2016-04-08",
  "events":[
    {
      "timestamp":1460168700000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You went to bed.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"GOT_IN_BED",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460169600000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You fell asleep.",
      "sleep_depth":100,
      "sleep_state":"SOUND",
      "event_type":"FELL_ASLEEP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    },{
      "timestamp":1460177040000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"You were moving around quite a bit.",
      "sleep_depth":2,
      "sleep_state":"AWAKE",
      "event_type":"GENERIC_MOTION",
      "valid_actions":["VERIFY","INCORRECT"]
    },{
      "timestamp":1460199000000,
      "timezone_offset":3600000,
      "duration_millis":60000,
      "message":"Good morning.",
      "sleep_depth":0,
      "sleep_state":"AWAKE",
      "event_type":"WOKE_UP",
      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
    }
  ],
  "metrics":[
    { "name":"total_sleep", "value":490, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"sound_sleep", "value":190, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"time_to_sleep", "value":15, "unit":"MINUTES", "condition":"IDEAL" },
    { "name":"times_awake", "value":2, "unit":"QUANTITY", "condition":"IDEAL" },
    { "name":"fell_asleep", "value":1460169600000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"woke_up", "value":1460199000000, "unit":"TIMESTAMP", "condition":"IDEAL" },
    { "name":"temperature", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"humidity", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"particulates", "value":null, "unit":"CONDITION", "condition":"WARNING" },
    { "name":"light", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
    { "name":"sound", "value":null, "unit":"CONDITION", "condition":"IDEAL"}
  ]
}

Other interesting resources

Pretty much everything displayed in the app is available as a REST resource. They clearly put some thought into API design, which makes it even more of a shame that they haven’t made it public yet. Some of the other interesting resources are:

/v1/alarms: the alarms set on the Sense.
/v1/room/current?temp_unit=c: gives the latest environmental readouts from the Sense.
/v1/room/all_sensors/hours?quantity=2&from_utc=<timestamp>: sensor history.
/v2/devices: the paired devices, including battery status of pills.
/v2/insights: JSON representation of the cards displayed on the main screen (recommendations, analysis, etc).
/v2/trends/LAST_WEEK (or MONTH or 3_MONTHS): gives graph data for sleep scores, duration and depths.

Now the API is figured out, all that’s left is to build something that uses it…