Chameth.com - posts like android-espresso-spoon, http2-and-sni, monitoring-power-with-wemo, why-you-should-be-using-https

Home Automation Without the Megacorps

2025-05-21T00:00:00Z

I first experimented with home automation in 2016, by picking up a Samsung “SmartThings” hub. It was terrible. The UI to configure things was slow and clunky, firmware updates were applied whether you wanted them or not, and everything stopped working if their cloud services stopped. You were also locked into whatever integrations they deigned to support, of course. After that broke for the umpteenth time I scaled back and for years the closest I got to home automation was a couple of Hue bulbs.

Recently I’ve been building it out again, though. This time using off-the-shelf components that interop using Zigbee, open-source software, and some code I wrote myself. It’s great; it runs entirely locally and has had basically zero downtime. The Zigbee ecosystem lets me integrate all sorts of things without having to spend lots of money on “smart” alternatives. I think I’ve spent less on this incarnation than I did on the original SmartThings hub all those years ago (even without adjusting for inflation!).

My current setup

I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based on the CC2652P chipset¹. Interfacing with the Zigbee stack is handled by zigbee2mqtt (z2m for short), an open-source project that basically bridges your devices to an MQTT broker². When a device reports some data, it will send a new message over MQTT; when you want to make a device do something you just post a message back. It’s incredibly lightweight, but supports a huge array of devices out of the box. And as it’s just using MQTT, it’s trivial to integrate with other software or build on top of.

I know a lot of people building out this kind of home automation use Home Assistant, but I don’t get on with it terribly well. It felt incredibly sluggish, and the entire project is just a bit… much. I want a binary or package or docker image I can just run, not an entire operating system. That’s not how computers are meant to work! In contrast, z2m is simple to set up, super light weight and responsive.

Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond this now, and I’m skipping some boring bits like error handling and JSON parsing, but at first I had something like:

for {
    message, topic, err := c.ReadSlices()
    
    if string(topic) == "zigbee2mqtt/desk-button" {
        if strings.Contains(message, "single") {
            c.Publish(nil, []byte("{\"state\": \"TOGGLE\"}"), "zigbee2mqtt/room-lights/set")
        }
    }
}

I think I actually made this harder to read by trying to simplify it here, but hopefully you can follow that it was relatively straight forward to listen for a particular action to happen (in this case me “single”-pressing on the device called “desk-button”) and then make another device do something in response (toggling the state of the “room-lights” device).

So what actually are these devices? At present in z2m I have the following:

Some generic buttons
An air quality sensor (USB powered), and a separate temperature sensor (battery powered)
A light switch
Some 240V relay modules, that I use to turn non-smart devices on and off
Some USB relay modules, for the same purpose
Some “smart” plugs that I mostly use to monitor power usage
A motorised blind roller

I think all of these are either Tuya or Moes branded, and were all sourced from AliExpress. You can get the same things from Amazon or elsewhere, but they tend to be 3-4x more expensive for the same product.

One of the cool things about Zigbee devices is that the powered ones work together to create a mesh network, so you don’t have to worry about network repeaters or signal strength like you do with Wi-Fi networks. z2m even makes a map showing the connections:

The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.

Some interesting automations

A Zigbee relay spliced into the power cable for the fan

So what do I actually do with all of this? Some of it is just what I’ll call “laziness automation”: I want to be able to turn on the lights without getting up and walking to the light switch. So there’s a Zigbee button on my desk that does it. Then there are some less manual automations: my blinds are automatically closed at sunset³, for example. It feels a bit trivial, but it’s surprisingly nice just not to have to think about that at all.

Another nice quality of life automation is the light I have above my monitor. It’s USB powered, and I now have it going through a Zigbee USB relay that lets it be toggled on and off. Then I have a small agent running on my desktop that turns the light on when the computer is unlocked, and off when it locks or shuts down. I keep meaning to make a “film mode” that detects when I’m watching a film, and turning all the lights and blinds down, but I haven’t got around to it yet.

The most complex automation is probably for a window fan. It’s not smart in any way, so I cut the power cable and inserted a Zigbee relay. The whole thing is housed in a little project box to keep it secure. The relay basically acts as a switch: the live wire leading to the fan runs to the “normally open”⁴ contact, while the upstream power goes to “common”. When the relay is told to turn the fan on, the two get connected and the fan gets power. Oh, it’s also not actually a window fan, but I designed and 3D printed some adapters so that it sits right in the window.

So now we have a window fan that can be turned off and on automatically. But I don’t want to have to press a button to do that. I’m lazy, remember? Instead, I made it so that the Go code constantly monitors the temperature reported by the air quality sensor, and queries the free OpenWeatherMap API to get the rough temperature outside. It can then turn the fan on if the room is too hot, and outside is cool enough to make a difference (there’s no point in blowing hotter air in!).

A graph of room temperature, showing the effect of the fan being turned on and off

I don’t want it to do that all the time though. The room doesn’t need to be cooled if I’m not around. There are various Zigbee presence sensors you can get, but the cheaper ones seem to be of dubious quality. Instead, I wrote some code to guess whether I’m present. It uses the state of the monitor light as a proxy for “is the computer in use”, and then makes some guesses based on the last time the computer was used and the current time (if I turn the computer off at midnight it probably means I’m going to sleep; if I turn it off at 2pm it probably means I’m going out somewhere).

You can see in the graph that the fan has a decent impact on temperature. I coded it to cool the room to 19 degrees, but then not turn on again until it was above 20. This prevents it flip-flopping on and off constantly. You can clearly see the pattern in the graph, as it actively cools and then slowly the room warms back up. This arrangement is much better than being woken up at 5am because it’s painfully cold. Trust me.

Bonuses: metrics and 3D printer

One of the things my custom Go code does is collate all the various stats reported by the Zigbee devices, and inserts them into a VictoriaMetrics database. I originally hosted this on the Pi itself, and it performed fine, but I’ve since moved it onto a server so that I can use it for some other things as well.

I set up Grafana to point to VM, and can create dashboards showing power usage, what devices are turned on when, and a bunch of environmental conditions. This also makes it easy to spot how good the data coming from the devices are. For example, the air quality sensor reports a figure for the amount of Carbon Dioxide in the air, as well as the amount of Volatile Organic Compounds (VOCs). The graphs are basically identical, but on a different scale. It turns out this particular device has no actual way of detecting CO₂, so it just “calculates” it from the VOCs figure. Useful to know if you want to actually use that data.

Recently I came across a go library for interacting with Bambu Lab printers, so I’ve also hooked that into my automation. It exports metrics about the printer, so I now have a way of seeing what’s going on when I’m not physically present⁵. I even added a HTTP endpoint (exposed over Tailscale) that shows pictures from the built-in camera.

On the automation front, I’ve made it automatically turn off the printer’s light when it finishes a print, and also added push notifications via PushOver whenever the state changes. No longer will I be sat in another room blissfully unaware it ran out of filament seconds after I walked out the door!

Was it worth it?

These days you could probably just buy some kind of ‘smart home’ hub that works well enough, and do most of what I’ve done with a lot less effort and no coding. I’m still of the opinion that for something so essentially local, it should itself be managed entirely locally. I don’t trust companies like Google or Amazon not to kill their products, or change or remove an API I rely on.

Obviously the “Not Invented Here” approach of coding everything yourself doesn’t suit everyone, but as someone who enjoys coding and enjoys having things work just so it works very well for me.

It turns out the chipset is important. I initially used a similar Sonoff dongle that used a CC2531 chipset, and its performance wasn’t great. It often failed to pair with new devices, and dropped links to existing ones. The zigbee2mqtt docs do explicitly advise against the CC2531 chips for that reason. At the time they recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d go with whatever their latest recommendation was. ↩︎
MQTT is basically a network protocol for publishing and subscribing to arbitrary messages. A broker is the thing that sits in the middle and routes the messages. I use Mosquitto but any will do. ↩︎
They originally also re-opened at sunrise but that was a terrible mistake. Who knew the sun rose so early?! ↩︎
“open” meaning “there’s a gap so it doesn’t work” not “open for business”. This confusion in terminology also extends to drawbridges. ↩︎
I use the printer in LAN mode, which means there’s no way to monitor it from a phone, even if you’re connected to the same network still. ↩︎

HTTP/2 and TLS Server Name Indication

2024-05-25T00:00:00Z

I was recently alerted to a bug in Centauri, a simple reverse proxy I wrote. The initial report was that it was serving completely the wrong website, but only sometimes, and it behaved differently in different browsers, and no-one else could reproduce it.

I use Centauri for all of my web-facing services (including this site!) so it’s a little surprising such a major bug would have escaped my notice. Shane, who first noticed the bug, was persistent though and eventually managed to figure out some exact reproduction steps.

A brief overview of Centauri and SNI

Centauri originally only proxied HTTPS requests¹. When it receives a HTTPS request, it first looks at the Server Name Indication (or SNI) field in the TLS ClientHello message. It uses this field to determine which TLS certificate to respond with (as one Centauri instance will typically serve many websites across many domain names, each with their own certificate). That’s what the field exists for: before SNI, if you wanted to host two HTTPS sites on the same machine you’d need separate IP addresses for them!

Once the TLS session was established Centauri would read in the HTTP request, select which backend it was going to be sent to based on the SNI field, and then proxy it on. The HTTP request itself contains a Host header which identifies which host the request is for, but that will always be the same as the SNI field… or so I thought.

HTTP connection reuse

When accessing a website, your browser will request dozens of resources in a short space of time: the webpage itself, some stylesheets, maybe some scripts, plus any images, fonts, videos, etc. It would be extremely inefficient to open a new connection for each individual request, as setting up the connection requires several round trips between the client and the server.

To address this issue, HTTP/1.1 formalised the idea of “persistent connections”, which allow the client to keep a connection open and send another request once the first has completed. HTTP/2 takes this a step much further and allows full multiplexing — sending multiple requests at once and allowing the server to respond out-of-order.

Obviously, you can only reuse the connection if you’re requesting further resources from the same host: if your browser makes a request to example.com and that includes a script from example.net, it has to open a new connection for the other domain. However, HTTP/2 expands this slightly:

For “https” resources, connection reuse additionally depends on
having a certificate that is valid for the host in the URI. The
certificate presented by the server MUST satisfy any checks that the
client would perform when forming a new TLS connection for the host
in the URI.

Putting it all together

The reproduction steps that Shane figured out involved visiting sites hosted on two subdomains. The first site to be visited got “stuck” and subsequent requests to the other site were routed there instead. This only worked for one specific domain, though, and it turns out because that domain was configured in Centauri to use a wildcard TLS certificate (i.e., the certificate served for the request to the first site was also valid for the second site).

The certificate being valid for both sites allowed the browser to use the same connection. This breaks my assumption that the SNI field would always match the HTTP host, as all requests are sent over the same TLS connection that had the SNI field set to the first site’s subdomain. While perfectly in spec, the behaviour is quite surprising.

The fix for this was trivial: Centauri now checks the HTTP Host header instead of routing based on the SNI field. I found the bug itself interesting though, as it has such an awkward set of conditions for it to occur:

There must be multiple sites that share a certificate (the default behaviour in Centauri is to obtain one certificate per site)
A user must visit two of those sites
The browser must still have a connection open to the first site when visiting the second

It’s also one of those rare bugs where everything is working as intended, it’s just that the intention was slightly wrong for some reason. In this case it was because I wasn’t aware of the fairly significant shift in behaviour introduced in HTTP/2 for that one tiny part of the spec².

Thanks again to Shane for the debugging he did to figure this all out!

It now also proxies HTTP requests but only if they come over a Tailscale connection. Otherwise, plain HTTP requests are redirected to HTTPS. ↩︎
I think it’s this kind of thing that drives software devs to become carpenters or farmers. You don’t suddenly get a Door 2.0 specification that invalidates all your assumptions about how hinges work when certain people try to open it. ↩︎

Android testing with Espresso and Spoon

2017-05-16T00:00:00Z

I’ve been spending some time recently setting up automated testing for our collection of Android apps and libraries at work. We have a mixture of unit tests, integration tests, and UI tests for most projects, and getting them all to run reliably and automatically has posed some interesting challenges.

Running tests on multiple devices using Spoon

Spoon is a tool developed by Square that handles distributing instrumentation tests to multiple connected devices, aggregating the results, and making reports.

As part of our continuous integration we build both application and test APKs, and these are pushed to the build server as build artefacts. A separate build job then pulls these artefacts down to a Mac Mini we have in the office, and executes Spoon with a few arguments:

java -jar spoon-runner.jar \
    --apk application.apk \
    --test-apk applicationTests.apk \
    --fail-on-failure \
    --fail-if-no-device-connected

Spoon finds all devices, deploys both APKs on them, and then begins the instrumentation tests. We use two physical devices and an emulator to cover the form factors and API versions that are important to us; if any test fails on any of those devices, Spoon will return an error code and the build will fail.

For library projects, you only have a single APK containing both the tests and the library itself. The current version of Spoon requires both --apk and --test-apk to be specified, so we simply pass in the same APK to both. It looks like future versions of Spoon will be more flexible in this regard.

Spoon produces HTML reports, showing the status of each test run on each device. We have the report output folder collected as a build artefact, so the reports can be seen right from the build server:

Spoon output summary, showing results of 171 tests run on 3 devices

Flake-free UI testing with Espresso

Espresso is an Android library that provides an API for interacting with and making assertions about the UI of Android applications. Espresso has a very simple interface, and does lots of clever things under the hood to ensure that your tests only execute code when the UI is idle (and hence stable). You shouldn’t ever need to make your code sleep or wait.

In order for Espresso’s magic to work, it needs to know whenever some background activity is going on that it needs to wait for. By default, it hooks in to Android’s AsyncTask executor so it can wait for those to finish. In our apps, there were a few cases where we used an explicit Thread to do some background work, which caused tests to fail intermittently (depending on whether the thread performed its UI update before or after Espresso executed the test code). Rewriting these cases to use an AsyncTask enabled Espresso to figure out what was happening and the tests started passing reliably.

Another, similar, problem occurred where we were using RxJava to load some data. There are two possible ways to deal with this… Espresso has a concept of an idling resource, which provides a way of telling Espresso when a resource is busy so it can belay interacting with or testing the UI until the resource is finished. In our case, the code in question was going to be rewritten soon, so we went for a quicker and dirtier option: force RxJava to use the same executor as AsyncTask.

To do this, we added a simple test utility class that registers an RxJavaPlugin that overrides the Schedulers used by Rx:

/**
 * Hooks in to the RxJava plugins API to force Rx work to be scheduled on the
 * AsyncTask's thread pool executor. This is a quick and dirty hack to make
 * Espresso aware of when Rx is doing work (and wait for it).
 */
public final class RxSchedulerHook {

    private static final RxJavaSchedulersHook javaHook =
            new RxJavaTestSchedulerHook();

    private RxSchedulerHook() {
        // Should not be insantiated
    }

    public static void registerHooksForTesting() {
        if (RxJavaPlugins.getInstance().getSchedulersHook() != javaHook) {
            RxJavaPlugins.getInstance().reset();
            RxJavaPlugins.getInstance().registerSchedulersHook(javaHook);
        }
    }

    private static class RxJavaTestSchedulerHook extends RxJavaSchedulersHook {
        @Override
        public Scheduler getComputationScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getIOScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getNewThreadScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }
    }
}

With the hook registered Rx does all of its work on the same thread pool as AsyncTask, which Espresso already knows about. It’s not the best long-term solution, but it means we don’t have to spend time integrating IdlingResource for code that doesn’t have long to live. With the hook in place, the tests that were flaking because of Rx started passing reliably as well.

Getting automatic screenshots of failures

Spoon provides a client library to, among other things, take a screenshot of the device. Espresso provides a hook that can be used to change how errors are handled. Putting the two together is very simple:

final FailureHandler defaultHandler =
        new DefaultFailureHandler(
                InstrumentationRegistry.getTargetContext());

Espresso.setFailureHandler(new FailureHandler() {
    @Override
    public void handle(Throwable throwable, Matcher<View> matcher) {
        try {
            Spoon.screenshot(
                    getActivity(),
                    "espresso-failure",
                    description.getClassName(),
                    description.getMethodName());
        } catch (Exception ex) {
            Log.e(TAG, "Error capturing screenshot", ex);
        }
        defaultHandler.handle(throwable, matcher);
    }
});

In our new error handler we simply ask Spoon to take a screenshot, then call Espresso’s original handler so that it can output its debugging information and fail the test. The Spoon runner automatically picks up the screenshot and adds it to the report:

Spoon output details, showing a screenshot captured of the failure

Having the screenshot, error message and logs all presented in a clean UI makes debugging failures much, much easier than searching through a huge build log to try and find the exception.

Filtering tests based on device capabilities

Some tests won’t work on every device you throw at them. We had two problems: some tests require a higher API version than some of our devices, and some of the UIs under test were designed to run only on certain resolutions.

The main cause for our dependence on newer API versions was the use of WireMock, a brilliant library for stubbing out web services. WireMock requires API 19, while our physical devices tend to run versions older than that. Stopping these tests running is simply a case of applying an annotation to the class:

@SdkSuppress(minSdkVersion=19)

Screen resolution is a bit more complicated. One of our apps is designed for a specific tablet device (and will never be used on anything else), and trying to render the UI on smaller screens results in items overlapping, important parts ending up below the fold, and other problems.

We’d still like all the other tests for that app to run on all of the devices, though, so we can test them on a variety of API versions and in other conditions. We just need to suppress the UI tests. To do this, we subclassed the Android ActivityTestRule and overrode the apply method:

@Override
public Statement apply(Statement base, final Description description) {
    if (!canRunUiTests(InstrumentationRegistry.getContext())) {
        // If we can't run UI tests, then return a statement that does nothing
        // at all.  With normal JUnit tests we'd just throw an assumption
        // failure and the test would be ignored, but that makes the Android
        // runner angry.
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                // Do nothing
            }
        };
    }

    return super.apply(base, description);
}

/**
 * Checks that the screen size is close enough to that of our tablet device.
 */
@TargetApi(Build.VERSION_CODES.HONEYCOMB_MR2)
private boolean canRunUiTests(Context context) {
    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.HONEYCOMB_MR2) {
        return false;
    }
    int screenWidth = dpToPx(context,
            context.getResources().getConfiguration().screenWidthDp);
    return screenWidth >= 600;
}

private int dpToPx(Context context, int dp) {
    DisplayMetrics displayMetrics = context.getResources().getDisplayMetrics();
    return Math.round(dp * displayMetrics.density);
}

When the rule is run, we check if the screen width meets a minimum number of pixels. If it doesn’t, an empty Statement is returned that turns the test into a no-op. These show up as passed rather than skipped in the output, but there doesn’t seem to be a nice way to signal to the Android JUnit runner that the test is being ignored programatically.

Why you should be using HTTPS

2016-06-17T00:00:00Z

The EFF’s HTTPS Everywhere logo

One of my favourite hobbyhorses recently has been the use of HTTPS, or lack thereof. HTTPS is the thing that makes the little padlock appear in your browser, and has existed for over 20 years. In the past, that little padlock was the exclusive preserve of banks and other ‘high security’ establishments; over time its use has gradually expanded to most (but not all) websites that handle user information, and the time is now right for it to become ubiquitous.

Why use HTTPS?

There are numerous advantages to using HTTPS, both for the users of a website and for the operator:

Privacy

The most obvious advantage is that HTTPS gives your users additional privacy. An insecure (HTTP) request can potentially be read by anyone on the same network, or the network operators, or anyone who happens to operate a network along the path between the user and the server.

Users on shared WiFi networks (such as those in coffee shops, hotels, or offices) are particularly vulnerable to passive sniffing by anyone else on that network. If the network is open (as is frequently the case) then anyone in radio range can see exactly what the user is up to.

Integrity

HTTPS also helps to maintain the integrity of your site. With a plain HTTP request, there’s nothing to stop anyone in between the server and the user from modifying the content of the request or the response. This is a frequent tactic used by annoying WiFi gateways (such as the ones you’d find in a hotel), dubious ISPs who want to serve you extra adverts, or just plain old nefarious attackers.

If you’re trying to convey some kind of information to users (and if you aren’t, why exactly are you running a website again?) it seems beneficial to both you and them if the information arrives as you intended, rather than in a modified form due to someone or something tampering with it.

Security

If your website has any kind of authentication, or session identifiers, it becomes extremely vulnerable to an attacker monitoring the traffic and stealing the credentials. This was starkly demonstrated in 2010 when Firesheep was released. This tool allowed anyone to quickly and automatically hijack social media accounts of anyone on the same network who was using HTTP to access them.

Even if your login pages are served over HTTPS, if you send a single session ID cookie over HTTP (such as a page you decided wasn’t particularly ‘important’) then an attacker can probably spoof the user’s session and gain full access to their account. Again, in the case of open WiFi networks that could be anyone in radio range.

Search engine rankings

Some search engines use HTTPS as a signal in their ranking algorithms. Google announced in 2004 that it was using the presence of HTTPS as a small positive signal, but that it may strengthen that signal over time as more and more websites switch to using a secure transport. It’s not unthinkable that at some point in the future there will be HTTPS-only search engines.

But… But… But…

There are lots of excuses for not implementing HTTPS. Most of them are either misguided or outdated.

It’s too expensive and/or complicated

In the past, getting HTTPS certificates was a pain. A number of free suppliers have existed for a while but the process for getting their certificates wasn’t particularly straight forward, and many imposed arbitrary restrictions on the certificate parameters. Even once you had the certificate, you had to fiddle about with your HTTP server configuration to make it work, remember to manually get a new certificate when the old one expired, and lots of other annoying busywork.

With the arrival of Let’s Encrypt, all that changed. You can retrieve and deploy a free HTTPS certificate with two or three commands. Renewal can be handled completely automatically with a single command executed by cron.

There’s no point; nothing on my site is sensitive

You might not think your content warrants privacy, but can you speak for everyone who accesses it? Even content that seems mundane to you — such as travel advice, or technical writing — could be used to build up a profile of a user. If an attacker is monitoring traffic in a coffee shop and sees a user looking at travel advice and weather forecasts for a foreign country, he could use that information to plan a burglary knowing that the user will be away. Similarly, some content which is perfectly mundane to you may actually be very sensitive in other countries with repressive governments. HTTPS makes it much harder for these people to snoop on traffic.

From another angle, if you’re offering any kind of information, instructions, or especially file downloads, there’s a severe risk to users if the content is modified on its way to them. An evil sysadmin could rewrite your travel advice to suggest visiting the local drug dealer’s hangout, or replace your download with a malware-infested version.

HTTPS is slower, uses more resources, etc

Back in 1995 this might have been a valid argument. Enabling HTTPS on a modern server will make an almost negligible difference to performance. If you also enable HTTP/2 (which most implementations only support over HTTPS), it’s likely to actually use fewer resources, and result in a faster, smoother experience for your users. HTTP/2 was designed to work with HTTPS, and designed with modern requirements and networking techniques in mind.

CloudFlare have an excellent demonstration of the benefits of HTTP/2, and it can show speed improvements of 2-3x in a typical environment. On top of being faster, HTTP/2 uses fewer connections which results in less resource overhead on both the server and the client.

So what are you waiting for?

If you run a website and aren’t using HTTPS, give it a try.

Monitoring power draw with WeMo Insight Switches

2016-05-02T00:00:00Z

A WeMo Insight Switch

I recently picked up a couple of Belkin’s WeMo Insight Switches to monitor power usage for my PC and networking equipment. WeMo is Belkin’s home automation brand, and the switches allow you to toggle power on and off with an app, and monitor power usage.

The WeMo Android app is pretty dismal. It’s slow, doesn’t look great, and crashed about a dozen times during the setup process for each of my two switches. It also doesn’t provide much information at all about power: you can see average power draw and current power draw, and that’s basically it.

Belkin has provided an option to e-mail yourself a spreadsheet with historical power data, and can even do it on a regularly scheduled basis, but that’s not really a nice solution if you want up-to-date power stats. Even if you were happy with data arriving in batch, having to get hold of an e-mail attachment and parse out a weirdly formatted spreadsheet doesn’t make for easy automation. It also relies on Belkin supporting the service indefinitely, which isn’t necessarily going to happen.

Enter the SOAP API

Fortunately, the devices themselves expose an API to get the data. Once a switch is setup with the WeMo app, it connects to a WiFi network. You can discover WeMo devices on the network using a UPnP broadcast, and then from there a list of supported services. Each service has a number of SOAP actions, with nice obvious names, and parameters documented in the XML service description.

For the insight switches, there’s a service at /upnp/control/insight1 which has a number of actions including GetPower and GetTodayKWH. Sending a SOAP request isn’t too difficult (albeit nowhere near as nice as a REST JSON API) — here’s a sample request I made using the DHC chrome extension:

POST /upnp/control/insight1 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.8
Content-Type: text/xml
Origin: chrome-extension://aejoelaoggembcahagimdiliamlcdmfm
SOAPACTION: "urn:Belkin:service:insight:1#GetPower"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.29 Safari/537.36

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
    </s:Body>
</s:Envelope>

The name of the action (in the SOAPACTION header) and the XML namespace in the body are both constructed from information in the service definition. The result that comes back is:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPowerResponse xmlns:u="urn:Belkin:service:insight:1">
            <InstantPower>92170</InstantPower>
        </u:GetPowerResponse>
    </s:Body>
</s:Envelope>

The current power draw in milliwatts is returned in the <InstantPower> argument.

Putting the data to work

So now we have a way to pull the current power draw, a nice thing to do would be to plot a graph of it to see the variation over time. This is the short of thing you’d expect to see in the Belkin app, but it’s sadly missing. After a bit of research I settled on the tried and true rrdtool to store data and generate graphs. I created a new database to store the values from the two switches:

rrdtool create power.rrd \
    --start now \
    --step 60 \
    DS:wemoComputer:GAUGE:120:U:U \
    DS:wemoNetworking:GAUGE:120:U:U \
    RRA:AVERAGE:0.5:1:1440 \
    RRA:AVERAGE:0.5:10:1008 \
    RRA:AVERAGE:0.5:30:1488 \
    RRA:AVERAGE:0.5:120:1488 \
    RRA:AVERAGE:0.5:360:1488 \
    RRA:AVERAGE:0.5:1440:36500

This creates a database file which expects values to be given every 60 seconds for two data series: ‘wemoComputer’ and ‘wemoNetworking’. These are gauge types (i.e., a value we read off a gauge, rather than a counter that keeps going up) with no minimum and maximum (‘U’).

It then defines a series of round-robin archives. Each of these stores a fixed number of entries, and the oldest is overwritten when they become full. The interesting arguments are the step count (second to last) and number of samples (last argument). The first one takes every sample and keeps 1,440 of them (i.e., a full day at 1-minute resolution); the last one has a step of 1,440 and keeps 36500 of them (i.e., 100 years at 1-day resolution). The use of RRAs ensures that rrdtool can retain enough data to create historical graphs, while providing a fixed, finite maximum file size. With sufficient RRA coverage you can graph most timescales and have data available at a reasonable resolution.

Next, I created a small python script to retrieve the power using SOAP:

#!/usr/bin/python3

import requests
from xml.etree import ElementTree

def get_power(ip_and_port):
    headers = {'Content-type': 'text/xml', 'SOAPACTION': '"urn:Belkin:service:insight:1#GetPower"'}
    payload = '''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
                             s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
                  <s:Body>
                   <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
                  </s:Body>
                 </s:Envelope>'''
    r = requests.post("http://%s/upnp/control/insight1" % ip_and_port, headers=headers, data=payload)
    et = ElementTree.fromstring(r.text)
    return et.find('.//InstantPower').text

I then have a dictionary of IP addresses to data series names, and the script polls each one in turn and then executes an rrdtool update query to add the items to the database. I have this script running every minute via cron.

After leaving the script to run for a bit and gather data, it’s time to make some graphs. I use the following to create a graph with a background gradient:

rrdtool graph desk-1d.png \
        -o -X0 -w800 -h500 \
        -u 2000 -l 20 -r \
        DEF:raw=power.rrd:wemoComputer:AVERAGE \
        CDEF:power=raw,1000,/ \
        CDEF:powerz=power,2000,LT,power,2000,IF CDEF:powerzNoUnk=power,UN,0,powerz,IF AREA:powerzNoUnk#ff0000 \
        CDEF:powery=power,1500,LT,power,1500,IF CDEF:poweryNoUnk=power,UN,0,powery,IF AREA:poweryNoUnk#ff0000 \
        CDEF:powerx=power,1000,LT,power,1000,IF CDEF:powerxNoUnk=power,UN,0,powerx,IF AREA:powerxNoUnk#ff0000 \
        CDEF:powerw=power,900,LT,power,900,IF CDEF:powerwNoUnk=power,UN,0,powerw,IF AREA:powerwNoUnk#ff0000 \
        CDEF:powerv=power,800,LT,power,800,IF CDEF:powervNoUnk=power,UN,0,powerv,IF AREA:powervNoUnk#ff1b00 \
        CDEF:poweru=power,700,LT,power,700,IF CDEF:poweruNoUnk=power,UN,0,poweru,IF AREA:poweruNoUnk#ff4100 \
        CDEF:powert=power,600,LT,power,600,IF CDEF:powertNoUnk=power,UN,0,powert,IF AREA:powertNoUnk#ff6600 \
        CDEF:powers=power,400,LT,power,400,IF CDEF:powersNoUnk=power,UN,0,powers,IF AREA:powersNoUnk#ff8e00 \
        CDEF:powerr=power,200,LT,power,200,IF CDEF:powerrNoUnk=power,UN,0,powerr,IF AREA:powerrNoUnk#ffb500 \
        CDEF:powerq=power,180,LT,power,180,IF CDEF:powerqNoUnk=power,UN,0,powerq,IF AREA:powerqNoUnk#ffdb00 \
        CDEF:powerp=power,160,LT,power,160,IF CDEF:powerpNoUnk=power,UN,0,powerp,IF AREA:powerpNoUnk#fdff00 \
        CDEF:powero=power,140,LT,power,140,IF CDEF:poweroNoUnk=power,UN,0,powero,IF AREA:poweroNoUnk#d7ff00 \
        CDEF:powern=power,120,LT,power,120,IF CDEF:powernNoUnk=power,UN,0,powern,IF AREA:powernNoUnk#b0ff00 \
        CDEF:powerm=power,100,LT,power,100,IF CDEF:powermNoUnk=power,UN,0,powerm,IF AREA:powermNoUnk#8aff00 \
        CDEF:powerl=power,90,LT,power,90,IF CDEF:powerlNoUnk=power,UN,0,powerl,IF AREA:powerlNoUnk#65ff00 \
        CDEF:powerk=power,80,LT,power,80,IF CDEF:powerkNoUnk=power,UN,0,powerk,IF AREA:powerkNoUnk#3eff00 \
        CDEF:powerj=power,70,LT,power,70,IF CDEF:powerjNoUnk=power,UN,0,powerj,IF AREA:powerjNoUnk#17ff00 \
        CDEF:poweri=power,60,LT,power,60,IF CDEF:poweriNoUnk=power,UN,0,poweri,IF AREA:poweriNoUnk#00ff10 \
        CDEF:powerh=power,50,LT,power,50,IF CDEF:powerhNoUnk=power,UN,0,powerh,IF AREA:powerhNoUnk#00ff36 \
        CDEF:powerg=power,40,LT,power,40,IF CDEF:powergNoUnk=power,UN,0,powerg,IF AREA:powergNoUnk#00ff5c \
        CDEF:powerf=power,30,LT,power,30,IF CDEF:powerfNoUnk=power,UN,0,powerf,IF AREA:powerfNoUnk#00ff83 \
        CDEF:powere=power,20,LT,power,20,IF CDEF:powereNoUnk=power,UN,0,powere,IF AREA:powereNoUnk#00ffa8 \
        CDEF:powerd=power,0,LT,power,0,IF CDEF:powerdNoUnk=power,UN,0,powerd,IF AREA:powerdNoUnk#00ffd0 \
        LINE:power#080

This seems a bit unweildy, but it’s fairly straight forward. The options tell rrdtool to create a graph with a canvas size of 800x500 pixels, a lower limit of 20W, upper limit of 2kW, and a log scale. The first CDEF divides our raw value (which was in millwatts) by 1000 to get a value in watts. rrdtool uses reverse polish notation like some calculators, so you provide the arguments before the operator.

The big block of CDEF/CDEF/AREA parameters creates a series of area plots in different colours according to the power level. They’re in descending order so the smaller areas are drawn on top of the larger layers. This results in a graph that looks like this:

Graph of power usage over a day

This graph shows the total power for all the things plugged in at my desk. You can see the idle power draw is around 60W. When I’m using the computer it jumps up to around 130W, and when the computer is under heavy load (playing games, for example) it goes up even further to the 200W mark. With a couple of small tweaks to the rrdtool command, I also have a graph showing the entire week:

Graph of power usage over a week

The two huge spikes near the start of the data are caused by a heater under my desk. They’re also one of the main reasons I chose to plot the graphs with a logarithmic scale. With a linear scale between 20W and 2kW, the graph would be completely dominated by these large spikes — in fact the difference in height between the two spikes would take almost half the graph! The relative difference between the normal values of 60-200W and the heater’s value of 1.8kW isn’t actually that interesting, and certainly not worth using about 80% of the graph to demonstrate. The log scale helps to compress this, and emphasises the difference in the smaller values more.