Chameth.com - posts like g15-ram-upgrade, monitoring-power-with-wemo, understanding-docker-volume-mounts

The Case of the Unchanging Config

2026-04-09T00:00:00Z

Last week I was attempting to make it so I could share pictures on IRC directly from my client. This sounds simple, but it involves a bouncer¹ that proxies the request to a standalone image hosting service that I had to modify to be compatible. At one point my testing loop was:

Make a change to the hosting service
Commit it
Tag a new release
Wait for it to build
Update the version I’m running on my server
Reconfigure the bouncer to pass the new parameter or change the URL or whatever
Try uploading a photo from my phone
Realise I’ve overlooked something and go back to step one

I could have set up a local copy of everything and tested it sensibly. I should have set up a local copy of everything and tested it sensibly. But it seemed like such a trivial change, and setting up the whole environment seemed like such a pain. After the third or so iteration of failure I was pretty annoyed with myself, computers, and basically everything.

My biggest annoyance was that my bouncer would not pick up the new URL from the config when I changed it. It’s meant to reload the config when it receives a SIGHUP, and it claimed to in the logs, but I could clearly see it was still hitting the old URL. Restarting the bouncer to update the config is a pain, as it disconnects me from all the IRC networks, and has to reconnect to them all, reauthenticate, etc. It also mildly spams everyone who shares a channel with me².

When I finally got everything working I had a look at the bouncer source, and thought I’d spotted the issue. I raised a bug report, ending in this remark:

It looks like the config is reloaded properly, but the handler for uploads is created once at startup and has its own copy of the uploader, so effectively snapshots the config to whatever it is at startup:

fileUploadHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    cfg := srv.Config()
    h := fileupload.Handler{
        Uploader:    cfg.FileUploader,
        DB:          db,
        Auth:        cfg.Auth,
        HTTPOrigins: cfg.HTTPOrigins,
    }
    h.ServeHTTP(w, r)
})

I’m not quite sure what I thought I saw there, but that’s not right. I want to call it a hallucination, but that feels like a very overloaded word these days. I guess I was just seeing what I wanted to see, and that was a good excuse to stop investigating. Still, even if I misidentified the cause, the bug was still present, right? … Right?

The Science of Deduction

When the developer got back and said he couldn’t reproduce my issue, I went back to my install and immediately reproduced it. With a calmer head, I figured it was probably something with my particular set up. First thing to check: am I actually running the version I think I am?

Unfortunately the bouncer doesn’t actually seem to expose the version anywhere that I can see. It’s not in the logs, it doesn’t have a -version flag, and none of the IRC-based status commands seem to include it. But I know it’s a Go app, and I know Go embeds the version information. go version -m <binary> will dump it all out, but the binary is inside a Docker image, and the Docker image is one of my nice, minimal, artisanal ones so doesn’t ship a go binary. No problem, docker compose cp bouncer:/bnc ./bnc yoinks the binary out of the container, and then dumping the version shows that, yes, I am running the version I thought I was. Hmm.

The next tool I reached for in my toolbox was strace. Maybe it’s not actually reading the file for some reason? I immediately executed strace -p <pid> -e trace=openat,open,read,pread64 -f without having to look any part of that up. Yep. Definitely. Then I edited the config, HUP’d the bouncer, and saw that it was… reading the config file. As it’s meant to. By default strace truncates strings to 32 bytes, so I couldn’t actually see the line I’d changed. Some more definitely-not-RTFMing later, and rerunning it with an extra -s 65536 let me see the full config. Surprise! The config hadn’t changed!

To confirm my findings, I used docker compose cp again, this time yoinking the config file from inside the container. The inside config file was definitely different to the outside config file. What? My hypothesis at this point was “something something Docker nonsense”. I mount the config as read-only, and was wondering if that meant that Docker was doing something weird instead of just bind mounting it. A quick trip to /proc/<pid>/mounts showed that it was, in fact, not doing anything weird, and was just bind mounting it.

If the file is bind mounted, then surely it’s the same file? I ran stat on the file on the host, noted the inode number, then pondered how to actually run stat on the file inside the container, given the aforementioned awkwardly minimal image. The solution was easy: access it via /proc/<pid>/root/. I could’ve saved myself a bunch of docker compose cp if I’d thought about that earlier. Oh well. The inode of that file was different. What?

The answer was DNS. It’s always DNS. Oh, sorry, force of habit. I meant the answer was vim. I was editing the config in vim, and when it saves files, by default, it writes the new content to a temporary file and does an atomic rename. That’s normally a good thing: it prevents corruption if the write fails midway through³. In this case, though, that means the new file has a new inode. Bind mounting a file binds to the inode, so the container just keeps clutching onto the original config from when it was started, blissfully unaware that the party has relocated down the street.

This problem is likely to happen whenever you bind mount a file into a container. When I mentioned this to a friend, he immediately responded “oh yeah, never do that”, and went on to describe the horrible hacks he’s had to add to Ansible to sidestep the issue. The nicer solution is to just bind mount an entire directory if you can, as then it doesn’t matter what happens to the files within it. I really like having the config files sat alongside the Docker compose files, though; having to create a directory just to work around some bind mount weirdness upsets me.

Now I knew what the problem was, I found there was an issue raised against Docker twelve years ago⁴. The first response was “that’s expected”, and I fully agree with the author: “respectfully, that might be expected by you, but it was not expected by me”!⁵ As a result of the issue they added a nice note to the docs:

Note: Many tools used to edit files including vi and sed --in-place may result in an inode change. Since Docker v1.1.0, this will produce an error such as “sed: cannot rename ./sedKdJ9Dy: Device or resource busy”. In the case where you want to edit the mounted file, it is often easiest to instead mount the parent directory.

But in the intervening twelve years, both the note and the functionality described have gone missing. Ho-hum.

I found it interesting how I’ve only just hit this problem, given how long I’ve used Docker. But I realised that almost everything I run I’m happy to just restart. Cattle, not pets, and so on. My IRC bouncer is one of the few exceptions to that. The only other thing I regularly hot reloaded was Centauri, my reverse proxy, but that had a whole config directory mounted because it was shared between containers, so nicely sidestepped the foot-gun.

So lesson learnt: check for weird bind mount issues before raising issues about config hot reloading. It’ll join the esteemed company of “maybe the drive is full and causing completely unrelated problems?”, “perhaps everything is dog slow because the kernel ran out of entropy?”, and “did systemd sneakily take over that functionality while you weren’t looking?” in the troubleshooting checklist.

An IRC bouncer is basically an always-on proxy. It connects to the IRC networks for you, then your clients connect to your bouncer. The bouncer can then send incoming messages to all your different clients, cache them when you’re offline and replay them later, and lots of other nice things people take for granted in their chat apps these days. ↩︎
Join/part/quit spam is part of IRC, and clients have ways of handling it, but I still feel bad about doing it excessively. ↩︎
You can disable this in vim by setting backupcopy=yes. Good luck redoing all the debugging if you ever accidentally remove that from your vimrc, though! ↩︎
Which is weird, because Docker can’t possibly be that old. That would make me much older than I’m prepared to accept. ↩︎
If you read the thread they didn’t actually mean to sound so dismissive, but it’s still pretty funny. “Yes, it’s expected that the foot-gun causes your foot to hurt. Duh. What did you expect?” ↩︎

Home Automation Without the Megacorps

2025-05-21T00:00:00Z

I first experimented with home automation in 2016, by picking up a Samsung “SmartThings” hub. It was terrible. The UI to configure things was slow and clunky, firmware updates were applied whether you wanted them or not, and everything stopped working if their cloud services stopped. You were also locked into whatever integrations they deigned to support, of course. After that broke for the umpteenth time I scaled back and for years the closest I got to home automation was a couple of Hue bulbs.

Recently I’ve been building it out again, though. This time using off-the-shelf components that interop using Zigbee, open-source software, and some code I wrote myself. It’s great; it runs entirely locally and has had basically zero downtime. The Zigbee ecosystem lets me integrate all sorts of things without having to spend lots of money on “smart” alternatives. I think I’ve spent less on this incarnation than I did on the original SmartThings hub all those years ago (even without adjusting for inflation!).

My current setup

I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based on the CC2652P chipset¹. Interfacing with the Zigbee stack is handled by zigbee2mqtt (z2m for short), an open-source project that basically bridges your devices to an MQTT broker². When a device reports some data, it will send a new message over MQTT; when you want to make a device do something you just post a message back. It’s incredibly lightweight, but supports a huge array of devices out of the box. And as it’s just using MQTT, it’s trivial to integrate with other software or build on top of.

I know a lot of people building out this kind of home automation use Home Assistant, but I don’t get on with it terribly well. It felt incredibly sluggish, and the entire project is just a bit… much. I want a binary or package or docker image I can just run, not an entire operating system. That’s not how computers are meant to work! In contrast, z2m is simple to set up, super light weight and responsive.

Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond this now, and I’m skipping some boring bits like error handling and JSON parsing, but at first I had something like:

for {
    message, topic, err := c.ReadSlices()
    
    if string(topic) == "zigbee2mqtt/desk-button" {
        if strings.Contains(message, "single") {
            c.Publish(nil, []byte("{\"state\": \"TOGGLE\"}"), "zigbee2mqtt/room-lights/set")
        }
    }
}

I think I actually made this harder to read by trying to simplify it here, but hopefully you can follow that it was relatively straight forward to listen for a particular action to happen (in this case me “single”-pressing on the device called “desk-button”) and then make another device do something in response (toggling the state of the “room-lights” device).

So what actually are these devices? At present in z2m I have the following:

Some generic buttons
An air quality sensor (USB powered), and a separate temperature sensor (battery powered)
A light switch
Some 240V relay modules, that I use to turn non-smart devices on and off
Some USB relay modules, for the same purpose
Some “smart” plugs that I mostly use to monitor power usage
A motorised blind roller

I think all of these are either Tuya or Moes branded, and were all sourced from AliExpress. You can get the same things from Amazon or elsewhere, but they tend to be 3-4x more expensive for the same product.

One of the cool things about Zigbee devices is that the powered ones work together to create a mesh network, so you don’t have to worry about network repeaters or signal strength like you do with Wi-Fi networks. z2m even makes a map showing the connections:

The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.

Some interesting automations

A Zigbee relay spliced into the power cable for the fan

So what do I actually do with all of this? Some of it is just what I’ll call “laziness automation”: I want to be able to turn on the lights without getting up and walking to the light switch. So there’s a Zigbee button on my desk that does it. Then there are some less manual automations: my blinds are automatically closed at sunset³, for example. It feels a bit trivial, but it’s surprisingly nice just not to have to think about that at all.

Another nice quality of life automation is the light I have above my monitor. It’s USB powered, and I now have it going through a Zigbee USB relay that lets it be toggled on and off. Then I have a small agent running on my desktop that turns the light on when the computer is unlocked, and off when it locks or shuts down. I keep meaning to make a “film mode” that detects when I’m watching a film, and turning all the lights and blinds down, but I haven’t got around to it yet.

The most complex automation is probably for a window fan. It’s not smart in any way, so I cut the power cable and inserted a Zigbee relay. The whole thing is housed in a little project box to keep it secure. The relay basically acts as a switch: the live wire leading to the fan runs to the “normally open”⁴ contact, while the upstream power goes to “common”. When the relay is told to turn the fan on, the two get connected and the fan gets power. Oh, it’s also not actually a window fan, but I designed and 3D printed some adapters so that it sits right in the window.

So now we have a window fan that can be turned off and on automatically. But I don’t want to have to press a button to do that. I’m lazy, remember? Instead, I made it so that the Go code constantly monitors the temperature reported by the air quality sensor, and queries the free OpenWeatherMap API to get the rough temperature outside. It can then turn the fan on if the room is too hot, and outside is cool enough to make a difference (there’s no point in blowing hotter air in!).

A graph of room temperature, showing the effect of the fan being turned on and off

I don’t want it to do that all the time though. The room doesn’t need to be cooled if I’m not around. There are various Zigbee presence sensors you can get, but the cheaper ones seem to be of dubious quality. Instead, I wrote some code to guess whether I’m present. It uses the state of the monitor light as a proxy for “is the computer in use”, and then makes some guesses based on the last time the computer was used and the current time (if I turn the computer off at midnight it probably means I’m going to sleep; if I turn it off at 2pm it probably means I’m going out somewhere).

You can see in the graph that the fan has a decent impact on temperature. I coded it to cool the room to 19 degrees, but then not turn on again until it was above 20. This prevents it flip-flopping on and off constantly. You can clearly see the pattern in the graph, as it actively cools and then slowly the room warms back up. This arrangement is much better than being woken up at 5am because it’s painfully cold. Trust me.

Bonuses: metrics and 3D printer

One of the things my custom Go code does is collate all the various stats reported by the Zigbee devices, and inserts them into a VictoriaMetrics database. I originally hosted this on the Pi itself, and it performed fine, but I’ve since moved it onto a server so that I can use it for some other things as well.

I set up Grafana to point to VM, and can create dashboards showing power usage, what devices are turned on when, and a bunch of environmental conditions. This also makes it easy to spot how good the data coming from the devices are. For example, the air quality sensor reports a figure for the amount of Carbon Dioxide in the air, as well as the amount of Volatile Organic Compounds (VOCs). The graphs are basically identical, but on a different scale. It turns out this particular device has no actual way of detecting CO₂, so it just “calculates” it from the VOCs figure. Useful to know if you want to actually use that data.

Recently I came across a go library for interacting with Bambu Lab printers, so I’ve also hooked that into my automation. It exports metrics about the printer, so I now have a way of seeing what’s going on when I’m not physically present⁵. I even added a HTTP endpoint (exposed over Tailscale) that shows pictures from the built-in camera.

On the automation front, I’ve made it automatically turn off the printer’s light when it finishes a print, and also added push notifications via PushOver whenever the state changes. No longer will I be sat in another room blissfully unaware it ran out of filament seconds after I walked out the door!

Was it worth it?

These days you could probably just buy some kind of ‘smart home’ hub that works well enough, and do most of what I’ve done with a lot less effort and no coding. I’m still of the opinion that for something so essentially local, it should itself be managed entirely locally. I don’t trust companies like Google or Amazon not to kill their products, or change or remove an API I rely on.

Obviously the “Not Invented Here” approach of coding everything yourself doesn’t suit everyone, but as someone who enjoys coding and enjoys having things work just so it works very well for me.

It turns out the chipset is important. I initially used a similar Sonoff dongle that used a CC2531 chipset, and its performance wasn’t great. It often failed to pair with new devices, and dropped links to existing ones. The zigbee2mqtt docs do explicitly advise against the CC2531 chips for that reason. At the time they recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d go with whatever their latest recommendation was. ↩︎
MQTT is basically a network protocol for publishing and subscribing to arbitrary messages. A broker is the thing that sits in the middle and routes the messages. I use Mosquitto but any will do. ↩︎
They originally also re-opened at sunrise but that was a terrible mistake. Who knew the sun rose so early?! ↩︎
“open” meaning “there’s a gap so it doesn’t work” not “open for business”. This confusion in terminology also extends to drawbridges. ↩︎
I use the printer in LAN mode, which means there’s no way to monitor it from a phone, even if you’re connected to the same network still. ↩︎

Upgrading the RAM in a Dell G15 laptop

2023-07-29T00:00:00Z

The Dell G15

I currently use a Dell G15 laptop for work. It has served me well for a little over a year, but recently it has been struggling a little with my day-to-day workload. It came with 32GB of RAM — the highest possible specification at the time¹ — but that is apparently no longer enough for me.

For a recent project, I was working on a Rust library used in an Android app. That meant running the usual glut of Android tools (Android Studio, an emulator and at least one Gradle daemon) alongside a normal IDE (IntelliJ IDEA). Throw in a web browser and a couple of electron apps, and I often managed to use all 32GB.

When you start swapping memory out to an encrypted disk — even an SSD — it doesn’t make for great performance. At first, I tried to work around this by enabling the Linux out-of-memory (OOM) killer, but it turns out that it’s not too good with Electron apps: it will kill the large browser process, but then the small Electron wrapper will just respawn it.

Can it be upgraded or not?

The obvious solution to not having enough RAM is to add more RAM. A quick look in the manual showed this might not be possible, though. The manual includes the following “Memory specifications” table:

Description	Values
Memory slots	Two SODIMM slots
Memory type	DDR5
Memory speed	4800
Maximum memory configuration	32GB
Minimum memory configuration	8GB
Memory size per slot	8GB or 16GB
Memory configurations supported	8 GB, 1 x 8 GB, DDR5, 4800 MHz 16 GB, 1 x 16 GB, DDR5, 4800 MHz 16 GB, 2 x 8 GB, DDR5, 4800 MHz, dual-channel 32 GB, 2 x 16 GB, DDR5, 4800 MHz, dual-channel

That unambiguously says that an upgrade from 32GB is not possible. I gave up.

Later though, I was complaining about memory issues to a friend, and he pointed out a Dell forum thread where a couple of people claim to have successfully installed dual-channel 32GB modules. Since the alternative was getting an entire new PC after only a year, I decided to give it a go.

The upgrade attempt

I ordered a pair of Crucial 32GB DDR5-4800 SODIMMs, and after they turned up dismantled the laptop. The G15 comes apart pretty normally: there are uncovered screws on the bottom holding the lower part of the case on. With those removed and some gentle prying, it pops off, and you get access to the battery, GPU and motherboard.

The first thing I saw was this:

The G15 motherboard adamantly proclaiming that it doesn’t want 32GB SODIMMs

Not one but two labels that indicate it will only accept 8GB or 16GB modules. Oh well, what’s the worst that can happen?

Oops?

I dutifully installed the new modules, reconnected the battery and put the case back together. I pressed the power button, and… nothing. None of the usual garish lights that immediately come on, no screen output, just a dead laptop. After reading some more of the user manual, I found that there is a tiny status LED on the side next to the ethernet port. Forcing the laptop to power off and back on again, the status LED blinked a distress code at me: 2 amber blinks, 4 white blinks. The manual says that is — unsurprisingly — a memory fault.

I figured at this point that the manual and labels on the motherboard were probably right. I took the laptop apart again, reinstalled the original 2x16GB modules, reassembled it, and pressed the power button. It didn’t boot. I don’t spend a lot of time fiddling inside computers, but I’ve done it enough that I’m reasonably confident I can’t entirely break a computer while swapping some RAM modules. I took to Googling², and found an interesting article that said Dell laptops don’t like to boot after RAM changes unless you clear the CMOS by popping out the battery for 15 minutes.

I opened the laptop up, and looked around for the CMOS battery. There wasn’t one. Turns out they don’t exist any more. I left the main battery disconnected for a while to see if it would help, and it didn’t.

Unexpected success

I started to get worried: if I couldn’t fix this, I wouldn’t be able to work until I got a new PC, and that wasn’t really in my budget at the minute. I sat reading old forum threads and help guides³, none of which were actually useful. Out of nowhere, though, the laptop booted up.

Naturally, I immediately shut the laptop down again, opened it up, and switched back to the new RAM modules. Then I turned it on again and sat waiting. After about 15 minutes of it looking totally dead, it turned on and showed a BIOS warning about the hardware configuration being changed. It then booted perfectly normally, and all 64GB of RAM was visible and usable.

My theory is that the forum threads were right: Dell laptops are funny about RAM upgrades. But somehow in removing the physical CMOS battery, they’ve kept the same “you have to wait 15 minutes” behaviour just without any indication that’s what’s happening. Regardless, I now have enough RAM even for the greediest of IDEs and Electron apps.

Bizarrely, the maximum spec has decreased to 16GB since then. ↩︎
In the generic sense. I use Kagi these days. ↩︎
On my phone because, y’know, the laptop was busted. ↩︎

Understanding Docker volume mounts

2019-04-01T00:00:00Z

The Docker project logo

One thing that always confuses me with Docker is how exactly mounting volumes behaves. At a basic level it’s fairly straight forward: you declare a volume in a Dockerfile, and then either explicitly mount something there or docker automatically creates an anonymous volume for you. Done. But it turns out there’s quite a few edge cases…

Changing ownership of the folder

Perhaps the most common operation done on a Docker volume other than simply mounting it is trying to change the ownership of the directory. If your Docker process runs as a certain user you probably want the directory to be writable by that user.

At first we might try something like:

FROM alpine
RUN adduser -D -u 1113 test123
USER test123
VOLUME /testing

But changing the user doesn’t seem to have any effect on the volume. Why? Checking the docs for the USER instruction shows that only affects certain future operations — namely RUN, CMD, and ENTRYPOINT. It doesn’t affect the VOLUME instruction; if it did, you’d probably just get a permission denied error unless the user you switch to had privileges to create mount points.

OK, so instead we might try using the good old chown command:

FROM alpine
RUN adduser -D -u 1113 test123
VOLUME /testing
RUN chown test123 /testing

But again, the directory is just owned by root at runtime. Back to the docs, this time for the VOLUME instruction and towards the bottom is this little tidbit:

Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.

As soon as Docker hits the VOLUME instruction the directory becomes a mount point, and anything we do to the temporary volume mounted there is discarded during the build process. So we have to change the ownership before the instruction, which may seem a little counter-intuitive:

FROM alpine
RUN adduser -D -u 1113 test123
RUN mkdir /testing && chown test123 /testing
VOLUME /testing

Now when the container runs, the /testing directory is owned by the test123 user. It’s not quite over, yet, though. This works if we let Docker create a volume automatically for us, or if we create a named volume and mount that; if we try and mount a host directory, though, it falls flat:

$ docker run --rm -it -v "$PWD/testing:/testing" testing ls -al /testing
total 8
drwxr-xr-x    2 1000     1000          4096 Apr  1 19:39 .
drwxr-xr-x    1 root     root          4096 Apr  1 20:44 ..

Docker handles mounting host directories differently to mounting volumes, even though the syntax is basically the same. Host directories are bind mounted directly into the container, so the permissions and ownership are the same as the directory on your host. The only way to fix them are to either change the permissions on the host, or have the container change them at runtime (assuming it has sufficient privileges).

One final wrinkle in all this happens when you use the same volume in multiple containers. Here we have two images built from the Dockerfile above, one with userid 1113 and one with userid 1114:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/testing testing1113 ls -nal /testing
total 8
drwxr-xr-x    2 1113     0             4096 Apr  1 19:49 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:51 ..

$ docker run --rm -it -v testing:/testing testing1114 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:47 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:52 ..

$ docker run --rm -it -v testing:/testing testing1114 touch /testing/Hello

$ docker run --rm -it -v testing:/testing testing1113 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:52 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:53 ..
-rw-r--r--    1 0        0                0 Apr  1 20:52 Hello

$ docker run --rm -it -v testing:/testing testing1114 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:52 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:52 ..
-rw-r--r--    1 0        0                0 Apr  1 20:52 Hello

Can you see what’s going on? When the volume is empty, the ownership changes based on the mount point in the container. Once it has something in it, the ownership is fixed.

So Docker behaves differently with regard to permissions:

when the folder is mounted from the host vs a volume
when the volume is empty vs having content

Pre-populating mounts with files from the image

One of the more esoteric features of the way Docker handles volume mounts is that in some cases files from the image are copied over into the container. For example:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/etc testing sleep 1

$ docker run --rm -it -v testing:/tmp testing ls -al /tmp
total 184
drwxr-xr-x   15 root     root          4096 Apr  1 20:58 .
drwxr-xr-x    1 root     root          4096 Apr  1 20:59 ..
-rw-r--r--    1 root     root             4 Jun  7  2018 TZ
-rw-r--r--    1 root     root             6 Dec 20 21:31 alpine-release
...

The first container we run mounts the newly created testing volume at /etc. Docker copies all the existing files and folders into the volume; when we then run the second container with the volume mounted at /tmp, we can see all the files that were in the first container’s /etc.

As with permissions, this behaviour is anything but consistent. Say we switch from a volume to a host directory:

$ mkdir testing
$ docker run --rm -it -v "$PWD/testing:/usr/bin" testing sleep 1
$ ls -al testing
total 8
drwxr-xr-x 2 root  root  4096 Apr  1 22:05 .
drwxr-xr-x 3 chris chris 4096 Apr  1 22:05 ..

Nothing is copied in, and inside the container the folder will be empty. Based on our discoveries with permissions, it’s reasonable to assume the same will happen with a non-empty volume too:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/testing testing touch /testing/Hello

$ docker run --rm -it -v testing:/usr/bin testing sleep 1

$ docker run --rm -it -v testing:/tmp testing ls -al /tmp
total 8
drwxr-xr-x    2 root     root          4096 Apr  1 21:09 .
drwxr-xr-x    1 root     root          4096 Apr  1 21:09 ..
-rw-r--r--    1 root     root             0 Apr  1 21:08 Hello

So at least that’s consistent. If you’re very observant, though, you might notice I switched from /etc/ to /usr/bin in the examples. That’s because within the container /etc/ has some files bind-mounted into it, such as /etc/resolv.conf, and these do always result in files being created in the mounted volumes or folders:

$ mkdir testing
$ docker run --rm -it -v "$PWD/testing:/etc" testing sleep 1
$ ls -al testing
total 8
drwxr-xr-x 2 chris chris 4096 Apr  1 22:12 .
drwxr-xr-x 3 chris chris 4096 Apr  1 22:12 ..
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 hostname
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 hosts
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 resolv.conf

Summary

Docker treats mounting host folders and mounting volumes differently. Don’t just assume that you can swap one for another and get the exact same behaviour.
Empty volumes will inherit permissions and files from the image they are mounted in; non-empty volumes and host folders will not.
Relying on Docker copying files into volumes is a very bad idea, as if you change those files in a future version of your image they will not be copied unless the volume is deleted and recreated.

I can’t find anywhere that these points are documented properly; if you know of anywhere, please drop me a message!

Monitoring power draw with WeMo Insight Switches

2016-05-02T00:00:00Z

A WeMo Insight Switch

I recently picked up a couple of Belkin’s WeMo Insight Switches to monitor power usage for my PC and networking equipment. WeMo is Belkin’s home automation brand, and the switches allow you to toggle power on and off with an app, and monitor power usage.

The WeMo Android app is pretty dismal. It’s slow, doesn’t look great, and crashed about a dozen times during the setup process for each of my two switches. It also doesn’t provide much information at all about power: you can see average power draw and current power draw, and that’s basically it.

Belkin has provided an option to e-mail yourself a spreadsheet with historical power data, and can even do it on a regularly scheduled basis, but that’s not really a nice solution if you want up-to-date power stats. Even if you were happy with data arriving in batch, having to get hold of an e-mail attachment and parse out a weirdly formatted spreadsheet doesn’t make for easy automation. It also relies on Belkin supporting the service indefinitely, which isn’t necessarily going to happen.

Enter the SOAP API

Fortunately, the devices themselves expose an API to get the data. Once a switch is setup with the WeMo app, it connects to a WiFi network. You can discover WeMo devices on the network using a UPnP broadcast, and then from there a list of supported services. Each service has a number of SOAP actions, with nice obvious names, and parameters documented in the XML service description.

For the insight switches, there’s a service at /upnp/control/insight1 which has a number of actions including GetPower and GetTodayKWH. Sending a SOAP request isn’t too difficult (albeit nowhere near as nice as a REST JSON API) — here’s a sample request I made using the DHC chrome extension:

POST /upnp/control/insight1 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.8
Content-Type: text/xml
Origin: chrome-extension://aejoelaoggembcahagimdiliamlcdmfm
SOAPACTION: "urn:Belkin:service:insight:1#GetPower"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.29 Safari/537.36

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
    </s:Body>
</s:Envelope>

The name of the action (in the SOAPACTION header) and the XML namespace in the body are both constructed from information in the service definition. The result that comes back is:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <s:Body>
        <u:GetPowerResponse xmlns:u="urn:Belkin:service:insight:1">
            <InstantPower>92170</InstantPower>
        </u:GetPowerResponse>
    </s:Body>
</s:Envelope>

The current power draw in milliwatts is returned in the <InstantPower> argument.

Putting the data to work

So now we have a way to pull the current power draw, a nice thing to do would be to plot a graph of it to see the variation over time. This is the short of thing you’d expect to see in the Belkin app, but it’s sadly missing. After a bit of research I settled on the tried and true rrdtool to store data and generate graphs. I created a new database to store the values from the two switches:

rrdtool create power.rrd \
    --start now \
    --step 60 \
    DS:wemoComputer:GAUGE:120:U:U \
    DS:wemoNetworking:GAUGE:120:U:U \
    RRA:AVERAGE:0.5:1:1440 \
    RRA:AVERAGE:0.5:10:1008 \
    RRA:AVERAGE:0.5:30:1488 \
    RRA:AVERAGE:0.5:120:1488 \
    RRA:AVERAGE:0.5:360:1488 \
    RRA:AVERAGE:0.5:1440:36500

This creates a database file which expects values to be given every 60 seconds for two data series: ‘wemoComputer’ and ‘wemoNetworking’. These are gauge types (i.e., a value we read off a gauge, rather than a counter that keeps going up) with no minimum and maximum (‘U’).

It then defines a series of round-robin archives. Each of these stores a fixed number of entries, and the oldest is overwritten when they become full. The interesting arguments are the step count (second to last) and number of samples (last argument). The first one takes every sample and keeps 1,440 of them (i.e., a full day at 1-minute resolution); the last one has a step of 1,440 and keeps 36500 of them (i.e., 100 years at 1-day resolution). The use of RRAs ensures that rrdtool can retain enough data to create historical graphs, while providing a fixed, finite maximum file size. With sufficient RRA coverage you can graph most timescales and have data available at a reasonable resolution.

Next, I created a small python script to retrieve the power using SOAP:

#!/usr/bin/python3

import requests
from xml.etree import ElementTree

def get_power(ip_and_port):
    headers = {'Content-type': 'text/xml', 'SOAPACTION': '"urn:Belkin:service:insight:1#GetPower"'}
    payload = '''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
                             s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
                  <s:Body>
                   <u:GetPower xmlns:u="urn:Belkin:service:insight:1"/>
                  </s:Body>
                 </s:Envelope>'''
    r = requests.post("http://%s/upnp/control/insight1" % ip_and_port, headers=headers, data=payload)
    et = ElementTree.fromstring(r.text)
    return et.find('.//InstantPower').text

I then have a dictionary of IP addresses to data series names, and the script polls each one in turn and then executes an rrdtool update query to add the items to the database. I have this script running every minute via cron.

After leaving the script to run for a bit and gather data, it’s time to make some graphs. I use the following to create a graph with a background gradient:

rrdtool graph desk-1d.png \
        -o -X0 -w800 -h500 \
        -u 2000 -l 20 -r \
        DEF:raw=power.rrd:wemoComputer:AVERAGE \
        CDEF:power=raw,1000,/ \
        CDEF:powerz=power,2000,LT,power,2000,IF CDEF:powerzNoUnk=power,UN,0,powerz,IF AREA:powerzNoUnk#ff0000 \
        CDEF:powery=power,1500,LT,power,1500,IF CDEF:poweryNoUnk=power,UN,0,powery,IF AREA:poweryNoUnk#ff0000 \
        CDEF:powerx=power,1000,LT,power,1000,IF CDEF:powerxNoUnk=power,UN,0,powerx,IF AREA:powerxNoUnk#ff0000 \
        CDEF:powerw=power,900,LT,power,900,IF CDEF:powerwNoUnk=power,UN,0,powerw,IF AREA:powerwNoUnk#ff0000 \
        CDEF:powerv=power,800,LT,power,800,IF CDEF:powervNoUnk=power,UN,0,powerv,IF AREA:powervNoUnk#ff1b00 \
        CDEF:poweru=power,700,LT,power,700,IF CDEF:poweruNoUnk=power,UN,0,poweru,IF AREA:poweruNoUnk#ff4100 \
        CDEF:powert=power,600,LT,power,600,IF CDEF:powertNoUnk=power,UN,0,powert,IF AREA:powertNoUnk#ff6600 \
        CDEF:powers=power,400,LT,power,400,IF CDEF:powersNoUnk=power,UN,0,powers,IF AREA:powersNoUnk#ff8e00 \
        CDEF:powerr=power,200,LT,power,200,IF CDEF:powerrNoUnk=power,UN,0,powerr,IF AREA:powerrNoUnk#ffb500 \
        CDEF:powerq=power,180,LT,power,180,IF CDEF:powerqNoUnk=power,UN,0,powerq,IF AREA:powerqNoUnk#ffdb00 \
        CDEF:powerp=power,160,LT,power,160,IF CDEF:powerpNoUnk=power,UN,0,powerp,IF AREA:powerpNoUnk#fdff00 \
        CDEF:powero=power,140,LT,power,140,IF CDEF:poweroNoUnk=power,UN,0,powero,IF AREA:poweroNoUnk#d7ff00 \
        CDEF:powern=power,120,LT,power,120,IF CDEF:powernNoUnk=power,UN,0,powern,IF AREA:powernNoUnk#b0ff00 \
        CDEF:powerm=power,100,LT,power,100,IF CDEF:powermNoUnk=power,UN,0,powerm,IF AREA:powermNoUnk#8aff00 \
        CDEF:powerl=power,90,LT,power,90,IF CDEF:powerlNoUnk=power,UN,0,powerl,IF AREA:powerlNoUnk#65ff00 \
        CDEF:powerk=power,80,LT,power,80,IF CDEF:powerkNoUnk=power,UN,0,powerk,IF AREA:powerkNoUnk#3eff00 \
        CDEF:powerj=power,70,LT,power,70,IF CDEF:powerjNoUnk=power,UN,0,powerj,IF AREA:powerjNoUnk#17ff00 \
        CDEF:poweri=power,60,LT,power,60,IF CDEF:poweriNoUnk=power,UN,0,poweri,IF AREA:poweriNoUnk#00ff10 \
        CDEF:powerh=power,50,LT,power,50,IF CDEF:powerhNoUnk=power,UN,0,powerh,IF AREA:powerhNoUnk#00ff36 \
        CDEF:powerg=power,40,LT,power,40,IF CDEF:powergNoUnk=power,UN,0,powerg,IF AREA:powergNoUnk#00ff5c \
        CDEF:powerf=power,30,LT,power,30,IF CDEF:powerfNoUnk=power,UN,0,powerf,IF AREA:powerfNoUnk#00ff83 \
        CDEF:powere=power,20,LT,power,20,IF CDEF:powereNoUnk=power,UN,0,powere,IF AREA:powereNoUnk#00ffa8 \
        CDEF:powerd=power,0,LT,power,0,IF CDEF:powerdNoUnk=power,UN,0,powerd,IF AREA:powerdNoUnk#00ffd0 \
        LINE:power#080

This seems a bit unweildy, but it’s fairly straight forward. The options tell rrdtool to create a graph with a canvas size of 800x500 pixels, a lower limit of 20W, upper limit of 2kW, and a log scale. The first CDEF divides our raw value (which was in millwatts) by 1000 to get a value in watts. rrdtool uses reverse polish notation like some calculators, so you provide the arguments before the operator.

The big block of CDEF/CDEF/AREA parameters creates a series of area plots in different colours according to the power level. They’re in descending order so the smaller areas are drawn on top of the larger layers. This results in a graph that looks like this:

Graph of power usage over a day

This graph shows the total power for all the things plugged in at my desk. You can see the idle power draw is around 60W. When I’m using the computer it jumps up to around 130W, and when the computer is under heavy load (playing games, for example) it goes up even further to the 200W mark. With a couple of small tweaks to the rrdtool command, I also have a graph showing the entire week:

Graph of power usage over a week

The two huge spikes near the start of the data are caused by a heater under my desk. They’re also one of the main reasons I chose to plot the graphs with a logarithmic scale. With a linear scale between 20W and 2kW, the graph would be completely dominated by these large spikes — in fact the difference in height between the two spikes would take almost half the graph! The relative difference between the normal values of 60-200W and the heater’s value of 1.8kW isn’t actually that interesting, and certainly not worth using about 80% of the graph to demonstrate. The log scale helps to compress this, and emphasises the difference in the smaller values more.