Chameth.com - posts like debugging-beyond-the-debugger, filament-weight-display, why-you-should-be-using-https but not adventures-in-3d-printing

Home Automation Without the Megacorps

2025-05-21T00:00:00Z

I first experimented with home automation in 2016, by picking up a Samsung “SmartThings” hub. It was terrible. The UI to configure things was slow and clunky, firmware updates were applied whether you wanted them or not, and everything stopped working if their cloud services stopped. You were also locked into whatever integrations they deigned to support, of course. After that broke for the umpteenth time I scaled back and for years the closest I got to home automation was a couple of Hue bulbs.

Recently I’ve been building it out again, though. This time using off-the-shelf components that interop using Zigbee, open-source software, and some code I wrote myself. It’s great; it runs entirely locally and has had basically zero downtime. The Zigbee ecosystem lets me integrate all sorts of things without having to spend lots of money on “smart” alternatives. I think I’ve spent less on this incarnation than I did on the original SmartThings hub all those years ago (even without adjusting for inflation!).

My current setup

I run everything on a Raspberry Pi 4, with a Sonoff USB Zigbee adapter based on the CC2652P chipset¹. Interfacing with the Zigbee stack is handled by zigbee2mqtt (z2m for short), an open-source project that basically bridges your devices to an MQTT broker². When a device reports some data, it will send a new message over MQTT; when you want to make a device do something you just post a message back. It’s incredibly lightweight, but supports a huge array of devices out of the box. And as it’s just using MQTT, it’s trivial to integrate with other software or build on top of.

I know a lot of people building out this kind of home automation use Home Assistant, but I don’t get on with it terribly well. It felt incredibly sluggish, and the entire project is just a bit… much. I want a binary or package or docker image I can just run, not an entire operating system. That’s not how computers are meant to work! In contrast, z2m is simple to set up, super light weight and responsive.

Anyway. z2m exposes Zigbee devices over MQTT, so I wrote some code in Go to connect to the MQTT broker, and listen to the messages. It’s grown a bit beyond this now, and I’m skipping some boring bits like error handling and JSON parsing, but at first I had something like:

for {
    message, topic, err := c.ReadSlices()
    
    if string(topic) == "zigbee2mqtt/desk-button" {
        if strings.Contains(message, "single") {
            c.Publish(nil, []byte("{\"state\": \"TOGGLE\"}"), "zigbee2mqtt/room-lights/set")
        }
    }
}

I think I actually made this harder to read by trying to simplify it here, but hopefully you can follow that it was relatively straight forward to listen for a particular action to happen (in this case me “single”-pressing on the device called “desk-button”) and then make another device do something in response (toggling the state of the “room-lights” device).

So what actually are these devices? At present in z2m I have the following:

Some generic buttons
An air quality sensor (USB powered), and a separate temperature sensor (battery powered)
A light switch
Some 240V relay modules, that I use to turn non-smart devices on and off
Some USB relay modules, for the same purpose
Some “smart” plugs that I mostly use to monitor power usage
A motorised blind roller

I think all of these are either Tuya or Moes branded, and were all sourced from AliExpress. You can get the same things from Amazon or elsewhere, but they tend to be 3-4x more expensive for the same product.

One of the cool things about Zigbee devices is that the powered ones work together to create a mesh network, so you don’t have to worry about network repeaters or signal strength like you do with Wi-Fi networks. z2m even makes a map showing the connections:

The network map produced by z2m. You can see how all the powered devices form a mesh that the lower power ones can connect to.

Some interesting automations

A Zigbee relay spliced into the power cable for the fan

So what do I actually do with all of this? Some of it is just what I’ll call “laziness automation”: I want to be able to turn on the lights without getting up and walking to the light switch. So there’s a Zigbee button on my desk that does it. Then there are some less manual automations: my blinds are automatically closed at sunset³, for example. It feels a bit trivial, but it’s surprisingly nice just not to have to think about that at all.

Another nice quality of life automation is the light I have above my monitor. It’s USB powered, and I now have it going through a Zigbee USB relay that lets it be toggled on and off. Then I have a small agent running on my desktop that turns the light on when the computer is unlocked, and off when it locks or shuts down. I keep meaning to make a “film mode” that detects when I’m watching a film, and turning all the lights and blinds down, but I haven’t got around to it yet.

The most complex automation is probably for a window fan. It’s not smart in any way, so I cut the power cable and inserted a Zigbee relay. The whole thing is housed in a little project box to keep it secure. The relay basically acts as a switch: the live wire leading to the fan runs to the “normally open”⁴ contact, while the upstream power goes to “common”. When the relay is told to turn the fan on, the two get connected and the fan gets power. Oh, it’s also not actually a window fan, but I designed and 3D printed some adapters so that it sits right in the window.

So now we have a window fan that can be turned off and on automatically. But I don’t want to have to press a button to do that. I’m lazy, remember? Instead, I made it so that the Go code constantly monitors the temperature reported by the air quality sensor, and queries the free OpenWeatherMap API to get the rough temperature outside. It can then turn the fan on if the room is too hot, and outside is cool enough to make a difference (there’s no point in blowing hotter air in!).

A graph of room temperature, showing the effect of the fan being turned on and off

I don’t want it to do that all the time though. The room doesn’t need to be cooled if I’m not around. There are various Zigbee presence sensors you can get, but the cheaper ones seem to be of dubious quality. Instead, I wrote some code to guess whether I’m present. It uses the state of the monitor light as a proxy for “is the computer in use”, and then makes some guesses based on the last time the computer was used and the current time (if I turn the computer off at midnight it probably means I’m going to sleep; if I turn it off at 2pm it probably means I’m going out somewhere).

You can see in the graph that the fan has a decent impact on temperature. I coded it to cool the room to 19 degrees, but then not turn on again until it was above 20. This prevents it flip-flopping on and off constantly. You can clearly see the pattern in the graph, as it actively cools and then slowly the room warms back up. This arrangement is much better than being woken up at 5am because it’s painfully cold. Trust me.

Bonuses: metrics and 3D printer

One of the things my custom Go code does is collate all the various stats reported by the Zigbee devices, and inserts them into a VictoriaMetrics database. I originally hosted this on the Pi itself, and it performed fine, but I’ve since moved it onto a server so that I can use it for some other things as well.

I set up Grafana to point to VM, and can create dashboards showing power usage, what devices are turned on when, and a bunch of environmental conditions. This also makes it easy to spot how good the data coming from the devices are. For example, the air quality sensor reports a figure for the amount of Carbon Dioxide in the air, as well as the amount of Volatile Organic Compounds (VOCs). The graphs are basically identical, but on a different scale. It turns out this particular device has no actual way of detecting CO₂, so it just “calculates” it from the VOCs figure. Useful to know if you want to actually use that data.

Recently I came across a go library for interacting with Bambu Lab printers, so I’ve also hooked that into my automation. It exports metrics about the printer, so I now have a way of seeing what’s going on when I’m not physically present⁵. I even added a HTTP endpoint (exposed over Tailscale) that shows pictures from the built-in camera.

On the automation front, I’ve made it automatically turn off the printer’s light when it finishes a print, and also added push notifications via PushOver whenever the state changes. No longer will I be sat in another room blissfully unaware it ran out of filament seconds after I walked out the door!

Was it worth it?

These days you could probably just buy some kind of ‘smart home’ hub that works well enough, and do most of what I’ve done with a lot less effort and no coding. I’m still of the opinion that for something so essentially local, it should itself be managed entirely locally. I don’t trust companies like Google or Amazon not to kill their products, or change or remove an API I rely on.

Obviously the “Not Invented Here” approach of coding everything yourself doesn’t suit everyone, but as someone who enjoys coding and enjoys having things work just so it works very well for me.

It turns out the chipset is important. I initially used a similar Sonoff dongle that used a CC2531 chipset, and its performance wasn’t great. It often failed to pair with new devices, and dropped links to existing ones. The zigbee2mqtt docs do explicitly advise against the CC2531 chips for that reason. At the time they recommended CC2652Ps, so that’s what I went with. If you’re starting new I’d go with whatever their latest recommendation was. ↩︎
MQTT is basically a network protocol for publishing and subscribing to arbitrary messages. A broker is the thing that sits in the middle and routes the messages. I use Mosquitto but any will do. ↩︎
They originally also re-opened at sunrise but that was a terrible mistake. Who knew the sun rose so early?! ↩︎
“open” meaning “there’s a gap so it doesn’t work” not “open for business”. This confusion in terminology also extends to drawbridges. ↩︎
I use the printer in LAN mode, which means there’s no way to monitor it from a phone, even if you’re connected to the same network still. ↩︎

Project log: Filament weight display

2023-12-03T00:00:00Z

The finished project

One problem¹ I have when 3D printing is that it’s hard to gauge whether there’s enough filament left on a roll to complete a print. Sometimes it’s obvious when the print is small or the roll is full, but often it’s not. If I’m unsure about it, I end up obsessing over the printer instead of just leaving it to do its thing.

The typical approach to this problem is to use a run-out sensor, which stops the printer when it detects that the filament is no longer running through it. I’d rather know in advance though: it’s no good stopping the print if I’m making something that has to look nice and now half of it is one colour and the remainder will be something else.

While browsing around for ideas, I came across a project that used a load cell to measure the weight of the filament roll². I didn’t really know what a load cell was, but decided to do some research and see if I could build my own. I didn’t actually read the article I found, intending to figure things out on my own if I could.

Load Cells 101

A load cell, its amplifier, and some random headers

At the heart of the project is a load cell. It turns out these are the things that do the lifting in most electronic scales. They’re basically a carefully machined metal bar with a strain gauge attached to it. The strain gauge is a long trace of conductive material that runs back and forth across the middle of the bar. When the bar bends very slightly (due to load), the conductor flexes with it and gets longer or shorter; that flex changes its resistance by a tiny amount, which can be measured using a Wheatstone bridge.

The Wheatstone bridge is a diamond of resistors, and requires an ’excitation’ voltage to be applied across it in one direction, and the resistance is then read from the perpendicular pair. As the resistance is tiny, and the changes even more tiny, you generally need an amplifier to read it and output something more useful. The HX711 is such an amplifier, and is so ubiquitous that most places that sell load cells will just bundle one with it.

The HX711 connects to the load cell using four wires: two for the excitation voltage, and two for the output. On the other end it takes a 5V input, ground, and then has two pins to deal with the onward communication: one for a clock signal, and one for data. We’ll get back to that later!

Early prototyping

The prototype, mid-debugging

I decided early on to base the project around an RP2040-based board, as I had previous experience with them and they’re cheap and easy to get hold of. I started off with a Raspberry Pi Pico, slotted it into a breadboard, and connected it to the HX711. A short search produced three or so different python libraries, but I couldn’t get any of them to work reliably.

If you look at the picture of the prototype, you’ll notice a large bundle of wires to the left. These are connected to a logic analyser I bought to try and understand what on earth was happening, because it seemed like the code should work, and there was definitely some communication with the HX711 happening, but none of the readings made any sense.

At one point I noticed that moving my head near to the Pico caused the readings to change, but I didn’t understand the implication of that³. I went to bed unable to figure out what was going on, and just before I dozed off realised that I must have the pins configured wrong. The next day I checked the spec sheet and discovered that the pin with the number “1” silk-screened next to it is in fact GPIO 0, not GPIO 1⁴. The pin the Pico was trying to read data from wasn’t actually connected to anything, which is why waving my hand near it was enough to induce a signal.

Updating the code to use the correct pins made everything work much better and I started to get sensible readings. At this point I came across the RP2040-Zero by Waveshare, which is not only a smaller, cuter, cheaper Pico, but has silk-screened numbers that actually match the GPIO pin numbers (plus an actual a reset button!). I immediately bought several, and switched the prototype over as soon as they arrived.

The rest of the prototyping was reasonably uneventful. I used a little OLED screen I had sitting around from a past Ali Express order, and once again there were several python libraries to handle it. Soon enough I had raw weight readings from the load cell (which was stuck to my desk with painters’ tape) being displayed on the screen.

The load cells don’t measure absolute weight, so you have to “calibrate” them by remembering the reading when there is no load, and then applying a scaling factor to turn the raw reading into a weight. I worked these out by putting a known weight on the business end of the load cell, and then hard-coding the numbers.

An interruption to deal with interrupts

One thing I noticed with the MicroPython libraries for the HX711 is that they relied on polling the data pin to see if there was any data to read. The way the HX711 communicates is by pulling the data line low when it has data; you then have to pulse the clock line to get it to shift a bit of the reading out over the data line. The HX711 I have operates at 10Hz by default⁵, and reading the data takes almost no time at all, so the most “efficient” thing to do is to sleep for 100ms between readings.

This is fine if you’re not doing anything else, but gets awkward if you also want to refresh a screen at certain points, or check for button inputs, etc. It also upset me to see the data line being low for so long when viewing the logic analyser traces. The RP2040 supports setting interrupts on the GPIO pins, but I couldn’t find anyone who was actually using them for the HX711.

I therefore did the only sensible thing: wrote my own library, using interrupts to monitor for data availability. It was pretty straightforward, but didn’t actually work consistently. Looking more closely at the other libraries, they all do something to try to increase the speed at which the clock signals are written⁶: some were disabling interrupts, others were using assembly. I, again, did the only sensible thing: I threw all the python away and wrote a new driver in TinyGo.

My new code is event driven: it adds interrupts for button presses, and one for the HX711 data line. When the data line is pulled low, it removes that interrupt (so it doesn’t fire for every bit of data we receive), does the reading logic, then re-adds the interrupt. Theoretically if you spam buttons at an inopportune time you could mess up a reading, but the reading 100ms after would be fine. So my code spends most of the time idle, which translates to very little power draw.

User experience

While prototyping I was using some standard small push buttons to navigate the user interface I was creating. I soon realised that if I needed to enter weights on the real device I wasn’t going to be happy poking an annoying little button hundreds of times, and I didn’t feel like coding anything more elaborate to make input easier. Instead, I switched to using a rotary encoder. This is a bit like a variable resistor, but it can rotate completely freely, and sends a signal when it’s turned. It also has a button built in, so you can press it down to select things.

It took me a while to get the hang of the rotary encoder: it has two data pins for turning, and you have to wait for a falling edge on one pin and then immediately read the other to figure out which way it’s turning. My initial attempts at doing this weren’t quite immediate enough⁷, so I ended up with somewhat random data. Trying to select a number when going right sometimes increases it and sometimes decreases it is an exercise in frustration.

The correctly functioning rotary encoder makes it much easier to input numbers and change settings. I ended up with a very simple menu system at the bottom of the screen which lets you cycle between three options: setting the spool weight, zeroing the scale, and calibrating it. Zeroing is the most straightforward option: it simply reads the current value, and saves it as the zero offset in the flash memory. Selecting spool weight enters a mode where turning the rotary encoder increases or decreases the spool weight by a gram at a time; pressing the encoder in saves the value to flash and exits the mode⁸.

Calibration is a bit more confusing. Changing the spool weight changes the value you see on the screen directly and predictably. Calibration instead affects the “scale” value you can’t actually see; you just see the effect of the scaling on the weight reading. Still, if you have a known weight on the scale you simply rotate the encoder until the right weight is displayed on the screen. Pressing the encoder saves the value to flash as with the other modes.

Putting it together

The final assembly

The final step was to assemble everything in a way where it would be usable, and wires wouldn’t keep falling out whenever you looked at it. I found some cute little prototype boards that would just about fit all the components on – I left the OLED screen and rotary encoder off so they could be mounted to the front of the box.

I obviously 3D printed the enclosure, because I own a 3D printer and therefore spend a lot of my time 3D printing parts for my 3D printer. In my first prototype I put raised areas on the back panel and put some heat-press inserts in them, intending to screw down the protoboard. The screw holes I was going to use, however, were underneath the soldered-down RP2040-Zero and HX711. I instead uninserted the inserts and simply put some screws through the back panel with the intention of slotting nuts under the components. This turned out to be both incredibly annoying and entirely unnecessary; the board sits at the back of the box on its own, and there’s no real danger from it having a bit of freedom.

While I was prototyping I was powering the circuit from the USB port on the microcontroller, but for the actual unit I added a buck converter and ran a cable from the 3D printer’s power supply (which outputs 24V). This has the handy side effect of turning the scale on and off with the printer. I’ve recently bought a bench power supply, which made testing this part a lot easier: being able to dial in a voltage and current limit and then just plug things in is great.

The other thing that needed to change between my prototype and the final version was the load cell. It needed to be attached to the printer in such a way that one end bore all the weight of the spool (and, you know, not stuck to my desk with painters’ tape). I was originally going to do something fancy with bearings, but settled on the easier option of a spool holder which uses the same threading as the original, so it slots straight in:

The load cell mounted as part of a spool holder on the printer.

With all of this done, I invented a complicated mounting system to keep it attached to the printer. Any resemblance to blue painters’ tape is entirely coincidental.

Bill of materials and sources

Here are the major components that I ended up using:

Part	Source	Cost
RP2040-Zero microcontroller	WaveShare	£3.15
5KG load cell + HX711	AliExpress	£1.89
Buck converter	AliExpress	£0.51
SH1106 OLED module	AliExpress	£1.70
Rotary encoder	AliExpress	£0.80
Mini PCB prototype board	Amazon	£1.49

Where things come in packs of more than one, I’ve listed the cost of one unit. I’ve not included things like wires, connectors, or the filament used in printing the enclosure as you can use whatever you have to hand.

The source code I wrote is available on GitHub, and the designs for the 3D printed parts are shared in OnShape.

Among many, many others, of course. ↩︎
I can’t actually find where I saw this original project now, but there are quite a few kicking around if you search. ↩︎
This happened at about 1am and I was tired and frustrated. That’s my excuse, anyway. ↩︎
At this point I realised I’d made the exact same mistake the last time I used a Pico. Fool me once, etc. ↩︎
You can change this to 80Hz by resoldering a resistor on the board, but 10Hz is already several orders of magnitude faster than I actually need. ↩︎
If the clock signal stays high for longer than 60ms it’s treated as a shutdown signal by the HX711. This is somewhat unideal when you’re trying to read data from it. ↩︎
Because of my very clever interrupt system. I wanted to do as little work in the interrupt handler as possible, so I was just enqueuing an event to be processed in the main loop. This introduced enough delay that I wasn’t reading the second pin at the right time. The solution was to read the pin in the interrupt handler and then just enqueue a “left” or “right” event instead of my original “turning somehow, you figure it out” event. ↩︎
I was initially planning on hard-coding the weights for all the spools I’m likely to use, and just having a way to switch between them. But the thought of having to re-flash the device every time I bought a new type of filament made me reconsider. Plus, it was way easier to not do that. ↩︎

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).

Why you should be using HTTPS

2016-06-17T00:00:00Z

The EFF’s HTTPS Everywhere logo

One of my favourite hobbyhorses recently has been the use of HTTPS, or lack thereof. HTTPS is the thing that makes the little padlock appear in your browser, and has existed for over 20 years. In the past, that little padlock was the exclusive preserve of banks and other ‘high security’ establishments; over time its use has gradually expanded to most (but not all) websites that handle user information, and the time is now right for it to become ubiquitous.

Why use HTTPS?

There are numerous advantages to using HTTPS, both for the users of a website and for the operator:

Privacy

The most obvious advantage is that HTTPS gives your users additional privacy. An insecure (HTTP) request can potentially be read by anyone on the same network, or the network operators, or anyone who happens to operate a network along the path between the user and the server.

Users on shared WiFi networks (such as those in coffee shops, hotels, or offices) are particularly vulnerable to passive sniffing by anyone else on that network. If the network is open (as is frequently the case) then anyone in radio range can see exactly what the user is up to.

Integrity

HTTPS also helps to maintain the integrity of your site. With a plain HTTP request, there’s nothing to stop anyone in between the server and the user from modifying the content of the request or the response. This is a frequent tactic used by annoying WiFi gateways (such as the ones you’d find in a hotel), dubious ISPs who want to serve you extra adverts, or just plain old nefarious attackers.

If you’re trying to convey some kind of information to users (and if you aren’t, why exactly are you running a website again?) it seems beneficial to both you and them if the information arrives as you intended, rather than in a modified form due to someone or something tampering with it.

Security

If your website has any kind of authentication, or session identifiers, it becomes extremely vulnerable to an attacker monitoring the traffic and stealing the credentials. This was starkly demonstrated in 2010 when Firesheep was released. This tool allowed anyone to quickly and automatically hijack social media accounts of anyone on the same network who was using HTTP to access them.

Even if your login pages are served over HTTPS, if you send a single session ID cookie over HTTP (such as a page you decided wasn’t particularly ‘important’) then an attacker can probably spoof the user’s session and gain full access to their account. Again, in the case of open WiFi networks that could be anyone in radio range.

Search engine rankings

Some search engines use HTTPS as a signal in their ranking algorithms. Google announced in 2004 that it was using the presence of HTTPS as a small positive signal, but that it may strengthen that signal over time as more and more websites switch to using a secure transport. It’s not unthinkable that at some point in the future there will be HTTPS-only search engines.

But… But… But…

There are lots of excuses for not implementing HTTPS. Most of them are either misguided or outdated.

It’s too expensive and/or complicated

In the past, getting HTTPS certificates was a pain. A number of free suppliers have existed for a while but the process for getting their certificates wasn’t particularly straight forward, and many imposed arbitrary restrictions on the certificate parameters. Even once you had the certificate, you had to fiddle about with your HTTP server configuration to make it work, remember to manually get a new certificate when the old one expired, and lots of other annoying busywork.

With the arrival of Let’s Encrypt, all that changed. You can retrieve and deploy a free HTTPS certificate with two or three commands. Renewal can be handled completely automatically with a single command executed by cron.

There’s no point; nothing on my site is sensitive

You might not think your content warrants privacy, but can you speak for everyone who accesses it? Even content that seems mundane to you — such as travel advice, or technical writing — could be used to build up a profile of a user. If an attacker is monitoring traffic in a coffee shop and sees a user looking at travel advice and weather forecasts for a foreign country, he could use that information to plan a burglary knowing that the user will be away. Similarly, some content which is perfectly mundane to you may actually be very sensitive in other countries with repressive governments. HTTPS makes it much harder for these people to snoop on traffic.

From another angle, if you’re offering any kind of information, instructions, or especially file downloads, there’s a severe risk to users if the content is modified on its way to them. An evil sysadmin could rewrite your travel advice to suggest visiting the local drug dealer’s hangout, or replace your download with a malware-infested version.

HTTPS is slower, uses more resources, etc

Back in 1995 this might have been a valid argument. Enabling HTTPS on a modern server will make an almost negligible difference to performance. If you also enable HTTP/2 (which most implementations only support over HTTPS), it’s likely to actually use fewer resources, and result in a faster, smoother experience for your users. HTTP/2 was designed to work with HTTPS, and designed with modern requirements and networking techniques in mind.

CloudFlare have an excellent demonstration of the benefits of HTTP/2, and it can show speed improvements of 2-3x in a typical environment. On top of being faster, HTTP/2 uses fewer connections which results in less resource overhead on both the server and the client.

So what are you waiting for?

If you run a website and aren’t using HTTPS, give it a try.