Chameth.com - posts like debugging-beyond-the-debugger, moving-back-to-a-dynamic-website but not g15-ram-upgrade, how-i-get-things-done, why-you-should-be-using-https

Monthly Meanderings: April 2026

2026-05-01T00:00:00Z

It’s apparently been a whole month since the last edition of Monthly Meanderings. Not sure when that happened. This month I’ve been to a LAN party, had an MRI, and have been trying unsuccessfully to get my doctor to change a prescription. The latter two are not doing much to help shake that middle-aged feeling I mentioned last month…

Website updates

Another pair of blog posts for this month: The Case of the Unchanging Config about a problem I had with a bind-mounted config in a Docker container, and Migrating from GitHub to Forgejo, a write-up of… well… migrating from GitHub to Forgejo. That one’s been on my to-do list for a while, so it’s nice to finally get it written up.

It’s not visible when you look at the site, but I’ve been doing a lot of refactoring this month. The existing design had a package for database operations, a package for the stylesheet, a package for shortcodes, and so on. As I bolt more and more things on this gets less and less useful: those packages get bigger and bigger, and trying to understand how, say, the music import functionality works means hopping all over the place. Instead, I’m moving towards having each feature contained in its own package. It defines its own stylesheet, database operations, HTTP handlers, and so on. They’re then all wired together at the top layer. There’s still some more to do on that front, but it’s progressing in a good direction, and I’m happier with the code.

The only actual visible change is the addition of a page dedicated to World of Warcraft that automatically imports some data from the game’s API. We’ll come back to that later!

Other projects

I made a trio of new tiny projects: pdsps, a reverse proxy for a Bluesky/ATProto PDS that overrides the age verification response so you don’t need to hand over ID; irc-adze, a plugin for irc-bot that receives and announces adze webhooks; and wow-spec-switch, a super-simple addon for World of Warcraft that lets you switch specialisation with a command.

I’ve also been working on a custom app launcher in the style of ulauncher but without requiring WebKit to render things. I’ve not yet polished it up enough to open source it, but it’s not far off.

Entertainment

Another single film month for me. It was a great one, though:

Project Hail Mary

2026-04-01

A film adaptation of a book I loved that’s actually good. Huh!

Ryan Gosling does great in what is often a solo act. Early on I thought the film was trying a bit too hard to be funny, but at some point it started working and I enjoyed it.

There are lots of parallels to The Martian, Arrival, and so on, but it’s a sufficiently different mash-up of the concepts that it’s not a problem.

I’m still enjoying SNL UK a lot more than I expected to, although I keep forgetting what day it airs. They should put it in the name or something.

The rest of my “entertainment” time has been spent playing World of Warcraft. I played an awful lot about 6-7 years ago, and have dipped in and out occasionally since. I got an MMORPG itch again recently and jumped back in, and I’m enjoying it a lot once again. I’ve been playing a healer properly for the first time (after enjoying healing in Final Fantasy XIV):

Level 90

Female Void Elf

Holy Priest

280 average item level

Methrica-Terenas

Professions

Midnight Alchemy		100/100
Midnight Engineering		92/100
Midnight Cooking		1/100
Midnight Fishing		4/300

View on blizzard.com View on raider.io

Around the web

git-history documentation

The latest release of git added this handy-seeming history subcommand. It allows you to reword a commit, or split a commit up interactively, without having to go through the interactive rebase song and dance.

Git has acquired a bunch of new commands in recent releases, but I tend to ignore them in favour of the old ones I know well. This one will definitely get use from me, though.

Git Koans by Steve Losh

Sticking with the same theme, this series of koans about Git and its various quirks had me laughing. If you’re reading this blog, you’re probably part of the target audience for it.

SQLite prefixes its temp files with etilqs_

I think I’d come across this before, but it’s still a fun anecdote. It reminds me of Daniel Stenberg’s e-mail collection; as the author of curl his name is listed on basically everything with the inevitable influx of people contacting him.

A dot a day keeps the clutter away

I love this approach to tracking how frequently things are used, and the weird sort of archaeological record you get from rotating the colours of the stickers used.

Monthly Meanderings: February 2026

2026-03-01T00:00:00Z

It doesn’t feel like a whole month has gone past since I wrote the last instalment of Monthly Meanderings, even allowing for how short a month February is. For more context on this series, you can check out the introduction to the first edition.

Website updates

I only wrote one new blog post this month: Just a nod, which is about the “nod” button I added to the bottom of most pages. I did a lot of other little bits and pieces, though:

My blog posts are now automatically posted to a dedicated Bluesky account, and posts now include syndication links at the bottom.
The site now tracks how long each request takes to generate along with how many SQL queries were used. You can see the details for the current request by hovering/tapping over the pi symbol at the bottom right of a page. To understand why it’s a pi symbol, you should watch The Net!
Speaking of films, I redid the star rating rendering so instead of unicode stars it uses some hand-drawn pixel art stickers. They’re somewhat randomised, so they don’t look repetitive. This is what it now looks like: .
I spent some time tweaking various styles and layouts so everything works nicely on smaller screens. I try to make sure these things work when I make them, but it’s easy to miss a particular interaction, or make an innocent-seeming change that breaks tiny screens.
I added a new page that shows stats about my walks, based on my Apple Health data. I had a private view of this for a while, but decided there’s no reason it couldn’t be more public.
The films page has gained a search box, to directly find my review of a particular film. I used CSS anchor positioning for the results panel. Modern CSS is fun!
I fixed the 3D prints page being utterly broken in numerous ways. It showed up on my shiny new “how many SQL queries does each page do?” dashboard, as it was doing multiple queries for each individual print instead of just getting all the data in one go. Then I noticed a badly written redirect rule had broken all the images and download links. Whoops!
I did a lot of under-the-hood cleaning, most of which won’t be visible. Besides just making me happy, this means the contact form and nod buttons now work with Javascript disabled.
Finally, I added a page that shows my board game plays. The data comes from exports from bgstats. I have a bunch of ideas for other things to do with that data, but haven’t got around to it yet.

Phew. After writing all that out it feels like a really long month, not a short one!

Other projects

Not much going on with my other projects. The only big thing was goplum, my self-hosted monitoring/alerting tool, getting a 1.0 release. It adds a reminder feature so you can get alerted repeatedly if a service is down. It also reworks how plugins work so they’re compile-time instead of trying to be clever and dynamic, as that never really gave any benefits.

I’m still continuing my migration of projects to a private Forgejo instance. All of my personal repos are now sourced there, and mirrored out to GitHub, except one: my Dockerfiles. I’m not sure how I want to approach that yet, as the action that does the updating is pretty involved. I also have some repositories split off into their own GitHub organisations which I haven’t got around to mirroring yet.

Entertainment

Unlike the past few months, I didn’t touch many video games in February. The ones I’d been dabbling with didn’t really hook me enough to make me keep going back. I made up for it with lots of other media forms though. First up, a whole bunch of films:

I won’t write too much about them here — you can click through to read my reviews if you want — other than to call out the three excellent horror films I hadn’t seen before: Black Swan, The Cabin in the Woods, and Get Out. Not so long ago I would’ve said that I didn’t enjoy horror films, now they’re some my favourites.

I don’t have any TV show tracking on my website (yet?), as I haven’t really watched much recently. This month I started watching The Pitt though, and it rapidly become one of my favourite shows. By “started watching” I mean “watched every aired episode of both season 1 and season 2”. I highly recommend it. The only problem is that it’s hard to find anything to watch afterwards, as nothing else can quite measure up.

I’ve also been binging a book series: Beneath the Dragoneye Moons. It’s a LitRPG series: basically a fantasy setting that borrows systems from an RPG, like classes and levelling. Something about the genre really works for me, and Beneath the Dragoneye Moons is one of the more interesting series I’ve come across in the genre. I’ve been going through the series at the rate of about a book a day.

Finally, board games! Nothing particularly notable, but now I can embed a pretty display of what I actually played:

6 plays
4 plays
4 plays
4 plays
2 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play
1 play

All of these were online via Board Game Arena. I haven’t really had time for in-person games with all the TV, films and reading!

Around the web

Start all of your commands with a comma

An old post that was doing the rounds again on Hacker News. It’s an interesting idea. I recently aliased cc to launch Claude Code with some custom parameters, and it horribly backfired when I tried to compile a C program and remembered that cc was part of GCC. I haven’t transitioned all my personal commands to use a prefix yet, but I might.

Tactical tornado is the new default

An interesting post about software developers who act as “tactical tornadoes”, and how that’s basically the role that all LLMs fill. Whenever I use coding agents, I often find myself following behind them in my IDE cleaning up bits and pieces. It’s often faster than prompting them to do it. I definitely get the “cleaning up after a tornado” metaphor.

Amusingly, the agents will sometimes notice that you’re tidying up after them and ascribe it to an automated process, outputting something like “the automated linter hasn’t run yet” if you fall behind. There’s probably a deeper philosophical discussion about what it means when the agent thinks that the human is the automated part of the system…

My smart sleep mask broadcasts users’ brainwaves to an open MQTT broker

Another one for the “the ’s’ in ‘IoT’ stands for security” books. I find it hard to actually comprehend how these things get to market. Does nobody actually think about it, or do they know full well how bad it is and just launch it anyway?

The IndieWeb and Small web

Ask three people to define ‘IndieWeb’ and ‘Small web’ and you’ll get about 20 different answers. This post by Christiano almost exactly nails my own views on the terms, and explains them well. I think most personal sites (my own included) straddle both terms to some extent, but having the labels is useful in discussions.

Moving back to a dynamic website

2025-10-28T00:00:00Z

For the past few weeks I’ve been working on converting chameth.com from a static site into a dynamically generated site backed by a database. This is the exact opposite process to one I went through maybe a decade ago. So why the change, and what’s different?

I’d actually been toying with the idea of switching for a while. Every now and then I’d have a thought along the lines of “it would be nice if I could add X to my website… but how would that work with static site generation?”. Comments, private drafts, dynamic themes based on the date, and many more little bits and pieces. You can definitely do them with a static site generator, but it felt like going against the grain. The beauty of a static site is that it’s… well… static. Shoehorning in dynamic features takes away a lot of the benefits, or leads to having lots of weird seams where you join the static and dynamic pieces together.

A bunch of things have changed since I originally switched to a static site generator. In no particular order:

The advent of containers, and managing them with config like in docker-compose, means standing up and managing a database for the site is trivial.
I’ve learnt and really like Golang, which makes writing HTTP-based servers a breeze, and makes it easy to deploy the site as a single binary (in a container).
LLM agents are good enough that I can have it do all the boring CRUD work needed to actually make a dynamic site work (so many admin functions…).

The code is open source if you’re interested. I’ve made no effort to make it generally useful outside of this website, but it might serve as a useful reference if you want to do something similar. Now that’s done, I can safely procrastinate on all those dynamic features I had planned!

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).