Chameth.com - posts like debugging-beyond-the-debugger, escaping-spotify-the-hard-way but not how-i-get-things-done

Monthly Meanderings: March 2026

2026-04-01T00:00:00Z

Since last month’s update I’ve been unpleasantly reminded that I’m middle aged, through the medium of a dodgy knee. There’s nothing quite like not being able to stand up without groaning to underscore that you’re not young any more. The ongoing game of “will this seemingly mundane activity make my knee go funny again?” is a delight to play, too. I’m great at it.

Website updates

Two new blog posts this month: the longest way to represent a date is a short thought experiment on absurd date formats, and modern CSS is fun goes through some neat new CSS features I’ve used recently for this site.

I did some design tweaks this month: headers now have a blue background behind them¹, and a lot of components have been redesigned to have a consistent style. You can see that component in action in the new “now playing” widget I’ve added:

Linkin Park — Cut the Bridge From Zero Scrobbled 14m ago

Instead of each component defining its own styles (which were mostly copied and pasted), there’s now a single class that handles the drop shadow, border, background colour, and the optional title. It also applies a consistent format to elements that are clickable: they use the blue accent colour for their borders, and have a glow effect shown on hover.

The now playing widget is part of a larger bit of work I did on importing some music stats. I’m pulling in the data from my Navidrome instance. A new music page shows my most listened albums and artists. Navidrome doesn’t store a complete play history, just the last play and the count, so I can’t do “what I listened to last month” stats until next month.

Other projects

Other than a minor bug-fix update to contempt, my Dockerfile templating/updating tool, I’ve not done much on my other open source projects. I’ve got a bit more work to do finish migrating all my repositories from GitHub to my private Forgejo instance. I’ve also been thinking about mirroring my public repositories to Codeberg so there’s a non-GitHub way to access them.

Entertainment

I apparently only watched a single film in March:

The Substance

2026-03-18

I really didn’t get on with this. It’s too long for what it is, and over-the-top in a way that just didn’t work for me.

Demi Moore and Margaret Qualley were great, but nothing else really was. The cinematography and sound design were too affected, the plot was pretty shallow and obvious, most of the body horror fell into the uncanny CGI valley for me.

It’s not completely without merit: there were around 40 minutes in the middle where I was really into it, but that’s not really enough given it’s closing in on a 2.5 hour runtime.

There’s something to be said for the message it’s presenting about women in Hollywood, too, but it was very heavy handed. Dennis Quaid’s character is even called Harvey; it’s not so much a subtle nod as a sledgehammer to the face.

Instead of films, I’ve been watching a bunch of TV. After devouring The Pitt last month, I struggled to find something to fill the hole it left. I settled on catching up on Chicago Fire and Chicago P.D.. They’re both mostly about getting things done without personal drama becoming the primary focus. They’re not quite as grounded as The Pitt, but at least they’ve not completely gone off the deep end like 9-1-1 where the first responders end up in space somehow?

I’ve also watched the first two episodes of SNL UK. I didn’t have very high expectations, and it gave me a pleasant surprise. Both episodes so far have some hilarious sketches, and are obviously written by British comics with the typical darker, more sardonic humour that separates us from the USA.

Last but not least, board games! I spent a weekend at a tabletop/LAN event with friends, so got to physically play some games for a change. I also kept up with some turn-based games on Board Game Arena, as usual.

6 plays
4 plays
4 plays
4 plays
3 plays
3 plays
3 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play

Around the web

Warranty Void if Regenerated

Some really cool speculative fiction about a future where all software is written by LLMs, and the social effects that may have. Unfortunately the story itself is written with an LLM, and that becomes painfully obvious about a third of the way in. By that point I was already hooked, though.

My 2-step process for AI-free blogging

Speaking of LLM-generated text, this short piece by Manuel Moreale made me laugh. I’m by no means anti-LLM, but I really hate people publishing the output like it’s their own work. It breaks the social assumption that the writer put more effort in than the reader, and makes me wonder how much is hallucinated.

25 Years of Eggs

An interesting applied use of coding agents. I’m a sucker for personal statistics. I have no explicit desire to keep my receipts, or generate graphs of my egg consumption, but seeing the graphs makes me question that. The journey to get there is a great read, too.

I made a one-page notebook

The clever paper folding is interesting, but I really like some of the small touches on the page. There’s a nice little “in reply to” callout providing context at the top of the page, you can click to rotate the image further down, and there’s a details element containing a textual version of it. It has that great ‘small web’ vibe of someone who really cares about what they’re doing.

Workers who love ‘synergizing paradigms’ might be bad at their jobs

I haven’t read the full paper, but the summary given by the article is equal parts interesting and amusing. They developed a “Corporate Bullshit Receptivity Scale”, what more is there to say?

It’s meant to resemble painter’s tape, but I’m not sure quite how well it works. ↩︎

Further Adventures in Music Organisation

2025-09-21T00:00:00Z

I wrote before about how I’d dropped Spotify in favour of locally stored music, but things have advanced a bit since. I had a few issues: Tauon would occasionally manage to lose its database and along with it all my carefully constructed playlists and song ratings¹, and the experience on my phone was not very fun.

I had to manually sync the music by plugging my phone in to the computer, and sometimes it just refused to mount the right partition. I don’t think there’s really a good way to debug an Apple phone not behaving properly when connected to a Linux desktop. Then I started wanting more than one playlist synced², and trying to find a way to make that work just broke me.

I spent a while looking at different ways of hosting the music centrally. Plexamp gets lots of good reviews, and I’ve used Plex a fair bit. I set about spinning up a Plex server, and just could not get it working. The server ran fine, served the web interface, but would neither associate with my account nor run standalone. The docs were contradictory and there was very little useful logging. After a lot of frustration, I stumbled across mentions that Plex block running on Hetzner. I assume that’s the cause of my issues, although I have no way to know for sure. I use Hetzner for all my servers and other hosted services, but Plex have decided I can’t run the self-hosted software that I have a lifetime subscription for there. What the actual fuck?

I could have probably worked around the arbitrary restriction, but I didn’t want to throw more time down the drain. Instead, I set up Navidrome, an open source music server. It supports the Subsonic protocol, which means you can use a whole slew of different clients with it (or even write your own). It also means there’s a nice way to get data in and out of it programmatically, which I recall being a bit of a fight with Plex.

Syncing and organising

All my music lived on my desktop, but now I wanted it on my server. At first I just used rsync to copy everything up. I’d maintain a local “master” copy, and periodically shove the changes to the server³. That quickly got old, so I set up Syncthing to keep the two folders in sync. That worked… for a while.

Concurrently, I was looking at improving the organisation of my music. It was mostly organised by some Go programs I’d thrown together to automate the importing, with no real validation of metadata or anything else. Some albums got split up into multiple folders because the tracks had different artists (and didn’t have an album artist set); some artists ended up with multiple folders with slight spelling or case variations. It was upsetting.

I’d used MusicBrainz Picard before to fix some issues, but I didn’t really like the UI, especially when trying to do bulk actions. The main alternative is beets, which describes itself as “the music geek’s media organiser”, and is entirely command line. I was intrigued.

I started playing around with beets, and quickly noticed a problem. Every time I changed a metadata tag in a music file, Syncthing had to upload the entire file. I was frequently changing tags across the whole library as I got beets set up how I wanted, and Syncthing was handling it by uploading gigabytes of files for every minor change. I was already a bit unhappy with having the content duplicated in two places, so now I was using a command line organiser I figured I could just get rid of my local copy and move everything to the server.

I set up a Docker container for beets, and then wrote some incredibly hacky shell scripts so I could run beet locally on my desktop and it would SSH to the server over Tailscale and exec into the container, passing the arguments along. Then I did one final sync of the music library, got rid of Syncthing, double checked I had a backup and deleted all of my local music.

Bears, Beets, Battlestar Galactica

Beets is amazing. It has a vast array of plugins that can do almost anything you could want with a music library, and the command line workflow works really well for me. It’s very well documented, and all the individual parts are pleasingly simple and easy to understand.

Here’s what it looks like when importing some new music:

Beets importing an album

If it doesn’t get a perfect match then it shows the closest matches, and summarises what’s different about them (missing tracks, different names, etc), and lets you decide what to do. Aside from the metadata matching, it’s doing a lot of things under the hood that aren’t necessarily apparent. It:

Looks up the genre from Last.fm
Analyses all the tracks and writes ReplayGain metadata to them
Fetches album art
Checks all the files to make sure they’re actually playable
Scrubs any existing metadata tags

It also maintains its own database, and you can query against it. The query language is both simple and quite powerful, like the rest of beets. As an example:

$ beet ls artist:"Linkin Park" year:2025 length:2:00..2:30 
Linkin Park - From Zero - Casualty

Almost all the other commands also let you use the query syntax, which makes the whole tool really powerful. Want to delete all country tracks that you added last month? No problem. Redownload all the album art for a certain band? Easy. You can even do smart playlists using the filters.

Actually playing music

With all this organisation, I’ve not actually mentioned one tiny detail: how I actually play music now. Navidrome has a web UI, which is perfectly usable, but I don’t really want my web browser involved as it makes balancing sound levels tricky, and getting media keys working is a pain. Thankfully there are loads of Subsonic clients. I’ve tried several, but eventually settled on Feishin on the desktop. It’s very pretty:

Feishin’s ’now playing’ screen

What attracted me to it, though, was its built-in support for Navidrome’s smart playlists. These aren’t even properly exposed in Navidrome’s web UI yet, but Feishin has a visual editor that lets you create and edit them. I ended up with an identical system to the playlists I had in Tauon: a “blacklist” playlist which are tracks I never want to play, a “favourites” playlist, and then an “everything” playlist which is the entire catalogue minus the blacklist.

On iOS I’ve settled on Arpeggi. It’s not actually on the App Store yet, but available via TestFlight. It’s one of those really nicely polished apps that are obviously a labour of love, not just out to do the minimum possible to get your money. It can cache songs offline, and automatically download entire playlists, and supports all of the standard Subsonic features like rating, reporting plays back to the server, and so on.

One unexpected benefit of using a central server is that it handles reporting plays to Last.fm and ListenBrainz instead of the clients. I don’t think I’ve ever bothered to configure my mobile clients to do that before, and now it just works automagically. I’m relying on those services more for recommendations and discovery, as that’s something you lose out on when self-hosting.

Making bad decisions about bitrates

With all the moving, copying, and organising, I started thinking about how large the collection was. The songs were all in different formats, depending on when and where I’d picked them up, and I couldn’t tell the difference between them. I did a test and transcoded an MP3 file down to 128kbps, and still couldn’t tell the difference. So armed with that sample size of 1, I transcoded the entire library and saved so much space.

I don’t remember which track I did that first test with, but it must have been a very unlucky pick. I quickly started to notice the distortions caused by the low bitrate. It was particularly bad in any song with a lot of treble, and started to really annoy me. I started a painful process of reimporting things in their original format. Beets came in clutch again, both with the import process (if you import a duplicate album, it asks if you want to replace the original, and shows the format and bitrate of them for comparison), and keeping track of what was left to fix (beet ls bitrate:..128000).

Having started paying more attention to the sound quality, I think I’ve been nibbled on by the audiophile bug. I’ve been trying to get new music in FLAC format where possible. I’m pretty sure I can’t tell the difference between a decent bitrate MP3 and a FLAC, but maybe if I got a better DAC and some nice headphones…? Someone please hide my credit card!

As for the library taking up a lot of space, I now realise it’s a price worth paying. I might need to swap out the server to one with more storage at some point in the future, but it’s generally worth upgrading every few years with rented dedicated servers anyway, as you often get more for the same amount of money. Next upgrade I’ll just make sure there’s an appropriately-sized disk instead of focusing entirely on RAM and CPU.

Writing more code

Naturally, this whole process spawned several side projects. When browsing albums in Navidrome, I was a bit upset that the album art wasn’t all a consistent size. I thought about scripting something to crop them consistently, but then I had a better idea, and jewelcase was born. It takes the album art, crops it down to a consistent size, and then renders it inside a jewel case. It applies some slight effects, like adjusting the colours, rounding the corners, and tweaking the edges so it looks a bit more real. I don’t know why⁴, but every time I see the effect it gives me a little spark of joy.

Album art, before and after jewelcase is applied

The other project is BASS, a tool that uses the Subsonic API to grab information about my music catalogue, and then generates a “Daily Mix” playlist. It uses a system of weights to select tracks semi-randomly, biasing towards favourite tracks, those that haven’t been played much, and a few other criteria. This gives me a nice balance between having my favourites on repeat, and exploring the full library at random.

All the weights in BASS are customisable, so I can tweak it to my heart’s desire, and anyone else can also run it and configure it entirely differently to me if they want to. I can’t imagine I would’ve been able to do that if I were using Plex!

I was running a version from git as the stable release had fun dependency issues on Arch, so it’s possible this wouldn’t be an issue in normal use. I also started backing up the database, but it’s still annoying to have to restore it. ↩︎
A “favourites” playlist for me, and a more “family friendly” playlist for when I’m playing music out loud that has a bit less screaming/swearing/etc. ↩︎
One of the reasons for starting this was being annoyed by the periodic sync to my phone, obviously it makes perfect sense to introduce a new, different periodic sync. ↩︎
Nostalgia? A little bit of OCD? Both? ↩︎

Escaping Spotify the hard way

2025-07-25T00:00:00Z

I realised my media consumption was too close to this for comfort.

For the longest time I used Spotify for all my music needs. And I listen to a lot of music: sometimes actively, but mostly passively as background noise. I cancelled my premium subscription last December, and stopped using the service entirely. Why? There’s a bunch of reasons.

Let’s talk about the money first. Spotify launched at £9.99/month, and stayed that way for over a decade. Then in 2023 it went up to £10.99/month. That’s probably fair: the economy was in the toilet, and they haven’t changed their price in so long. Then in 2024 they upped it again to £11.99/month. Hmm. The service they were providing me didn’t improve in that time. I didn’t want audiobooks in my music player, I didn’t want an AI DJ that spouted inane comments at me. Paying them more money to do things I didn’t want seemed silly, and the money that actually goes to the artists is both so small and gets split in such a convoluted way that it’s not worth even thinking about.

My bigger concern was how uninvolved I became in choosing what to listen to. I leaned hard on their algorithmic playlists like “Discover Weekly” instead of manually curating playlists. But they ended up developing this weird feedback loop where it kept playing a country band¹ that I didn’t particularly like, but also didn’t dislike enough to skip. That acted as positive reinforcement, and they kept coming up everywhere. There were definitely ways to solve that by changing how I used Spotify, but the realisation that I was basically just consuming whatever was fed to me made me want to try something completely different. So I decided to go back to buying music instead. Which is where things got… complicated.

Getting some MP3s will be easy, right?

I figured with the money I was no longer giving to Spotify, I could just buy an album or two a month and slowly build up a nice music collection. I already had some music kicking around from before the streaming era, so there would at least be some variety in my listening while I got started. Where to buy music from, though?

Amazon used to do DRM-free music, but as far as I can tell it all got folded up into their own streaming service. Some smaller bands are on Bandcamp, but you won’t be able to grab Taylor Swift’s latest album from there. Qobuz have a good catalogue, but some of the stuff is pricey². It’d be cheaper to buy CDs and rip them like it was 1998 again.

I finally stumbled across an unlikely answer: the iTunes store³. This is still somehow separate from the behemoth of Apple Music, and if you fire up the iTunes software you can indeed buy reasonably priced, DRM free music. For comparison, 1989 (Taylor’s Version)⁴ is £12.89 (plus delivery) on CD from Taylor’s official website, £16.79 in CD quality from Qobuz, and £12.99 from iTunes.

Now, I use Linux, so using iTunes isn’t exactly easy. I ended up running a virtual machine with Windows on it. Every time I wanted to buy an album, I’d boot the VM, open iTunes, search for the album, press the buy button, enter my password, wait for it to download, copy all the files to a shared drive, and then back on Linux copy those files into my music directory. Smooth, right? But it worked. Earlier in the year I switched computers, though, and didn’t want to bother with the VM rigamarole. Spoiler alert: I didn’t make things any simpler.

Do you guys not have phones?

I tried a lot of things to avoid setting up a Windows VM again. I’m pretty sure I went through most of the 7 stages of grief in the process. I tried to get Apple Music to work, because that was obviously where all their development was focused, but you can’t actually buy music there (at least not on any version I could run on Linux). I tried running iTunes under Wine, but it just wouldn’t work. I looked into third-party clients, but unsurprisingly they didn’t go anywhere near payment flows.

I eventually realised that I have an Apple device in my pocket. After a bit of searching, it turns out there is actually still an iTunes app for iOS⁵. I bought an album⁶ to see if I could exfiltrate it. I connected my phone to the computer, paired it with idevicepair and then mounted the storage using ifuse⁷. Looking around, I could see a bunch of m4a files, and I could play them. Success!

Except… the file names were random hex strings, and they had no metadata. Yuck. I grabbed a copy of MusicBrainz Picard and had it sort out the mess. I stuck with this process for a while, but matching albums in Picard was sometimes a pain. MusicBrainz does a great job of figuring out what the tracks are, but it struggles to tie those back to a single edition of an album. I wasn’t aware how many subtly different versions of albums there were until I had to manually find the one iTunes had given me. Not fun. Eventually I got so fed up with this that I did something about it.

Plists all the way down

Armed with an instance of Claude Code, I set out to see if I could write some Go code to import the files properly. There were plists in the directory with the m4a files, and I know enough to know that they’re where all the juicy metadata lives on iOS. Maybe we can parse them and do something useful?

There’s a nice plist parser written in Go already, so I got Claude to write a program to dump the contents of one of the many plist files in the directory. Inside was all the metadata for one of the tracks. Yay! Just one small problem: there’s no way to link the plist file to the audio file. There’s one plist file that has a real name, though: StorePurchasesInfo.plist. I had Claude translate it to JSON and found that it contained a data field with a bunch of binary data. Huh.

I thought this was probably a dead end, but stuck the base64’d data into CyberChef and fiddled around a bit to get a hexdump of the binary. Six characters jumped out at me: bplist. I went back to Claude and suggested this might be a plist-in-a-plist scenario. Claude then got very confused because it couldn’t understand where the base64 was coming from⁸. After some LLM-induced suffering, I eventually got the contents of the inner-plist, and for each track it had a list of the audio file, the metadata, and the album art. Yay!

With the data figured out, it was fairly simple to get Claude to write code to parse everything and link them together, and after that move the files into their correct place in my library. I also had it take the metadata from the track’s plist file and write it into tags on the m4a so my media player didn’t have to understand Apple nonsense. I looked around for a Go library to do this, but couldn’t find one supporting m4a files, so ended up just shelling out to exiftool.

As a bonus, I made the tool automatically make a temporary directory and use ifuse to mount the phone, then unmount it when it’s finished. So now all I have to do is plug my phone in, and run import-music on the command line⁹. It’s a very nice quality-of-life upgrade that I probably wouldn’t have bothered with if I didn’t have Claude Code to speed the process up¹⁰.

Sometimes I actually play music too

With all this talk about buying and sorting music, I almost forgot about the most important bit: listening! I use Tauon, and it looks like this:

Tauon, showing off a wide range of different music in my library.

I tried a bunch of different music players, but Tauon hits the sweet spot of being simple enough to use but capable enough in the right places. I particularly like the playlist generators. My “Music” playlist just auto-imports all music in my “Music” folder, then I have a “Classical” playlist with a generator of s"Music" g"classical" auto. That makes it use the “Music” playlist as a source, filter on tracks that have “classical” in their genre, and automatically update when the base playlist changes¹¹. The “Disliked” playlist does the same but filters on rating: s"Music" a rat=0.5 auto. That takes all tracks (a) from the Music playlist, then filters it down to only those that have a rating of 0.5 out of 5¹². Finally, the “Normal” playlist imports everything from “Music” that’s not in one of the others. It sounds a bit clunky, but it works well in practice and I like the way it works.

Tauon in general just seems to work the way I’d expect it to, which is refreshing. You can just start typing anywhere to search for a track, hit enter, and it starts playing. It has keyboard shortcuts for other operations, and standard features like reporting to last.fm and other sites.

Tauon also supports transcoding and copying tracks to a music player or a phone. I use this to sync my main playlist to my phone. I’ll give you a minute to just bask in the glory of the rube goldberg machine I’ve created to move music from my phone, to my computer, and ultimately back to my phone. This does actually serve a purpose, though: it lets me control which tracks are on my phone. I don’t have to deal with excluding genres or tracks I dislike on the phone in addition to the computer. In the future I might opt to just sync a subset of my tracks to keep the storage size down. I’m telling you it makes sense! Stop looking at me like that!

Paying Attention Pays Off

I mentioned one of my aims for getting away from Spotify was to feel less like I was just consuming whatever’s in front of me. The corollary to that is how do you discover new music without an algorithm to feed it to you? I’ve ended up subscribing to a few blogs, paying more attention to music-related news, and being more aware of songs on the radio and from other sources.

When I find a song I like these days, I tend to listen to a few other tracks on the album (via YouTube, normally, as it’s the least friction) and decide if I want to buy it. When I do, I sit down and listen to the whole thing in order. It’s a radically different experience to just hearing a song now and then. There are themes and connections between tracks that you just miss if you listen to songs at random. Of course, you could do that on Spotify, but I don’t think it really lent itself to it.

My favourite example of this is my rediscovery of Linkin Park¹³. I’m of the age where they were around as I was growing up. I knew a couple of their big hits but younger-me wasn’t that into rock¹⁴, so I didn’t pay them much attention. Then I caught Up From the Bottom on the radio, and from there ended up buying and listening to all of From Zero. It’s an amazing album. The Emptiness Machine has become my favourite song: not only is it a great tune, performed well, it’s also a fantastic introduction to a new era for the band and to Emily Armstrong as the new vocalist. The rest of the album then builds on it to show off what they can do.

I’ve since gone back and reconnected with a lot of Linkin Park’s older albums, too. And from there discovered a lot of other rock bands I really like. Could this have happened with Spotify? Maybe, but it seems unlikely. It had me in a country-band-I-can’t-remember filter bubble, it seems pretty unlikely it’d drop some Linkin Park on me out of nowhere. And the passivity that came along with it would have meant that I probably wouldn’t have paid enough attention when I heard it from elsewhere.

Would I recommend that everyone runs out and cancels their Spotify subscriptions? Eh, probably not. If it works for you, that’s fine. Discovering recursive plists is fun for me, but I appreciate it’s not everyone’s cup of tea. What I would recommend is being a bit more deliberate in your choice of music. Don’t just tune in to a random playlist, or let the ‘DJ’ decide what to play. Find something, and actually listen to it. Then recommend it to me. I want more music!

I can’t even remember the name of the band, that’s how much I was into them. ↩︎
And they’re annoyingly deceptive about it. All the prices are “From £x”, where x is the discounted price you’d get if you paid them £15/month. ↩︎
I’d link to it, but… there isn’t a website. That’s not how things were done in the 1990s, silly. ↩︎
I don’t know why I’m so hung up on Taylor Swift in this post. ↩︎
Again, unrelated to the Apple Music app. ↩︎
It probably wasn’t a Taylor Swift album. Maybe. ↩︎
The Arch wiki has a useful page on this (of course). ↩︎
We were translating the plists to JSON, and if you serialise a byte array to JSON in Go it ends up as a base64-encoded string. Claude was then adamant it needed to base64 decode the binary plist. It did not. ↩︎
I just had the sudden realisation that I can probably automate running the import-music tool whenever I plug my phone in using systemd… ↩︎
I feel like I say that a lot lately. ↩︎
I like classical music, but it’s a mood, and I’m often not in that mood. ↩︎
I basically use this to filter out those introduction/bridge tracks on albums that are just talking and not singing. ↩︎
Surprise! My library isn’t actually just Taylor Swift songs. ↩︎
Younger-me was a dumbass, in some ways. ↩︎

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).