Chameth.com - posts like a-year-of-boardgames, debugging-beyond-the-debugger, utility-user-stories, why-you-should-be-using-https

If all you have is a hammer…

2025-06-18T00:00:00Z

This, dear reader, is a hammer. It is almost entirely irrelevant to the article. Enjoy.

I presume everyone is familiar with the idiom “if all you have is a hammer, everything looks like a nail”. If not, well, there it is. It’s generally used pejoratively about being single-minded, but I think it also gives a glimpse into something more interesting: mental and perceptual sets.

Before I explain, let me tell you a story about a person who bought a 3D printer¹. When they were first thinking about getting one, they weren’t sure if it was worth it. They could print a few board game accessories, but then what? After they got it, though, a whole new world opened up: everywhere they looked there were opportunities to improve things by adding 3D printed plastic. Broken appliances were repaired with 3D printed parts, all sorts of shelves, organisers, hooks and other things were made. But when they talked about this to other people, most often the response was “That’s nice, but I don’t think I’d use one”. How could they not see the truth in all its glorious layer lines?!

The answer is in the concept of a ‘set’², and instead of trying to explain it, I’m just going to quote Wikipedia:

In psychology, a set is a group of expectations that shape experience by making people especially sensitive to specific kinds of information. A perceptual set, also called perceptual expectancy, is a predisposition to perceive things in a certain way. […] A mental set is a framework for thinking about a problem. It can be shaped by habit or by desire. Mental sets can make it easy to solve a class of problem, but attachment to the wrong mental set can inhibit problem-solving and creativity.

This perfectly captures what happened. They’d developed a ‘3D printing mental set’: a predisposition to see problems that can be solved with their ~~hammer~~ 3D printer. Once I started noticing this pattern, I started seeing it all over the place.

Sets at work

Aside from the “could this be improved with some 3D printed plastic” perceptual set, I’ve noticed a couple of other sets: DIY, and writing puzzles.

DIY is probably the most obvious one. Once you become a little familiar with the basics of what you can fix and how, it seems like you start noticing the opportunities more. Before, you might have ignored that loose door handle, but now you notice it and realise with a screwdriver and a sex bolt³ you can fix it and stop it coming loose again anytime soon.

As you do more things, you build up your set of mental sets as well. You learn how to attach things to different types of walls, how to safely work with electricity, and so on, and suddenly installing a new outside light goes from a seemingly complex job to just exercising some of those concepts you’re already familiar with.

This works across ‘disciplines’ as well: I wanted to automatically cool a room down, using a fan in the window to blow in cool air. The 3D printing mental set gave me a way to make it physically go where I wanted it to, the home automation set gave me a plan for how to automate it, and the electricity set helped me figure out how to wire in the relay. It sounds a bit stilted when I spell it out like that, but the catch is I didn’t have to sit down and work any of this out. I was already primed to think in a certain way. I had a whole selection of hammers, and could pick the right combination of them to beat this particular nail⁴.

Puzzles are a bit different. I enjoy taking part in puzzle hunts⁵, and at first I was bewildered at how the authors came up with all the puzzles. After I started thinking about writing some myself, though, I began to notice all kinds of things that could maybe be turned into a puzzle: a specialised database, or a song with repetitive lyrics, or set of things with unique ID numbers. Writing the actual puzzles is still tricky, but once you’ve equipped the “look out for puzzle material” perceptual set it’s much easier to gather a range of ideas.

For example, one puzzle I wrote came about because I stumbled upon Sir Patrick Stewart’s Cowboy Classic Sampler⁶. I think everyone will agree that it’s a fairly unique collection. I tied some samples of the songs together with some screen captures of Sir Patrick as Picard in Star Trek: The Next Generation, and puzzlers had to find the season and episode numbers of the pictures, then use those numbers to select a track and a word from that track in the audio mix. Those words spell out a question, and the answer to that question is the answer to the puzzle.

There were other hints mixed in to point you in the right direction, it wasn’t quite as opaque as it might sound! But the point is without being in that “puzzle finding” mindset when I came across the album, there’s no way I could have come up with that puzzle. Like with the fan, I was primed to think a certain way, and managed to create something I otherwise would have struggled with.

The curse of noticing things

Figuring out how to make stuff isn’t the only place sets appear, though. Perceptual sets are about how we perceive things. When you read something with lots of bullet points, em-dashes, and the word “delve” do you automatically think it’s LLM generated?⁷ Then you have an LLM text perceptual set! It’s kind of annoying, as it’s difficult to “turn off” and stop noticing these things once you’ve seen them. Similarly, I think I have a “bad grammar” set. I know it doesn’t bother a lot of people, but if I’m reading a book and there’s a rogue apostrophe it will throw me right out of the moment. It’s like tripping up when walking and being torn out of a daydream. There’s some use to these — being able to notice grammar errors easily helps with my writing — but for the most part they’re a negative input: they point out things are broken or bad.

The “edge case set” takes this to another level. After about 20 years of writing software⁸ I have a finely honed ability to spot problems. Give me something to implement and my brain will immediately rattle off all the different edge cases and under-specified parts. Then some other bit of my brain gets the job of deciding which we actually care about, which are worth raising, and so on. It’s definitely useful to spot them up-front, rather than when you’re deep in the weeds implementing something, and tends to make everything go a lot smoother if you can hammer the major ones out when planning work. It doesn’t turn off after work though; it comes up when reading board game rule books, or descriptions of the magic system in fantasy novels, or news reports, and so on. It’s a lot less useful then.

Speaking of news reports, here’s a fun one: Gell-Mann amnesia is a cognitive bias where even when you critically assess media reports about something you’re knowledgeable about (say, technology), you continue to trust similar reporting in other areas (say, medicine). But once you know about Gell-Mann amnesia, you might start to actively notice when it happens. You scoff at the article about how the Internet is a “series of tubes” and then start reading the next one about a cancer treatment, but — wait a minute — your Gell-Mann amnesia set kicks in. I’m not sure what you then do with this information. Dismiss basically all the media and news as useless? It’s honestly pretty annoying.

The meta set

I’ve had this blog post on my ideas list since I first noticed the “3D printing mindset”. I didn’t know sets were an actual thing, and I didn’t have most of the examples I do now. As time went on with this idea rattling around in the back of my head, I started noticing the patterns come up more and more. There’s lots of more examples I didn’t include because I didn’t want to just write a listicle of mental patterns. But the fact that I noticed them all seems to mean that I’ve developed a set set, or a meta-set, if you will. It’s fascinating to think about how all this happens inside our mushy bag of neurons, but also a bit terrifying. If thought processes are that malleable, what stops you accidentally breaking them? Actually, thinking about things like Q-Anon, flat earthers, and a lot of contemporary “politics”, maybe that’s quite easy to do?

Image credits

Photo	Creator	Licence	Source
Hammer	Ian Talmacs	Unsplash License	Unsplash

Surprise! I am that person. Bet you didn’t see that coming! ↩︎
I feel like they should have consulted the mathematicians and/or computer scientists before they picked a name. ↩︎
Yes, that’s a real thing. No, I won’t ever use any of the alternative names for it. ↩︎
OK, I’ve strained the metaphor enough. I’m sorry. I’ll stop. ↩︎
If you’re not familiar, a puzzle hunt is a competition where you try to solve puzzles faster than other teams. The sort of puzzles involved are perfectly described by this quote from Mike Develin: “Imagine a word search. Now imagine you aren’t told what words to look for. Now imagine you aren’t told it’s a word search. Now imagine it isn’t a word search.” ↩︎
Go and give it a listen. I’ll wait. ↩︎
I hereby certify all my em-dashes are hand-crafted. Relatedly, Claude Code keeps writing comments that use hyphens when they should actually be em-dashes. Maybe they’ve trained it to avoid them? ↩︎
I just made myself sad by working this out. I still feel young at heart! … Or maybe that’s cholesterol? ↩︎

A year of boardgames

2024-07-26T00:00:00Z

I realised the other day that it’s a little over a year since I started building out a collection of board games¹, and thought it’d be fun to go through the collection and what I think about the games in hindsight.

The beginning

Ticket to Ride: Europe

I don’t remember how I discovered the concept of “modern” boardgames, but just over a year ago I purchased Ticket to Ride: Europe as my first. Prior to that I thought board games were like Scrabble, Monopoly, Trivial Pursuit, etc: they might provide a distraction occasionally, but didn’t amount to actual fun in the same way a video game might. I’d characterise those “classic” games as generally being extremely dependent on luck, not presenting many interesting choices, and generally being punishing if you’re behind.²

Ticket to Ride came as a breath of fresh air. You get to decide where to route your lines, whether you want to take on more cards, how aggressive to be in blocking the other players, and more. There’s still some luck involved, but it’s managed by being able to draw from a pool of visible cards, or being able to select which route cards you keep.

Shortly after I also picked up Carcassonne, and was similarly enamoured. I’ve not played either of them recently, though: they both fall into a weird little gap for me. While I enjoy the games, I don’t like playing them with certain groups³, but I don’t love them enough to prioritise them over other games when playing with other groups.

Pandemic

The next “gateway” game I picked up was Pandemic, which was my first co-operative game. I loved everything about it, and still do. The feeling when you managed to claw back a victory after setback after setback is great, but so is the shock when everything goes even worse than you thought possible!

Pandemic was also the first game where one of the mechanics caught my attention. The way epidemics work is that you take the bottom card from the infection deck, which becomes the center of the new outbreak, then shuffle that and all the discards back onto the top of the deck. It sounds simple, but it has big implications: everywhere that was previously infected can now be re-infected, and you’re guaranteed some tense rounds while you work back through those cards. It still blows my mind a little how shuffling and re-adding some card can mix up the game so much.

I still play Pandemic when I can, and it generally has a spot in my board game bag when I’m going away. I’ve since picked up the On the Brink expansion which adds some fun new twists⁴ to the base game, and Pandemic: Hotzone – Europe which plays similarly but is a bit quicker and smaller. I’ve also recently ordered Iberia, but it hasn’t materialised yet.

More gateway games

The Quacks of Quedlinburg

Over the next few months I picked up The Quacks of Quedlinburg, Azul (the mini variant), and Splendor. Quacks is probably my least favourite out of all the games so far: it’s not terrible, but it straddles deck-building and push-your-luck in a way that I don’t find particularly satisfying. It’s fun to play the combos you’ve invested in, and it’s fun to try and push your luck, but it’s not really fun when failing at the luck element means you don’t get to use the stuff you were building.

Quacks does bear the distinction of being the first game I customised, though: it got a little plastic organiser to hold all the ingredients instead of having to sort through a massive plastic bag with them all mixed in. My appreciation for game boxes with good built-in storage has gradually increased over the year, as have the tools I have for fixing those that don’t. But we’ll get back to that…

Azul is fun when you’re in the right mood; I’m not terribly into abstract strategy games, but Azul is light enough that it’s enjoyable. Splendor, on the other hand, I really like. So much so that I wrote a digital re-implementation of it. Both Azul and Splendor allow you to mess with your opponent a little, but not too much, which keeps it interesting but also means it still plays well in groups that don’t like too much head-on confrontation.⁵

I think the thing that really caught my attention in Splendor, though, is the quasi-engine building you do. It’s not actually classed as an engine builder on BoardGameGeek, but as you collect cards you get discounts on other cards, which lets you get better cards and further discounts, and so on. I love that kind of positive feedback loop in video games, and I apparently love it in board games as well.

Terraforming Mars

A recent game of Terraforming Mars

I’d heard lots of good things about Terraforming Mars, but I was worried it sounded a bit too complicated so put off buying it for a while. I mean, its weight on BoardGameGeek is orange. That’s scary, right?! Well… no. Terraforming Mars is easily my favourite board game, and I’ve played it pretty consistently since buying it. It has such a huge stock of cards that no two games feel the same: one game you might have an arsenal of cards that generate and use microbes; another you might have a plethora of science tags that let you play some of the most powerful project cards; next you might be spamming forests on Mars, or drowning in credits because you have too much income. It has the engine building element that I love, but I also really enjoy that it’s not always obvious who’s doing well and who’s not until the final score is tallied; that keeps everyone engaged and working on things for the entire game.

I’ve added in the Prelude expansion, which I wouldn’t play the game without now, and the Venus Next expansion, which adds some interesting bits but is nowhere close to being a hard requirement. I’ll probably pick up Colonies next. Prelude adds a new type of card that you take at the start of the game, which gives you a fairly hefty boost; these help you get off to a faster start, and may also affect what strategies you try to run with for the rest of the game. Venus Next adds another resource track, but you don’t have to complete it to end the game; it also adds a bunch of project cards that have “floater” resources, but they don’t feel that much different to the base game microbes⁶. One interesting thing that Venus Next adds, though, is a “solar phase” where one of the global parameters is increased at the end of each generation. The active player decides which parameter, but gets none of the benefits for it, so you get to decide entirely tactically.

There are two drawbacks to Terraforming Mars: it’s a little lengthy, and some of the components are a bit rubbish. Timing-wise, I find it takes a good hour and a half for a two player game, while adding a third player can almost double that. Obviously it depends who you’re playing with and what’s happening in the game, but that’s enough of a commitment that you more-or-less have to plan when to play instead of just spontaneously bringing it to the table. Component-wise, the game comes with basically no storage for anything other than little plastic bags that slosh around in the box; horrible plastic cubes for resource tracking that somehow jump out of your hand constantly, and flimsy printed sheets for putting those cubes on.

My custom player boards

The reason I got a 3D printer was in part so that I could make my own upgraded player boards that actually held the pieces in place instead of letting them fly everywhere if the table was slightly knocked. You can buy third party ones but I figured it was a teach-a-man-to-fish kind of situation, and dropping £200 on a 3D printer would save me money in the long run vs buying £50 of player boards someone else had made.⁷ About seven versions later, I have some boards which I think look and feel great. It’s 3D printed in two halves, then has a laminated 2D print sandwiched in the middle for all the markings and artwork.

As well as the player mats, I 3D printed some boxes, card holders, and so on, but have since replaced them with an insert from Folded Space. These are made of foam, and are perfectly sized to the game box and all the components in it. I could have eventually got to that point with 3D prints, but it would have taken a lot of time to design and print everything. Terraforming Mars was also the game that I first sleeved, as trying to shuffle the huge decks was an exercise in frustration. With all of the cards sleeved, not only are they more protected against wear and tear, you can now shuffle the deck just by gently mashing stacks of cards together. It makes shuffling so easy I now sleeve basically all my games as I hate having to riffle shuffle unsleeved cards.

Marrakech and HeroQuest

Later in the year I bought Marrakech after playing a giant version of it at a games expo. It sits somewhere between Azul and Splendor for me: it’s fun, and good when you want a short game, but it’s not high enough on my list to want to take it away with me. Playing with felt rugs does make a nice change from flipping cards or dealing with plastic components, though.

I also picked up HeroQuest at one point when it was on offer. It’s a good concept, and I felt like I would like it, but it was a bit of a let-down. The biggest problem for me is that you either need one player to basically be the DM, and manage the board, or you use an app to do that and have to deal with a bunch of admin and duplicating things because you’re half playing on the board and half in the app. I also don’t like relying on an app because I’ve been around long enough to know how easily computer software can become defunct, get removed from stores, get “updated” to remove features you need, and so on. I feel like the game would work well with five people: one to DM, and four to control the four heroes. But if you’ve got five people who want to play a dungeon crawler why aren’t you just playing D&D or a similar TTRPG? I’ll probably sell this at some point.

Party games

Codenames

These are all smaller games, and with most of them I’d played them before and decided I wanted my own copy. Codenames is my favourite party game, and I also picked up Codenames: Duet and they now live in the same box with a nice 3D printed insert. Codenames seems to just naturally produce fun moments: when an obscure clue is understood completely it feels amazing, but when it’s totally misunderstood it’s also generally hilarious. There’s also a great tension as the spymaster when you give a clue, then immediately realise it can be misinterpreted in a way that points straight at the assassin.

Poetry for Neanderthals is fun in the same party-word-game sort of way, but basically has no replayability. There are few enough cards that if you play a few times you can remember the “harder” word and just immediately guess it once you get the “easier” one being hinted (you can also sometimes just do this outright, which feels great but it kind of sucks from a gameplay point of view). The best bits are the inflatable “no stick” you get to bop people with, and how everyone speaks like a neanderthal for a little bit out of habit even after the game has ended. I seem to be weirdly good at this game, and I’m not sure what that says about me.

Love Letter has to take the spot of best game per cubic centimeter. It’s basically just a small set of cards, but it enables some really fun emergent gameplay. I really love how small the game is, and how easy it is to set up and play. It’s the perfect time filler. I’ve also played Lovecraft Letter, which adds a lot more complexity and was great fun, but is sadly out of print and I haven’t been able to find a copy at a sensible price.

Fluxx

Star Trek mega-Fluxx

My arbitrary 12 month period ends not long after I went to a tabletop event and played, amongst other things, Cthulu Fluxx, Doctor Who Fluxx, and Star Trek: The Next Generation Fluxx. Each game has the same core Fluxx elements, but adds slight twists along with the unique themes. I ordered a copy of Star Fluxx, but then found out that you can combine the four Star Trek variants (TOS, TNG, DS9 and VOY) together with some expansions to make a mega game so ordered all of those as well. While putting together my mega-Fluxx order I noticed they had a Firefly Fluxx that was out of print so I snapped up one of the last copies of that, too.

I ordered most of the games direct from Looney Labs, as the out of print ones weren’t readily available anywhere else. Unfortunately, UK Border Force managed to misread the import label⁸ and demanded £300 of VAT and import duties on them! They refunded it eventually, but this small set of tiny games were for a little while the most expensive things in my collection.

Fluxx is fun enough, but the mega-deck version is brilliant. It gets much more chaotic, with multiple captains, teleporters, and other doodads in play to steal cards back and forth. I was worried it wouldn’t play well with non-Trekkies, but the cards from each series use a different font and the weird people that can’t identify Star Trek series based on uniform variations seem happy enough relying on that.

Future

A few more months have passed since then, and my collection is still growing. Board game stores have summer sales, it turns out! If my physical wallet was in any way involved in the purchasing flow I’m sure it’d be crying.

I’ve recently started logging my plays with BG Stats (which synchronises the information up to Board Game Geek). I’m interested to see if the hard numbers actually align with my conceptions of how often I play certain games. If I do another roundup in a year’s time, expect it to have a lot more stats in…

Other words that may be appropriate here: “hoarding”; “compulsively obtaining”; “spending all my money on”, etc ↩︎
Think about a game of Monopoly: what do you actually get to decide? It basically comes down to “buy or not” if you land on an available property, then later on “buy more houses or not”. ↩︎
e.g. a family member doesn’t like the farmer mechanic in Carcassonne, and I don’t find the game interesting enough without them. Nothing wrong with the game or the people, we’re just not going to both enjoy it the way we like when playing together. ↩︎
Which is definitely why I bought it. Not because it comes with amazing little petri dishes to keep the disease cubes in. Certainly not. ↩︎
There’s a different edge to taking a card that someone else wanted in Splendor — there’s twelve others they can pick from, or they can take a face down one, etc; it might be an inconvenience but you haven’t totally upended their plans — versus blocking a really long train line they were trying to build in Ticket to Ride. ↩︎
Floaters are meant to be big pieces of infrastructure that float in the atmosphere of Venus. The fact that they feel exactly the same way as tiny little microbes when played seems wrong. ↩︎
Having since bought a more expensive printer, I think I’m past the point of convincing myself it’s “cheaper”. It’s a lot more fun, though, and if you design parts yourself you end up with exactly what you want! ↩︎
Turns out the “weight (lbs)” column is not the “quantity” column. Who knew? ↩︎

On the utility of user stories

2021-10-16T00:00:00Z

User stories are a staple of most agile methodologies. You’d be hard-pressed to find an experienced software developer that’s not come across them at some point in their career. In case you haven’t, they look something like this:

As a frequent customer,
I want to be able to browse my previous orders,
So that I can quickly re-order products.

They provide a persona (in this case “a frequent customer”), a goal (“browse my previous orders”) and a reason (“so that I can quickly re-order products”). This fictitious user story would probably rank among one of the better ones I’ve seen. More typically you end up with something like:

As a user,
I want to be able to login,
So that I can browse while logged in.

This doesn’t really provide a persona or any proper reasoning. It’s just a straight-forward task pretending to be a user story. If this is written in an issue then it provides no extra information over one that simply says “Allow users to login”. In fact, because it’s expressed so awkwardly I’d argue that it’s worse.

This kind of task-disguised-as-a-user-story problem becomes more obvious when people try to write technical tasks in the same way:

As a developer,
I want to refactor the JobFactory,
So that I can work with it more efficiently in the future.

This just says “Refactor the JobFactory”. If you wrote that in a ticket you’d probably feel bad for not describing it very well, but somehow when it’s dressed up as a user story it feels more valuable.

Does the user really want that?

One thing that really irks me about user stories is that it lets you twist your business objectives into sounding like they’re the user’s idea: the story becomes a post-hoc justification for a task you decided was required.

Say you’re making a mobile app for an online book store, and your team for whatever reason has a target of increasing the number of users who view the daily book-of-the-day offer. Maybe you do some interviews and users tell you they forget to check in each day to see what the offer is. How can you funnel more users there?

As a mobile app user,
I want to receive a push notification whenever a new book-of-the-day is available,
So that I have the chance to buy the book.

Some users probably do want such a thing, but I’d argue the vast majority of them do not. Imagine if every app on your phone alerted you whenever there was a new deal, or popped up a dialog whenever you went near a physical shop they had vouchers for…

Obviously in the cold reality of capitalism businesses make money by doing things not strictly in the interests of users¹. When written up like this it becomes so painfully disingenuous, though. Despite writing a user story that starts with the words “As a user”, you’re not really putting yourself in the user’s shoes.

Who even is the user?

One of the big benefits of user stories comes from using personas to describe users. It’s also one of the things that’s rarely actually done, at least in my experience. Looking at Google Trends you can see the rise of searches for “user story” as agile slowly takes over the world, but the movement in searches about personas is very slight:

Graph showing relative search frequency for user stories against user personas

A lot of the time stories are just written with “As a user”, or have some adjectives tacked on to the start (“as a mobile user”, “as a logged-in user”). The best I’ve ever seen in the wild was specifying the class of user like in the example at the start: “as a frequent customer”, or “as a visually impaired user”.

The problem with using these classes is it requires you to come up with them when you’re writing the story. Maybe one day you think about visually impaired users, but the next you don’t. Maybe Bob thinks about certain classes of users, but Alice concentrates on different ones.

The ideal way to solve this is to come up with personas that all the team understand. For example:

Kiera is addicted to books. She reads across genres, and often buys books to gift to her friends and family. She has piles and piles of books to read, but that doesn’t stop her ordering more if she sees a good deal. She likes receiving new books almost as much as reading them, and opts for the fastest delivery available.
Sharon is a slow, methodical reader. She buys one book at a time when she’s close to finishing her current one, and gifts or resells her old books. She almost exclusively reads Science Fiction, and will generally read complete series from start to finish. She’s short-sighted and often struggles when using mobile phones or computers.

You’d probably want 2-4 personas that collectively represent most facets of your userbase. They can be a lot more fleshed out than these - if you search for example user personas you’ll find many beautifully presented examples that have complete backgrounds including hobbies, education levels, and so on. Even with this minimal level of detail, though, I’d argue they’re more useful than just writing things in a standard user story form.

Thinking back to the story about push notifications, writing it to use one of these two personas forces you to think about the trade-offs involved. Kiera probably would like a notification, but it would annoy and possibly confuse Sharon. This then leads you down the path of considering how to accommodate both types of user — maybe adding it as an option, or doing some fancy machine learning, etc. The conversation is now focused around the users, rather than steamrolling over them to reach a business objective.

You’re holding it wrong

The problems I’ve described are not a problem with user stories per se, but rather common issues with how they’re used. But there’s only so many times you can tell people they’re holding a tool wrong before you have to accept that maybe the tool was badly designed.

Considering features from the perspective of multiple personas is the single best thing you can possibly do to ensure you’re providing value to your users. You don’t even need to write things in the typical user story style to benefit from this.

Tacking “As a user,” to the start of all your JIRA tickets isn’t being agile, and isn’t good for users, even if it lets you tick a box somewhere. In some cases this lip-service to users is actively detrimental to them. We should be valuing users and personas over stilted templates and check-box exercises (remind you of anything?).

As a user, I want to pay more money for the things that I buy, so that the company’s CEO can afford to go to space. ↩︎

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).

Why you should be using HTTPS

2016-06-17T00:00:00Z

The EFF’s HTTPS Everywhere logo

One of my favourite hobbyhorses recently has been the use of HTTPS, or lack thereof. HTTPS is the thing that makes the little padlock appear in your browser, and has existed for over 20 years. In the past, that little padlock was the exclusive preserve of banks and other ‘high security’ establishments; over time its use has gradually expanded to most (but not all) websites that handle user information, and the time is now right for it to become ubiquitous.

Why use HTTPS?

There are numerous advantages to using HTTPS, both for the users of a website and for the operator:

Privacy

The most obvious advantage is that HTTPS gives your users additional privacy. An insecure (HTTP) request can potentially be read by anyone on the same network, or the network operators, or anyone who happens to operate a network along the path between the user and the server.

Users on shared WiFi networks (such as those in coffee shops, hotels, or offices) are particularly vulnerable to passive sniffing by anyone else on that network. If the network is open (as is frequently the case) then anyone in radio range can see exactly what the user is up to.

Integrity

HTTPS also helps to maintain the integrity of your site. With a plain HTTP request, there’s nothing to stop anyone in between the server and the user from modifying the content of the request or the response. This is a frequent tactic used by annoying WiFi gateways (such as the ones you’d find in a hotel), dubious ISPs who want to serve you extra adverts, or just plain old nefarious attackers.

If you’re trying to convey some kind of information to users (and if you aren’t, why exactly are you running a website again?) it seems beneficial to both you and them if the information arrives as you intended, rather than in a modified form due to someone or something tampering with it.

Security

If your website has any kind of authentication, or session identifiers, it becomes extremely vulnerable to an attacker monitoring the traffic and stealing the credentials. This was starkly demonstrated in 2010 when Firesheep was released. This tool allowed anyone to quickly and automatically hijack social media accounts of anyone on the same network who was using HTTP to access them.

Even if your login pages are served over HTTPS, if you send a single session ID cookie over HTTP (such as a page you decided wasn’t particularly ‘important’) then an attacker can probably spoof the user’s session and gain full access to their account. Again, in the case of open WiFi networks that could be anyone in radio range.

Search engine rankings

Some search engines use HTTPS as a signal in their ranking algorithms. Google announced in 2004 that it was using the presence of HTTPS as a small positive signal, but that it may strengthen that signal over time as more and more websites switch to using a secure transport. It’s not unthinkable that at some point in the future there will be HTTPS-only search engines.

But… But… But…

There are lots of excuses for not implementing HTTPS. Most of them are either misguided or outdated.

It’s too expensive and/or complicated

In the past, getting HTTPS certificates was a pain. A number of free suppliers have existed for a while but the process for getting their certificates wasn’t particularly straight forward, and many imposed arbitrary restrictions on the certificate parameters. Even once you had the certificate, you had to fiddle about with your HTTP server configuration to make it work, remember to manually get a new certificate when the old one expired, and lots of other annoying busywork.

With the arrival of Let’s Encrypt, all that changed. You can retrieve and deploy a free HTTPS certificate with two or three commands. Renewal can be handled completely automatically with a single command executed by cron.

There’s no point; nothing on my site is sensitive

You might not think your content warrants privacy, but can you speak for everyone who accesses it? Even content that seems mundane to you — such as travel advice, or technical writing — could be used to build up a profile of a user. If an attacker is monitoring traffic in a coffee shop and sees a user looking at travel advice and weather forecasts for a foreign country, he could use that information to plan a burglary knowing that the user will be away. Similarly, some content which is perfectly mundane to you may actually be very sensitive in other countries with repressive governments. HTTPS makes it much harder for these people to snoop on traffic.

From another angle, if you’re offering any kind of information, instructions, or especially file downloads, there’s a severe risk to users if the content is modified on its way to them. An evil sysadmin could rewrite your travel advice to suggest visiting the local drug dealer’s hangout, or replace your download with a malware-infested version.

HTTPS is slower, uses more resources, etc

Back in 1995 this might have been a valid argument. Enabling HTTPS on a modern server will make an almost negligible difference to performance. If you also enable HTTP/2 (which most implementations only support over HTTPS), it’s likely to actually use fewer resources, and result in a faster, smoother experience for your users. HTTP/2 was designed to work with HTTPS, and designed with modern requirements and networking techniques in mind.

CloudFlare have an excellent demonstration of the benefits of HTTP/2, and it can show speed improvements of 2-3x in a typical environment. On top of being faster, HTTP/2 uses fewer connections which results in less resource overhead on both the server and the client.

So what are you waiting for?

If you run a website and aren’t using HTTPS, give it a try.