Chameth.com - posts like debugging-beyond-the-debugger, g15-ram-upgrade, modern-css-is-fun, why-you-should-be-using-https but not sense-api, travel-toolkit

Monthly Meanderings: March 2026

2026-04-01T00:00:00Z

Since last month’s update I’ve been unpleasantly reminded that I’m middle aged, through the medium of a dodgy knee. There’s nothing quite like not being able to stand up without groaning to underscore that you’re not young any more. The ongoing game of “will this seemingly mundane activity make my knee go funny again?” is a delight to play, too. I’m great at it.

Website updates

Two new blog posts this month: the longest way to represent a date is a short thought experiment on absurd date formats, and modern CSS is fun goes through some neat new CSS features I’ve used recently for this site.

I did some design tweaks this month: headers now have a blue background behind them¹, and a lot of components have been redesigned to have a consistent style. You can see that component in action in the new “now playing” widget I’ve added:

Gabrielle — Tell Me What You Dream Rise Scrobbled 0m ago

Instead of each component defining its own styles (which were mostly copied and pasted), there’s now a single class that handles the drop shadow, border, background colour, and the optional title. It also applies a consistent format to elements that are clickable: they use the blue accent colour for their borders, and have a glow effect shown on hover.

The now playing widget is part of a larger bit of work I did on importing some music stats. I’m pulling in the data from my Navidrome instance. A new music page shows my most listened albums and artists. Navidrome doesn’t store a complete play history, just the last play and the count, so I can’t do “what I listened to last month” stats until next month.

Other projects

Other than a minor bug-fix update to contempt, my Dockerfile templating/updating tool, I’ve not done much on my other open source projects. I’ve got a bit more work to do finish migrating all my repositories from GitHub to my private Forgejo instance. I’ve also been thinking about mirroring my public repositories to Codeberg so there’s a non-GitHub way to access them.

Entertainment

I apparently only watched a single film in March:

The Substance

2026-03-18

I really didn’t get on with this. It’s too long for what it is, and over-the-top in a way that just didn’t work for me.

Demi Moore and Margaret Qualley were great, but nothing else really was. The cinematography and sound design were too affected, the plot was pretty shallow and obvious, most of the body horror fell into the uncanny CGI valley for me.

It’s not completely without merit: there were around 40 minutes in the middle where I was really into it, but that’s not really enough given it’s closing in on a 2.5 hour runtime.

There’s something to be said for the message it’s presenting about women in Hollywood, too, but it was very heavy handed. Dennis Quaid’s character is even called Harvey; it’s not so much a subtle nod as a sledgehammer to the face.

Instead of films, I’ve been watching a bunch of TV. After devouring The Pitt last month, I struggled to find something to fill the hole it left. I settled on catching up on Chicago Fire and Chicago P.D.. They’re both mostly about getting things done without personal drama becoming the primary focus. They’re not quite as grounded as The Pitt, but at least they’ve not completely gone off the deep end like 9-1-1 where the first responders end up in space somehow?

I’ve also watched the first two episodes of SNL UK. I didn’t have very high expectations, and it gave me a pleasant surprise. Both episodes so far have some hilarious sketches, and are obviously written by British comics with the typical darker, more sardonic humour that separates us from the USA.

Last but not least, board games! I spent a weekend at a tabletop/LAN event with friends, so got to physically play some games for a change. I also kept up with some turn-based games on Board Game Arena, as usual.

6 plays
4 plays
4 plays
4 plays
3 plays
3 plays
3 plays
2 plays
2 plays
2 plays
2 plays
2 plays
1 play

Around the web

Warranty Void if Regenerated

Some really cool speculative fiction about a future where all software is written by LLMs, and the social effects that may have. Unfortunately the story itself is written with an LLM, and that becomes painfully obvious about a third of the way in. By that point I was already hooked, though.

My 2-step process for AI-free blogging

Speaking of LLM-generated text, this short piece by Manuel Moreale made me laugh. I’m by no means anti-LLM, but I really hate people publishing the output like it’s their own work. It breaks the social assumption that the writer put more effort in than the reader, and makes me wonder how much is hallucinated.

25 Years of Eggs

An interesting applied use of coding agents. I’m a sucker for personal statistics. I have no explicit desire to keep my receipts, or generate graphs of my egg consumption, but seeing the graphs makes me question that. The journey to get there is a great read, too.

I made a one-page notebook

The clever paper folding is interesting, but I really like some of the small touches on the page. There’s a nice little “in reply to” callout providing context at the top of the page, you can click to rotate the image further down, and there’s a details element containing a textual version of it. It has that great ‘small web’ vibe of someone who really cares about what they’re doing.

Workers who love ‘synergizing paradigms’ might be bad at their jobs

I haven’t read the full paper, but the summary given by the article is equal parts interesting and amusing. They developed a “Corporate Bullshit Receptivity Scale”, what more is there to say?

It’s meant to resemble painter’s tape, but I’m not sure quite how well it works. ↩︎

Modern CSS is fun

2026-03-17T00:00:00Z

I’ve been doing a bunch of CSS tweaking recently, and keep being surprised by how nice modern CSS is to work with. As someone grey-haired enough to remember writing HTML without CSS, it’s amazing to think how far along web technology has come¹. I wanted to demonstrate some of the handy bits and pieces I’ve used recently.

:has

The :has selector allows you to effectively query for child elements. While a span will match a <span> within an <a>, a:has(span) will match an <a> that contains a <span>. This really shines when combined with more complex selectors, for example:

input {
    border-radius: var(--border-radius);

    &:has(+ .results:not(:empty)) {
        border-bottom-left-radius: 0;
        border-bottom-right-radius: 0;
    }
}

This is part of the styling for my film search box. When the search box is presented initially it has fully rounded corners. But when it is followed by a non-empty results element, it removes the rounding on the bottom corners so that the border continues in a straight line down into the results. You can see it in action below; just enter a few characters (like “the”) to get some results:

You could achieve the same effect by having JavaScript add a class to the <input> element, but I’ll take a CSS solution over a JavaScript solution any day.

Nested rules

You probably spotted this in the example above. It’s what finally made me switch from SCSS to plain CSS. If you have a rule for .foo and a rule for .foo .bar you can just nest them. Not only does it save repeating yourself an awful lot, it keeps everything organised nicely. For example, the CSS for my film list embeds looks like this:

a.film-list {
    display: grid;
    grid-template-areas: "header images" "description images" "meta images";
    grid-template-columns: 1fr auto;
    grid-template-rows: auto 1fr auto;
    row-gap: var(--small-space);

    @media (width <= 800px) {
        grid-template-areas: "header" "description" "images" "meta";
        grid-template-columns: 1fr;
        grid-template-rows: auto auto auto auto;
    }

    & + & {
        margin-top: var(--medium-space);
    }

    h3 {
        font-size: var(--font-size-xxlarge);
    }

    .description {
        /* ... */
    }
}

Everything is wrapped up in a single bundle, including the @media rules for changing the layout on smaller screens, the lovely little & + & rule that adds some extra margin if there are two lists in a row, etc. The & syntax refers to the parent selector, so & + & in this case is the same as a.film-list + a.film-list: it’s very handy! The best part about this is that it’s the exact same syntax as SCSS, so converting is easy². The styled element looks like this:

Watched films ranked

Every film I’ve watched since I started logging, ranked.

Obviously super subjective, and subject to change often and arbitrarily.

264 films

Media range syntax

Once again, you might have spotted this in the previous snippet. I’m not deliberately teasing things, I promise! Back in the day, you did media queries like so:

@media (min-width: 1000px) and (max-width: 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I always hated this syntax. I always got muddled up as to whether I wanted “min” or “max”, and whether they were inclusive or not. It’s easy to reason through, but it never came naturally. Fortunately you can now just use ranges:

@media (1000px <= width <= 2000px) {
    /* some rules that work for screens at least 1000px wide and at most 2000px wide */
}

I find this style so much easier to write and understand.

Anchor positioning

Trying to dynamically position one element next to another used to exclusively fall within the purview of JavaScript. Fortunately anchor positioning fixes all this. I use this in the film search field I showed above:

.film-search {
    input {
        anchor-name: --film-search-box;
    }
    .results {
        position: absolute;
        position-anchor: --film-search-box;
        position-area: bottom center;
        position-visibility: always;
        width: anchor-size();
    }
}

This does lots of fun things. The .results element is anchored to the input field (via the --film-search-box name), and it’s positioned on the bottom, directly under the anchor. The position-area rule describes a 3x3 grid, with the anchor at the center, so top left would put it diagonally adjacent to the anchor. Next, position-visibility keeps the .results element visible when the anchor isn’t; otherwise when you scroll the anchor off the screen the results would immediately vanish. Finally, the special anchor-size() function makes the .results element take up the same exact width as the anchor.

All-in-all, this puts the results exactly where I want them, without having to deal with any JavaScript at all. I can see how it’d also be amazingly useful trying to do tooltips or other forms of ‘floating’ content too. You can even leave the positioning up to the browser, giving it hints about which order to try, or what property to optimise for (e.g. you can say “anchor to the left or right, whichever has more horizontal space”).

attr() function

This one blew my mind a little. In some of the elements on the site, I’ve added a little label to the top. It looks something like this:

It’s meant to just be a visual flourish, not a semantic title or anything, so I initially just added a ::before selector to each element, customising the content to have the right value. I didn’t like the duplication, though. Now, instead, I do this:

&::before {
    content: attr(data-title);
}

Each component defines its title in a data attribute: <div class="raised-box" data-title="an example">, and the attr function plucks it out and puts it in the content rule. It’s worth noting that this is only widely supported for the content rule, you can’t yet use it for colours or dimensions or other things. It still feels a bit magical, though. Like you’re making a reusable, customisable component with just HTML and CSS.

layers

I’d come across layers a few times, but I never found a need for them. It seemed like something you’d only really need with complicated design systems, or something. Then I had a problem and layers were the perfect solution!

The headings on this site have a whole bunch of CSS attached to them. Currently they look a bit like blue painter’s tape with handwriting on them. This style is applied to all headers, and then the various places that don’t want it had to manually reset everything. I got fed up with that, so decided to add a plain-header class which would “disable” the extra styling:

h2,
h3,
h4,
h5,
h6 {
    /* common styles for all headers, regardless */
    font-family: "Chris Hand", sans-serif;
    font-size: var(--font-size-xxlarge);
    color: var(--text-highlight-colour);
    margin: var(--medium-space) 0;
    line-height: 1;

    &:not(.plain-header) {
        /* do fancy stuff if it _doesn't_ have the .plain-header class */
        transform: rotate(-0.15deg);
        padding: var(--smedium-space);
        font-weight: 800;
        filter: drop-shadow(3px 3px 5px rgb(0, 0, 0));

        a {
            background: none;
            text-decoration-skip-ink: auto;
        }
    }
}

This might seem a bit backwards but I want the default to be styled, as those are far more common. I added this change, sprinkled around some “plain-header” classes, tidied up a lot of duplicate CSS, and things mostly worked. Mostly. Some things stopped working. In a few places the headers had slight tweaks, and they stopped working entirely.

It turns out in moving the rules from a plain h2 selector to a h2:not(.plain-header), I’d made them more specific. Previously a selector like h2.special would have been more specific, so its properties would override those from the less specific h2 selector. My first reaction was to try and hack around it. Changing all the overrides to body h2.special would make them more specific again, but I didn’t want to have to remember to do that forever more.

Instead, I defined some custom layers, and put the header definitions in one:

@layer reset, links, headings

@layer headings {
    h2,
    h3 {
        /* etc */
    }
}

The first @layer rule defines some layers that will be handled in a specific order: first the “reset” layer, then the “links” layer, then the “headings” layer, and then anything not in a layer after that. Rules in one layer don’t have to worry about beating the specificity of rules in another layer, because they’re handled separately.

Note that I couldn’t just put the headings in a layer and call it a day: their rules have to come after the CSS reset, and after the normal link styles, as headers have a bit of extra styling for nested links. If the CSS reset weren’t in a layer, then the * { margin: 0; padding: 0; } type reset would apply over the rules from the headings layer.

Using layers is definitely a bit fiddly, and is not necessary for a lot of sites, but it’s a much cleaner alternative to ugly specificity hacking.

Things I’m looking forward to

There are a few things that aren’t (widely) available yet, that particularly interest me:

sibling-index() and sibling-count()

These new functions are available in WebKit and Blink based browsers, but not Firefox. They give you the index of the element within its siblings, or the count of siblings. For the film list component above, I currently have this abomination for the overlapping posters:

&:nth-child(1) {
    z-index: 5;
    left: 0;
}
&:nth-child(2) {
    z-index: 4;
    left: calc(133px * var(--overlap) * 1);
}
&:nth-child(3) {
    z-index: 3;
    left: calc(133px * var(--overlap) * 2);
}
&:nth-child(4) {
    z-index: 2;
    left: calc(133px * var(--overlap) * 3);
}
&:nth-child(5) {
    z-index: 1;
    left: calc(133px * var(--overlap) * 4);
}

These could be replaced with something like:

z-index: calc(1 + sibling-count() - sibling-index());
left: calc(133px * var(--overlap) * (sibling-index() - 1));

As an added bonus, it would scale to any number of posters. index being 1-based not 0-based is a bit unfortunate as it makes using it in calculations awkward (see 1 offset in both of those rules!), but it’s leagues better than writing 5 separate rules.

random()

Currently only available in Safari. There are a few places where I’d like to have slight random variations of the style. Things like elements that are rotated slightly for aesthetics; they look a bit silly if they’re all identically positioned. I also randomise the icons and positions of my rating stars to break up the visual monotony. Here are a few examples:

Currently I do the stars by hardcoding a bunch of classes and having the backend randomly apply one when it generates the markup for stars. Being able to do it in CSS would be great, though.

Mixins

This one’s so far off it’s not even listed on caniuse.com yet. Mixins are another handy feature of SCSS, that allow you to define reusable blocks of rules, then import them when needed:

@mixin fancy-background {
    background: /*...*/;
}

.some-element {
    @include fancy-background;
}

#other-element {
    @include fancy-background;
}

This would be particularly useful if you can’t easily control the markup to add classes to everything. For example, my footnotes are rendered by a markdown plugin, and don’t easily have a way to add extra classes to them. To style it the same as another element, I currently duplicate a bunch of rules between them both. Mixins would allow me to define those rules once, and then import them in both places.

Aside from JavaScript, which seems to just endlessly reinvent new frameworks and ways to make the most sprawling dependency tree possible. ↩︎
It also means that if your website’s syntax highlighting library doesn’t seem to understand nested rules, you can just pretend your perfectly valid CSS is actually SCSS and it’ll magically work. Grumble, grumble. ↩︎

Upgrading the RAM in a Dell G15 laptop

2023-07-29T00:00:00Z

The Dell G15

I currently use a Dell G15 laptop for work. It has served me well for a little over a year, but recently it has been struggling a little with my day-to-day workload. It came with 32GB of RAM — the highest possible specification at the time¹ — but that is apparently no longer enough for me.

For a recent project, I was working on a Rust library used in an Android app. That meant running the usual glut of Android tools (Android Studio, an emulator and at least one Gradle daemon) alongside a normal IDE (IntelliJ IDEA). Throw in a web browser and a couple of electron apps, and I often managed to use all 32GB.

When you start swapping memory out to an encrypted disk — even an SSD — it doesn’t make for great performance. At first, I tried to work around this by enabling the Linux out-of-memory (OOM) killer, but it turns out that it’s not too good with Electron apps: it will kill the large browser process, but then the small Electron wrapper will just respawn it.

Can it be upgraded or not?

The obvious solution to not having enough RAM is to add more RAM. A quick look in the manual showed this might not be possible, though. The manual includes the following “Memory specifications” table:

Description	Values
Memory slots	Two SODIMM slots
Memory type	DDR5
Memory speed	4800
Maximum memory configuration	32GB
Minimum memory configuration	8GB
Memory size per slot	8GB or 16GB
Memory configurations supported	8 GB, 1 x 8 GB, DDR5, 4800 MHz 16 GB, 1 x 16 GB, DDR5, 4800 MHz 16 GB, 2 x 8 GB, DDR5, 4800 MHz, dual-channel 32 GB, 2 x 16 GB, DDR5, 4800 MHz, dual-channel

That unambiguously says that an upgrade from 32GB is not possible. I gave up.

Later though, I was complaining about memory issues to a friend, and he pointed out a Dell forum thread where a couple of people claim to have successfully installed dual-channel 32GB modules. Since the alternative was getting an entire new PC after only a year, I decided to give it a go.

The upgrade attempt

I ordered a pair of Crucial 32GB DDR5-4800 SODIMMs, and after they turned up dismantled the laptop. The G15 comes apart pretty normally: there are uncovered screws on the bottom holding the lower part of the case on. With those removed and some gentle prying, it pops off, and you get access to the battery, GPU and motherboard.

The first thing I saw was this:

The G15 motherboard adamantly proclaiming that it doesn’t want 32GB SODIMMs

Not one but two labels that indicate it will only accept 8GB or 16GB modules. Oh well, what’s the worst that can happen?

Oops?

I dutifully installed the new modules, reconnected the battery and put the case back together. I pressed the power button, and… nothing. None of the usual garish lights that immediately come on, no screen output, just a dead laptop. After reading some more of the user manual, I found that there is a tiny status LED on the side next to the ethernet port. Forcing the laptop to power off and back on again, the status LED blinked a distress code at me: 2 amber blinks, 4 white blinks. The manual says that is — unsurprisingly — a memory fault.

I figured at this point that the manual and labels on the motherboard were probably right. I took the laptop apart again, reinstalled the original 2x16GB modules, reassembled it, and pressed the power button. It didn’t boot. I don’t spend a lot of time fiddling inside computers, but I’ve done it enough that I’m reasonably confident I can’t entirely break a computer while swapping some RAM modules. I took to Googling², and found an interesting article that said Dell laptops don’t like to boot after RAM changes unless you clear the CMOS by popping out the battery for 15 minutes.

I opened the laptop up, and looked around for the CMOS battery. There wasn’t one. Turns out they don’t exist any more. I left the main battery disconnected for a while to see if it would help, and it didn’t.

Unexpected success

I started to get worried: if I couldn’t fix this, I wouldn’t be able to work until I got a new PC, and that wasn’t really in my budget at the minute. I sat reading old forum threads and help guides³, none of which were actually useful. Out of nowhere, though, the laptop booted up.

Naturally, I immediately shut the laptop down again, opened it up, and switched back to the new RAM modules. Then I turned it on again and sat waiting. After about 15 minutes of it looking totally dead, it turned on and showed a BIOS warning about the hardware configuration being changed. It then booted perfectly normally, and all 64GB of RAM was visible and usable.

My theory is that the forum threads were right: Dell laptops are funny about RAM upgrades. But somehow in removing the physical CMOS battery, they’ve kept the same “you have to wait 15 minutes” behaviour just without any indication that’s what’s happening. Regardless, I now have enough RAM even for the greediest of IDEs and Electron apps.

Bizarrely, the maximum spec has decreased to 16GB since then. ↩︎
In the generic sense. I use Kagi these days. ↩︎
On my phone because, y’know, the laptop was busted. ↩︎

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).

Why you should be using HTTPS

2016-06-17T00:00:00Z

The EFF’s HTTPS Everywhere logo

One of my favourite hobbyhorses recently has been the use of HTTPS, or lack thereof. HTTPS is the thing that makes the little padlock appear in your browser, and has existed for over 20 years. In the past, that little padlock was the exclusive preserve of banks and other ‘high security’ establishments; over time its use has gradually expanded to most (but not all) websites that handle user information, and the time is now right for it to become ubiquitous.

Why use HTTPS?

There are numerous advantages to using HTTPS, both for the users of a website and for the operator:

Privacy

The most obvious advantage is that HTTPS gives your users additional privacy. An insecure (HTTP) request can potentially be read by anyone on the same network, or the network operators, or anyone who happens to operate a network along the path between the user and the server.

Users on shared WiFi networks (such as those in coffee shops, hotels, or offices) are particularly vulnerable to passive sniffing by anyone else on that network. If the network is open (as is frequently the case) then anyone in radio range can see exactly what the user is up to.

Integrity

HTTPS also helps to maintain the integrity of your site. With a plain HTTP request, there’s nothing to stop anyone in between the server and the user from modifying the content of the request or the response. This is a frequent tactic used by annoying WiFi gateways (such as the ones you’d find in a hotel), dubious ISPs who want to serve you extra adverts, or just plain old nefarious attackers.

If you’re trying to convey some kind of information to users (and if you aren’t, why exactly are you running a website again?) it seems beneficial to both you and them if the information arrives as you intended, rather than in a modified form due to someone or something tampering with it.

Security

If your website has any kind of authentication, or session identifiers, it becomes extremely vulnerable to an attacker monitoring the traffic and stealing the credentials. This was starkly demonstrated in 2010 when Firesheep was released. This tool allowed anyone to quickly and automatically hijack social media accounts of anyone on the same network who was using HTTP to access them.

Even if your login pages are served over HTTPS, if you send a single session ID cookie over HTTP (such as a page you decided wasn’t particularly ‘important’) then an attacker can probably spoof the user’s session and gain full access to their account. Again, in the case of open WiFi networks that could be anyone in radio range.

Search engine rankings

Some search engines use HTTPS as a signal in their ranking algorithms. Google announced in 2004 that it was using the presence of HTTPS as a small positive signal, but that it may strengthen that signal over time as more and more websites switch to using a secure transport. It’s not unthinkable that at some point in the future there will be HTTPS-only search engines.

But… But… But…

There are lots of excuses for not implementing HTTPS. Most of them are either misguided or outdated.

It’s too expensive and/or complicated

In the past, getting HTTPS certificates was a pain. A number of free suppliers have existed for a while but the process for getting their certificates wasn’t particularly straight forward, and many imposed arbitrary restrictions on the certificate parameters. Even once you had the certificate, you had to fiddle about with your HTTP server configuration to make it work, remember to manually get a new certificate when the old one expired, and lots of other annoying busywork.

With the arrival of Let’s Encrypt, all that changed. You can retrieve and deploy a free HTTPS certificate with two or three commands. Renewal can be handled completely automatically with a single command executed by cron.

There’s no point; nothing on my site is sensitive

You might not think your content warrants privacy, but can you speak for everyone who accesses it? Even content that seems mundane to you — such as travel advice, or technical writing — could be used to build up a profile of a user. If an attacker is monitoring traffic in a coffee shop and sees a user looking at travel advice and weather forecasts for a foreign country, he could use that information to plan a burglary knowing that the user will be away. Similarly, some content which is perfectly mundane to you may actually be very sensitive in other countries with repressive governments. HTTPS makes it much harder for these people to snoop on traffic.

From another angle, if you’re offering any kind of information, instructions, or especially file downloads, there’s a severe risk to users if the content is modified on its way to them. An evil sysadmin could rewrite your travel advice to suggest visiting the local drug dealer’s hangout, or replace your download with a malware-infested version.

HTTPS is slower, uses more resources, etc

Back in 1995 this might have been a valid argument. Enabling HTTPS on a modern server will make an almost negligible difference to performance. If you also enable HTTP/2 (which most implementations only support over HTTPS), it’s likely to actually use fewer resources, and result in a faster, smoother experience for your users. HTTP/2 was designed to work with HTTPS, and designed with modern requirements and networking techniques in mind.

CloudFlare have an excellent demonstration of the benefits of HTTP/2, and it can show speed improvements of 2-3x in a typical environment. On top of being faster, HTTP/2 uses fewer connections which results in less resource overhead on both the server and the client.

So what are you waiting for?

If you run a website and aren’t using HTTPS, give it a try.