Chameth.com - posts like g15-ram-upgrade, http2-and-sni, utility-user-stories

HTTP/2 and TLS Server Name Indication

2024-05-25T00:00:00Z

I was recently alerted to a bug in Centauri, a simple reverse proxy I wrote. The initial report was that it was serving completely the wrong website, but only sometimes, and it behaved differently in different browsers, and no-one else could reproduce it.

I use Centauri for all of my web-facing services (including this site!) so it’s a little surprising such a major bug would have escaped my notice. Shane, who first noticed the bug, was persistent though and eventually managed to figure out some exact reproduction steps.

A brief overview of Centauri and SNI

Centauri originally only proxied HTTPS requests¹. When it receives a HTTPS request, it first looks at the Server Name Indication (or SNI) field in the TLS ClientHello message. It uses this field to determine which TLS certificate to respond with (as one Centauri instance will typically serve many websites across many domain names, each with their own certificate). That’s what the field exists for: before SNI, if you wanted to host two HTTPS sites on the same machine you’d need separate IP addresses for them!

Once the TLS session was established Centauri would read in the HTTP request, select which backend it was going to be sent to based on the SNI field, and then proxy it on. The HTTP request itself contains a Host header which identifies which host the request is for, but that will always be the same as the SNI field… or so I thought.

HTTP connection reuse

When accessing a website, your browser will request dozens of resources in a short space of time: the webpage itself, some stylesheets, maybe some scripts, plus any images, fonts, videos, etc. It would be extremely inefficient to open a new connection for each individual request, as setting up the connection requires several round trips between the client and the server.

To address this issue, HTTP/1.1 formalised the idea of “persistent connections”, which allow the client to keep a connection open and send another request once the first has completed. HTTP/2 takes this a step much further and allows full multiplexing — sending multiple requests at once and allowing the server to respond out-of-order.

Obviously, you can only reuse the connection if you’re requesting further resources from the same host: if your browser makes a request to example.com and that includes a script from example.net, it has to open a new connection for the other domain. However, HTTP/2 expands this slightly:

For “https” resources, connection reuse additionally depends on
having a certificate that is valid for the host in the URI. The
certificate presented by the server MUST satisfy any checks that the
client would perform when forming a new TLS connection for the host
in the URI.

Putting it all together

The reproduction steps that Shane figured out involved visiting sites hosted on two subdomains. The first site to be visited got “stuck” and subsequent requests to the other site were routed there instead. This only worked for one specific domain, though, and it turns out because that domain was configured in Centauri to use a wildcard TLS certificate (i.e., the certificate served for the request to the first site was also valid for the second site).

The certificate being valid for both sites allowed the browser to use the same connection. This breaks my assumption that the SNI field would always match the HTTP host, as all requests are sent over the same TLS connection that had the SNI field set to the first site’s subdomain. While perfectly in spec, the behaviour is quite surprising.

The fix for this was trivial: Centauri now checks the HTTP Host header instead of routing based on the SNI field. I found the bug itself interesting though, as it has such an awkward set of conditions for it to occur:

There must be multiple sites that share a certificate (the default behaviour in Centauri is to obtain one certificate per site)
A user must visit two of those sites
The browser must still have a connection open to the first site when visiting the second

It’s also one of those rare bugs where everything is working as intended, it’s just that the intention was slightly wrong for some reason. In this case it was because I wasn’t aware of the fairly significant shift in behaviour introduced in HTTP/2 for that one tiny part of the spec².

Thanks again to Shane for the debugging he did to figure this all out!

It now also proxies HTTP requests but only if they come over a Tailscale connection. Otherwise, plain HTTP requests are redirected to HTTPS. ↩︎
I think it’s this kind of thing that drives software devs to become carpenters or farmers. You don’t suddenly get a Door 2.0 specification that invalidates all your assumptions about how hinges work when certain people try to open it. ↩︎

Upgrading the RAM in a Dell G15 laptop

2023-07-29T00:00:00Z

The Dell G15

I currently use a Dell G15 laptop for work. It has served me well for a little over a year, but recently it has been struggling a little with my day-to-day workload. It came with 32GB of RAM — the highest possible specification at the time¹ — but that is apparently no longer enough for me.

For a recent project, I was working on a Rust library used in an Android app. That meant running the usual glut of Android tools (Android Studio, an emulator and at least one Gradle daemon) alongside a normal IDE (IntelliJ IDEA). Throw in a web browser and a couple of electron apps, and I often managed to use all 32GB.

When you start swapping memory out to an encrypted disk — even an SSD — it doesn’t make for great performance. At first, I tried to work around this by enabling the Linux out-of-memory (OOM) killer, but it turns out that it’s not too good with Electron apps: it will kill the large browser process, but then the small Electron wrapper will just respawn it.

Can it be upgraded or not?

The obvious solution to not having enough RAM is to add more RAM. A quick look in the manual showed this might not be possible, though. The manual includes the following “Memory specifications” table:

Description	Values
Memory slots	Two SODIMM slots
Memory type	DDR5
Memory speed	4800
Maximum memory configuration	32GB
Minimum memory configuration	8GB
Memory size per slot	8GB or 16GB
Memory configurations supported	8 GB, 1 x 8 GB, DDR5, 4800 MHz 16 GB, 1 x 16 GB, DDR5, 4800 MHz 16 GB, 2 x 8 GB, DDR5, 4800 MHz, dual-channel 32 GB, 2 x 16 GB, DDR5, 4800 MHz, dual-channel

That unambiguously says that an upgrade from 32GB is not possible. I gave up.

Later though, I was complaining about memory issues to a friend, and he pointed out a Dell forum thread where a couple of people claim to have successfully installed dual-channel 32GB modules. Since the alternative was getting an entire new PC after only a year, I decided to give it a go.

The upgrade attempt

I ordered a pair of Crucial 32GB DDR5-4800 SODIMMs, and after they turned up dismantled the laptop. The G15 comes apart pretty normally: there are uncovered screws on the bottom holding the lower part of the case on. With those removed and some gentle prying, it pops off, and you get access to the battery, GPU and motherboard.

The first thing I saw was this:

The G15 motherboard adamantly proclaiming that it doesn’t want 32GB SODIMMs

Not one but two labels that indicate it will only accept 8GB or 16GB modules. Oh well, what’s the worst that can happen?

Oops?

I dutifully installed the new modules, reconnected the battery and put the case back together. I pressed the power button, and… nothing. None of the usual garish lights that immediately come on, no screen output, just a dead laptop. After reading some more of the user manual, I found that there is a tiny status LED on the side next to the ethernet port. Forcing the laptop to power off and back on again, the status LED blinked a distress code at me: 2 amber blinks, 4 white blinks. The manual says that is — unsurprisingly — a memory fault.

I figured at this point that the manual and labels on the motherboard were probably right. I took the laptop apart again, reinstalled the original 2x16GB modules, reassembled it, and pressed the power button. It didn’t boot. I don’t spend a lot of time fiddling inside computers, but I’ve done it enough that I’m reasonably confident I can’t entirely break a computer while swapping some RAM modules. I took to Googling², and found an interesting article that said Dell laptops don’t like to boot after RAM changes unless you clear the CMOS by popping out the battery for 15 minutes.

I opened the laptop up, and looked around for the CMOS battery. There wasn’t one. Turns out they don’t exist any more. I left the main battery disconnected for a while to see if it would help, and it didn’t.

Unexpected success

I started to get worried: if I couldn’t fix this, I wouldn’t be able to work until I got a new PC, and that wasn’t really in my budget at the minute. I sat reading old forum threads and help guides³, none of which were actually useful. Out of nowhere, though, the laptop booted up.

Naturally, I immediately shut the laptop down again, opened it up, and switched back to the new RAM modules. Then I turned it on again and sat waiting. After about 15 minutes of it looking totally dead, it turned on and showed a BIOS warning about the hardware configuration being changed. It then booted perfectly normally, and all 64GB of RAM was visible and usable.

My theory is that the forum threads were right: Dell laptops are funny about RAM upgrades. But somehow in removing the physical CMOS battery, they’ve kept the same “you have to wait 15 minutes” behaviour just without any indication that’s what’s happening. Regardless, I now have enough RAM even for the greediest of IDEs and Electron apps.

Bizarrely, the maximum spec has decreased to 16GB since then. ↩︎
In the generic sense. I use Kagi these days. ↩︎
On my phone because, y’know, the laptop was busted. ↩︎

On the utility of user stories

2021-10-16T00:00:00Z

User stories are a staple of most agile methodologies. You’d be hard-pressed to find an experienced software developer that’s not come across them at some point in their career. In case you haven’t, they look something like this:

As a frequent customer,
I want to be able to browse my previous orders,
So that I can quickly re-order products.

They provide a persona (in this case “a frequent customer”), a goal (“browse my previous orders”) and a reason (“so that I can quickly re-order products”). This fictitious user story would probably rank among one of the better ones I’ve seen. More typically you end up with something like:

As a user,
I want to be able to login,
So that I can browse while logged in.

This doesn’t really provide a persona or any proper reasoning. It’s just a straight-forward task pretending to be a user story. If this is written in an issue then it provides no extra information over one that simply says “Allow users to login”. In fact, because it’s expressed so awkwardly I’d argue that it’s worse.

This kind of task-disguised-as-a-user-story problem becomes more obvious when people try to write technical tasks in the same way:

As a developer,
I want to refactor the JobFactory,
So that I can work with it more efficiently in the future.

This just says “Refactor the JobFactory”. If you wrote that in a ticket you’d probably feel bad for not describing it very well, but somehow when it’s dressed up as a user story it feels more valuable.

Does the user really want that?

One thing that really irks me about user stories is that it lets you twist your business objectives into sounding like they’re the user’s idea: the story becomes a post-hoc justification for a task you decided was required.

Say you’re making a mobile app for an online book store, and your team for whatever reason has a target of increasing the number of users who view the daily book-of-the-day offer. Maybe you do some interviews and users tell you they forget to check in each day to see what the offer is. How can you funnel more users there?

As a mobile app user,
I want to receive a push notification whenever a new book-of-the-day is available,
So that I have the chance to buy the book.

Some users probably do want such a thing, but I’d argue the vast majority of them do not. Imagine if every app on your phone alerted you whenever there was a new deal, or popped up a dialog whenever you went near a physical shop they had vouchers for…

Obviously in the cold reality of capitalism businesses make money by doing things not strictly in the interests of users¹. When written up like this it becomes so painfully disingenuous, though. Despite writing a user story that starts with the words “As a user”, you’re not really putting yourself in the user’s shoes.

Who even is the user?

One of the big benefits of user stories comes from using personas to describe users. It’s also one of the things that’s rarely actually done, at least in my experience. Looking at Google Trends you can see the rise of searches for “user story” as agile slowly takes over the world, but the movement in searches about personas is very slight:

Graph showing relative search frequency for user stories against user personas

A lot of the time stories are just written with “As a user”, or have some adjectives tacked on to the start (“as a mobile user”, “as a logged-in user”). The best I’ve ever seen in the wild was specifying the class of user like in the example at the start: “as a frequent customer”, or “as a visually impaired user”.

The problem with using these classes is it requires you to come up with them when you’re writing the story. Maybe one day you think about visually impaired users, but the next you don’t. Maybe Bob thinks about certain classes of users, but Alice concentrates on different ones.

The ideal way to solve this is to come up with personas that all the team understand. For example:

Kiera is addicted to books. She reads across genres, and often buys books to gift to her friends and family. She has piles and piles of books to read, but that doesn’t stop her ordering more if she sees a good deal. She likes receiving new books almost as much as reading them, and opts for the fastest delivery available.
Sharon is a slow, methodical reader. She buys one book at a time when she’s close to finishing her current one, and gifts or resells her old books. She almost exclusively reads Science Fiction, and will generally read complete series from start to finish. She’s short-sighted and often struggles when using mobile phones or computers.

You’d probably want 2-4 personas that collectively represent most facets of your userbase. They can be a lot more fleshed out than these - if you search for example user personas you’ll find many beautifully presented examples that have complete backgrounds including hobbies, education levels, and so on. Even with this minimal level of detail, though, I’d argue they’re more useful than just writing things in a standard user story form.

Thinking back to the story about push notifications, writing it to use one of these two personas forces you to think about the trade-offs involved. Kiera probably would like a notification, but it would annoy and possibly confuse Sharon. This then leads you down the path of considering how to accommodate both types of user — maybe adding it as an option, or doing some fancy machine learning, etc. The conversation is now focused around the users, rather than steamrolling over them to reach a business objective.

You’re holding it wrong

The problems I’ve described are not a problem with user stories per se, but rather common issues with how they’re used. But there’s only so many times you can tell people they’re holding a tool wrong before you have to accept that maybe the tool was badly designed.

Considering features from the perspective of multiple personas is the single best thing you can possibly do to ensure you’re providing value to your users. You don’t even need to write things in the typical user story style to benefit from this.

Tacking “As a user,” to the start of all your JIRA tickets isn’t being agile, and isn’t good for users, even if it lets you tick a box somewhere. In some cases this lip-service to users is actively detrimental to them. We should be valuing users and personas over stilted templates and check-box exercises (remind you of anything?).

As a user, I want to pay more money for the things that I buy, so that the company’s CEO can afford to go to space. ↩︎