Chameth.com - posts like http2-and-sni, shoring-up-sshd, why-you-should-be-using-https

HTTP/2 and TLS Server Name Indication

2024-05-25T00:00:00Z

I was recently alerted to a bug in Centauri, a simple reverse proxy I wrote. The initial report was that it was serving completely the wrong website, but only sometimes, and it behaved differently in different browsers, and no-one else could reproduce it.

I use Centauri for all of my web-facing services (including this site!) so it’s a little surprising such a major bug would have escaped my notice. Shane, who first noticed the bug, was persistent though and eventually managed to figure out some exact reproduction steps.

A brief overview of Centauri and SNI

Centauri originally only proxied HTTPS requests¹. When it receives a HTTPS request, it first looks at the Server Name Indication (or SNI) field in the TLS ClientHello message. It uses this field to determine which TLS certificate to respond with (as one Centauri instance will typically serve many websites across many domain names, each with their own certificate). That’s what the field exists for: before SNI, if you wanted to host two HTTPS sites on the same machine you’d need separate IP addresses for them!

Once the TLS session was established Centauri would read in the HTTP request, select which backend it was going to be sent to based on the SNI field, and then proxy it on. The HTTP request itself contains a Host header which identifies which host the request is for, but that will always be the same as the SNI field… or so I thought.

HTTP connection reuse

When accessing a website, your browser will request dozens of resources in a short space of time: the webpage itself, some stylesheets, maybe some scripts, plus any images, fonts, videos, etc. It would be extremely inefficient to open a new connection for each individual request, as setting up the connection requires several round trips between the client and the server.

To address this issue, HTTP/1.1 formalised the idea of “persistent connections”, which allow the client to keep a connection open and send another request once the first has completed. HTTP/2 takes this a step much further and allows full multiplexing — sending multiple requests at once and allowing the server to respond out-of-order.

Obviously, you can only reuse the connection if you’re requesting further resources from the same host: if your browser makes a request to example.com and that includes a script from example.net, it has to open a new connection for the other domain. However, HTTP/2 expands this slightly:

For “https” resources, connection reuse additionally depends on
having a certificate that is valid for the host in the URI. The
certificate presented by the server MUST satisfy any checks that the
client would perform when forming a new TLS connection for the host
in the URI.

Putting it all together

The reproduction steps that Shane figured out involved visiting sites hosted on two subdomains. The first site to be visited got “stuck” and subsequent requests to the other site were routed there instead. This only worked for one specific domain, though, and it turns out because that domain was configured in Centauri to use a wildcard TLS certificate (i.e., the certificate served for the request to the first site was also valid for the second site).

The certificate being valid for both sites allowed the browser to use the same connection. This breaks my assumption that the SNI field would always match the HTTP host, as all requests are sent over the same TLS connection that had the SNI field set to the first site’s subdomain. While perfectly in spec, the behaviour is quite surprising.

The fix for this was trivial: Centauri now checks the HTTP Host header instead of routing based on the SNI field. I found the bug itself interesting though, as it has such an awkward set of conditions for it to occur:

There must be multiple sites that share a certificate (the default behaviour in Centauri is to obtain one certificate per site)
A user must visit two of those sites
The browser must still have a connection open to the first site when visiting the second

It’s also one of those rare bugs where everything is working as intended, it’s just that the intention was slightly wrong for some reason. In this case it was because I wasn’t aware of the fairly significant shift in behaviour introduced in HTTP/2 for that one tiny part of the spec².

Thanks again to Shane for the debugging he did to figure this all out!

It now also proxies HTTP requests but only if they come over a Tailscale connection. Otherwise, plain HTTP requests are redirected to HTTPS. ↩︎
I think it’s this kind of thing that drives software devs to become carpenters or farmers. You don’t suddenly get a Door 2.0 specification that invalidates all your assumptions about how hinges work when certain people try to open it. ↩︎

Shoring up SSHd configuration

2016-10-18T00:00:00Z

I recently came across a useful tool on GitHub called ssh-audit. It’s a small Python script that connects to an SSH server, gathers a bunch of information, and then highlights any problems it has detected. The problems it reports range from potentially weak algorithms right up to know remote code execution vulnerabilities.

This is the kind of output you get when running ssh-audit. In this particular example, I’m looking at GitHub’s SSH server and have filtered the output to just warnings and failures:

Output of ssh-audit pointing at GitHub’s SSH servers

GitHub’s a bit of a special case, as they’re trying to cope with scores of developers pushing code: they can’t disable weaker algorithms without also stopping lots of people being able to use their service. Still, from the output you can see that ssh-audit has spotted a known vulnerability (CVE-2016-076) and has a lot to say about the various types of supported algorithms.

Background: crypto algorithms used by SSH

Establishing an SSH connection is a moderately complex endeavour, and various parts involve the use of a number of different cryptographic algorithms:

The first such algorithm is the key exchange algorithm. This is the process by which the client and the server agree on a shared key that will be used later. Next comes the host-key algorithm; this is how the server proves its identity to the client. Most SSH users will be familiar with warnings like the following:

$ ssh server.example.com
The authenticity of host 'server.example.com (11.22.33.444)' can't be established.
ED25519 key fingerprint is SHA256:rPVMho1fhEkJqvgce/8iAl353dX5QkGT9F3uCFndsa.
Are you sure you want to continue connecting (yes/no)?

The warning means that the SSH client doesn’t recognise the server’s key, and is asking the user to confirm it. If the key changes later, the SSH client will refuse to connect. In the warning above you can see the algorithm used by the server was ED25519.

Next up is the encryption algorithm, which handles actually encrypting the data sent over the connection. Finally comes the message authentication code algorithm, commonly referred to as ‘mac’. The mac algorithm is effectively responsible for signing each message as a proof that it came from the other party.

Following the recommendations

ssh-audit’s recommendations are pretty easy to follow. It points and shouts at a particular algorithm, and you configure SSHd to not allow it. This is a snippet from my new SSHd config, which gets no complaints from ssh-audit:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

KexAlgorithms curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

What’s more interesting is the reasoning behind some of the algorithms removed. The ecdh-sha2-nistp series of key exchange algorithms are subject to a sidechannel attack described in a paper in 2014. Some people are also concerned about the involvement of NIST, and the potential for backdoors. Various other key exchange algorithms use too small a number of bits in the key exchange (e.g. diffie-hellman-group1-sha1, which uses 1024). Others still use known-bad hash algorithms (e.g. diffie-hellman-group14-sha1, which uses an acceptable 2048 bit modulus, but relies on SHA1 hashes). ssh-audit only treats the use of SHA1 as a warning, but there’s no compelling reason to keep it around if you’re using remotely modern clients to connect. Similarly the host-key DSA algorithm uses a 1024 bit key, so should be disabled.

Many of the rejected encryption algorithms use basically-broken algorithms (3des-cbc and arcfour for example). Some of the remaining are block ciphers with small block sizes, which makes them weak (e.g. blockfish-cbc uses a block size of 64 bits).

Many of these concerns also apply to mac algorithms (e.g. eliminating hmac-md5, hmac-sha1-etm@openssh.com, etc, as they use weak hash algos). Of particular note, OpenSSH supports the hmac-ripemd160 and hmac-ripemd160-etm@openssh.com algorithms. RIPEMD160 isn’t that common but, like SHA1, is considered to be weak. One other concern with mac algorithms is the order in which the encryption and mac attachment are performed. Encrypt-then-mac is the preferred way of doing it (i.e., the message is encrypted, then a MAC of the ciphertext is attached). The default used in SSH is encrypt-and-mac, where the mac of the plaintext is attached after encryption. Attaching the plaintext mac potentially leaks information (a mac is designed to provide integrity, not confidentiality, after all). The encrypt-then-mac algorithms are indicated by the -etm suffix.

Other changes

In addition to the ssh-audit inspired changes, I took the time to review the rest of my standard SSH configuration. The config touches on a few areas; I’m only going to highlight a couple of them:

  PubkeyAuthentication yes
  RhostsRSAAuthentication no
  HostbasedAuthentication no
  ChallengeResponseAuthentication no
  PasswordAuthentication no

Here all authentication methods other than public key are disabled. A decent key (used in combination with good crypto algorithms!) is drastically harder to brute force than a very good password. It’s also less prone to accidentally being copied into the wrong place, provided to the wrong server, etc.

- UsePrivilegeSeparation yes
+ UsePrivilegeSeparation sandbox

Switching UsePrivilegeSeparation from ‘yes’ to ‘sandbox’ tells OpenSSH to employ kernel sandbox mechanisms on the unprivileged process. This adds another layer of defence in case there’s a severe exploit in OpenSSH itself.

An unexpected side effect

After reconfiguring OpenSSH, all of my servers stopped reporting SSH brute force attempts. Every day prior to the change saw hundreds of connections and, after rate limiting and automatic banning blocked a fair chunk, about two dozen unsuccessful login attempts. With the new algorithm selections in place, there were still hundreds of connections, but no failed login attempts at all. A closer look at the logs showed this:

fatal: Unable to negotiate with 1.2.3.4 port 55025:
  no matching key exchange method found. Their offer:
    diffie-hellman-group14-sha1,
    diffie-hellman-group-exchange-sha1,
    diffie-hellman-group1-sha1

Apparently not a single one of the clients trying to bruteforce their way in supported the one key exchange algorithm I now allow. I guess at some point they’ll be updated with a modern crypto stack, but until then it’s going to be oddly peaceful…

Why you should be using HTTPS

2016-06-17T00:00:00Z

The EFF’s HTTPS Everywhere logo

One of my favourite hobbyhorses recently has been the use of HTTPS, or lack thereof. HTTPS is the thing that makes the little padlock appear in your browser, and has existed for over 20 years. In the past, that little padlock was the exclusive preserve of banks and other ‘high security’ establishments; over time its use has gradually expanded to most (but not all) websites that handle user information, and the time is now right for it to become ubiquitous.

Why use HTTPS?

There are numerous advantages to using HTTPS, both for the users of a website and for the operator:

Privacy

The most obvious advantage is that HTTPS gives your users additional privacy. An insecure (HTTP) request can potentially be read by anyone on the same network, or the network operators, or anyone who happens to operate a network along the path between the user and the server.

Users on shared WiFi networks (such as those in coffee shops, hotels, or offices) are particularly vulnerable to passive sniffing by anyone else on that network. If the network is open (as is frequently the case) then anyone in radio range can see exactly what the user is up to.

Integrity

HTTPS also helps to maintain the integrity of your site. With a plain HTTP request, there’s nothing to stop anyone in between the server and the user from modifying the content of the request or the response. This is a frequent tactic used by annoying WiFi gateways (such as the ones you’d find in a hotel), dubious ISPs who want to serve you extra adverts, or just plain old nefarious attackers.

If you’re trying to convey some kind of information to users (and if you aren’t, why exactly are you running a website again?) it seems beneficial to both you and them if the information arrives as you intended, rather than in a modified form due to someone or something tampering with it.

Security

If your website has any kind of authentication, or session identifiers, it becomes extremely vulnerable to an attacker monitoring the traffic and stealing the credentials. This was starkly demonstrated in 2010 when Firesheep was released. This tool allowed anyone to quickly and automatically hijack social media accounts of anyone on the same network who was using HTTP to access them.

Even if your login pages are served over HTTPS, if you send a single session ID cookie over HTTP (such as a page you decided wasn’t particularly ‘important’) then an attacker can probably spoof the user’s session and gain full access to their account. Again, in the case of open WiFi networks that could be anyone in radio range.

Search engine rankings

Some search engines use HTTPS as a signal in their ranking algorithms. Google announced in 2004 that it was using the presence of HTTPS as a small positive signal, but that it may strengthen that signal over time as more and more websites switch to using a secure transport. It’s not unthinkable that at some point in the future there will be HTTPS-only search engines.

But… But… But…

There are lots of excuses for not implementing HTTPS. Most of them are either misguided or outdated.

It’s too expensive and/or complicated

In the past, getting HTTPS certificates was a pain. A number of free suppliers have existed for a while but the process for getting their certificates wasn’t particularly straight forward, and many imposed arbitrary restrictions on the certificate parameters. Even once you had the certificate, you had to fiddle about with your HTTP server configuration to make it work, remember to manually get a new certificate when the old one expired, and lots of other annoying busywork.

With the arrival of Let’s Encrypt, all that changed. You can retrieve and deploy a free HTTPS certificate with two or three commands. Renewal can be handled completely automatically with a single command executed by cron.

There’s no point; nothing on my site is sensitive

You might not think your content warrants privacy, but can you speak for everyone who accesses it? Even content that seems mundane to you — such as travel advice, or technical writing — could be used to build up a profile of a user. If an attacker is monitoring traffic in a coffee shop and sees a user looking at travel advice and weather forecasts for a foreign country, he could use that information to plan a burglary knowing that the user will be away. Similarly, some content which is perfectly mundane to you may actually be very sensitive in other countries with repressive governments. HTTPS makes it much harder for these people to snoop on traffic.

From another angle, if you’re offering any kind of information, instructions, or especially file downloads, there’s a severe risk to users if the content is modified on its way to them. An evil sysadmin could rewrite your travel advice to suggest visiting the local drug dealer’s hangout, or replace your download with a malware-infested version.

HTTPS is slower, uses more resources, etc

Back in 1995 this might have been a valid argument. Enabling HTTPS on a modern server will make an almost negligible difference to performance. If you also enable HTTP/2 (which most implementations only support over HTTPS), it’s likely to actually use fewer resources, and result in a faster, smoother experience for your users. HTTP/2 was designed to work with HTTPS, and designed with modern requirements and networking techniques in mind.

CloudFlare have an excellent demonstration of the benefits of HTTP/2, and it can show speed improvements of 2-3x in a typical environment. On top of being faster, HTTP/2 uses fewer connections which results in less resource overhead on both the server and the client.

So what are you waiting for?

If you run a website and aren’t using HTTPS, give it a try.