Chameth.com - posts like dns-over-tls-on-edgerouter-lite, http2-and-sni

HTTP/2 and TLS Server Name Indication

2024-05-25T00:00:00Z

I was recently alerted to a bug in Centauri, a simple reverse proxy I wrote. The initial report was that it was serving completely the wrong website, but only sometimes, and it behaved differently in different browsers, and no-one else could reproduce it.

I use Centauri for all of my web-facing services (including this site!) so it’s a little surprising such a major bug would have escaped my notice. Shane, who first noticed the bug, was persistent though and eventually managed to figure out some exact reproduction steps.

A brief overview of Centauri and SNI

Centauri originally only proxied HTTPS requests¹. When it receives a HTTPS request, it first looks at the Server Name Indication (or SNI) field in the TLS ClientHello message. It uses this field to determine which TLS certificate to respond with (as one Centauri instance will typically serve many websites across many domain names, each with their own certificate). That’s what the field exists for: before SNI, if you wanted to host two HTTPS sites on the same machine you’d need separate IP addresses for them!

Once the TLS session was established Centauri would read in the HTTP request, select which backend it was going to be sent to based on the SNI field, and then proxy it on. The HTTP request itself contains a Host header which identifies which host the request is for, but that will always be the same as the SNI field… or so I thought.

HTTP connection reuse

When accessing a website, your browser will request dozens of resources in a short space of time: the webpage itself, some stylesheets, maybe some scripts, plus any images, fonts, videos, etc. It would be extremely inefficient to open a new connection for each individual request, as setting up the connection requires several round trips between the client and the server.

To address this issue, HTTP/1.1 formalised the idea of “persistent connections”, which allow the client to keep a connection open and send another request once the first has completed. HTTP/2 takes this a step much further and allows full multiplexing — sending multiple requests at once and allowing the server to respond out-of-order.

Obviously, you can only reuse the connection if you’re requesting further resources from the same host: if your browser makes a request to example.com and that includes a script from example.net, it has to open a new connection for the other domain. However, HTTP/2 expands this slightly:

For “https” resources, connection reuse additionally depends on
having a certificate that is valid for the host in the URI. The
certificate presented by the server MUST satisfy any checks that the
client would perform when forming a new TLS connection for the host
in the URI.

Putting it all together

The reproduction steps that Shane figured out involved visiting sites hosted on two subdomains. The first site to be visited got “stuck” and subsequent requests to the other site were routed there instead. This only worked for one specific domain, though, and it turns out because that domain was configured in Centauri to use a wildcard TLS certificate (i.e., the certificate served for the request to the first site was also valid for the second site).

The certificate being valid for both sites allowed the browser to use the same connection. This breaks my assumption that the SNI field would always match the HTTP host, as all requests are sent over the same TLS connection that had the SNI field set to the first site’s subdomain. While perfectly in spec, the behaviour is quite surprising.

The fix for this was trivial: Centauri now checks the HTTP Host header instead of routing based on the SNI field. I found the bug itself interesting though, as it has such an awkward set of conditions for it to occur:

There must be multiple sites that share a certificate (the default behaviour in Centauri is to obtain one certificate per site)
A user must visit two of those sites
The browser must still have a connection open to the first site when visiting the second

It’s also one of those rare bugs where everything is working as intended, it’s just that the intention was slightly wrong for some reason. In this case it was because I wasn’t aware of the fairly significant shift in behaviour introduced in HTTP/2 for that one tiny part of the spec².

Thanks again to Shane for the debugging he did to figure this all out!

It now also proxies HTTP requests but only if they come over a Tailscale connection. Otherwise, plain HTTP requests are redirected to HTTPS. ↩︎
I think it’s this kind of thing that drives software devs to become carpenters or farmers. You don’t suddenly get a Door 2.0 specification that invalidates all your assumptions about how hinges work when certain people try to open it. ↩︎

DNS-over-TLS on the EdgeRouter Lite

2017-12-17T00:00:00Z

An EdgeRouter Lite

DNS-over-TLS is a fairly recent specification described in RFC7858, which enables DNS clients to communicate with servers over a TLS (encrypted) connection instead of requests and responses being sent in plain text. I won’t ramble on about why it’s a good thing that your ISP, government, or neighbour can’t see your DNS requests…

I use an EdgeRouter Lite from Ubiquiti Networks at home, and recently configured it to use DNS-over-TLS for all DNS queries. Here’s how I did it.

Installing unbound

Out of the box, the ERL uses dnsmasq to service DNS requests from local clients. To get DNS-over-TLS support I switched to using Unbound, an open source DNS resolver with support for many modern features such as DNSSEC and DNS-over-TLS.

Installing unbound on the ERL is a simple case of SSHing in, and then:

sudo apt-get update
sudo apt-get install unbound

And then configuring the ERL to use the new local resolver for DNS requests, turn off dnsmasq, and tell DHCP clients to send DNS requests to it (obviously substituting network names and subnets as appropriate):

set system name-server 127.0.0.1
set service dhcp-server shared-network-name lan1 subnet 192.168.1.0/24 dns-server 192.168.1.1
set service dhcp-server use-dnsmasq disable
set service dns

At this point DNS should still work, but Unbound will still be sending requests out in plaintext.

Configuration

The unbound configuration lives in /etc/unbound/unbound.conf, here’s a basic example that I use to enable DNS-over-TLS:

# Unbound configuration file for Debian.
#
# See the unbound.conf(5) man page.
#
# See /usr/share/doc/unbound/examples/unbound.conf for a commented
# reference config file.

server:
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    verbosity: 1
    interface: 0.0.0.0
    interface: ::0
    port: 53
    do-ip4: yes
    do-ip6: yes
    do-udp: yes
    do-tcp: yes
    access-control: 192.168.0.0/16 allow
    access-control: 127.0.0.0/8 allow
    access-control: 10.0.0.0/8 allow
    root-hints: "/var/lib/unbound/root.hints"

    hide-identity: yes
    hide-version: yes
    harden-glue: yes
    harden-dnssec-stripped: yes

    cache-min-ttl: 900
    cache-max-ttl: 14400
    prefetch: yes
    rrset-roundrobin: yes
    ssl-upstream: yes
    use-caps-for-id: yes

    private-address: 192.168.0.0/16
    private-address: 172.16.0.0/12
    private-address: 10.0.0.0/8

    logfile: "/var/lib/unbound/unbound.log"
    verbosity: 0
    val-log-level: 3

forward-zone:
    name: "."
    forward-addr: 9.9.9.9@853

Notice the server directive ssl-upstream, and that the forward zone specifies the quad9 resolver on its TLS port (853).

DNSSEC

To enable Unbound to validate DNSSEC signatures, we need to provide it with some information about the root nameservers that we trust. First, download the list of root nameservers to the root-hints file specified in the unbound config:

wget ftp://FTP.INTERNIC.NET/domain/named.cache -O /var/lib/unbound/root.hints

Then we need to add the root keys to the auto-trust-anchor-file. The trust anchor at the time of writing is below, but you can get the latest values from the IANA.

.       172800  IN      DNSKEY  257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQu
.       172800  IN      DNSKEY  257 3 8 AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjFFVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoXbfDaUeVPQuY

Redirecting unencrypted requests

I have a slew of devices on my network that, over time, I have configured to use 8.8.8.8 as a DNS server. They’re not going to care about the DHCP reply, and I don’t really feel like going around checking every weird and wonderful internet-connected device in the house, so I decided to just intercept requests to 8.8.8.8 and send them to Unbound. A simple NAT rule does the trick:

set service nat rule 1 description "HonestDNS Redirect"
set service nat rule 1 destination address 8.8.8.8
set service nat rule 1 destination port 53
set service nat rule 1 inbound-interface eth0
set service nat rule 1 inside-address address 192.168.1.1
set service nat rule 1 inside-address port 53
set service nat rule 1 log disable
set service nat rule 1 protocol tcp_udp
set service nat rule 1 type destination

You could of course redirect any traffic to port 53, but that would prevent you from explicitly querying any other DNS server. By just intercepting traffic to 8.8.8.8 I’m taking care of the vast majority of my statically configured devices, and can still issue manual queries to other resolvers when needed.

Validating

To check that everything is working, you can use tcpdump on the router to inspect packets on the WAN interface directed at port 53:

sudo tcpdump -Xi eth0 port 53

You should, hopefully, not see anything.