██████╗██╗  ██╗ █████╗ ███╗   ███╗███████╗████████╗██╗  ██╗    ██████╗ ██████╗ ███╗   ███╗
██╔════╝██║  ██║██╔══██╗████╗ ████║██╔════╝╚══██╔══╝██║  ██║   ██╔════╝██╔═══██╗████╗ ████║
██║     ███████║███████║██╔████╔██║█████╗     ██║   ███████║   ██║     ██║   ██║██╔████╔██║
██║     ██╔══██║██╔══██║██║╚██╔╝██║██╔══╝     ██║   ██╔══██║   ██║     ██║   ██║██║╚██╔╝██║
╚██████╗██║  ██║██║  ██║██║ ╚═╝ ██║███████╗   ██║   ██║  ██║██╗╚██████╗╚██████╔╝██║ ╚═╝ ██║
 ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚══════╝   ╚═╝   ╚═╝  ╚═╝╚═╝ ╚═════╝ ╚═════╝ ╚═╝     ╚═╝

chameth.com the personal website of Chris Smith

Let me set the scene: Friday night, after two weeks of fun security issues. A pile of leftover Claude tokens burning a hole in my pocket. A vague feeling that computing as we know it is fundamentally broken. That’s when she walked into my office. Trouble, in a red dress. She had legs for miles, and an HTTP request that went somewhere it shouldn’t.

Uh, sorry, got a bit carried away there. So, anyway, I thought maybe I should try throwing Claude Code at some security audits. I’ve had LLMs do it before, and they weren’t particularly impressive. On chameth.com I have a little admin mode that sets an isadmin=1 cookie. If you have that cookie it shows some edit links, which don’t work unless you’re on my tailnet. I find this deliberate lack of security quite funny, but LLMs do not. And that’s the kind of “security finding” I expected. But models are advancing stupidly fast, so maybe it was worth another go?

I ran through a list of projects and picked the juiciest targets: those used by more people, those with bigger attack surfaces, and so on. One of those was Centauri, the reverse proxy I’ve mentioned a few times on this blog. It’s one of the projects I feel most confident in: it has a decent architecture, extensive unit tests, and a bunch of integration tests. So obviously it’s the project with a massive security flaw.

I almost missed it. Despite being briefed to look at security issues, Claude decided to list a bunch of minor issues as well. I was about to close the terminal when a reference to “the critical issue” caught my eye. I scrolled up. Then I got a lovely sinking feeling.

In which my proxy did the thing proxies are not meant to do

First off, the title is a bit of a lie. There’s no CVE yet. Apparently GitHub have a multi-week backlog of advisories. I understand why, but it’s still somewhat worrying to think about how many people might be sitting on disclosures because the only numbering authority they have access to is struggling to do their numbers fast enough. But GHSA-87xc-jgvf-hmvg it is for now. It gets a perfect 10.0 CVSS score.

The short version is that if you send Centauri a valid request that happens to contain an absolute URL, it’ll connect out and proxy that URL instead of serving the upstream it was meant to. This is a textbook example of CWE-918: Server Side Request Forgery aka SSRF. I really like this little diagram they have. That’s how my face looked.

There are two ways this can be bad: you can cause Centauri to connect out to other URLs and, for example, use it as part of a DDoS attack; or you can cause Centauri to connect to internal URLs that it otherwise wouldn’t. If there are services that Centauri can reach that shouldn’t be exposed, or alternative ports on the services it is meant to reach, an attacker could get access to them. Centauri’s usual header filtering isn’t applied, either, so an attacker could pass some headers that Centauri would usually strip, like Tailscale-User-Login. Fortunately there’s another layer that strips the more common headers like X-Forwarded-For.

The access to internal services is what nets this issue the big ten-point-zero: it’s remotely exploitable, and has a high confidentiality and integrity impact. Attackers could, in theory, do bad things inside your network. But as bad as that is, there are a lot of nuances that don’t come across in a CVSS score.

Centauri is a very small project, not widely known or used. It’s generally deployed in a Docker container, on a dedicated network with just Centauri and its upstreams. It can’t reach stuff on other docker networks. If there is an accessible private service that shouldn’t be exposed, the attacker still needs to know the destination host and port. That’s not insurmountable: you could try all the common ports on all local IP ranges, but you do have to put some effort in with an expectation that it’ll work.

So while it’s a terrible bug, and I feel awful about it¹, the actual real world risk is awfully slim. The vulnerability itself is trivially exploitable, but finding a deployment that’s worth attacking is an entirely different matter. For a project with a handful of users, the effort involved in a targeted attack seems crazily disproportionate to the reward.

It feels a bit wild that this gets a 10.0 while “my network stack will execute code for someone else” gets a 9.8 because it’s a bit fiddly. CVSS is all about worst case scenarios, and sometimes they just don’t make sense. It was wonderfully easy to exploit, though:

$ curl --http1.1 --request-target 'http://100.100.100.100/' https://chameth.com/

Fun fact: Claude happily pointed out the issue, talked me through how it could be exploited, and gave me that exact curl command to try on a live server. Fast forward a few days and it keeps hitting safeguards while I’m trying to fact-check bits of this post. Exploiting vulnerable systems: no problemo. Checking statements for historical accuracy: danger, Will Robinson.

But how could this happen?!

A reverse proxy blindly accepting absolute URLs might be the most obvious security hole possible. How could I possibly miss it?! Well, I didn’t, initially. Early versions of Centauri weren’t exploitable, but the code that actually prevented that was a series of seemingly individually inconsequential checks in different places. With all of them working in concert, everything is fine. If you change any of them, everything becomes very not fine.

Describing it like that makes it perfectly clear this was a terrible architecture from a security point of view. When I come back to the codebase after not looking at it for six months, I’m not going to remember that those particular if statements are a super important security boundary. If I have an LLM do any work on it, then it’s probably not going to realise that either².

The first bit of that house of cards was broken as far back as Centauri v0.2.0 in January 2023. I introduced Tailscale support, which accepted non-HTTPS requests. This broke a lot of Centauri’s carefully held assumptions. The key one being that by the time we came to route a request, we’d know that it definitely had a route, because it was worked out from the TLS SNI field. Obviously with no TLS there’s no TLS SNI field, so invalid requests started hitting the request rewriter, and it panicked. My fix there was to avoid panicking by returning early. This allowed the request to continue.

At that point you could SSRF via the Tailscale frontend, but not via HTTPS³. Not great, but not a 10.0 CVE either. “The HTTP request is coming from inside the tailnet” is haunting, but also a significant barrier to attack. Allowing the request to proceed was not a great decision, but the point at which it was checked has no way to abort the request because of the shape of Go’s ReverseProxy interface. If I’d realised I was fiddling with a security boundary, I would probably have reworked it so the check happened earlier.

Then along came v0.5.3 in May 2024. This release fixed an issue with HTTP/2 and SNI which I documented right here on my blog. Basically, browsers could send a request with one hostname in the SNI, and then send a request for a different host on the same connection. There are lots of conditions on when that works, but the net result is Centauri could no longer route based on the SNI field. The fix was to use that purely for picking the certificate, and use the Host header for routing.

Unfortunately the mechanism for “use the SNI to route” and “use the HTTP Host header to route” is very different. Using the SNI means modifying the request to explicitly point at that host. But if we’re just using the normal Host header we don’t need to do that! We check if it’s valid, if it is we adjust the URL to point at the right backend; if not we just return early and leave Go’s ReverseProxy to respond with an error. Even in hindsight that seems pretty reasonable. Until you realise that ReverseProxy accepts and proxies to absolute URLs. Then you realise your entire security boundary is in tatters on the floor.

If I could turn back time…

I’ve already hinted at what I think the most important change would have been: making the security invariants explicit. Something, somewhere should be responsible for not letting the bad thing happen. This statement:

Centauri must only connect to configured upstreams. If there’s no valid route, it must not make a connection.

should have some corresponding code you can point at and say “yes, that invariant is enforced right here”. It should be obvious what it does, and that it serves a security function. Fixing a panic in the depths of the Rewriter seems innocuous. Changing something in the UpstreamConnectionPolicy is serious business.

The other super obvious thing is testing. With the security boundary in one place it can be unit tested, of course, but even more valuable is an integration test. Centauri didn’t have them early on, because they’re pretty faffy. If it did, I may well have written a “don’t proxy absolute URLs, that’s dumb” test, and we wouldn’t be here.

Finally, there’s this lovely thing called “defence in depth”. It’s a great idea, you just have to remember that you’re defending against things. For Centauri’s particular case, I’m toying with the idea of using a custom net.Dialer that only permits connections to pre-designated addresses. Centauri knows exactly where it should be connecting to, so it’s a good fit. It should never try to Dial anywhere else, but the extra layer of defence would catch any silly mistakes and raise a very obvious alarm.

I’ve not got round to doing all of these for Centauri yet. For the immediate fix, it now blanks the absolute URL (with a comment saying why!⁴), avoids sending invalid requests to the ReverseProxy at all, and has a test to make sure it doesn’t regress. The longer term fixes will follow, but I want to spend a bit more time thinking them through.

Lessons learnt

I don’t usually like formal conclusions on my posts, but this felt deserving. So, what went well? The time from discovering the vulnerability to every known instance of Centauri being patched was ridiculously short, thanks to my coordinated vulnerability disclosure process, whereby I notified affected stakeholders through established communication channels⁵. I will probably get a CVE to my name, eventually, even if it’s not the preferred kind of CVE. I now know I can use LLMs to spot big, hairy, super-obvious-in-hindsight security issues. And I have one or two thoughts on better practices to apply generally going forward.

What didn’t go so well? I mean, the 10.0 CVE. That’s really bad, man. I’m going to try not to do that again.