// Personal website of Chris Smith

Recently a friend was deploying Centauri, my reverse proxy service that can obtain TLS certificates using ACME DNS-01 challenges. He asked a seemingly innocent question: “When Centauri is ‘waiting for record propagation’ is there an option to increase the time?” But that’s a question with a hidden, concerning XY problem. We both use the same DNS provider, and it definitely shouldn’t take more than a minute to see a DNS record after it’s been created. Plus, I don’t have issues with it timing out!

Centauri uses the excellent lego library to do the actual ACME dance, and I’ve dug into it before pretty extensively to debug other DNS issues. When I last looked, it used the local recursive DNS resolver to check for CNAMEs and find the authoritative server, then checked for record propagation there. So, naturally, I blamed Simon’s local resolver doing something weird. Turns out that was surprisingly close to correct, but not for the reasons I thought.

Unfortunately for my snap judgements, Simon had a perfect rejoinder: he’d deployed the exact same VM image before and Centauri worked fine. And also while we’d been speaking it had managed to get a certificate after waiting 58 seconds for DNS propagation. Simon suggested that maybe it was systemd-resolved caching something stale, but I dismissed that based on what I knew of how Lego worked.

Diving back into Lego

Of course, after I dismissed the idea, I immediately started doubting myself. Last time I was looking at DNS issues I traced the code all the way across the Lego codebase. This time I just had a coding agent explore and summarise for me. Turns out it DOES use the recursive resolver to check for the TXT challenge record.

The DNS functionality looks like this:

• CNAME query on the _acme-challenge record using the recursive resolver.
• Maybe an SOA query on the same, depending on the DNS provider, using the recursive resolver.
• Actually create the challenge record using the DNS provider.
• Query records recursively to discover CNAMEs using the recursive resolver. It doesn’t matter if this fails.
• Query the TXT record on every configured recursive resolver, blocking until they all work.
• Query the TXT record on the authoritative nameserver.

So yes, a stale response from the recursive resolver can indeed ruin the whole thing. Simon’s domain had a minimum TTL of 60 seconds, so that links in nicely with it working at 58 seconds and failing most of the time: the 1 minute timeout is racing the 1 minute TTL, somehow. But my domains use a much longer TTL and I can see renewals happening in my logs basically immediately. What gives?

NXDOMAIN caching and RFC fun

So I had a theory as to what was happening, but it still had some loose ends that I didn’t understand. I did get a pretty big hint when I started doing lookups from various locations, though. On my desktop if I do a CNAME query then a TXT query for a record that doesn’t exist, I get this:

$ drill CNAME nonexistent.chameth.com
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 12630
chameth.com.	3600	IN	SOA	chameth.com. root.chameth.com. 2026339162 7200 3600 86400 3600

$ drill TXT nonexistent.chameth.com
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 56402
chameth.com.	3590	IN	SOA	chameth.com. root.chameth.com. 2026339162 7200 3600 86400 3600

And from my server:

$ drill CNAME nonexistent.chameth.com
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 7807
chameth.com.	3600	IN	SOA	chameth.com. root.chameth.com. 2026339162 7200 3600 86400 3600

$ drill TXT nonexistent.chameth.com 
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 5069
chameth.com.	3600	IN	SOA	chameth.com. root.chameth.com. 2026339162 7200 3600 86400 3600

Notice how the TTL (the first number) decreases locally, but not on the server? That implies my local resolver is reusing the same result when I do the second query, and the resolver on the server isn’t. It turns out that on my desktop I use systemd-resolved, which sends requests to Tailscale, which passes them off to NextDNS. On the server they go straight to Tailscale and then onwards to NextDNS. So systemd-resolved is what’s causing the behaviour. Like Simon guessed right at the start.

But why is it actually doing that? It’s kind of annoying. Well, it’s doing what it’s meant to. RFC2308 says:

A negative answer that resulted from a name error (NXDOMAIN) should be cached such that it can be retrieved and returned in response to another query for the same <QNAME, QCLASS> that resulted in the cached negative response.

A negative answer that resulted from a no data error (NODATA) should be cached such that it can be retrieved and returned in response to another query for the same <QNAME, QTYPE, QCLASS> that resulted in the cached negative response.

Because the record doesn’t exist at all (there’s no other records for _acme-challenge: why would there be?) we get an NXDOMAIN response, and that “should” be cached for any query type. So when Lego queries for the CNAME record, we’re priming the cache with “_acme-challenge doesn’t exist” with a potentially huge TTL on that cache entry. When we then try and query the TXT record we created we get the cached NXDOMAIN back.

If the TXT record was created first then the CNAME query would get a NODATA response instead, which wouldn’t affect us as that is understandably cached per query type. But the point in Lego’s CNAME check is to decide where to put the TXT record, so it can’t just flip the ordering.

I apparently dodged this problem entirely on my servers, because nothing in that resolution path implements this particular caching behaviour.

But why now?

But Simon had this working before. It’s possible that his existing deployments are having this issue now, and are just retrying enough to win the race, but he would’ve noticed if they failed this much when he set them up.

So I did some archaeology. Here’s the timeline:

• February 2016: CNAME check added to deal with, well, CNAMEs.
• February 2016: Immediate follow-up making a failed CNAME check non-blocking.
• September 2024: Option added to check propagation using custom recursive resolvers.
• February 2026: That option now defaults to on.

And the mystery is solved: Lego v5 enabled the recursive check by default. That also handily explains why I thought its behaviour was different. It used to be! There were still two things I didn’t understand: why was it enabled, and why didn’t I realise when I upgraded to Lego v5? Looking back, I think this line of the changelog is meant to cover the behaviour:

[lib,dnsprovider] New approach of the DNS calls

The migration guide also mentions the flag being renamed, but not the fact that the default setting also flipped. Feels like burying the lede a little, but hey ho.

As for why, there doesn’t seem to be any documented rationale. Given the recursive resolver check was added to fix a fairly niche problem, it seems like an odd decision to me. The new defaults mean that any recursive resolver that caches NXDOMAIN responses across query types will delay the propagation check, and likely fail it entirely if the TTL is longer than the timeout. That’s… unfortunate to say the least. I opened a discussion suggesting the default gets flipped back but it doesn’t sound like that will happen before v6.

For Centauri, at least, I’ve fixed it by unconditionally disabling the recursive resolver check, taking it back to the Lego v4 behaviour. And after all this, Simon’s original hunch turned out to be spot on: it was systemd-resolved caching, but it was doing the correct thing. Pesky old-man Lego was the one responsible for dropping a rake on the ground then immediately stepping on it.