Published at 2022-08-04
Today, I finally added AAAA DNS records for all the domains I manage.
After adding these, I was able to ping each domain with
ping -6 <domain> and everything seemed to work fine.
Later that day, my mailserver greeted me with this:
I’m using mailcow as my mailserver’s software because I’ve fiddled with my custom mail config too often in the past and mailcow just works™. This includes automatically renewing Let’s Encrypt certificates before they expire. That means in theory, you should never run into a situation like the one from my screenshot above.
Unless something is very, very wrong.
Pretty much all servers I use and manage are Hetzner servers - always been satisfied with them. This particular mailserver is one that I started renting almost 4 years ago, in August 2018. It went through two debian dist upgrades just fine and has been handling all my mails ever since.
So, after getting the already mentioned tls error, I looked at the logs of the acme container. They contained a bunch of messages like this:
Detecting IP addresses…
OK: 159.xxx.xxx.xxx, 0000:0000:0000:0000:0000:0000:0000:0000
Found AAAA record for mail.kolaente.de: 2a01:xxxx:xxxx:xxxx::1 - skipping A record check
Cannot match your IP 0000:0000:0000:0000:0000:0000:0000:0000 against hostname mail.kolaente.de (DNS returned 2a01:xxxx:xxxx:xxxx::1)
I then checked the mailcow forums for someone with the same problem and found a bunch of similar results. Unfortunately most solution were along the lines of “Yeah I just did an upgrade and everything worked fine after that” which didn’t work for me.
More digging it was.
I’ve tried a few other things, none of them worked:
Eventually I figured out IPv6 didn’t seem to work at all on this server: I could ping the server ip from that server itself but wasn’t able to ping any IPv6 target or the server IP from another IPv6 target.
But I was used to IPv6 to just work™ on Hetzner vps, something had to be off here.
Next, I just got a new Hetzner Cloud vps and checked its network interface config. And sure enough, it was different.
Here is the relevant part of my config at
iface eth0:0 inet6 static
post-up route add -A inet6 default gw fe80::1%eth0 || true
pre-down route del -A inet6 default gw fe80::1%eth0 || true
And here is the one from the new server:
iface eth0 inet6 static
dns-nameservers 2a01:4ff:ff00::add:1 2a01:4ff:ff00::add:2
There’s three differences between the two worth pointing out:
:0 suffix. I don’t know anything about what the name should look like but that does not seem correct.
pre-down routes which do something with the IPv6 address. AFAIK they add iptable rules?
Once I changed my config file to match the one on the new server and rebooted, everything went back to work again. A bit disappointing of a solution, but I’m not going to complain.
I don’t know enough about networking under debian to understand this completly. My guess is Hetzner changed the rules at some point and the old ones don’t work with some newer part of the infrastructure. Remember, that server is almost 4 years old.
If you have a good explanation, I’d love to hear it!