Getting IPv6 to work on an older Hetzner Cloud server

Today, I finally added AAAA DNS records for all the domains I manage. After adding these, I was able to ping each domain with ping -6 <domain> and everything seemed to work fine. Later that day, my mailserver greeted me with this:

I’m using mailcow as my mailserver’s software because I’ve fiddled with my custom mail config too often in the past and mailcow just works™. This includes automatically renewing Let’s Encrypt certificates before they expire. That means in theory, you should never run into a situation like the one from my screenshot above.

Unless something is very, very wrong.

Pretty much all servers I use and manage are Hetzner servers - always been satisfied with them. This particular mailserver is one that I started renting almost 4 years ago, in August 2018. It went through two debian dist upgrades just fine and has been handling all my mails ever since.

So, after getting the already mentioned tls error, I looked at the logs of the acme container. They contained a bunch of messages like this:

Detecting IP addresses…
OK: 159.xxx.xxx.xxx, 0000:0000:0000:0000:0000:0000:0000:0000
Found AAAA record for mail.kolaente.de: 2a01:xxxx:xxxx:xxxx::1 - skipping A record check
Cannot match your IP 0000:0000:0000:0000:0000:0000:0000:0000 against hostname mail.kolaente.de (DNS returned 2a01:xxxx:xxxx:xxxx::1)

I then checked the mailcow forums for someone with the same problem and found a bunch of similar results. Unfortunately most solution were along the lines of “Yeah I just did an upgrade and everything worked fine after that” which didn’t work for me.

More digging it was.

I’ve tried a few other things, none of them worked:

• Upgrading Docker to 20.x. This was a prerequisite to use the docker-native ipv6 nat bridge.
• To upgrade to a new Docker version, I had to first upgrade to Debian buster because stretch is EOL since a few days.
• During that upgrade, the debian upgrade guide reccomended changing interface names. I tried that as well but I messed it up and locked myself out of the server completely. Fixed it with the rescue shell and got access again.

Eventually I figured out IPv6 didn’t seem to work at all on this server: I could ping the server ip from that server itself but wasn’t able to ping any IPv6 target or the server IP from another IPv6 target.

But I was used to IPv6 to just work™ on Hetzner vps, something had to be off here.

Next, I just got a new Hetzner Cloud vps and checked its network interface config. And sure enough, it was different.

Here is the relevant part of my config at /etc/network/interfaces.d/50-cloud-init.cfg:

auto eth0:0
iface eth0:0 inet6 static
    address 2a01:xxxx:xxxx:xxxx::1/64
    gateway fe80::1
    post-up route add -A inet6 default gw fe80::1%eth0 || true
    pre-down route del -A inet6 default gw fe80::1%eth0 || true

And here is the one from the new server:

auto eth0
iface eth0 inet6 static
    address 2a01:xxxx:xxxx:xxxx::1/64
    dns-nameservers 2a01:4ff:ff00::add:1 2a01:4ff:ff00::add:2
    gateway fe80::1

There’s three differences between the two worth pointing out:

1. The interface name has a :0 suffix. I don’t know anything about what the name should look like but that does not seem correct.
2. My server does not have an IPv6 nameserver configured. I think as long as there’s an IPv4 server this should not be a problem.
3. There are two post-up and pre-down routes which do something with the IPv6 address. AFAIK they add iptable rules?

Once I changed my config file to match the one on the new server and rebooted, everything went back to work again. A bit disappointing of a solution, but I’m not going to complain.

I don’t know enough about networking under debian to understand this completly. My guess is Hetzner changed the rules at some point and the old ones don’t work with some newer part of the infrastructure. Remember, that server is almost 4 years old.

If you have a good explanation, I’d love to hear it!