Palo’s Mgmt-Intf is not usable with IPv6 anymore

Wow, that was unexpected: With PAN-OS 11.1 the out-of-band management interface of Palo Alto Networks firewalls doesn’t accept an IPv6 default route pointing to one of its own data interfaces anymore. That is: In most setups, you can’t use IPv6 for management purposes anymore. “Works as expected.” Wow. Really?

Setup

This is how we normally connect the management interface to one of the internal layer 3 interfaces/VLANs from the firewall itself (at least in small environments where the out-of-band management is not within its own dedicated infrastructure):

The default gateway for the management interface points to one of the IP addresses of a data interface of the same firewall. In my lab, the management interface settings look as follows:

Error

I upgraded a PA-440 from PAN-OS 11.0.3 to 11.1.1. After the reboot, the auto-commit was not able to succeed. It ended in an auto-commit loop. I did a “commit force” from the CLI which showed the following error:

In virtual-router default: Mgmt IPv6 GW 2a00:6020:5004:2621:0:0:0:1/128 is conflict with address 2a00:6020:5004:2621:0:0:0:1/64 on interface ethernet1/3.21.(Module: routed)
client routed phase 1 failure
Commit failed

After I changed the default gateway of the mgmt-interface (IPv6) to an unused IPv6 address, the commit succeeded. However, when changing it back to the correct default gateway (which is a layer 3 subinterface from the same firewall itself), the same error appears again.

Some notes:

• I am using a logical router (not a virtual router). Hence I’m confused about this error since it reads “In virtual-router default: …”
• This configuration was fully valid and running with PAN-OS 11.0.3 and previous versions for the last decade!
• The IPv6 default gateway from the mgmt interface is the IPv6 global unicast address from ethernet1.3/21. Same for IPv4 where it’s still working.
• I also tried to change the IPv6 default gateway address to the link-local address of ethernet1.3/21, which resulted in the same error.

Reply from PANW

Of course, I opened a support ticket with Palo Alto Networks. It turned out that this behaviour is already known and working as expected starting with PAN-OS 11.1. WHAT?

I also requested and got the bug ID, but since it is internal only, I won’t publish it here.

At least the engineering team was instructed to write documentation about this. Hahaha.

Some Notes

• Isn’t it a real “out-of-band management” since the very beginning? If it were a real separate management plane, this would not happen, because the management and data plane would be *separate*. Hence it’s not the case. Ups.
• To my mind, this setup (routing the management traffic through the firewall itself) is quite common if not the de facto standard. Why is Palo Alto Networks refusing this setup?
• Isn’t Palo Alto Networks testing PAN-OS upgrades before releasing them enough? I would have expected this error to be noticed. I’m talking about the upgrade from PAN-OS 11.0 to 11.1. Either through a pre-check (before installing the upgrade) or at least through release notes.
• Though Palo Alto Networks was a little behind concerning IPv6 completeness for the last years (e.g., DHCPv6-PD was introduced with PAN-OS 11.0 while other vendors had it for a couple of years), the quality of their implementation was always very good. But now they have introduced a big deal when it comes to IPv6. Bugs like this are not helping while deploying IPv6 at the customers’ sites.
• Why is it still working with legacy IP? That is: Which “kernel” upgrade brought them to this error?
• As workarounds, you must use “Interface Mgmt” profiles to access the firewall via HTTPS/SSH/SNMP/API on data interfaces rather than on the management interface.
• Furthermore, you must rely on legacy IP for all other services such as DNS, NTP, syslog, authentication servers, etc. <– That’s exactly what was IPv6-only in my lab. D’oh!

Just as an example, this is how my NTP went dead (status: error, reachable: no):

weberjoh@pa> show ntp

NTP state:
    NTP not synched, using local clock
    NTP server: 2001:470:1f0b:16b0::dcfb:123
        status: error
        reachable: no
        authentication-type: none
    NTP server: 2a00:6020:5004:2600:85da:7fd8:1a80:8c8c
        status: error
        reachable: no
        authentication-type: none

Too bad.

Soli Deo Gloria!

Photo by Jussara Paulo on Unsplash.