Palo Alto Syslog via TLS

As we have just set up a TLS capable syslog server, let’s configure a Palo Alto Networks firewall to send syslog messages via an encrypted channel. While it was quite straightforward to configure I ran into a couple of (unresolved) problems as I added and deleted some syslog servers and their certificates. Uhm.

Syslog & Certificate Configuration

I am using a PA-220 with PAN-OS 10.0.7 for this setup. My syslog-ng server with version 3.25.1 is running on a Ubuntu 20.04.3 LTS.

Basically, you just have to set the transport of a syslog server to “SSL”. (For whatever reason it’s still named SSL. It should read TLS since a couple of years.) I am using the “IETF” format nowadays. This seems to be the more modern standard, refer to RFC 5424 Appendix A.1. Though I don’t see any obvious differences between the BSD and the IETF type.

If you’re doing an immediate commit you will see these system logs:

Syslog  SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed'

Screenshot:

–> This is because I am using a self-signed x.509 certificate on my syslog-ng server which is not trusted indeed. Good so far!

The same errors are appearing at the /var/log/syslog log file on the syslog-ng server itself:

Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection accepted; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca', location='/etc/syslog-ng/conf.d/generic.conf:28:17'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Error reading RFC6587 style framed data; fd='42', error='Connection reset by peer (104)'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection closed; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)'

You have to import the certificate of the syslog-ng server into the Palo as well:

Now it’s working. ;)

Note that for some reason the Palo does NOT use IPv6 for this outgoing syslog connection, though my FQDN had an AAAA record at the time of writing and the syslog server itself was accessible.

weberjoh@nb15-lx:~$ host test2.weberlab.de
test2.weberlab.de has address 194.247.5.27
test2.weberlab.de has IPv6 address 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

This is what the legacy IP TLS connection looks like with Wireshark:

Some maloperations?!?

Since I wanted to use IPv6, I configured the IPv6 address of the syslog-ng server manually:

The TCP handshake to the syslog server worked, but no TLS connection was established. I got the same errors in the system log:

And this is it in Wireshark:

After trying that I decided to go back to the FQDN of my syslog-ng server to let the Palo use it with legacy IP. However, I was not able to get it working again at all! I tried the following:

1. without certificate -> not working
2. imported certificate -> working
3. deleted certificate -> still working (!!!???)
4. change of the syslog server to a dummy
5. change back to my real one -> not working (ok)
6. imported certificate again -> still not working (not ok!)
7. reboot of the whole device -> still not working

Hence I suppose some kind of bugs here. :(

Anyway, lessons learned:

Photo by Claudio Testa on Unsplash.