Quantcast
Channel: Johannes Weber – Weberblog.net
Viewing all articles
Browse latest Browse all 311

Using NetFlow with nProbe for ntopng

$
0
0
nProbe ntopng featured image

This blog post is about using NetFlow for sending network traffic statistics to an nProbe collector which forwards the flows to the network analyzer ntopng. It refers to my blog post about installing ntopng on a Linux machine. I am sending the NetFlow packets from a Palo Alto Networks firewall.

My current ntopng installation uses a dedicated monitoring ethernet port (mirror port) in order to “see” everything that happens in that net. This has the major disadvantage that it only gets packets from directly connected layer 2 networks and vlans. NetFlow on the other hand can be used to send traffic statistics from different locations to a NetFlow flow collector, in this case to the tool nProbe. This single flow collector can receive flows from different subnets and routers/firewalls and even VPN tunnel interfaces, etc. However, it turned out that the “real-time” functionalities of NetFlow are limited since it only refreshes flows every few seconds/bytes, but does not give a real-time look at the network. It should be used only for statistics but not for real-time troubleshooting.

Some Pre Notes

I am using a Ubuntu 14.04.5 LTS (GNU/Linux 3.16.0-77-generic x86_64) server. At the time of writing, nProbe had version v.7.4.160802 while ntopng was in version v.2.4.160802. Furthermore note that nProbe requires a license.

For general information about NetFlow use Wikipedia or Cisco or RFC 3954. For the other tools, use the official web sites: nProbe and ntopng. The nProbe site offers a detailed documentation PDF. A similar tutorial for installing nProbe is this one.

Installation of nProbe

(Since I already showed how to install ntopng, I will only show how to use nProbe here.) The stable builds for nProbe and ntopng are listed here. That is, to install nProbe, I used the following commands:

wget http://apt-stable.ntop.org/14.04/all/apt-ntop-stable.deb
sudo dpkg -i apt-ntop-stable.deb
sudo apt-get update

sudo apt-get install nprobe

Since I want to receive NetFlow packets and forward them to ntopng, nProbe must run in Collector Mode. That is, I am using the following configuration file:

sudo nano /etc/nprobe/nprobe-none.conf

with these entries:

--zmq="tcp://*:5556"
--collector-port=2055
-n=none
-i=none

Note the naming of the config file: “nprobe-none.conf“. This is mandatory due to the documentation of nProbe: “When nProbe is used in probe mode it is not bound to any interface as its job is to collect NetFlow from some other device. In this case the configuration file to be created is: nprobe-none.conf.” (To my mind, this is a spelling mistake because it should read “When nProbe is NOT used in probe mode…”. However, it is working.)

Furthermore, an empty “start” file is needed to tell the init process to use this configuration file:

sudo touch /etc/nprobe/nprobe-none.start

After a start of the service with

sudo service nprobe start
, ntopng must be configured to use this nProbe instance. Open the configuration file:
sudo nano /etc/ntopng/ntopng.conf

and add the following interface (= localhost):

--interface="tcp://127.0.0.1:5556"

Finally, restart the ntopng process:

sudo service ntopng restart
.

A netstat view should indicate the listening 2055 UDP port for nProbe, the 5556 TCP port for the connection between nProbe and ntopng, as well as the common 3000 TCP port from the ntopng WebGUI:

weberjoh@jw-nb10-syslog-mirror:~$ sudo netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      107        12714       1184/redis-server 1
tcp        0      0 0.0.0.0:5556            0.0.0.0:*               LISTEN      0          15260       1641/nprobe
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          12157       1017/sshd
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      65534      14983       1676/ntopng
tcp6       0      0 :::22                   :::*                    LISTEN      0          12159       1017/sshd
udp        0      0 0.0.0.0:2055            0.0.0.0:*                           0          15261       1641/nprobe
udp        0      0 192.168.120.10:123      0.0.0.0:*                           0          14413       1526/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           0          14412       1526/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           0          14405       1526/ntpd
udp        0      0 0.0.0.0:161             0.0.0.0:*                           0          12958       1224/snmpd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           0          12684       1157/syslog-ng
udp        0      0 0.0.0.0:55059           0.0.0.0:*                           0          12943       1224/snmpd
udp6       0      0 :::2055                 :::*                                0          15262       1641/nprobe
udp6       0      0 2003:51:6012:120::1:123 :::*                                0          14416       1526/ntpd
udp6       0      0 fe80::21d:92ff:fe53:123 :::*                                0          14415       1526/ntpd
udp6       0      0 ::1:123                 :::*                                0          14414       1526/ntpd
udp6       0      0 :::123                  :::*                                0          14406       1526/ntpd
udp6       0      0 ::1:161                 :::*                                0          12959       1224/snmpd

Since all services are now configured within configuration files that are referenced in the init scripts, they are started automatically after a system reboot. Great.

Palo Alto NetFlow

I am using a Palo Alto Networks firewall (version 7.1.3) to send NetFlow statistics to the nProbe collector. (More information about NetFlow on Palo.) This is configured in the following way: Adding of a NetFlow Server Profile and referencing this profile on all needed Network Interfaces, such as:

New NetFlow Server Profile. Referenced in some (or all) Network Interfaces. Indicated by these icons at the summary page.

I am using quite fast values for the Template Refresh Rate as well as the Active Timeout. On interfaces with huge amount of traffic other values are probably better.

A small tcpdump capture shows some samples of the NetFlow packets sent by the Palo Alto. The following Wireshark screenshots show a NetFlow template as well as a sample flow:

NetFlow Template 258. Sample flow using template 258: SMTP between two IPv6 host using 27 packets.

ntopng Usage

Now here is the usage within ntopng. Simply choose the tcp://127.0.0.1:5556 interface at the upper right side. All features of ntopng remain the same, such as using the Dashboard, the Flows or the Hosts pages. (Refer to my post to see some features.)

Choosing the nProbe interface. Using ntopng as normal.

However, here comes the problem with NetFlow: It is NOT a real-time application that lets ntopng show every single flow and its bandwidth correctly. It can be used to see a rough view of all flows during the past few seconds, but not its actual throughput at the moment.

Refer to the following two dashboard screenshots from ntopng. The first shows the Realtime Top Application Traffic from the NetFlow probe, while the second one shows the same from the mirror port eth1. The 54 MBit/s peak in the first screenshot is not true at all. In fact, it was a constant download over a few minutes. Whereas the second screenshot from eth1 shows the correct real-time bandwidth usage.

Peak from nProbe not correct on this time basis. Real-time view of a mirror port are correct.

Conclusion

nProbe for ntopng can be used quite easily. It is possible to receive flows from different locations which can be displayed in a single instance of ntopng. However, if the primary goal is to have a real-time look at the network, e.g., which hosts or flows are consuming bandwidth, this approach does not fit. NetFlow data must be used with statistical applications that can report traffic stats, but not with real-time analyzers such as ntopng.


Viewing all articles
Browse latest Browse all 311

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>