Design
Illustration 2: Architecture overview showing actions from the frontend to display a graph
The back end is responsible for delivering data to the front end and to keep the stored data up to date.
For specifying the sources and ports to work with, there is a config file. It is also used to set which database to use and where the captured data is located.
The display of graphs needs to be almost instantaneous, no matter what time range is selected. With the nfcapd files easily exceeding 1 GB per month and source, this kind of speed would not be possible without the use of a database. NfSen relies on RRDtool for this purpose. RRDtool is a time-series database which stores the data in a circular buffer, meaning that its size won’t grow when inserting new elements. It was firstly released in 1999 and is written and maintained by Tobi Oetiker (Oetiker, 2017).
The database layout that has been developed for the nfsen-ng project is similar to NfSen, but it allows to store three years of data instead of one. Because RRDtool is not very flexible in terms of additional metadata to be saved, the following structure has been created:
- one database per source (e.g. source1.rrd) to display sources and protocols
- one database per port (e.g. 80.rrd) to display ports
- one database per port per source (e.g. source1_80.rrd) to display ports data per source
Due to each database only weighing in at 5.5 MB, this is not a problem, but rather an inconvenience. But still, it was reason enough to build the back end in a way to support other databases through a database interface.
As there are more modern time-series databases than RRDtool by now, the initial wish was to use Akumuli for this part. Due to very limited documentation and not being in the standard repositories, the team decided to keep using RRDtool, but to provide an additional interface that can be implemented to be able to switch the used database with minimal effort. This interface defines all the needed methods that a database class needs to implement to be compatible with nfsen-ng.
If there is existing captured data (nfcapd files), the latter can be imported into the database using the import command of the CLI. This will execute an nfdump command (-I to get a statistics summary of a file) for every 5-minute slot per day and source, and write the parsed result to the chosen database.
The CLI is also responsible for starting and stopping the daemon, which regularly imports the latest nfacpd file data into the database.
To streamline dealing with nfdump, there is an nfdump class. This class is able to set any available nfdump options but also cares about escaping all inputs and returning the output in an array. When an error occurs, the interface tries to translate the error code to a more meaningful error message. It also fails, if the number of already running nfdump commands exceeds the number set in the configuration.
The single point of access from the outside is the API. It has its own error handling and returns errors for example if the request has not enough parameters, or a parameter’s data type is wrong. After checking the inputs, it forwards the attributes to the nfdump class (for flows and statistics) or the database class (for the graph), and returns the resulting output in JSON (or CSV if specified).
To stay up to date, it is needed to regularly write new data to the database. This is where the daemon comes in. Using a lock file (specifying the process id) prevents the user from starting multiple daemons, and allows the user to end the daemon more comfortable.
The front end is responsible for getting user inputs and displaying corresponding graphs, flows listing and statistics. It receives its starting configuration from the back end. The components below are used by the front end:
- Bootstrap v3
- dygraphs
- ion.rangeSlider
- FooTable
- nfsen-ng.js
Bootstrap (Bootstrap: The world's most popular mobile-first and responsive front-end framework., 2017) is a HTML, CSS, and JavaScript framework for developing responsive, mobile-first projects. It is the second most-starred project on GitHub and contains various HTML and CSS templates for typography, forms, buttons, error alert and much more. The whole front end structure is built with Bootstrap CSS classes.
Dygraphs (dygraphs library, 2017) is a fast, flexible open source JavaScript charting library that allows to display a chart with a little amount of code. It can handle millions of data points and provides interactive features like zoom. Moreover, it is customizable and provides the use of options and custom callbacks. Additionally, it works on tablets and mobile phones. The graph mode in nfsen-ng relies on Dygraph to display the graph and provide further interactions (e.g. zoom) and options (e.g. logarithmic scale).
Ion.RangeSlider (Ineshin, 2017) is a cross-browser JavaScript plugin that brings a responsive and customizable range slider. It also supports touch-devices. This plugin is used by the front end to propose a date/time range selection for the three last years.
FooTable (FooTable: jQuery plugin to make HTML tables responsive, 2017) is a responsive table plugin. nfsen-ng uses it to display lists of flows or statistics.
nfsen-ng.js contains all the JavaScript code needed to initialize the GUI and the used components.
Illustration 3: Initial mock-up for the graph view