Navigation

Main Documentation Download FAQ

Contribute

Architecture

This section will provide an insight into how we have structured the system and where you need to go if you want to make advanced changes. Our structure is representative of the many installations we have performed and the logical seperation individual NSM components (ie. sensors and servers) with regard to their respective configuration and the data they collect. We will explain it in the following order:

Introduction
System Functionality
1. Server
2. Sensor
3. Client
Logical Connectivity
System Layout
1. Server
2. Sensor
3. Client

Introduction

The intent of this section is to describe the different applications and tools brought together by NSMnow. We will discuss each tool/application with repect to its role and then detail where you can find the configuration files associated with them. It is our intent to ensure everything is structured in a logical way and for this documentation to make it easy to find the appropriate files. If you wanted to perform some more advanced modificaitons it is important to know where to look, particularly if you choose not to use the NSM administration scripts provided.

System Functionality

NSMnow brings together a number of tools that all contribute to the NSM framework, in this section we will highlight the role of each tool and how it fits into the bigger picture.

Server

The Server is were all the action occurs, it retrieves the collected data from the sensor(s), stores it in the database and sends real time alerts to the analyst console(s) at the client(s). The server's job is not done there, it also sits between the client and the sensor in order to retrieve stream data from the full content data collected.

The following tools make up the server:

Tool	Description
mysql	This provides the primary data storage for all sguil related information.
sguild	This provides the real time alerting mechanism for the clients and the interface to the mysql database.

Sensor

The sensor is responsible for collecting the data and passing it to the server for real time alerting and post analysis, with the exception of full content data. The full content data is stored on the sensor and streams are retrieved when called by the analyst console.

The following tools make up the sensor:

Tool	Description
snort	There are a number of instances of snort that are configured to collect a variety of forms of data. Once instance has snort collecting Alert data, which is accompanied with its own performance statistics (protocol breakdown, packets dropped, etc) and is outputted in a unified2 formatted file. A second instance is configured to collect Full Content data that captures all network traffic on the specified interface.
sancp	This process is reponsible for collecting Statistic data regarding network traffic. Based on its configured rule set it wull identify, record, and tag traffic of interest. We use it to track all statitics regarding network connections.
snort_agent	This agent receives information directly from barnyard2, in real time, providing communication back to the sguild daemon on the server.
sancp_agent	This agent receives information from the sancp data files, processes them, and passes all relevant information back to the sguild daemon.
pcap_agent	This agent receives information from the snort log files, processes them, and passes all relevant information back to the sguild daemon.
barnyard2	This process is responsible for reading the unified2 formatted files from snort, containing the Alert Data, and generating sguil formatted information directly to the snort_agent.

Client

The client is where the analyst will spend all of their time analysing the incoming alerts. This may mean anything from pulling down data from the statistical logs, alerts logs and full content data so they can make sense of the information and accurately report on what is happening.

The following tools make up the client:

Tool	Description
sguil.tk	Analyst console, interface to alert data, session data and full content data.
wireshark	This is called from within the sguil client for viewing full content data.

Logical Connectivity

The diagram below summarises everything that has been discussed highlighting how all the tools plug together to provide the NSM framework, hopefully it makes sense.

System Layout

This section will explain the layout of the systemr. By that we mean the directory structure and where raw data and configuration files can be found. Some of directories are specific to Debian/Ubuntu systems and variations for Fedora/RHEL/CentOS systems are are noted.

Before we talk about the different components of NSM (server, sensor, client) lets first explain the core layout of the system. Below you will see a list of the main directories that are used throughout this section by the various applications:

Directory	Purpose
/nsm	Directory used by both the server and sensor to store raw data. The intent of this directory is to allow a separate partition to be mounted for the storage of all the NSM raw data.
/etc/nsm	Location for all NSM related configuration files.
/usr/local/bin	Path for all runtime applications and scripts. NOTE: This will be /usr/bin on Fedora/RHEL/CentOS systems.
/usr/local/lib/nsmnow	Location for various library files used within NSMnow and its sub components. NOTE: This will be /usr/lib/nsmnow on Fedora/RHEL/CentOS systems.
/var/log/nsm	Path for all the log files associated with the tools/applications and scripts. All stdout traffic is routed to a specific log file in these directories. They are categorised by the sensor/server name.
/var/run/nsm	Path for all the running applications process ID. This allows for tracking and status of all running applications. They too are categoriesd by the sensor/server name and process name respectively.

i. Server

The following table outlines the directories used by the server and what you will find in each of them:

Directory	Purpose
/nsm/server_data/XXX	Root directory used to store all the data for server defined as XXX. It contains a number of sub-directories including archive, load, load/failed and rules.
.../archive	This directory is used to store raw pcap files that are passed to the sguil console for transcript and wireshark views. Files are stored in subsequent directories by date they where accessed.
.../load	This directory is used as a temporary store for data that is about to be parsed into the database. For example sancp stats will sit here while they are being processed and loaded into the database.
.../load/failed	Files that fail to be parsed into the database will be moved from the load directory into here. This is commonly seen when sancp files fail to be parsed into the mysql database. Two of the most common problems for that are not using the FILE directive in mysql privileges or if you have apparmor configured on Ubuntu.
.../rules/sensorname	This directory stores the rules that are in use by the sensor "sensorname". Rules are stored in a sub-directory with the "sensorname" and are used to provide the rule in the sguil console. If rules are not copied here when a sensor is created you will not be able to use the show rule option in the sguil client.
/etc/nsm/XXX	This directory is where the server defined as XXX stores all the sguild related configuration files and SSL certificates.
/usr/local/lib/sguild	Location of the sguild library files. Generally do not need to access/modify files in here unless you are troubleshooting some big problems. NOTE: This will be /usr/lib/sguild on Fedora/RHEL/CentOS systems.

Sensor

The following table outlines the directories used by the sensor and what you will find in each of them:

Directory	Purpose
/nsm/sensor_data/YYY	Root directory used to store all the data for sensor defined as YYY. It contains a number of sub-directories including dailylogs, portscans and sancp. In the root of this directory Snort stores alert data in unified2 format for barnyard2 to process and performance statistics for processing by the snort_agent.
.../dailylogs	Location for full packet capture data by snort. A directory is created for each day. pcap_agent processes the data on request from a console user (via sguild).
.../portscans	Snort stores portscan data here for processing by the snort_agent.
.../sancp	Sancp logs session data here for processing by the sancp_agent.
/etc/nsm/YYY	This is where you will find all the configuration files used by the sensor defined as YYY. The configuration files are for the sensor specific tools such as snort (including rules), sancp, barnyard2, and the sguil agents (snort_agent, sancp_agent, pcap_agent).

Client

The following table outlines the directories used by the sguil console:

Directory	Purpose
/etc/sguil	Location for the sguil client configuration files. Due to the paths being hardcoded in sguil.tk, they haven't been moved. You can override configuration options for each user by copying it to the path they run sguil.tk (eg. home dir).
/usr/local/lib/sguil	Location of the sguil.tk library files. Generally do not need to access/modify files in here unless you are troubleshooting some big problems. NOTE: This will be /usr/lib/sguil on Fedora/RHEL/CentOS systems.