pfSense log parsing in Graylog (including suricata/snort)

This guide is the second part in a series which looks at setting up a grafana dashboard for your pfSense network, the first part should be completed before following these steps.

1. Setting up indices

Graylog stores log in a series of indices and we’ll be splitting out our logs into 3 main areas. The reason for this is twofold. First, Suricata/Snort and filterlog have different attributes from the rest of the logs when parsed. Second, Suricata/Snort and filterlog can generate a lot of data and you may wish to choose different retention strategies (beyond the scope of this guide).

For simplicity, we’ll use the Graylog UI to create these indices (there are other tools if you should so wish to use them). New indicies are setup from System / Overview > Indices and then using the Create index set button. For simplicity in this guide, use the default values, which should mean you’ll just need to enter the title, Description and Index prefix for each:

Screenshot of Graylog index settings

Hit Save and do the same for the other two indices pfSense / Suricata (pfsense_suricata) and pfSense (pfsense).

2. Setting up Service and GeoIP look ups

Next up we’ll configure the ability for Graylog to automatically convert port numbers to the service names and for any external IPs to do the Geo lookup (this allows you to plot events on a map).

2.1. Port Mappings

opc40772 has some additional pfSense/grafana content - whilst that isn’t specifically used in this tutorial it may provide some interesting background reading and also provides a CSV that can be used by the content pack for service lookups.

For the time being, just download the port mappings to your graylog server - from the command line and move it to where Graylog can easily access it:

cp service-names-port-numbers.csv /etc/graylog/server

2.2 Geo IP look ups

Similar to the Port/Service lookup we’ll download and setup the Maxmind Geo IP database. You’ll need to register at the Maxmind website and get your license key from My Account > My License Key, then run the following from the command line (and replace with your license key):

wget -O geo-city.tar.gz
mv GeoLite2-City_20200630/GeoLite2-City.mmdb /etc/graylog/server/GeoLite2-City.mmdb

Once you’ve compeleted the downloads, activate the Geo IP database in Graylog System > Configurations. At the bottom of the screen is the configuration for plugins - enabled the Geo-Location Processor:

Screenshot of Graylog Geo-Processor settings

3. Setting up the pfSense content pack

Now all the scaffolding is complete, the last piece of Graylog configuration is to import the c0ontent pack with the various pipelines to fill the indices we setup at the of this guide.

3.1. Download the content pack

Download the content pack and then visit System > Content Pack > Upload.

3.2 Set the order of processors

Graylog is specific about the order that processors run, these settings work for me and this guide and if you don’t set them this way you’ll find your pipelines don’t parse any messages so set them as follows unless you know what you’re doing. Under System > Configurations > Message Processors COnfiguration click the update button and make sure it is set to teh following order:

Screenshot of Graylog Processors

3.3 Configure the streams

The pipelines are the bit that perform the magic to parse the logs we shipped in the first part of the guide. Now the content pack is uploaded, the next step is to assign the pipelines to the correct streams and indices.

For each of the 3 pfSense streams, the process is the same. Edit the stream under Streams > pfSense > More Actions > Edit:

Screenshot of Graylog stream

Then assign the new index stream (remove from the ‘All Messages’ is optional, but recommended), save and then activate the stream if needed:

Screenshot of Graylog stream edit settings

Repeat this for each of the remaining pfSense streams.

4. Testing

At this point you should now start to see logs from pfSense and Suricata/Snort parsed in your Graylog server. Click on the filterlog stream you have just configured and you should see messages flowing the the dst_ip_configuration_code and dst_service fields competed:

Screenshot of Graylog filterlog

Read More

pfSense log consolidation to Graylog (including suricata/snort)

This guide is an overview of how to push logs from pfSense (an Open Source firewall) into Graylog (an Open Source log aggregated and parser).

It’s one of the foundational building blocks to building a Threat Intellegence toolset using Open Source software.

These instructions are one way of getting data from pfSense and Suricata (tested on pfSense 2.4.5) into Graylog (tested 3.2). They should also work for Snort given the same log formats.

1. Installing Graylog

Rather than reproduce extensively maintained installation instructions, you should be able to dollow the instructions from Graylog to get a basic installation up and running.

Once you have this, then follow on to the next step.

2. Setting Up Graylog

To make things easy, I keep a seperate UDP input for pfSense. This allows me to easily sperate pfSense’s logs as they are not in the standard format Graylog expects and to keep some simple extractors which then parses the correct application from the input and runs a couple of pipelines to further parse messages and store them in the approriate streams.

2a. Setup a new input

First up set up a new UDP stream to receive all pfSense logs. Visit System / Inputs > Inputs at the top select Syslog UDP and click Launch new input.

Fill out the values below and replace sg1 in Override source with the hostname you use for your pfSense firewall hostname

Screenshot of Graylog input settings

2b. Add an extractor to your new input

This step sets our first variable in Graylog we’ll use in Part 2 to build our streams and pipelines

  1. Visit Github to download extractors.json
  2. Visit System / Inputs > Inputs then Manage extractors (the blue button next the stream you have just setup) and then Actions > Import extractors.
  3. Copy the text from extractors.json and click Add extractors to input

3. Sending pfSense Logs

We’ll be sending both the default pfSense logs and the Suricata/Snort logs to Graylog. This is done in two parts.

3a. pfSense Syslog Logs

First of all from your pfSense firewall visit Status > System Logs > Settings. From there send the logs to Graylog by replacing your.server:4514 with the hostname or IP address of your Graylog Server and leave :4514 unless you decided to digress from the instructions and used a different port. Then hit Save.

Screenshot of pfSense Syslog settings

3b. Suricata/Snort Logs

Next, we’ll send Suricata or Snort logs. These instuctions assume you already have Suricata or Snort set up, check out the pfSense IDS / IDP insturctions if you haven’t. I’m using Suricata but the following should be similar for Snort. Visit Services > Suricata and click the edit icon next to the interface you wish to send logs for.

Screenshot of pfSense Suricata interfaces

We’ll then use Barnyard2 to send the logs - these are sent straight to the graylog server input detail above. Using Barnyard2 has the added benefit that the start of the log includes the Suricata/Snort tag, rather than a random number stream if you just use local logging without Barnyard2. First of all make sure that Barnyard 2 is enabled at the top and then complete the following details:

Screenshot of pfSense Barnyard2 settings

Click Save and then follow the instructions that tell you [re]start Barnyard 2

4. Testing

At this point you should now start to see logs from pfSense and Suricata/Snort in your Graylog server. Click on the ‘Search’ tab in Graylog to check - it’ll probably have a lot of filterlog entries if you’re logging firewall events.

Read More

Site Cookies

Yes our site uses cookies, it’s just to provide website stats with Google Analytics as I’m a data junkie. If you’re uncomfortable with that then you don’t need to accept them and the site will continue to work.

Read More

What does your Smart City look like?

As our World becomes more connected with every passing second, what will our Smart Cities become as they start to work to be better, safer places for us?

Council’s focus currently on optimising their existing services, but what would you do if you could design your ultimate Smart City? Discuss.

Read More

To Be a Tech Leader, Kill the Tech Talk

When I moved from startups to big business I used to get frustrated that my technical know-how wasn’t immediately understood and valued — what was the point of having to explain myself when we could just be getting on with work?

Read More

5 Ways to Stay Strategic on Your Way to the C-Suite

I remember they day it happened. At first all I felt was shame and disappointment; it was the day my software developers said to me, “We’re not letting you write code anymore.”

Read More

Don’t be an idiot — encapsulate unique business knowledge in your digital twinss

For me the title says it all but if you still need to be convinced, first of all catch up on Eight Ways Digital Twins Can Action Real Change and Why you need to invest in heroes to build Digital Twins, then read on.

Read More

Why you need to invest in heroes to build Digital Twins

Successful Digital Twins should be enablers within your organisation, increasing safety/productivity and encapsulating unique business knowledge.

Read More

5 Ways Digital Twins Can Action Real Change

In the buzzword heavy industry that is technology Digital Twins have reached unicorn status; used within organisations and dropped into conversations to show street cred and innovation in an era laden with version numbers: Industry 4.0, Digital 4.0, etc.

Read More

2 tips to avoid Siri’s fate in your digital transformation

I’ve spoken previously about a passion to be the best and why it is important to set big hairy, audacious goals when it comes to digital teams and measuring success.

Read More

3 Keys to Successful Digital Transformation

The knowledge data brings is a key foundation to Digital Transformation, Industry 4.0/X.0.

Read More

Work on a shire horse not a unicorn startup

Life is too short to spend time on something you don’t love so concentrate on making your startup a shire horse, not a unicorn (although you might get that too!).

Read More

10 reasons an Enterprise Architect is like being a new parent

I have been a first time father for almost 3 weeks. Whilst chatting to one of my mates we had a laugh about some of the similarities between parenthood and being an EA:

Read More

10 things you need to do as a Vanguard Enterprise Architect

As Enterprise Architecture matures two roles are emerging — foundational and vanguard. If you’re looking to focus on the later, here are my top 10 focus points.

Read More

Why you must use Open Source at your Enterprise

It’s November 1989 and Tim Berners Lee has just viewed the first ever web page on the system he invented. He has two choices - turn it into a proprietary system and sell it, or give it away as an open standard. Had he chosen the former we may not have the likes of Amazon, Netflix or iPhone to enhance our life.

Read More

DynamoDB Outage, The Library of Alexandria and BCP

The Library of Alexandria was one of the largest and most significant libraries of the ancient world until its destruction in 48BC (or maybe AD 270 – 275, AD 391 or AD 642).

Read More