This guide is the second part in a series which looks at setting up a grafana dashboard for your pfSense network, the first part should be completed before following these steps.
Graylog stores log in a series of indices and we’ll be splitting out our logs into 3 main areas. The reason for this is twofold. First, Suricata/Snort and filterlog have different attributes from the rest of the logs when parsed. Second, Suricata/Snort and filterlog can generate a lot of data and you may wish to choose different retention strategies (beyond the scope of this guide).
For simplicity, we’ll use the Graylog UI to create these indices (there are other tools if you should so wish to use them). New indicies are setup from
System / Overview > Indices and then using the
Create index set button. For simplicity in this guide, use the default values, which should mean you’ll just need to enter the title, Description and Index prefix for each:
Save and do the same for the other two indices pfSense / Suricata (pfsense_suricata) and pfSense (pfsense).
Next up we’ll configure the ability for Graylog to automatically convert port numbers to the service names and for any external IPs to do the Geo lookup (this allows you to plot events on a map).
opc40772 has some additional pfSense/grafana content - whilst that isn’t specifically used in this tutorial it may provide some interesting background reading and also provides a CSV that can be used by the content pack for service lookups.
For the time being, just download the port mappings to your graylog server - from the command line and move it to where Graylog can easily access it:
wget https://raw.githubusercontent.com/opc40772/pfsense-graylog/master/service-names-port-numbers/service-names-port-numbers.csv cp service-names-port-numbers.csv /etc/graylog/server
Similar to the Port/Service lookup we’ll download and setup the Maxmind Geo IP database. You’ll need to register at the Maxmind website and get your license key from
My Account > My License Key, then run the following from the command line (and replace with your license key):
wget https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOUR_LICENSE_KEY&suffix=tar.gz -O geo-city.tar.gz mv GeoLite2-City_20200630/GeoLite2-City.mmdb /etc/graylog/server/GeoLite2-City.mmdb
Once you’ve compeleted the downloads, activate the Geo IP database in Graylog
System > Configurations. At the bottom of the screen is the configuration for plugins - enabled the Geo-Location Processor:
Now all the scaffolding is complete, the last piece of Graylog configuration is to import the c0ontent pack with the various pipelines to fill the indices we setup at the of this guide.
Download the content pack and then visit
System > Content Pack > Upload.
Graylog is specific about the order that processors run, these settings work for me and this guide and if you don’t set them this way you’ll find your pipelines don’t parse any messages so set them as follows unless you know what you’re doing. Under
System > Configurations > Message Processors COnfiguration click the update button and make sure it is set to teh following order:
The pipelines are the bit that perform the magic to parse the logs we shipped in the first part of the guide. Now the content pack is uploaded, the next step is to assign the pipelines to the correct streams and indices.
For each of the 3 pfSense streams, the process is the same. Edit the stream under
Streams > pfSense > More Actions > Edit:
Then assign the new index stream (remove from the ‘All Messages’ is optional, but recommended), save and then activate the stream if needed:
Repeat this for each of the remaining pfSense streams.
At this point you should now start to see logs from pfSense and Suricata/Snort parsed in your Graylog server. Click on the filterlog stream you have just configured and you should see messages flowing the the dst_ip_configuration_code and dst_service fields competed:
It’s one of the foundational building blocks to building a Threat Intellegence toolset using Open Source software.
These instructions are one way of getting data from pfSense and Suricata (tested on pfSense 2.4.5) into Graylog (tested 3.2). They should also work for Snort given the same log formats.
Once you have this, then follow on to the next step.
To make things easy, I keep a seperate UDP input for pfSense. This allows me to easily sperate pfSense’s logs as they are not in the standard format Graylog expects and to keep some simple extractors which then parses the correct application from the input and runs a couple of pipelines to further parse messages and store them in the approriate streams.
First up set up a new UDP stream to receive all pfSense logs. Visit
System / Inputs > Inputs at the top select
Syslog UDP and click
Launch new input.
Fill out the values below and replace
Override source with the hostname you use for your pfSense firewall hostname
This step sets our first variable in Graylog we’ll use in Part 2 to build our streams and pipelines
System / Inputs > Inputsthen
Manage extractors(the blue button next the stream you have just setup) and then
Actions > Import extractors.
Add extractors to input
We’ll be sending both the default pfSense logs and the Suricata/Snort logs to Graylog. This is done in two parts.
First of all from your pfSense firewall visit
Status > System Logs > Settings. From there send the logs to Graylog by replacing
your.server:4514 with the hostname or IP address of your Graylog Server and leave
:4514 unless you decided to digress from the instructions and used a different port. Then hit
Next, we’ll send Suricata or Snort logs. These instuctions assume you already have Suricata or Snort set up, check out the pfSense IDS / IDP insturctions if you haven’t. I’m using Suricata but the following should be similar for Snort. Visit
Services > Suricata and click the edit icon next to the interface you wish to send logs for.
We’ll then use Barnyard2 to send the logs - these are sent straight to the graylog server input detail above. Using Barnyard2 has the added benefit that the start of the log includes the Suricata/Snort tag, rather than a random number stream if you just use local logging without Barnyard2. First of all make sure that Barnyard 2 is enabled at the top and then complete the following details:
Save and then follow the instructions that tell you [re]start Barnyard 2
At this point you should now start to see logs from pfSense and Suricata/Snort in your Graylog server. Click on the ‘Search’ tab in Graylog to check - it’ll probably have a lot of filterlog entries if you’re logging firewall events.
As our World becomes more connected with every passing second, what will our Smart Cities become as they start to work to be better, safer places for us?
Council’s focus currently on optimising their existing services, but what would you do if you could design your ultimate Smart City? Discuss.
I remember they day it happened. At first all I felt was shame and disappointment; it was the day my software developers said to me, “We’re not letting you write code anymore.”
For me the title says it all but if you still need to be convinced, first of all catch up on Eight Ways Digital Twins Can Action Real Change and Why you need to invest in heroes to build Digital Twins, then read on.
Successful Digital Twins should be enablers within your organisation, increasing safety/productivity and encapsulating unique business knowledge.
In the buzzword heavy industry that is technology Digital Twins have reached unicorn status; used within organisations and dropped into conversations to show street cred and innovation in an era laden with version numbers: Industry 4.0, Digital 4.0, etc.
I’ve spoken previously about a passion to be the best and why it is important to set big hairy, audacious goals when it comes to digital teams and measuring success.
I have been a first time father for almost 3 weeks. Whilst chatting to one of my mates we had a laugh about some of the similarities between parenthood and being an EA:
As Enterprise Architecture matures two roles are emerging — foundational and vanguard. If you’re looking to focus on the later, here are my top 10 focus points.
It’s November 1989 and Tim Berners Lee has just viewed the first ever web page on the system he invented. He has two choices - turn it into a proprietary system and sell it, or give it away as an open standard. Had he chosen the former we may not have the likes of Amazon, Netflix or iPhone to enhance our life.
The Library of Alexandria was one of the largest and most significant libraries of the ancient world until its destruction in 48BC (or maybe AD 270 – 275, AD 391 or AD 642).