Filter Syslog Messages Before They Are Forwarded To Azure Sentinel

If you are deploying Azure Sentinel as your SIEM, one of the items you will need to deploy is a syslog forwarder. Microsoft only supports using Linux as the syslog forwarder and my current favorite Linux distribution is Ubuntu. I hope they create a syslog forwarder for a Windows Server or develop a hardened appliance. Seems silly that they don't provide either option.

I recommend installing the forwarder in a DMZ as it will access to the internet so it can forward syslog and CEF logs it collects to Azure Sentinel. I also recommend installing the Windows Defender ATP agent on the forwarding server if you are licensed to use it. This will ensure that the box is getting updated on a regular basis and also alert you if someone has compromised the box.

When you start forwarding logs to the syslog relay, you will notice that the syslog messages will contain a lot of noise. You will want to filter the noise as it can be costly if meaningless logs are stored in Azure Sentinel. Azure Sentinel currently does not have the ability to ignore or drop logs built into the Azure Sentinel cloud console. You will need to perform the filtering on the syslog server.

You will need to SSH into the Linux machine you have installed the log forwarder. Once you have connected via SSH, you will need to go to the following directory, /etc/rsyslog.d. If you do not have a file called 60-cef.com, you will want to create one. This file is used to look for messages that are in the CEF format or are coming from Cisco ASA devices. The file should contain the following information.

$rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
& stop

Make sure that you have & stop at the end of the file, if you don't, then the log forwarder will send the logs to the syslog parser and it could be processing logs twice and sending them up to Azure Sentinel, which will cost you money. I learned this the hard way as once I made this change, the number of logs it was ingesting was cut in half.

Save the file and run the following command to restart the syslog server.

service rsyslog restart

Once that is in place, you will need to  create a file with a lower number, I chose the name 55-filter.conf. Pick your favorite editor and edit the file.

I chose to filter the raw message and look for syslog messages that contained the keyword I wanted filtered out.  

I have some examples listed below


:rawmsg, contains, "Login request is not expected to be encrypted" ~
:rawmsg, contains, "send_authenticate_pdu: Sending Preamble" ~
:rawmsg, contains, "In update_aaa_cntr:" ~
:rawmsg, contains, "GwInsight" ~
:rawmsg, contains, "Line protocol on Interface" ~

Additional methods exist to perform filtering, so you can create more advanced filters. (This video is a good source on how to create some of the advanced filters, Azure Sentinel webinar: Log Forwarder deep dive | Filtering CEF and Syslog

Save this file and restart the syslog server again, using service rsyslog restart.

If you wait 30 minutes,  you go to the Azure Sentinel Log analytics workspace  go to log  --> query syslog, you should not see any log entries with the keywords above.

Querying the syslog log in Azure Sentinel is a good way to find out additional logs you might want to filter.

I did not realize you could filter out noisy syslog messages. Not only does it make writing queries much easier, it also will have you some money as you are not having to ingest and store meaningless log files.

This information should be much easier to find. It is a very easy process to filter out the syslog messages, but not very well documented.

I hope this helps you avoid the mistake I made with my deployment of Azure Sentinel.

TBJ Consulting

TBJ Consulting