Information Security Stack Exchange is a question and answer site for information security professionals. It only takes a minute to sign up.
Sign up to join this community
As everyone knows each company wants to save as much money as possible. I have been tasked with pre-filtering logs on less important status/health messages before they get to the SIEM.
There are two reasons why pre-filtering is done: to save on the EPS (Events Per Second) rate and to reduce the noise. Every SIEM technology has a maximum EPS analysis capability which are restricted either by the hardware or by the software license. When events exceed that limitation, the SIEM server begin to drop the packets. That is why you would only want to bring in only those events which have significant security implications.
But even more important than the EPS is to reduce the events so that your analyst(s) can analyse those events manually. Although SIEM technology can automate many event correlation activities, initially you need an analyst to go through each of the received log or traffic event manually in order to determine whether it is true positive or false positive. When you overwhelm the system with a lot of noisy events that do not have a security implication, not only are you burdening the system hardware and software resources but you are actually over burdening the analyst in analysing those events as well.
Based on my experience, you need to understand that an effective SIEM solution is the one which are analysing only security events. Many a times people want to let the SIEM analyse and co-relate many non security events as well. At times that might be useful such as correlating the system health information with the received traffic logs in case of a denial of service attacks, however, in most of the cases pre-filtering all non security logs and events and forwarding only the relevant logs to your SIEM will make the system as well as the SIEM analyst more efficient.
I work for the company that is developing SIEM solution for last 5 years, so I constantly have to talk to new and existing clients, helping them pick best way to integrate what we offer.
What comes to my mind about SIEM and pre-filtering is if you need to keep your logs intact when entering system because of various country regulations. This might or might not be important for your company.
Secondly, if your SIEM offers $$ plan that concentrates on stored data and not data transferred, you might filter what actually enters storage (retention level, pre-indexing part) so you don't spend too much storage space and you dont kill SIEM alerting with too much junk data. Configuring what enters SIEM and what is discarded at SIEM level and not on OS level gains you more control on what happens if someone quits, you change OS or similar. Look at SIEM rules as documentation for your infrastructure. If it is all in one place it is easier to look at then if its distributed at non standard places.
TLDR I would collect everything and make few rules in retention level so data does not enter indexing level (alerting, correlation etc). If there are regulatory compliances you must follow i would just leave like it as is.
