How do you handle log analysis in your business?
I want to establish a good way of finding exceptions and alerts in the server logs, either automatically or semi automatic (having to review exceptions and alerts).
Is this functionality that is usually handled by an IDS/IPS?
Right now I am collecting all log data from web servers, databases, syslog and such into Splunk, and I'm considering to make Splunk dashboards which presents me weird and non-normal log entries. Is this the right way to go?
OSSEC is a great FOSS HIDS that works well to reduce the number of log events you need to check out to something potentially manageable.
It supports all of the log sources you mentioned: http://www.ossec.net/main/supported-systems
We built an in-house app that allows us to do a quick “daily check” of all the alerts OSSEC fires in a day; it typically takes us less than 20 minutes of time to call out an all clear or to kick-off an incident investigation. For one environment we receive ~70,000 OSSEC alerts per day, but most of the alerts/events are clusters of auth events or app errors spamming the logging system, for example:
Since you have Splunk going you could do the Splunk + OSSEC integration: http://www.ossec.net/main/splunk-ossec-integration
The other day I saw a couple enterprise SIEMs in action – I was stunned to learn they apparently have no canned reports detailing unusual or rare log events. Unusual and rare events are frequently the best indicators of something bad and can serve as your short list of events to chase down until the leads turn cold.
We also use Logwatch as a secondary log analyzer.
This functionality is generally achieved with the help of SIM or SIEM solutions. There are several popular vendors for these CA, ArcSight, Loglogic, splunk etc.
You can collect the Logs, be informed about the Alerts and take action based on Correlation of rules. You can collect logs from all possible sources, Databases, Unix, Windows, Firewalls etc.
Methods often differ depending on scale and location, for example perimeter logging for an enterprise is usually too much for an internal alert team to cope with, so this is usually outsourced to one of the managed service suppliers, who will then pass on relevant alerts.
For smaller organisations, or for specific host logging the problem is much more manageable, however you are still going to have to look at how you train and tune the alerting process. Splunk and others will do the job, but you need to plan for the amount of resource required to look after during the first few weeks, on any upgrade or change to the logged environment, on any change in attack profiles, and in fact continuously over time.
Having that out the way, the usual mechanisms are to use both signature based and statistical alerting - signatures are fast and can be updated by service providers so require low effort, however they need to be created so typically do not identify new types of attack, whereas statistical alerting responds to any change in logging - so you can end up overloaded with false positives until you tune.
From a security viewpoint (there's lots of other things which are worth checking logs for) then the logs will only reliably show where the security is working as expected. If someone has bypassed the security measures then it probably won't be detected by the logs / the logs may be compromised - so they are of limited use.
Also, if you can construct mechanism to analyse the logs for potential violations, then, in most cases it's just as easy to enforce the policy rather than wait for a violation to occur.
Log analysis is no substitute for a host-based file integrity checker (such as Lids, tripwire)
Having said that, I would recommend using fail2ban where appropriate.
Also the log analysis should take into account the organisation's change management routine - or more specifically the software/hardware update process - at least in the sense of whether file integrity has changed.
