What techniques and tools do you use to relate security events?

Question

You have central logging going, detailed app logging/alerting (e.g. modsec), network based security alerting (e.g. snort), and whatever else feeding your observation deck.

Do you have any cool techniques you’d like to share for how you relate security events?

How about tools? (in-house is fine, just describe what it does)

Answer 1

Enterprise tools which correlate related security and logs events are generally called Security Information and Event Management (SIEM) systems. Most are designed to accept data from common log formats, IDS alerts, antivirus, firewall rule changes etc.

• LogRhythm
  http://www.logrhythm.com
  Used by many companies around the world.

• IBM QRadar
  http://www-03.ibm.com/software/products/en/qradar-siem

• Cisco MARS
  http://www.cisco.com/c/en/us/products/security/security-monitoring-analysis-response-system/index.html

• Intel McAfee NitroSecurity
  http://www.mcafee.com/
  

• ArcSight(purchased by HP)
  http://www.arcsight.com/
  Very expensive. It's a beast to setup, a beast to use, but the correlation engine is really impressive. A few years ago when I used it, it was able to correlate multi-stage attacks (ie trojan on host a, allows host b to be reachable to external ip, and also know host b firewall rule allows snort attack seen on IDS c to occur.
  Looking at arcsight express (and asking my vendor for the price), it may be a nice fit for smaller shops (under 500 hosts) but it needs to be backed up by people able to interpret and analyze alerts.
  

Other tools that I know about:

• OSSIM, The Open Source SIEM
  http://sourceforge.net/projects/os-sim/
  One of my interns has played around with this, with generally favorable reviews. For a project, or a small shop, this is probably your best bet.

• Intellitactics (aparrently now owned by trustwave)
  http://www.intellitactics.com/int/
  They were another big player when SEIM first started, they were unique in that they were using a text database (as opposed to Oracle like ArcSight and the rest). I heard they were going to stop using a text db and move to something else, but never heard anything from then in a long while.

• RSA enVision
  http://www.rsa.com/node.aspx?id=3170
  This was regarded as a pile of crap, talking to the new product manager of the team at defcon, they were re-banging the product, using business intelligence tools and techniques (column store databases, etc). What I was told was EMC (the parent company of RSA) bought Greenplum (a ZFS based BI system) for this project. I would be watching them closely, and maybe get under an NDA and get the real story.

I'm using older information here, would love to hear whats correct and whats not.

Answer 2

• http://chuvakin.blogspot.com/2011/03/updated-free-log-management-tools.html

The above link lists a few tools which are really going to stick out above others, such as that

1. Novell Sentinel Log Manager 25 appears to be better than Splunk if you are starting out and aren't funded by a Global 200 company
2. Q1Labs appears to be attempting to compete with AlienVault (OSSIM) by providing a mid-market solution that even startups and companies considering SIEM on-trial will want to leap to
3. log2timeline implements a good concept not usually found in IDR methodologies or any commercial/free tools that I've seen in that it can specifically pinpoint gaps in time that could have been due to log file deletion (purposeful or not)

I am partial to Suricata over snort and I do not currently like any of the existing vulnerability management systems (e.g. OpenVAS, Arachni) or application monitoring systems (e.g. ModSecurity, AppSensor) -- however I am very likely to implement these along with OSSEC via OSSIM given the modern limitations of IT/Ops and AppDev shops.

It also doesn't appear that old school tools such as Beltane or Cerebus have kept up to today's needs. I guess the new freshness would be adopting integration for OSSIM and OSSEC in a Vagrant environment.

Answer 3

Here, we're switching from Intellitactics to Splunk, which seems to be the up-and-coming contender in the area. One of the nice things about Splunk is its extensibility; it's pretty easy to write data-mining or reporting modules for it. Another is the built-in "regex wizard" that lets you build a regex which corresponds to selected example text.

I personally have grand plans which seem to be perpetually in the "almost started" phase to use machine learning techniques for multiclass classification and clustering to organize the interesting-but-non-critical stuff that comprises 99% of logged data.