The problem will usually be defining "normal" traffic against which you can detect the anomalies. From past research, ensemble classifiers seem to be most efficient in lowering your false positives. I myself worked researching a lot of these algorithms and it's still under heavy research. Another area is Artificial Immune Systems in a combination of IDS and Incident Response technology.
Check out multiple classifiers in use: http://roberto.perdisci.com/projects/mcpad Here's a paper on Artifical Immune Systems: http://www.cs.unm.edu/~forrest/publications/hofmeyr_forrest.pdf
How good is the technology behind the fraud detection tools that the financial sector deploy in terms of anomaly detection?