2

I wanted to know if anyone had experience sensoring single flows that generate 90 kpps or upward of traffic. My conundrum is that I use tools which I would like to be able to properly see an entire flow with (Zeek, Suricata), however both of these utilities are not capable of handling larger traffic volumes if a single connection exceeds a "worker thread's" maximum processing rate. Because a single connection is able to generate this volume of traffic, I am not aware of a load balancing method and accompanying software that would produce similar resulting information from Zeek & Suricata(Understandably this matters a lot less with Zeek, as seeing the entire connection isn't strictly necessary, however having a giant connection fill the rx ring results in losing subsequent connections).

For referencing I'm currently using the AF_Packet family sockets for both Suricata in Zeek, which are both in their own fanout. I get roughly 48,000pps running suricata with 4 capture threads with my current configuration and ~26,000pps per Zeekctl cluster.

I'm aware the most sensible thing to do would be to strictly analyze the flow, and then filter it out pre-sensor, however I'm doubtful policy in my org will allow me to do this, but I've been unlucky in my search for a satisfactory tool or reconfiguration.

I am hoping someone might have experience handling this type of situation or knows of software which has a less restrictive processing per "worker/thread" rate that would produce similar results(or honestly any results that aren't massive amounts of packet loss).

Improve this question

1 Answer 1

Reset to default
1

First thing I would look into is switching to use PF_RING sockets.

If that's not sufficient then it's probably a job for dedicated network packet broker hardware. You will want to look at vendors like Arista, Gigamon or Ixia. Some high-end switches may have the functionality built-in as well.

If you are really and truly CPU-bound by single flows and not total flow volume then short of buying CPUs with extreme single-core performance you are probably out of luck with Zeek and will have to switch to e.g. a commercial flow processor that supports on-NIC hardware packet processing.

Improve this answer
6
  • Thanks for the answer. I actually have a second server processing the same traffic with pf_ring zc with comparably similar struggles. Packet capture would definitively suggest that it is indeed single flow. Along with the fact that flow/symmetric-based hashing on both my pf_ring configuration and af_packet configuration are having relatively identical issues keeping up. Could you suggest any commercial solutions/documentation/analysis on their single flow rates? EDIT: To clarify I have a gigamon hc1 feeding both servers duplicate traffic.
    – ReedGhost
    Feb 9, 2021 at 4:18
  • @ReedGhost If you have an HC1 as your central aggregator, you can distribute the packet flow among more processing machines. The Gigamon is flow-aware so it will distribute the packets such that all packets part of the same "stream" will go to the same Zeek box. Our implementation used e.g. 15x capture boxes, each with a 10Gb link, all downstream from the Gigamon, and then the software (we used Moloch, now Arkime) would aggregate and present the unified results.
    – matoro
    Feb 9, 2021 at 16:27
  • Thanks, I'm aware of this, my issue is that no matter how far I divide up total packet volume. Flow hashing still means that one zeek worker and one suricata capture thread will always bear the one large flow. I could break the flow/connection apart but then the usefulness of information from zeek and suricara would be limited, as each part would be considered a different connection programmatically. Since both Zeek and Suricata are expected to process entire flows/connections, and neither can handle doing distributed work on a single flow/connection, unless I'm mistaken.
    – ReedGhost
    Feb 9, 2021 at 16:44
  • 1
    @ReedGhost No, you're correct, you can't distribute any more granular than a single flow without losing metadata. That is a very unusual traffic pattern for sure though. Anyway my only other suggestion would be in Arkime, you can use rules that evaluate BPF expressions that will allow you to stop processing packets after some metadata criterion is matched (which happens early in the flow). This will allow you to at least get the important info (host X downloaded Y GB from site Z) without chewing every single byte.
    – matoro
    Feb 9, 2021 at 16:56
  • Thats incredibly useful, I'll have to look at Arktime, thanks for the suggestions!
    – ReedGhost
    Feb 9, 2021 at 17:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .