Is there a trusted independent firewall review company?

Question

I don't know how to write this without it sounding like a plug (lol), but it's not. So please don't close-vote it, even if you really want your editor badge. Here goes:

There's this site called av-comparatives that I use and trust to provide me independent antivirus reviews. You probably do, too, actually. While I keep myself somewhat knowledgeable about infosec, my specialties lay elsewhere and it's so useful for me to have something like that as a resource.

I don't know of anything like that for firewalls, and when I go out to find one, I'm accosted with millions, all claiming that one product or another is the best. Cross-referencing them often leads to one or two that I've never heard of. The (I think) fairly obvious conclusion there is that these are all just fake third party review sites set up for marketing purposes.

Unfortunately, I don't have the resilience to wade through such murky waters for so many more hours :P

IMPORTANT: This is NOT an open invitation to fake firewall review sites to post their crap. I will respond appropriately to spam.

I'm looking for legit review sites that you, as security professionals, trust (ie: on par with av-comparatives, which you probably trust too)

Answer 1

To your initial question, the only thing I've seen that purports to really do this is the Gartner Magic Quadrant. However, given the overall complexity of firewall solutions, it's hard to make an apples to apples comparison in my opinion. Often they are used for their basic layer 3 and layer 4 filtering capabilities, and as such are treated like commodity devices like a switch. In this case the most important thing is simply finding a reputable vendor that can handle the scale you operate at for the cheapest price you can find.

I am making the assumption you're talking about hardware appliances, but in general this would go for any firewall solution. It's really tough to get a real answer on a firewall, since there are so many variables. There's how much bandwidth it can push, how many connections per second it can handle, the rule chain processing, application level stuff, etc. There often are IDS features bundled into firewalls, but again testing is needed to use these at scale.

Like with any technology, scaling is always a concern. However, unlike AV a firewall generally has a key point in the infrastructure and so if it runs slowly it causes real business impact immediately. There are so many features available in various firewalls, it's also very hard to do an apples to apples comparison. I've done some pretty large scale firewall implementations and testing with aggregate bandwidth in the 100s of Gbps. The way we performed it was testing several vendors using simulated traffic similar to our production traffic. We found that in our case some vendors just could not handle our traffic patterns, despite them being "rated" for our environment.

Answer 2

There is NSS Labs which contrary to Gartner actually tests the devices.

But, talking about firewalls is just about as specific as taking about means of transport. What you actually need varies between a hand barrow and a container ship. Even within the same class (like passenger car) you will find lots of different solutions which not only differ by price but also by feature.

And given that your expertise is not in firewalls you better don't just buy anything yourself because it will be either provide nearly no security or you will not be able to maintain it because you don't understand what it does. Real firewalls are complex beasts and to get actually a benefit from them you need to tightly adapt them to your network and maybe also to restructure your network with security in mind. If anybody talks about simple firewalls they are probably selling only a simple packet filter with no real inspection capabilities. While this might be adequate to separate networks from each other it does not protect a company against current attacks from the outside.

Actually, no firewall is adequate to do this by its own. A firewall is only part of a larger security policy and can help to implement parts of this policy.

In summary: Don't buy just any device because you will not be able to make effective use of it. There are independent companies who offer a range of products from different vendors and do also security management and counseling. Talk to them to find out how to get the most security for your money in your specific situation.