Is it practical and worthwhile to alternate penetration test vendors?

Question

Is it considered a best practice to alternate the vendor used to perform penetration tests? For example if it is your policy to conduct a third-party penetration test semi-annually you would have vendor A conduct the penetration test in the first half of the year and vendor B conduct another penetration test in the second half of the year.

[Edit]
Ideally you would be alternating between two trusted partners. The idea being that no one person/company is perfect. With that said is this practical and do the benefits justify expending the time/effort/money required to build two solid relationships.

Answer 1

Many (maybe most?) people consider rotating providers a best practice.

The most worthwhile benefit I typically hear is it enables you to compare the quality and value.

I also think there is room for you to be creative; for example, hiring a firm specializing in application penetration testing for one round followed by one specializing in social engineering, or soliciting multiple providers per round, or some combination thereof.

In all cases paying attention to the names and respective skill sets of the individuals performing the actual work is key to ensuring you are getting what you expect (i.e. you want to avoid bait and switch tactics and you want to track individuals that deliver high quality).

Answer 2

@Tate's answer is good on the benefit of rotating, but I'll point out another important point:

The disadvantage of rotating is that you lose a lot of the context and knowledge that your provider already built up. Both knowledge of your business context, requirements, custom rules, how the app works, etc, and historical familiarity with issues you've had in the past. If you rotate, you have to start all over.

Of course, if you only have the external type of pentester, comes in once in a while to do a blind scan, well then they haven't built up much knowledge about you anyway... but then why bother with them? You'd be better off with a "partner" - someone who learns your business, works according to that, and can identify trends, root causes, and repeated mistakes that happened before - and works with you to fix them. But that's hard to build if you rotate every 6 months...

I guess its a tradeoff, verification (and originality) against better, more efficient work (but you'd have to trust).

Answer 3

I am going to make some completely different arguments that go directly against PCI DSS and CIP CVA.

My first argument is that it is stupid to ever hire external penetration-testers. Penetration-testing should be a "bug hunt" day that happens during infrastructure or iteration demos (i.e. while in QA/staging). Everyone, including all consultants/contractors/QA-people/devs/managers/etc should be invited and allowed to participate. They should work together in teams (usually pairs) and the teams should be different minds (e.g. trained vs. untrained).

My second argument is that it is stupid to work with one partner company "occasionally". If you are going to put in the effort to acquire a trusted adviser, you are wasting both their time and yours if you only engage them once a year, or when the regulations dictate that you should. It is important to be in constant contact with the people you build trust with.

Hire an appsec consulting company and treat them like employees even if they aren't at a desk everyday of the work week. Go into the engagement knowing it's going to be 3 years before you show progress, but do set goals/objectives and metrics.

Answer 4

From experience I would say the times rotation works best is at the global enterprise scale, where you may have 3 - 6 vendor organisations on a panel continually working for you so they all build up long term knowledge of the environment, and you can rotate them through a range of areas, eg internal, external, 3rd party etc., but the real value only comes if you can build up the meta-data around testing so you can normalise and track trends:

• Vendor x always rates a particular issue as Minor, but Vendor y rates it as Significant.
• Tester a provides more info on app test results than tester b.
• Vendor z was less skilled but improving faster than the others.

Without rotation, the cross comparison on skills, and the ability to renegotiate price can be difficult to manage, but be aware you do need to budget for time to bring new vendors up to speed on your working methods, delivery expectations, reporting styles, thresholds etc.