Is it practical and worthwhile to alternate penetration test vendors?

Question

Is it considered a best practice to alternate the vendor used to perform penetration tests? For example if it is your policy to conduct a third-party penetration test semi-annually you would have vendor A conduct the penetration test in the first half of the year and vendor B conduct another penetration test in the second half of the year.

[Edit]
Ideally you would be alternating between two trusted partners. The idea being that no one person/company is perfect. With that said is this practical and do the benefits justify expending the time/effort/money required to build two solid relationships.

Answer 1

Many (maybe most?) people consider rotating providers a best practice.

The most worthwhile benefit I typically hear is it enables you to compare the quality and value.

I also think there is room for you to be creative; for example, hiring a firm specializing in application penetration testing for one round followed by one specializing in social engineering, or soliciting multiple providers per round, or some combination thereof.

In all cases paying attention to the names and respective skill sets of the individuals performing the actual work is key to ensuring you are getting what you expect (i.e. you want to avoid bait and switch tactics and you want to track individuals that deliver high quality).

Answer 2

@Tate's answer is good on the benefit of rotating, but I'll point out another important point:

The disadvantage of rotating is that you lose a lot of the context and knowledge that your provider already built up. Both knowledge of your business context, requirements, custom rules, how the app works, etc, and historical familiarity with issues you've had in the past. If you rotate, you have to start all over.

Of course, if you only have the external type of pentester, comes in once in a while to do a blind scan, well then they haven't built up much knowledge about you anyway... but then why bother with them? You'd be better off with a "partner" - someone who learns your business, works according to that, and can identify trends, root causes, and repeated mistakes that happened before - and works with you to fix them. But that's hard to build if you rotate every 6 months...

I guess its a tradeoff, verification (and originality) against better, more efficient work (but you'd have to trust).

Answer 3

I am going to make some completely different arguments that go directly against PCI DSS and CIP CVA.

My first argument is that it is stupid to ever hire external penetration-testers. Penetration-testing should be a "bug hunt" day that happens during infrastructure or iteration demos (i.e. while in QA/staging). Everyone, including all consultants/contractors/QA-people/devs/managers/etc should be invited and allowed to participate. They should work together in teams (usually pairs) and the teams should be different minds (e.g. trained vs. untrained).

My second argument is that it is stupid to work with one partner company "occasionally". If you are going to put in the effort to acquire a trusted adviser, you are wasting both their time and yours if you only engage them once a year, or when the regulations dictate that you should. It is important to be in constant contact with the people you build trust with.

Hire an appsec consulting company and treat them like employees even if they aren't at a desk everyday of the work week. Go into the engagement knowing it's going to be 3 years before you show progress, but do set goals/objectives and metrics.

Answer 4

From experience I would say the times rotation works best is at the global enterprise scale, where you may have 3 - 6 vendor organisations on a panel continually working for you so they all build up long term knowledge of the environment, and you can rotate them through a range of areas, eg internal, external, 3rd party etc., but the real value only comes if you can build up the meta-data around testing so you can normalise and track trends:

• Vendor x always rates a particular issue as Minor, but Vendor y rates it as Significant.
• Tester a provides more info on app test results than tester b.
• Vendor z was less skilled but improving faster than the others.

Without rotation, the cross comparison on skills, and the ability to renegotiate price can be difficult to manage, but be aware you do need to budget for time to bring new vendors up to speed on your working methods, delivery expectations, reporting styles, thresholds etc.

Answer 5

I just wrote an article on this as it always seems to be a common question for clients. I have summed up 3 considerations for both perspectives:

REASONS FOR ALTERNATING VENDORS:

1. Complacency Application security engineers just like developers can be blind to some aspects of their work. For software developers, it can be very hard to find their own bugs, hence the practice of peer code review and the slew of quality assurance controls that are put in place such as unit tests, manual and automated quality assurance tests.

One of the questions you need to ask your pentest vendor is what do they do to overcome this problem.

2. Quality and value Intelligence It might be worth to consider other vendors to understand what service you are getting at what price point. Additionally, not all penetration testing providers are created equal, so one vendor could be really good at identifying vulnerabilities but post-report support might not be great, while another vendor really excels at the services provided post-report.

3. Leveraging different expertise It might be worth exploring different vendors for different expertise areas. For example, one vendor could be very strong in performing a pentest, but not as much in social engineering. So leveraging different vendors for different areas of strengths can be beneficial.

CONSIDER BEFORE SWITCHING:

1 .  Losing Context and application knowledge: there is most likely 3 levels of depth to any pentest:

Level 1: The low hanging fruit: which is basically what any scanner can find.

Level 2: The medium range issues: bugs that scanners can’t find but an engineer can, they are still easy to find, it just needs someone looking for them, agile enough with their approach.

Level 3: The Difficult Bug: these are only found when the engineer gets intimate enough with the application to understand exactly how it works. For a 2–3 week long penetration test, it is very hard for a new vendor to reach Level 3 in such a short amount of time. Depending on the nature of your application, there might not be a lot of business logic to the application and hence there is no need for Level 3. Another thing to consider is who are your primary threat actors? Are you up mostly against script kiddies or professional hackers and cyber gangs?

2 .  Losing the Partner relationship: it takes time for a pentest service to understand your business, application, and the dynamics of your team. Working with a pentest provider should not stop at the delivery of the report. A good pentest provider should be able to identify really good bugs, and more importantly help you mitigate those bugs and further fortify the application. If you are not getting that help from your current vendor, it might be worth it to have a conversation with that vendor or search for a new one.

3 .  Losing Motivation: for most companies in the professional services space, it is (or at least it should be) their top priority to go the extra mile to keep their clients. Knowing from the get go that it is a temporary relationship might not motivate that vendor to go above and beyond for you. There are several factors that go into choosing a pentest partner: skills, history, price, and processes among others. Building a solid relationship with them and getting the most value requires communication, trust and transparency.