What is a satisfactory result of penetration testing assessment?

Question

I'd like to ask you what should be satisfactory result of pen-testing job?

My main concern is that pen-testing is hard and it won't always result in gaining remote shells or roots. However, it is much easier to list potential vulnerabilities.

For example, if there's PHP version 4 from 2007 I can list it as potential vector but I may be unable to exploit it. Is successful exploitation a requirement for pen-testing job? Would vulnerability scan be good result of the job as well if there's some successful exploitation included (but accounts for less than 1% of all possible issues).

Answer 1

As someone who contracts pen-testers more than I act as a pen-tester, what I'm looking for is that you did more than run Nessus/ZAP/Burp - I can do that myself (though I expect that you do that as well). I expect you watch the dataflows in the app/website and look for those loose threads that indicate there is a logic error that might be exploitable. I expect that you are able to tell me what you can glean from the outside, that you can tell me things that cause concern that couldn't be found with a scan.

I'm looking for indications that you looked at, for instance, password reset screens and considered whether the flow is exploitable. I want to see that you've considered whether privileged information is available to unprivileged users (ie, is the app just using css to hide it or something daft like that).

Ideally, I've done the easy stuff before I contract you - I've done the scan, I've done the patches and I've picked all the low-hanging fruit. I hire a pen-tester for the hard stuff.

Really, if you don't manage an exploit, I want to see that you've worn your fingernails down scratching at the outside looking for a crack.

Answer 2

Is successful exploitation a requirement for pen-testing job?

Following a strict definition of penetration testing, you have to actually attack the target system and keep a record of your successful and failed attempts. It's not sufficient to simply conclude that a server should be vulnerable because your fingerprinting tools revealed an outdated software version. You are explicitly taking the perspective of an attacker and have to demonstrate how the system can be penetrated.

The SANS Penetration Testing paper makes the following distinction (although definitions vary):

Pen-Testing vs. Vulnerability Assessment

[There] is often some confusion between penetration testing and vulnerability assessment. The two terms are related but penetration testing has more of an emphasis on gaining as much access as possible while vulnerability testing places the emphasis on identifying areas that are vulnerable to a computer attack. [...] A vulnerability assessor will stop just before compromising a system, whereas a penetration tester will go as far as they can within the scope of the contract.

That said, your average customer is probably unaware of this distinction and maybe doesn't really want you to spend too much time going "as far as you can". It might be more important to them to receive clear instructions on what exactly needs to be fixed rather than getting a list of all your root shells. You will have to find out beforehand what they effectively want to achieve by letting you test the system. Your customer should be aware that a penetration test is not equal to a comprehensive security assessment.

Answer 3

I guess it depends on what you've tasked to find and what the scope of work is. Some pen tests I've done have only wanted to see the theoretical findings, others wanted me to actual break in and create some havoc. Pen testing is only hard to those who don't truly enjoy it. Get creative, have some fun...but stay within scope. ;)

Answer 4

Let me put it this way: unless the developer of the application under test is themselves a security expert, then I absolutely do expect you to find at least some exploit. I would be very unhappy if you did not find anything at all, because in my experience your average developer does not have a thorough enough understanding of security issues to be able to avoid all possible holes from the get-go.

Note that I would lift the "security by obscurity" restriction for your test. That is, you would get access to the machine (shell etc.) or even to the sourcecode for gaining knowledge about the application. Your penetration must of course work without that, just like a real attacker's.

Answer 5

It's really hard to define the quality of performed pentest, and anything can be a satisfactory result. It really depends on the system. If the pentest is performed on a relatively simple system or website, then it's highly likely that there will not be any high risk findings.

Moreover, even if they are very skilled, they might not have enough time to exploit the system, and show a proof-of-concept. They must prioritize the pentest to cover all the areas included in the contract.

Let's assume pentester discovers a SQL injection or vulnerable software (based on version number). It's definitely good to try to exploit and show vulnerability for a client, but it might be very hard to exploit it, and it often would not make sense to spend majority of planned pentest time to produce a proof-of-concept for a single vulnerability.

Pentester showing that specific software version is vulnerable based on CVEs, or pentester showing SQL error output based on user input should be good enough reason for a client to patch and fix their systems.

Answer 6

This is a good question as I think many places approach the pen test incorrectly. This has become even worse because the executive level of many organisations tend to believe the pen test is a silver bullet - if you have a pen test and the results show that there was no successful penetration of any system, then your security is OK and we can all tick that box. On the other hand, if the pen test fails, then the security team are not doing their job. This simply isn't the case.

A pen test is just one tool which can be used to assess the effectiveness of your security controls. It does not tell you that all your systems are secure and the level of confidence you can place in that pen test depends heavily on both the skills of the pen tester and how well you have specified and planned the test. The absolute worse thing you can do is simply call some security company and say "Hi, I want a pen test, when can you do it". Prior to engaging a pen tester, you need to have a clear idea of what you expect from the pen test, what your priorities are and what information you want in the final report. You also want to approach more than one provider. What you are looking for is someone who is able to demonstrate they have the right skills, who can understand your requirements and who is able to provide you with a result which can assist you in improving your security posture. A pen test should not be thought of as a 'test' in the sense of either a pass or a failure. Essentially, you need to have a clear understanding before the test of what the grades are and on completion of the test, have sufficient details and information to help you focus on how to get a better grade in the next test.

One of the most difficult aspects of pen testing is that the quality of the pen test is vary much determined by the individual performing the pen test. I have switched security companies primarily because an employee of the first company has changed employers and that individual was someone we knew could do a good pen test for the organisation (usually because they were talented and good at their job and because they understood our business).

Engaging a pen tester is in itself a skill. You will get better at it each time you do it provided you approach the task with clear objectives. Knowing your current security posture is only one part of the equation. You need to have a clear idea of what your final target state is. What you want from the pen test is increased clarity regarding your current posture and sufficient information to help you develop plans to move the organisation to the desired state.

A good pen test provides you with details of what was done, what succeeded and what failed. It should not just be a list of possible vulnerabilities or systems which are not at the latest patch level or examples of 'poor practice'. All of that can be obtained with simple vulnerability assessments. It should provide full details of how the environment was penetrated and suggestions of what controls could be implemented to prevent or reduce the likelihood of repeat occurances or the impact from these occurances. The pen test report should provide sufficient details that you can assess whether a failure to gain access was because the controls are adequate or because there was not sufficient time allocated or because the scope was too limited etc. In many respects, what you are looking for is a partnership with the pen tester where they are working with you to improve your security.

Answer 7

A satisfactory result of a pen test is when the client and/or target audience of the reporting understands the messages and takes appropriate action based on decision science. Please reference How to Measure Anything in Cybersecurity Risk for more information on decision science as it relates to a positive-penetration test.