50

I'd like to ask you what should be satisfactory result of pen-testing job?

My main concern is that pen-testing is hard and it won't always result in gaining remote shells or roots. However, it is much easier to list potential vulnerabilities.

For example, if there's PHP version 4 from 2007 I can list it as potential vector but I may be unable to exploit it. Is successful exploitation a requirement for pen-testing job? Would vulnerability scan be good result of the job as well if there's some successful exploitation included (but accounts for less than 1% of all possible issues).

2
  • 8
    It is satisfactory to fullfill a contract. Having actual workable exploits VS showing with good certainty that exploits could be achieved at a particular location depends on what your customer wants. Exploiting a flaw can cost more and is not necessarily what your customer asks for, as he might have a limited budget
    – niilzon
    Nov 8 '16 at 13:03
  • 1
    Finding a sub7 trojan on the domain controller? [Hey! The testing was a success! Now you've got worse problems.]
    – Joshua
    Nov 11 '16 at 0:18
76

As someone who contracts pen-testers more than I act as a pen-tester, what I'm looking for is that you did more than run Nessus/ZAP/Burp - I can do that myself (though I expect that you do that as well). I expect you watch the dataflows in the app/website and look for those loose threads that indicate there is a logic error that might be exploitable. I expect that you are able to tell me what you can glean from the outside, that you can tell me things that cause concern that couldn't be found with a scan.

I'm looking for indications that you looked at, for instance, password reset screens and considered whether the flow is exploitable. I want to see that you've considered whether privileged information is available to unprivileged users (ie, is the app just using css to hide it or something daft like that).

Ideally, I've done the easy stuff before I contract you - I've done the scan, I've done the patches and I've picked all the low-hanging fruit. I hire a pen-tester for the hard stuff.

Really, if you don't manage an exploit, I want to see that you've worn your fingernails down scratching at the outside looking for a crack.

2
  • 5
    I'd like to add that there is also an audit part. There is a reason why tools like metasploit value the reporting side so much: You not only want to know that the pentester did the hard work but also have a set of documents (database, text, JSON, whatever) that describe which tests were run. The next time you will contract a pentester and he finds something, he can also track what changes to the system triggered the problem based on the previous report.
    – grochmal
    Nov 9 '16 at 13:31
  • 1
    Echo this answer in spades. The company I work for just had some external pen-testing done, but only after we'd run automated tools a couple of times and made some less-formal, non-expert attempts to penetrate ourselves. What we wanted was what human expertise and "out-of-the-box" thinking could find that we / automated systems couldn't find. Also, while listing all potential vulnerabilities is "the right thing to do", you also need to give context ("theoretical, conceivable attack" vs. "begging to be exploited") so we can make appropriate calls on what to defend against.
    – TripeHound
    Nov 10 '16 at 9:48
28

Is successful exploitation a requirement for pen-testing job?

Following a strict definition of penetration testing, you have to actually attack the target system and keep a record of your successful and failed attempts. It's not sufficient to simply conclude that a server should be vulnerable because your fingerprinting tools revealed an outdated software version. You are explicitly taking the perspective of an attacker and have to demonstrate how the system can be penetrated.

The SANS Penetration Testing paper makes the following distinction (although definitions vary):

Pen-Testing vs. Vulnerability Assessment

[There] is often some confusion between penetration testing and vulnerability assessment. The two terms are related but penetration testing has more of an emphasis on gaining as much access as possible while vulnerability testing places the emphasis on identifying areas that are vulnerable to a computer attack. [...] A vulnerability assessor will stop just before compromising a system, whereas a penetration tester will go as far as they can within the scope of the contract.

That said, your average customer is probably unaware of this distinction and maybe doesn't really want you to spend too much time going "as far as you can". It might be more important to them to receive clear instructions on what exactly needs to be fixed rather than getting a list of all your root shells. You will have to find out beforehand what they effectively want to achieve by letting you test the system. Your customer should be aware that a penetration test is not equal to a comprehensive security assessment.

1
  • 1
    I'd like to add that in addition to this that it depends on the goal. Usually what people want is breadth-first testing, which is a VA. They want a wide picture of risk rather than a deep picture of risk. Of course, that's not always the case, so clarifying the goal is important to determining the appropriate outcome.
    – h4ckNinja
    Nov 9 '16 at 1:22
1

I guess it depends on what you've tasked to find and what the scope of work is. Some pen tests I've done have only wanted to see the theoretical findings, others wanted me to actual break in and create some havoc. Pen testing is only hard to those who don't truly enjoy it. Get creative, have some fun...but stay within scope. ;)

5
  • 4
    Sorry, but I have to disagree. If a pen test's deliverable is only theoretical findings, that's not a pentest.
    – h4ckNinja
    Nov 9 '16 at 1:23
  • 1
    You can disagree, as I did when I got the request, but the scope of this customer's pen test was to find the holes and NOT punch through them.
    – cyb3ard
    Nov 9 '16 at 12:13
  • @hamm3rh3ad That's Vulnerability Assessment, not PenTesting
    – pguetschow
    Nov 11 '16 at 7:31
  • I'm aware of the difference between a vulnerability assessment and a Pen Test. Some people/customers call it a Vulnerability Assessment, some may consider it a 'Passive Pen Test'. In my scenario I didn't simply run a vulnerability scanning tool against their products and call it a day. There is a difference between performing a vulnerability assessment and performing full reconnaissance in prep for attempting to gain access to a system or product.
    – cyb3ard
    Nov 11 '16 at 12:03
  • I guess it depends on the customer's terminology when you get down to it. Yes what I described is by definition a "Vulnerability Assessment", but in the context the customer framed it, it read and was classified as a "Passive Penetration Test". I just signed the paper and did the work, I don't care what they call it as long as they pay me. ;)
    – cyb3ard
    Nov 11 '16 at 12:06
1

Let me put it this way: unless the developer of the application under test is themselves a security expert, then I absolutely do expect you to find at least some exploit. I would be very unhappy if you did not find anything at all, because in my experience your average developer does not have a thorough enough understanding of security issues to be able to avoid all possible holes from the get-go.

Note that I would lift the "security by obscurity" restriction for your test. That is, you would get access to the machine (shell etc.) or even to the sourcecode for gaining knowledge about the application. Your penetration must of course work without that, just like a real attacker's.

2
  • You don't know how many times the software has been pen-tested by different auditors and patched. If it's a first time ever, sure...
    – kubanczyk
    Nov 9 '16 at 13:31
  • Well, sure, if I had previous knowledge, like already passed pen-tests, then my answer would probably be different. But by default, that's what I would expect. It is hard to write secure software, so my assumption that a non-security-savvy developer will leave holes which an up-to-date security expert will find. Call it a "litmus test" for penetration tests.
    – AnoE
    Nov 9 '16 at 15:13
1

It's really hard to define the quality of performed pentest, and anything can be a satisfactory result. It really depends on the system. If the pentest is performed on a relatively simple system or website, then it's highly likely that there will not be any high risk findings.

Moreover, even if they are very skilled, they might not have enough time to exploit the system, and show a proof-of-concept. They must prioritize the pentest to cover all the areas included in the contract.

Let's assume pentester discovers a SQL injection or vulnerable software (based on version number). It's definitely good to try to exploit and show vulnerability for a client, but it might be very hard to exploit it, and it often would not make sense to spend majority of planned pentest time to produce a proof-of-concept for a single vulnerability.

Pentester showing that specific software version is vulnerable based on CVEs, or pentester showing SQL error output based on user input should be good enough reason for a client to patch and fix their systems.

1

This is a good question as I think many places approach the pen test incorrectly. This has become even worse because the executive level of many organisations tend to believe the pen test is a silver bullet - if you have a pen test and the results show that there was no successful penetration of any system, then your security is OK and we can all tick that box. On the other hand, if the pen test fails, then the security team are not doing their job. This simply isn't the case.

A pen test is just one tool which can be used to assess the effectiveness of your security controls. It does not tell you that all your systems are secure and the level of confidence you can place in that pen test depends heavily on both the skills of the pen tester and how well you have specified and planned the test. The absolute worse thing you can do is simply call some security company and say "Hi, I want a pen test, when can you do it". Prior to engaging a pen tester, you need to have a clear idea of what you expect from the pen test, what your priorities are and what information you want in the final report. You also want to approach more than one provider. What you are looking for is someone who is able to demonstrate they have the right skills, who can understand your requirements and who is able to provide you with a result which can assist you in improving your security posture. A pen test should not be thought of as a 'test' in the sense of either a pass or a failure. Essentially, you need to have a clear understanding before the test of what the grades are and on completion of the test, have sufficient details and information to help you focus on how to get a better grade in the next test.

One of the most difficult aspects of pen testing is that the quality of the pen test is vary much determined by the individual performing the pen test. I have switched security companies primarily because an employee of the first company has changed employers and that individual was someone we knew could do a good pen test for the organisation (usually because they were talented and good at their job and because they understood our business).

Engaging a pen tester is in itself a skill. You will get better at it each time you do it provided you approach the task with clear objectives. Knowing your current security posture is only one part of the equation. You need to have a clear idea of what your final target state is. What you want from the pen test is increased clarity regarding your current posture and sufficient information to help you develop plans to move the organisation to the desired state.

A good pen test provides you with details of what was done, what succeeded and what failed. It should not just be a list of possible vulnerabilities or systems which are not at the latest patch level or examples of 'poor practice'. All of that can be obtained with simple vulnerability assessments. It should provide full details of how the environment was penetrated and suggestions of what controls could be implemented to prevent or reduce the likelihood of repeat occurances or the impact from these occurances. The pen test report should provide sufficient details that you can assess whether a failure to gain access was because the controls are adequate or because there was not sufficient time allocated or because the scope was too limited etc. In many respects, what you are looking for is a partnership with the pen tester where they are working with you to improve your security.

0

A satisfactory result of a pen test is when the client and/or target audience of the reporting understands the messages and takes appropriate action based on decision science. Please reference How to Measure Anything in Cybersecurity Risk for more information on decision science as it relates to a positive-penetration test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.