Specifically considering client websites where we have been asked to execute a pen test; at what point do we stop and say we're done?

We have access to various tools (some automated, some manual); but if we say "we tried all our tools, and couldn't make any progress", that could be construed as us saying that we're not clever enough (and there's always some hacker out there who could be cleverer).

So; how do we protect ourselves against upset clients who claim that we didn't work with due diligence? Is there a standard report framework we can work within?

  • 22
    Short answer: you specify the scope and expected output in the initial contract for services – schroeder Feb 1 '17 at 18:12
  • 2
    A good term to look for is: Managing client expectations. In general for this kind of jobs: You will work together with the client to make the application safer. You don't sell: A fully 100% safe application. – Luc Franken Feb 2 '17 at 8:48

So this is actually a very interesting question for the industry in general. The way I would suggest you handle it is

  • Have something in your contract that disclaims liability for vulnerabilities not noted during testing. Reason for this is, it's basically impossible to be sure that you've found every exploitable issue in a website, or any other system. To pick one example, think of all the sites that were sitting vulnerable to shellshock for years and years, should all the pen test companies who touched one of those sites be liable for not telling their customers?

  • Have a methodology, saying what you will do. This should cover the general areas of testing that will be completed. For websites, consider basing on something like the OWASP Top 10 as a starting point. This gives you some common ground with the customer on what you'll be looking at.

  • Make sure your company covers the basics with a checklist. as @rapli says above document all the little things, but don't overblow the severity. Whilst it's important to make sure your test isn't just a checklist, using one can avoid embarassing mistakes where basic tests get missed.

The problem you might/will run into is unrealistic expectations from customers. that one is a case by case to address. If you get a customer that expects that their complex application will be completely reviewed in like 5 person-days of testing, well you should explain why that's not a practical concept :)

  • 5
    You should disclaim liability for any vulnerabilities, period. Even if you noted an issue, there is no guarantee that your client will fix it. – DepressedDaniel Feb 2 '17 at 2:15
  • Rory - you could add in a link to PTES which has a section on this: pentest-standard.org/index.php/Main_Page – Rory Alsop Feb 2 '17 at 8:23
  • @DepressedDaniel I was mainly thinking of liability for you not telling them things, generally I'd say if you tell a customer about something, it's obviously down to them whether they fix it (I'd be surprised if a court thought otherwise), whereas some customers may get the mistaken idea that you will find "all the vulns" and it's that which needs a disclaimer – Rоry McCune Feb 2 '17 at 11:23
  • @RоryMcCune it's not entirely "obviously down to them whether they fix it" - for example, I have certainly seen contracts (or even mandatory audit requirements) where it's explicitly in scope to verify the corrections and re-test the system after they have been implemented. – Peteris Feb 2 '17 at 12:35
  • 1
    but it's not for the pen test company to require a fix. Their regulators may require a fix, their customers may require a fix, but the pen test company has no say here (at least in my experience). Sure testing companies get asked to do re-tests, but they can't mandate that the customer actually fix the issues mentioned. To me that's entirely ok as the pentest company is only covering (in most cases) technical risk and doesn't have a background on the wider business risk position – Rоry McCune Feb 2 '17 at 12:52

Specify in contract which security aspects you investigate and only take responsibility for those. You wont always find vulnerabilities. But I guarantee you will find few minor things, and I suggest you to include every little detail you can in the report, missing HSTS in headers, weak ciphers, etc. So they see that you did something.

There are some reporting tools I know of, but they are either not publicly available or paid products.


As addition to the other answers:

When your contract states your scope and which vulnerabilities to test, you could write down that you actually tested cases in scope against known vulnerabilities defined in the contract. Something like:

SQL injection:

SQL-Injections are ...

We identified 42 Input fields in the Webpages in Scope. We identified 0 
Input fields vulnerable to SQL injection (with our used method).

Drawback is: in case there actually is a SQL-Injection possible in one of the listed fields appended to the report; you didn't promise 100% security but promised that this isn't a vulnerability, but then you surely made something wrong - that's why you should somehow state which kind of tests you used, what kind of SQL-Injections are state of the art in the introduction to SQL-Injections, so you will not be blamed for not finding attacks which will be found in future.

This also works for general Website-Security stuff:

HTTPS: https is ... recommended to use TLS X.Y ...

We determined the usage of https for all websites in Scope with TLS X.Y.
Certificates expire Dates are set to Date X which is in the recommended
certificate expire time range. Certificates hold an 4096-Bit key which is
acceptable for current usage.

Also try to create templates of your reports since you don't want to rewrite everything about SQL-Injection and HTTPS/TLS etc. over and over again, so you only have to fill the results of your tests. This will also ensure you made all tests and did not miss any, when you see some paragraph not written to be done and not found anything nor have any findings.

  • A word of caution on stating that you've covered 'x' fields of 'y' vulnerable to SQL Injection. a) do you know exactly what vectors all your automated tools use, b) if you're blackbox making sure you even have a list of all the valid parameters (let alone their use in combination) is very tricky c) having valid test data to excercise all business processes and make sure you hit all code paths can be very tricky in blackbox tests... – Rоry McCune Feb 2 '17 at 12:55
  • This is true, i was focused on the web-app specified pentest. But i'd indeed say i should know what my tools test, else how could i think i did test anything for anything at all? Having an application to test in maschine code is imo not comparable since you don't have those standard does and don'ts like OWASP top 10. – SchreiberLex Feb 2 '17 at 13:30
  • ah I was talking about web app. pen tests, sorry, by black box I mean you only have remote access (e.g. no code and no server access). Interesting statement about the tools, given that I know some of them don't explicitly say what vectors they use, so it can be some work to work it out... – Rоry McCune Feb 2 '17 at 13:34
  • That's true, but i won't argue with code coverage on remote tests at all, hell if your customer wants to have his entire system checked he has to give you the docs and anything. Testing remote is to test what you see, so document what you see and what kind of tests you run at it. And you should know if your tool recognizes an SQL injection when servers timing is wrong even without any suspicios DB-like response. On the other hand you may not be able to really know everything, but how can you then ever be sure what you did really test? – SchreiberLex Feb 2 '17 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.