4

I have done a number of vulnerability assessments and I'm noticing a trend. In the first assessment the clients are impressed and grateful for the massive security holes I find. During the second and third assessments, they feel frustrated because they were hoping to get a "clean report" that they could tuck into their marketing package and brag about to their potential clients.

Even though they are doing a good job in mitigating the issues, the problem is that I'm better at my job so I always find more. On top of all that, there is no such thing as perfect security. I believe the only "clean report" is one where the assessor didn't dig deep enough. By the very nature of vulnerability assessments, I'm working against them - and yet I run a business and I need to find a way to work with them at the same time.

I also pride myself at being very helpful in teaching developers how to mitigate issues. The reality is that more often than not the theory doesn't get translated into practice. The developers cite tight deadlines and tight budgets from their management. I know the industry so, it's not hard to believe.

This puts me into awkward situation. Clients are scared to do an assessment with me because they feel they are not ready and haven't patched enough. They are (rightfully scared) of what I will find. Some even tell me "don't go so deep this time, ok?". And because I don't want to lose their business I find myself holding back my full abilities. Needless to say, this rubber stamping is not fun at all. It may be profitable, but not fun.

I'm also worried that this will backfire because inevitably someone else will re-assess what I did and find massive holes. They they will conclude that I have no idea what I'm doing because I missed such obvious flaws. Reputation is everything in this industry, so this is a big risk.

One solution that I can think of is that I keep finding fresh clients and staying away from 2nd and 3rd assessments. Finding fresh clients is something that I find frustrating and inefficient. I like repeat business where my relationship gets deeper over the years. It's the long term relationships that I enjoy in my business. The closeness is what creates the problem in the first place. Has anyone found an alternative where you can keep exposing never-ending flaws while keeping a client happy? Or am I just dreaming?

The other solution I tried is restricting scope. If I only test a small part of the stack, the client could get what they are looking for: a "clean report". But it's rare that they agree to that. They usually want everything tested. They can't have the cake and eat it too, but I have trouble communicating that to them.

I think I know the answer to my own question. I'm just wondering if I'm missing something here or if this is just the way it is and I have to deal with it.

  • 4
    "Hey doc, please give me a full physical, but don't tell me if I have cancer. My spouse wouldn't like to hear that." – schroeder May 11 '18 at 22:58
  • 3
    Keep in mind that there is no balance with integrity, it’s why it’s called integrity ;-) – John Keates May 12 '18 at 1:55
  • Perhaps you mistaken security assessment with security implementation feasibility. Anyway, it seems @Mark Buffalo help clarify your question with a good answer. In short, all business come with a cost , it is up to them to carry out the step to fix stuff that within they deem important. Perhaps the most important things is not how detail you can report, but ability to show them the priorities. You just can't fix everything and discover everything in a single run. – mootmoot May 15 '18 at 15:57
  • You might want to ask this on one of the SE sites dedicated to interpersonal exchanges. – forest May 15 '18 at 23:46
6

Sounds like you're doing security assessments for compliance reasons, but I see no mention of compensating controls, acceptable risks, remediation plans in action, etc.

This whole question is really weird to me because from the information you've provided in your question, it seems like neither side of the business relationship -- either you or the client telling you to cover things up -- is qualified to perform compliance assessments. Not trying to be harsh, but there's simply way too many red flags. I'll explain why below.

For some things, the client will not, cannot, and should not be forced to fix in order to get a "Clean Report," but let's [Mass Dispel] some things first.


Security Assessments

A security assessor's job is to find vulnerabilities, and help explain away vulnerabilities that are not exploitable. Something may be vulnerable, but there could be multiple reasons why it isn't exploitable.

It will then be your job to examine the evidence and determine whether or not the evidence is correct, and then exclude those irrelevant items from the report. In the case of PCI compliance, evidence of this will need to be provided to a third-party Qualified Security Assessor.

If they're hiring you to do the explaining, you'll need to dig deep on each vulnerability you find and attempt to assess whether or not that vulnerability can be exploited. Work with the engineers (sysadmins, developers, network engineers) to get the information you need.

It could be something simple such as nmap/Nessus/etc. returning the wrong version, not realizing that a backported version exists, or something to do with multiple compensating controls for common TLS "misconfigurations."

You can check for the exact version with dpkg -l | grep 'package' for example, then determine if a backported security patch was applied. This problem is really common with many approved scanning vendors.

These are businesses who can't stop what they're doing to fix every little thing that isn't exploitable. It's not a sustainable security model. Should they still be made aware? Absolutely. But they need to prioritize fixes and manage risk.


Clean Reports vs. Fraud

Correct me if I'm wrong, but it sounds like they're looking for a clean report with no remediation. Something's really fishy here. This doesn't make sense.

Any reputable company, especially those involved in required compliance audits, be they quarterly or yearly, will fix all of the vulnerabilities which prevent a passing score for their compliance report. In many cases, they are able to circumvent fixing these vulnerabilities if they can provide documentation for compensating controls, acceptable risks, remediation plan in progress, etc., and you will be required to accept these circumventions if their evidence is correct, but you will need to examine it yourself.

If they attempt to pass off a clean report that is not really clean, then it's fraud - especially if they're pushing these lies to their clients. That's fraud in the highest order, and you do not want to be complicit in this. There will be severe penalties for both you and your client.


Fraudulent Activities

My responses in this section will be made with the assumption that no compensating controls, acceptable risks, remediation plans in action, etc. exist, and they're trying to cover up vulnerabilities.

They are (rightfully scared) of what I will find. Some even tell me "don't go so deep this time, ok?"

Assuming what you're saying is true and not taken out of context, then I'm pretty sure that's illegal -- unless you're going out of scope. The company should provide you a scope, and you must stay within whatever is determined to be in scope for the audit. If you intentionally go outside of your defined scope, I'd be surprised if they continue to hire your company to perform assessments as that's a betrayal of trust.

The other solution I tried is restricting scope. If I only test a small part of the stack, the client could get what they are looking for: a "clean report". But it's rare that they agree to that. They usually want everything tested.

I'm having trouble with this. Almost every single client will provide a scope, especially those going through compliance audits. Not everything is going to be in scope for them. If you want to go out of scope because you think you found something dangerous, you need to ask permission and get it in writing.

I'm also worried that this will backfire because inevitably someone else will re-assess what I did and find massive holes. They they will conclude that I have no idea what I'm doing because I missed such obvious flaws. Reputation is everything in this industry, so this is a big risk.

If you're within scope, then... do you want to be complicit in fraud? You're worried you'll lose clients if people are asking you not to dig too deep, but think about what will happen when a qualified auditor shows up and discovers your company is complicit in fraud.

This is an extreme ethical violation. If you have an (ISC)² certification, then prepare for revocation if anyone finds out. This can also amount to a federal crime if you're in the United States, or anywhere else where fraud is illegal.

Believe me, as someone who has dealt with multiple penetration testing and security assessment firms, if you hold back on glaring flaws like this, many of us will no longer do business with you. If you have a reputation for doing this, it will spread quickly. I've seen consultation firms dropped for not finding very basic things that we already knew about.


Clean report with Remediation

However, it's not uncommon for companies to provide a clean report after they fix your proven vulnerabilities. There should be no need to provide penetration test and security assessment reports that contain vulnerabilities, but only the report after the company remediates them.

If you find more vulnerabilities afterwards (provided they're paying you to do so), that's actually great! Any reputable company will be glad to fix these problems (or have you or themselves explain compensating controls, acceptable risks, remediation plan in progress, etc.) so they can pass their audit.

If they're hiring you too late in their compliance audit process and are worried it won't be completed fast enough, then that's on them and not you -- unless you're too slow at getting your job done within a reasonable time frame.

Where's the remediation coming into play here?


Bringing Balance to the Force

How to maintain balance between integrity and client satisfaction in vulnerability assessments

There is no balance in the case of fraud or other crimes.

Do your job and do it right. Don't violate anyone's trust and do not commit fraud. Clients who ask you to commit crimes are not the type of clients you're looking for, or they shouldn't be.


TLDR

Get some more experience with assessment and compliance auditing. Manage risk, but don't commit fraud.

  • 1
    I too, am confused by some of the wording in the question. What external party cares if the pentester "found things"? Every finding needs context and a Management Response. To "keep the client", the OP needs to work with them on the secondary report, and not try to make the primary report palatable to a 3rd party. – schroeder May 11 '18 at 23:01
  • 1
    Luckily I haven't released the results of the assessments yet, so I can still go as deep as I should. Also these are not for official compliance. These assessments are internal / marketing material. But thanks to your harsh words I understand it doesn't matter. By all counts its a horrible idea. Company A says "I will use your web application only if you share your internal VA with me" Company B says "OK here it is" Company A says "yay or ney" – user3280964 May 11 '18 at 23:28
  • Ah, I do apologize as I did not intend to be harsh. Your question and tag seemed to indicate that this is for compliance reasons, but there's nothing in your question that indicates anyone had compliance training on either side. ¯\_(ツ)_/¯ You could make money and gain more business if you can help them with a final report after everything is fixed or proven to be "not a problem." I'm a bit concerned about the client too. – Mark Buffalo May 12 '18 at 0:04
2

I work in a similar job as you do and I know these inquiries.

There is one thing that you and your clients have to understand: the management of risks that you identify is not your job. It is the job of your clients. As Mark pointed out in his excellent answer, you have an obligation to them (and to yourself) to tell them truthfully about the vulnerabilities of the systems within the presented scope. But your clients then have to decide what to do about these vulnerabilities. If they do not have all the information they need, they can not come to a sound decision. Furthermore, it is their choice, what information they pass on to third parties (if not legally bound to otherwise).

Make it clear to your clients where your and their duties begin and where they end. Only if that is clear they will stop asking for "clean" reports.

1

Like you said, if you "hold back" I'm fairly sure that could potentially lead to some legal issues. But I think that the best thing to do would probably be to write out your talking points and then sit down with them and fully explain your situation. Then to elaborate on a feasible level of security for them and maybe give examples of larger companies and how even they aren't completely immune but they still try their best.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.