How can I measure the effectiveness of a Penetration Testing carried out in a Cloud Environment (IaaS/PaaS) by a third Party Vendor ? Should I ask for a CVSS score or something else ?
Any advise will be highly appreciated.
TL;DR - You should receive a report from the penetration-tester which will evaluate the security of the systems you're running, usually they will use a CVSS score to judge how vulnerable something is.
If you've had a professional pen-test they ideally will produce a report on the issues they found, how they exploited the issues and of course how critical those issues are. This is typically done with a CVSS score as it's a good way of measuring how bad the vulnerabilities are.
The Common Vulnerability Scoring System (CVSS) is a free and open industry standard for assessing the severity of computer system security vulnerabilities. CVSS attempts to assign severity scores to vulnerabilities, allowing responders to prioritize responses and resources according to the threat. - Wikipedia
So in short, yes ideally they should be giving you CVSS scores, you should also receive their report which will detail their findings from that you will be able to work out how to fix it. Obviously some common-sense is required, the pen-tester can only evaluate what they know; like every human they can be wrong, so when reading through their report just make sure it all makes sense & you can evaluate the security yourself (to some extent) obviously the pen-tester is more qualified than yourself in identifying vulnerabilities so don't second guess everything they've told you, just make sure it makes sense.
Like everything, a lot of this depends on your threat-model if you're a small company and assume that you won't be getting hit by nation-states and he points out a flaw in a crypto suite you're running which only a nation-state could attack (and you know it will cost too much to fix) then maybe it's best you carry out an internal risk assessment on that. Just because they've produced a report doesn't mean you cannot produce your own and identify risks yourself - in-fact I'd encourage it. It makes no sense for a company to patch everything (if they cannot afford it) just be aware of the pen-testers level of expertise.
In essence, it all comes down to Risk Analysis