13

Apart from the conventional email phishing tests, what other security Key Performance Indicators can be used to measure end user security awareness in an Organization?


Looking at the SANS critical security control #9:

  • 9.4 Validate and improve awareness levels through periodic tests to see whether employees will click on a link from suspicious e-mail or provide sensitive information on the telephone without following appropriate procedures for authenticating a caller; targeted training should be provided to those who fall victim to the exercise.

I am trying to come with an indication that looks beyond the phishing emails or providing sensitive information over the phone. For example, how would I know people are aware of the dangers of using infected USBs in the organisation? Disabling the use of the USB prevents the security breach but does not benifit end user awareness of the issue. Users usually see it as an annoying control that stops them from performing their tasks. USB is just one example, how would I know that users are aware of the dangers of sharing their usernames? and similarly other wrong practices. getting their answers on the subject by surveys is one part, are there other indications?

  • 4
    www.awarity.at is providing a plattform for e-learning and incident reporting, where you are able to get certain KPIs. – Dr.Ü Dec 3 '14 at 10:46
  • 1
    Useful question! I encourage you to flesh out your question a bit more. You might start by spelling out the acronym "KPI" (to help others find your question via search) and explaining what research you've done and what you've found. – D.W. Dec 4 '14 at 1:11
  • How about comparing the number of infections from USB before and after the awareness session? That would give an indication how successful was the awareness session and how aware the users are. – AdnanG Dec 11 '14 at 4:01
6

A rudimentary training program should minimally educate users on critical issues. Measuring its effectiveness provides the opportunity to ensure that users are getting the relevant information they need to do their jobs safely and effectively.

Send out a survey that assesses awareness of job-specific information security issues and see how your employees score.

  1. How often should you change your password?
  2. What are appropriate methods for transmitting confidential information to a business partner?
  3. Whom it is safe to discuss your personal details with over the phone?
  4. How can you identify a secure site?
  5. How can you identify a secure e-mail?
  6. What is "phishing"?
  7. What are the security risks in using social networks?
  8. What information have you (or would you) put on a social network?
  9. How can computer viruses be avoided?
  10. What types of incident should be reported?

etc.

  • 2
    Surveys are a good example of a KPI that can be used. – AdnanG Dec 3 '14 at 11:00
  • As long as the surveys are not graded (i.e. turn it into an exam). You want to get a snapshot of the awareness so that you know how you are doing, not how they are doing. – schroeder Nov 23 '17 at 15:54
6

Going out on a limb here, I believe in testing by regular drills and Red Team simulated attacks. If the Red Team succeeds in social engineering or other kinds of compromise, the awareness is nil.

Reasoning: KPIs and surveys and whatnot measure and report averages. The attacker is not interested in averages, he's going to attack (citing from a recent comment by one of the Sec.SE denizens) the laptop belonging to the CFO's secretary, or whatever weakest spot he finds.

You are also looking for anti-complacency measures. I cannot find a better way to keep users on their toes than regular drills, spot checks and the very real threat of an actual attack.

enter image description here

  • While your comment about the attacker is true, I am interested in an indication on how good we are. No attack does not mean we are in good shape, similarly an attack does not mean we are in a total mess. – AdnanG Dec 3 '14 at 12:15
  • @AdnanG - the Red Team will find the weakest link. If it's good enough, you're OK, if it breaks, you're in a mess. – Deer Hunter Dec 3 '14 at 13:18
  • Trying to find an indication of "how good we are" (and using the acronym KPI) already suggests that it's a failure, though. As pointed out by Deer Hunter, it absolutely doesn't matter how good "we" are, it matters how bad one person is (the single, worst person). It doesn't matter for your organizational security if 120k people wordwide change passwords twice per week when at the same time everybody in a particular south-east-asian office will share credentials and "borrow" RSA tokens to anyone who asks for them (which is something I've actually seen!). – Damon Dec 3 '14 at 15:21
  • 4
    Today I learned a new English word: complacency - a feeling of quiet pleasure or security, often while unaware of some potential danger, defect, or the like (dictionary.reference.com/browse/complacency). – Ismael Miguel Dec 3 '14 at 16:46
  • It is said that since a squad always stays together, a squad is also only as fast as its slowest member. – Panzercrisis Dec 3 '14 at 17:40
2

What we decided for one KPI is the amount of incidents reported. In the current phase, we say we want more incidents to be reported: not enough people report/are aware they should. When the amount rises, this will mostly be because of increased awareness. From a certain point (no real steady increase in reports) we will switch to Less incident reports means less incidents. This means that we do not rely on reports yet, but will in the future.

Another KPI might be the amount of people actually trained: track them. Third KPI could be the amount of policies people remember by doing a survey. However, I found out people only fill in multiple choice surveys with enough dedication to take the response seriously.

  • I like the idea of keeping track of how many people are trained. However, one should look at how recent their training is. And perhaps the training should only count if the participant passed a test. – S.L. Barth Nov 23 '17 at 15:11
  • @S.L. Barth I agree with you on the "when is he/she trained" addition. However, I can imagine someone fails a test. a test is always a snapshot and will probably be different than reality. When one fails multiple tests after multiple training sessions, that would be a useful KPI to me indeed. – johan vd Pluijm Nov 23 '17 at 15:25
  • @S.L.Barth tests suffer the same weaknesses as mandatory completion of training. There is little guarantee that the student retained any knowledge. That's why the 'survey' suggestion in another answer here is such a good idea. It tracks how good the training is, not how good the user is at bluffing through a easy multiple choice exam. – schroeder Nov 23 '17 at 16:01
2

SANS Secure The Human now publishes lists of KPIs for you to consider:

https://securingthehuman.sans.org/resources/metrics

They split it up into "Tracking" metrics (who has completed the training) and "Impact" metrics to try to measure the impact training has had.

  • number of people successfully phished over time
  • number of people reporting phishing
  • number of infected computers
  • number of people posting sensitive info on social networking sites
  • etc.

Basically, any behaviour that you train for can be turned into a metric (tracking number of people, not the number of instances, for obvious reasons).

In my book (shameless plug) I list a number of non-traditional metric types that can be considered. For example:

  • number of false positive reports for users in the first 2 weeks after training or an awareness push
  • numbers of repeat offenders
  • number of unique visitors to the internal awareness site
  • number of queries to the security team
  • number of reports of errors in the training material (after you include a couple purposeful errors)
  • etc.

You really need to treat this like you would for tracking users on a sales website (ask your web user tracking team, they can help with ideas). You train behaviours and you need to track the change in behaviour over time.

1

With awareness, what we seek to achieve is a basic understanding of threats and responses. It differs from training and education which pushes for more detailed understanding and knowing the rationale behind the various measures taken.

Therefore, the key to promoting awareness is simply through exposure. The more the staff are exposed to such security messages, the greater the awareness. One easy and cheap way to promote awareness is through printing security messages on mugs, mousepads, magnets etc. and giving them out to the staff.

You can hold contests for designing the above mentioned trinkets and give out prizes. To measure KPI, you can base it on the quantity/quality of submissions. If you are feeling adventurous, you could even plant clues inside these trinkets and award prizes to employees who manage to solve the riddle or problem.

By generating hype and getting more people to talk about it, you would have achieved your objective of awareness.

I am not a fan of surveys or questionnaires as most people simply study to pass it and do not really internalise the knowledge.

  • It's difficult to use a questionnaire both as a survey (i.e. a measurement of the general level of knowledge) and a test (to assess individuals). But a survey need not be something you “pass”, it's even best to make it anonymous. – Relaxed Dec 3 '14 at 13:28
  • There are issues with anonymous surveys or those that do not requires a pass. There would be a subset of employees who don't take it seriously and just answer yes to all qns. – limbenjamin Dec 3 '14 at 13:35
  • 1
    There are many issue with surveys and also many techniques to deal with these issues and an entire field of study devoted to them. But a mandatory test isn't really a survey and hardly solves any of these issues… In my view, confusing evaluation and measurement and hoping you can get good data by tying the survey with career consequences of some kind is a hundred times worse than any issue you might face with a proper survey. – Relaxed Dec 3 '14 at 13:38
  • 1
    I don't know, I'm a little skeptical about the value of exposure, beyond a certain point. It comes across as "thought police" tactics to some people. – David Z Dec 3 '14 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.