You are essentially substituting characters for images. So the complexity and security increase as you require more images and there are more images to choose from. The entropy increases if the images are not similar and are equally likely to be chosen. If you also require a specific order, there are more possibilities (combinations vs permutations).
I would approach this the same as passwords - you need more security, you should make the challenge harder. However, since this is for self-service reset its needs to be balanced with ease. This is an interesting method because you are now using something the user knows (a secret) but not information that can be looked up (e.g., high school mascot).
Things to consider (specific to image reset, we should spin another question for general self service reset).
- If you have 20 pictures, is there an equal likelihood that the common user would select any one of them? If certain pictures are more likely to be chosen, a brute force attack can be better focused. Experiments with password schemes where user's click on parts of a picture have been shown to be poor since many users tends to click the same areas of interest within a picture.
- Are the photos generic, or if you know something about an individual, would you be able to guess what an individual is going to pick
- The user should only remember the images they pick, so I think having a library of large number of options only helps - this is using recognition, if the user doesn't recognize an image later, then it unlikely they picked it
- Thinking about the above, you may be able to offer a small sub-set of images to the user during the selection process, but during the reset process you can offer a super set. This would be like requiring only requiring lower case characters in a password, but convincing the attacker he needs to set his brute forcer to check for upper, lower, numbers, and special. Also, if a large number of pictures are chosen which were not in the set available for the initial password setting this may help identify an attacker [If this is not already a feature, add it]
For the actual number, it should be directly proportional to the risk. A key system or application should require more complexity and options to increase the number of possibilities. This must also be balanced against the users. You may need to do per organization testing to see where the mins, max, and happy mediums are. Perhaps you stratify based on the risk of a particular account being compromised or the accounts value.