Outsourcing security evaluation

Question

Does anyone have any experience with outsourcing security code review via freelance sites (like Rent-a-coder, Elance, Guru, Getafreelancer, etc.)? Is this effective? What are the best practices? Are there any pitfalls? Does anyone have any recommendations or lessons-learned? Are there any resources or readings to learn more about others experience with this?

Some example questions: Do you have any advice for how to screen for freelancers with security knowledge? (Is there a standard pre-test to screen candidate freelancers?) For security review, is it more effective to ask for open-ended review, or is it more effective to create many smaller tasks with a narrow scope (e.g., review this codebase for SQL injection vulnerabilities)? Pricing-wise, are there any standard rates or pay ranges? Is it more effective to charge a fixed price, or are bonuses effective (e.g., a bonus for finding a vulnerability)? Is it a good use of money to try to hire multiple people for the same code review task, to maximize the chances of finding problems, and/or to cross-compare their results and identify the most effective freelancers?

One issue you can take as out of scope: you can assume that I understand the risks of sharing such code with unknown individuals, and am comfortable with that.

Answer 1

Trying to cut the code review up into multiple pieces suffers from two problems. The first is the infamous and ever present problem of enumerating badness, and the second is the overhead of recruiting and managing the remote freelancers.

Going for an open ended review will avoid the issues of splitting it up, but you will need to take care ensuring that the freelancers don't just stop at the low hanging fruit because they feel that they have written as much as you paid for. Any remote outsourcing needs to be very carefully specified - being very specific about how you want the review reported will manage the risk of freelancers stopping early. Developing this specification interactively with the freelancer - maybe even starting with a complete blank slate - will provide a great opportunity to assess the scope and depth of their knowledge better than a quiz would allow.

Deliberately seeding the code for review with problems is also a possible approach to assessing reviewers, but you get the feedback too late in the process to be of use unless you're looking for reviewers to use on a repeat basis.

While getting multiple reviews will probably be worth your money it may not be worth the increased process overhead. The trade-off will vary between organisations - if technical staff are doing all the recruitment and management their time is likely better spent on code and internal review, if project management time is plentifully available the overhead may be acceptable.

Answer 2

While I don't have any experience hiring someone from a freelance site to do code reviews, thinking about the question I believe that you are really looking for someone who can just be another eye to catch some omission. Sounds like a 'duh moment I know. But, if what you are asking if you can slap something together then just get a secure code editor to pretty it up and make it nice and secure whilst all you do is concentrate on the logic ... I think you're going to be disappointed.

You need to know and probably have your code 95-97% ready to go already using some secure coding principles, you build that in from the beginning can't tack it on at the end. What about befriending a JUG or other local programming group, perhaps establishing a relationship there and seeing if you can't get a code review that way or at least maybe someone can pass someone along word of mouth.

I think you want to be specific if you are going to go the freelance route. Portion up pieces of your program and hire out to look for a specific type of vulnerability. Code analysis is a long process and you may not get a good product if someone who values their time by the hour knows that looking for a bug is only worth it for X hours but you've gotten paid for some time less than what it requires so the motivation is gone. It's an interview, maybe it would be a good idea to toss them something you know is a bug to make sure they can find it.

I'm curious as to how people with more experience in this will answer your question.