Why shouldn't we roll our own?

Question

Why shouldn't we create our own security schemes?

I see a lot of questions around here about custom crypto and custom security mechanisms, especially around password hashing.

With that in mind, I'm looking for a canonical answer, with the following properties:

• Easy for a newbie to understand.
• Clear and explicit in why rolling your own is a bad idea.
• Provides strong examples.

Obligatory xkcd.

Answer 1

You can roll your own, but you probably will make a major security mistake if you are not an expert in security/cryptography or have had your scheme analyzed by multiple experts. I'm more willing to bet on an open-source publicly known encryption scheme that's out there for all to see and analyze. More eyes means less likely that the current version doesn't have major vulnerabilities than something developed in-house by non-experts.

From Phil Zimmermann's (PGP creator) Introduction to Cryptography (Page 54):

When I was in college in the early 70s, I devised what I believed was a brilliant encryption scheme. A simple pseudorandom number stream was added to the plaintext stream to create ciphertext. This would seemingly thwart any frequency analysis of the ciphertext, and would be uncrackable even to the most resourceful government intelligence agencies. I felt so smug about my achievement.

Years later, I discovered this same scheme in several introductory cryptography texts and tutorial papers. How nice. Other cryptographers had thought of the same scheme. Unfortunately, the scheme was presented as a simple homework assignment on how to use elementary cryptanalytic techniques to trivially crack it. So much for my brilliant scheme.

From this humbling experience I learned how easy it is to fall into a false sense of security when devising an encryption algorithm. Most people don’t realize how fiendishly difficult it is to devise an encryption algorithm that can withstand a prolonged and determined attack by a resourceful opponent.

(This question has more discussion of the above quote.)

If you are not convinced of "Don't Roll Your Own [Cryptography/Security]", then you probably are not an expert and there are many mistakes you likely will make.

Is your application robust against:

• Timing Attacks. E.g., to the nanoseconds do completely-bad keys and partially-bad keys take the same amount of time in the aggregate to fail? Otherwise, this timing information can be exploited to find the correct key/password.

• Trivial Brute Force Attacks; e.g., that can be done in within seconds to years (when you worry about it being broken within a few years). Maybe your idea of security may be a 1 in a billion (1 000 000 000) chance of breaking in (what if someone with a bot net tries a few billion times?). My idea is to aim for something like 1 in ~2^128 ( 34 000 000 000 000 000 000 000 000 000 000 000), which is roughly ten million billion billion times more secure and completely outside the realm of guessing your way in.

• Attacks on user accounts in parallel; e.g., you may hash passwords with the same (or worse no 'salt') on all password hashes in the db like what happened with the leaked linkedin hashes.

• Attack any specific account trivially simply. Maybe there was a unique random salt with each simply hashed (e.g., MD5/SHA1/SHA2) password, but as you can try billions of possible passwords on any hash each second, so using common password lists, dictionary attacks, etc. it may only take an attacker seconds to crack most accounts. Use strong cryptographic hashes like bcrypt/PBKDF2 to avoid or key-strengthen regular hashes by a suitable factor (typically 10^(3-8).

• Attacks on guessable/weak "random" numbers. Maybe you use microtime/MT-rand or too little information to seed the pseudo-random number like Debian OpenSSL did a few years back.

• Attacks that bypass protections. Maybe you did hashing/input validation client side in web application and this was bypassed by the user altering the scripts. Or you have local application that the client tries running in a virtual machine or disassembles to reverse engineer it/alter the memory/ or otherwise cheat somehow.

• Other attacks, including (but not attempting to be a complete list) CSRF, XSS, SQL injection, network eavesdropping, replay attacks, Man in the Middle attacks, buffer overflows, etc. Best protections very quickly summarized. CSRF: require randomly generated CSRF tokens on POST actions; XSS: always validate/escape untrusted user-input before inputting into db and displaying to user/browser; SQLi: always use bound parameters and limit how many results get returned; eavesdropping: encrypt sensitive network traffic; replay: put unique one-time nonces in each transaction; MitM: Web of Trust/Same as site last visited/Certificate issued by trusted CA; buffer overflows: safe programming language/libraries/executable space protection/etc).

You are only as strong as your weakest exploitable link. Also just because you aren't rolling your own scheme, doesn't mean your scheme will be secure. Its quite likely that the person who created what you rolled out was not an expert, or created an otherwise weak scheme.

Answer 2

There is a house in my area with a really nice deck outside the second story family room. It looks swell, until you go underneath and see how it was constructed. It seems the homeowner decided he did not need to pay big bucks to a builder or architect to tell him how to build a deck. He built it himself and it looks like a chaotic spider web of 2x4’s underneath. It PROBABLY will be fine. Personally, I would rather not risk life and limb on an amateur construction job like that.

I think that if you want to develop an algorithm to do encryption, you should do so and have a good time of it. I would not recommending using it to hide your online bank statements but if you want to encrypt your girl friend’s love letters on your home computer, that should be fine—provided your wife is not a cryptanalyst.

There is a story in “The American Black Chamber”* about the Navy developing their own ciphers. The Navy would show their new cryptosystem, pleased with themselves and Yardley, the Army analyst, would promptly break the code, explaining what they had done wrong. They would offer to fix the code but Yardley pointed out that while they could fix specific weaknesses, without a solid understanding, they were going to always have a problem. Their system was intrinsically flawed. It is a little like patching a leaky roof. You can patch forever but the water is still going to find its way in. If you don’t want to get wet, the roof needs to be constructed by somebody that knows more than a little about roofs.

Did I ever tell you about the do-it-yourself brain surgery I performed on my late mother-in-law? Everything went fine until she went and died. Seriously, few of us would trust our health to an amateur doctor; do you really want to trust your secrets to amateur software? I hate to admit it but I buy a lottery ticket every few months. I fully expect to lose but the potential payout is huge. I can play the odds and maybe I will come out ahead. If I don’t, I am out a buck. Why play the odds on encryption? The payout is not there.

Regards, /Bob Bryan

• Recommended: Herbert O. Yardley, “The American Black Chamber” -- A book as interesting today as when it was written in 1931. “The American Black Chamber was filled with good stories well told, as well as frank descriptions of Yardley's successes in cryptanalysis. It was a best-seller in 1932 -- overseas as well as domestically.” From NSA: Pearl Harbor Review - The Black Chamber

Answer 3

The original post asked for an example:

There is a good story of a bad cryptosystem causing problems. Mary Queen of Scots was imprisoned by her cousin Queen Elizabeth I and was communicating with people on the outside via encrypted letters. The alphabet was replaced with a cryptoalphabet of squiggles, crossed circles and triangles with extra letters assigned for common letters, like e,t,i and o so the meaning of letters could not be found quickly by frequency analysis. They also added a few null characters which were ignored on the decryption to throw off analysts. The problem was that The Queen had a very competent cryptanalyst on her staff in the person of Thomas Phelippes who was able to decrypt the messages as they were intercepted.

As things progressed, Mary went along with a plot to have her escape and take over the throne. When The Queen’s agents intercepted the last letter from Mary before passing it on, they added an encrypted sentence asking for the names of those involved in the plot “so they can be properly rewarded.” Mary’s correspondent dutifully answered and The Queen’s agents had everybody involved executed.

When my kids were small, I would send them cryptograms with their lunch (with a key (using a Vernam cipher)). Generally, they were jokes but they were never of any importance. In a case like that, roll your own is fine. If you are plotting to overthrow The Queen of England (or the Shah of Iran or the slowly reforming Thug-ocracy in Myanmar), I would suggest that you make sure that what you using cannot be easily decrypted. I think it was Bruce Schneier that said, anybody can come up with a cryptosystem they cannot decrypt but coming up with one nobody else can decrypt is harder.

Answer 4

Bruce Scheier wrote back in 1998:

Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can't break. It's not even hard. What is hard is creating an algorithm that no one else can break, even after years of analysis. And the only way to prove that is to subject the algorithm to years of analysis by the best cryptographers around.

Cory Doctrow dubbed this concept "Schneier's Law" in a 2004 speech as follows:

Any person can invent a security system so clever that she or he can't think of how to break it.

As a follow-up, this again from Schneier:

When someone hands you a security system and says, "I believe this is secure," the first thing you have to ask is, "Who the hell are you?" Show me what you've broken to demonstrate that your assertion of the system's security means something.

Phil Zimmmerman also wrote in his original PGP papers:

When I was in college in the early seventies, I devised what I believed was a brilliant encryption scheme. A simple pseudorandom number stream was added to the plaintext stream to create ciphertext. This would seemingly thwart any frequency analysis of the ciphertext, and would be uncrackable even to the most resourceful Government intelligence agencies. I felt so smug about my achievement. So cock-sure.

Years later, I discovered this same scheme in several introductory cryptography texts and tutorial papers. How nice. Other cryptographers had thought of the same scheme. Unfortunately, the scheme was presented as a simple homework assignment on how to use elementary cryptanalytic techniques to trivially crack it. So much for my brilliant scheme.