Is my developer's home-brew password security right or wrong, and why?

Question

Our developer, let's call him 'Dave', insists on using a home-brew script for hashing passwords. See Dave's proposal below.

We have already researched and adopted an industry standard protocol using Bcrypt. Our protocol is not new, but is based on tried and tested implementations that support millions of users.

EDIT: To be clear, there isn't anything new about the implementation in our proposal. The "protocol" is simply set of specifications detailing the current state of the art, software components used, and how they should be implemented. The implementation is based on a known-good implementation. Thanks @agf for raising the question.

Dave has argued against this proposal from day one. His reasoning is that algorithms like Bcrypt, because they are published, have greater visibility to hackers, and are more likely to be targeted for attack. He also contests that the protocol itself is too bulky and difficult to maintain, but I believe Dave's primary hangup is the fact that Bcrypt is published.

What I'm hoping to accomplish by sharing his code here (besides a lesson in humility), is to generate consensus on:

1. Why home-brew is not a good idea, and
2. What specifically is wrong with his script

Thank you in advance. Your feedback is sincerely appreciated.

/** Dave's Home-brew Hash */

// user data
$user = '';
$password = '';

// timestamp, random #
$time = date('mdYHis');
$rand = mt_rand().'\n';

// crypt
$crypt = crypt($user.$time.$rand);

// hash
function hash_it($string1, $string2) {
    $pass = md5($string1);
    $nt = substr($pass,0,8);
    $th = substr($pass,8,8);
    $ur = substr($pass,16,8);
    $ps = substr($pass,24,8);

    $hash = 'H'.sha1($string2.$ps.$ur.$nt.$th);
    return $hash
}

$hash = hash_it($password, $crypt);

Answer 1

/** Dave's Home-brew Hash^H^H^H^H^Hkinda stupid algorithm */

// user data
$user = '';
$password = '';

// timestamp, "random" #
$time = date('mdYHis'); // known to attackers - totally pointless
// ^ also, as jdm pointed out in the comments, this changes daily. looks broken!

// different hashes for different days? huh? or is this stored as a salt?
$rand = mt_rand().'\n'; // mt_rand is not secure as a random number generator
// ^ it's even less secure if you only ask for a single 31-bit number. and why the \n?

// crypt is good if configured/salted correctly
// ... except you've used crypt on the username? WTF.
$crypt = crypt($user.$time.$rand); 

// hash
function hash_it($string1, $string2) {
    $pass = md5($string1); // why are we MD5'ing the same pass when crypt is available?
    $nt = substr($pass,0,8); // <--- BAD BAD BAD - why shuffle an MD5 hash?!?!?
    $th = substr($pass,8,8);
    $ur = substr($pass,16,8);
    $ps = substr($pass,24,8); // seriously. I have no idea. why?
    // ^ shuffling brings _zero_ extra security. it makes _zero_ sense to do this.
    // also, what's up with the variable names?

    // and now we SHA1 it with other junk too? wtf?
    $hash = 'H'.sha1($string2.$ps.$ur.$nt.$th); 
    return $hash
}

$hash = hash_it($password, $crypt); // ... please stop. it's hurting my head.

summon_cthulhu();

Dave, you are not a cryptographer. Stop it.

This home-brew method offers no real resistance against brute force attacks, and gives a false impression of "complicated" security. In reality you're doing little more than sha1(md5(pass) + salt) with a possibly-broken and overly complicated hash. You seem to be under the illusion that complicated code gives better security, but it doesn't. A strong cryptosystem is strong regardless of whether the algorithm is known to an attacker - this fact is known as Kerckhoff's principle. I realise that it's fun to re-invent the wheel and do it all your own way, but you're writing code that's going into a business-critical application, which is going to have to protect customer credentials. You have a responsibility to do it correctly.

Stick to tried and tested key derivation algorithms like PBKDF2 or bcrypt, which have undergone years of in-depth analysis and scrutiny from a wide range of professional and hobbyist cryptographers.

If you'd like a good schooling on proper password storage, check out these links:

• How to store salt?
• Do any security experts recommend bcrypt for password storage?
• How to securely store passwords?

Answer 2

Advantages of a public protocol:

• Probably written by smarter people than you
• Tested by a lot more people (probably some of them smarter than you)
• Reviewed by a lot more people (probably some of them smarter than you), often has mathematical proof
• Improved by a lot more people (probably some of them smarter than you)
• At the moment just one of those thousands of people finds a flaw, a lot of people start fixing it

Disadvantages of a public protocol:

• If indeed a flaw is found,
• AND made public
• AND not fixed fast enough

then attackers will start searching for vulnerable sites/applications. BUT:

• They'll probably go after more important targets first
• When the flaw is exposed, you know and can lock-down
• Not all flaws are "critical" (most are like "collisions are possible under certain conditions" or "the time to bruteforce is not 10^12 years, but 10^6 years")

Security by obscurity is deemed outright bad. Inventing your own falls under that category.

Answer 3

If Dave is really "your" developer, as in you have the authority to fire him, then you have the authority to direct him to use a more well-documented scheme, and you should.

In cryptography, the fewer secrets that are required to be kept, the better. This applies especially to "hard-coded" secrets, such as the hash function itself, which are not secrets as soon as the code containing the secret leaves your building.

This is why open standards for cryptography are a good thing; if the scheme has been around for years or even decades, with the basic algorithm and various implementations publicly known, and still nobody's found a way to get in, it's a good scheme.

Case in point, Polynomial provided a very good breakdown of what's wrong with the scheme, but here are the major failings of the proposed hash algorithm:

• MD5 is completely broken; While a preimage attack isn't made much easier by the primary ways that MD5 is broken (it's extremely vulnerable to known-plaintext strong collisions; knowing the message and the digest, one can produce a collision in 2^24 time), the hashing algorithm is way too fast to be secure against modern distributed cracking hardware. It should never be used in applications requiring data security.

• SHA-1 is considered vulnerable; Again, there's not an elegant vector for a preimage attack, but there is a strong collision attack (2^61 time for a 160-bit hash), and the hashing algorithm is fast enough and the keyspace small enough that current hardware can feasibly brute-force it.

• More hashes don't necessarily mean better hashing. Hashes are a deterministic process; they stand in for a "random oracle" in theoretical cryptography, but since they must always produce the same output given the same input, they cannot add any entropy to what is already inherent in the input.

  Stated simply, using a secret combination of multiple hashes as Dave is doing, even if that combination is unknown, doesn't add a significant amount of work to a brute-force (the relative order of three hashes has 6 possibilities, increasing the complexity of trying all possibilities by less than 3 powers of 2), and that extra complexity disappears as soon as an attacker can decompile your code and discover the relative order of the hash functions.

• Passwords are inherently low-entropy. The best "user-consumable" (non-gibberish) passwords max out in complexity at about 50 bits of entropy, meaning they can be cracked, if you have some idea of what you're looking for, in 2^50 or better time. That makes passwords easier to crack than other things you'd normally hash (like certificate digests), requiring additional "proof of work" to increase their security.

• This scheme is not adding any significant proof of work; three hashes instead of one, in the grand scheme of things, adds the equivalent of about 1.25 bits of entropy to the scheme in extra required work; you could do better by just requiring passwords to be one character longer.

BCrypt, in contrast to all of the above, has as its main advantage an extremely slow key derivation function or KDF. Blowfish, the encryption cipher that forms the basis of BCrypt, uses an "SBox"; a large array of predetermined initial values, which is modified by the key and/or IV to produce the initial state of the cipher through which the plaintext is fed. In BCrypt, this SBox setup is made more complex by taking the smaller password and salt values and running them through the "unkeyed" cipher a predetermined number of times, "warming up" the cipher into a state that theoretically could only be produced by performing the same number of iterations of setup, using the same password and salt. This "warmed-up" cipher is then fed a constant plaintext to produce the hash value. In CS terms, the hash forms a "proof of work"; a computational task that is difficult (time- and/or resource-consuming) to perform, but easy to check for correctness (does the produced hash match the one we already have?).

That "predetermined number" of iterations of SBox setup is configurable; this is BCrypt's real power. A number of "rounds" of key setup are specified when hashing, and for n rounds, 2^n iterations of key setup are performed. That means that incrementing the number of rounds makes the hash perform twice as much work and take twice as long on the same hardware to produce the requisite proof of work. BCrypt therefore can easily keep up with Moore's Law, which has been the bane of many hash functions that came before.

BCrypt's main theoretical weakness is its low memory usage; while computation time can be exponentially increased, the amount of memory required remains effectively constant (the SBox doesn't get any bigger as the iterations increase and no "intermediate results" are required to be kept around). As such, while BCrypt stymies the pure pipelined computational power of a GPU, it remains vulnerable to cracking by more sophisticated "field-programmable gate arrays" (basically a massive, highly modular set of "soft-wired" logic units, that are the current state-of-the-art in highly distributed computing), because the low memory constraints mean you can just throw more relatively-inexpensive circuit boards at the problem.

A newer algorithm, SCrypt, combats FPGA crackers by exponentially increasing the memory demands as well as the computational expense of calculating the hash, so the limited memory available to each FPGA quickly makes them infeasible as well (distributed cracking would basically require large numbers of CPU/FSB/RAM combinations, essentially full computers, hooked together). The only problem there is that SCrypt is only 2 or 3 years old, so it doesn't have the cryptanalytic pedigree of BCrypt (13 years old). And really, if you have to worry about an FPGA-armed cracker getting a password in a feasible amount of time (like before you can change all of them anyway), you have pissed off some very powerful people, and have already exposed another serious vulnerability (allowing an attacker to obtain your stored password hashes in the first place, so they can crack one "offline").

Bottom line, use BCrypt for password hashing. It's 13 years old, based on a 20-year-old cipher algorithm, an in that time no known vulnerabilities have been found that would aid a password cracker. It's slow as molasses, and can be configured to always be so, provided you combine it with a requirement for users to change their password every 90 days or so, so new passwords keep getting hashed with more expensive configurations.

Answer 4

To be fair to Dave, in terms of homebrew password security this is one of the better cases as all it just a little obsfuscation (and really not much) masking hash = SHA1(salt + MD5'(Password)) where MD5' does a reversible swap of the order of the bytes of the MD5 hash. Now the username/time/random/crypt-part is just used to generate a salt, and the only requirement we have of salts is that they just need to have a very good likelihood of uniqueness; so while its over-complicated there's really no use talking about it.

Again hash = sha1($salt+md5'($password)). Rearranging the MD5 adds no security (swap(00112233445566778899aabbccddeeff) becomes ccddeeff8899aabb0011223344556677) the swap does not prevent you from using md5 rainbow tables after (e.g., look at the code and reverse the swap). However, the presence of the unique salt makes the rainbow tables infeasible.

Now to be critical, this boils down to a simple cryptographic hash function applied twice; which is better than storing plaintext password (see plaintext offenders ), and better than storing an unsalted hash (see linkedin leak ). However, in the age of cheap massively-parallel GPUs, this is too weak for modern use. Anyone with a bit of general-purpose GPU programming experience that got on the live server somehow (to obtain the hashes with their salt) can likely see his source code, try out their own password as a test case and then can brute-force any particular password at a rate of billions of attempts per second per GPU.

So if any user is using a password on a list of a million or so previously leaked password (say from linkedin), the attacker can near instantaneously crack it. If some user's password is a random 8-letter from the char set A-Za-z0-9; it would take about 60 hours on average to break per GPU (so if you had 60 GPUs; would take 1 hour). Using common cracking techniques exploiting forms of common passwords could speed it up dramatically. Its also worth noting that since $password passes through a 128-bit MD5 hashing function, there is absolutely no benefit in using more than 128-bits of entropy in the passphrase (though to be fair that's a very secure password; like a 10 word diceware passphrase or random 22 character alphanumeric password).

Really, you should be using iterated cryptographic hash functions; that is something like bcrypt or PBKDF that slows down the brute-force rate of attackers by a large constant factor (say 10⁵) (so instead of 60 hours to crack a random 8-char password from a 62-char set (A-Za-z0-9); it takes 6 million hours (~700 years to likely be broken) with a single GPU, and this gets better with stronger passwords (e.g., a 10 char password would take ~3 million years; so even with a million GPUs it would take 3 years). So a little keystrengthening moves relatively weak password (8 random chars from 62-charset) out of the range of feasibility of being cracked by an attacker. For more on the upper limits of using a simple keystrengthened password see this answer.

Collision Attacks vs Pre-Image Attacks (or Why Collision Attacks on a hash function are irrelevant for password hashing)

KeithS's answer, while it gives good advice (use bcrypt and not a simple cryptographic hash function to hash your password) originally criticized MD5 and SHA1 for the wrong rationale (don't use MD5 as its vulnerable to collision attacks). There's a subtle difference between pre-image and collision attacks and the arguing in the comments was too condensed, so I am elaborating here. I highly suggest reading the wikipedia article on pre-image attacks.

A pre-image attack says given a specific 128-bit hash written in hexadecimal: h=ad2baf26a87795b3c8a8366a08b44112, a specific hash function H, please find any message m such that h=H(m); note that there are N=2¹²⁸ different distinct hashes. Now if your cryptographic hash function is not broken, each bit in your hash has a 50% chance of being 0 or 1 for a random message m. Then I will need on average to generate hashes for roughly N=2¹²⁸ hashes before I am lucky enough to find any message m such that h=H(m) (this message may not be the same input that was used to originally generate the hash -- but this still falls under the category of 'pre-image' attack).

A collision attack says find me any two messages m1 and m2 such that H(m1) = H(m2). Note that m1, m2, (and H(m1)) are all free to change. This case is a much easier problem since if I generate M hashes for M different messages, I am not just comparing the M messages to one specific hash (so have M chances of a finding a collision), now I have M*(M-1)/2 pairs of hashes, so roughly M^2 chances of having a collision. So in this case, I will need roughly to generate roughly sqrt(N)~2⁶⁴ hashes before its likely that one of them will collide with another one on an ideal 128-bit hash.

Let's look at the two types of birthday problems. The collision problem translates into the common birthday "paradox"; how many people do you need in a room before its quite that two people share a birthday with N=365 days in a year. The answer is a paradox as you only need about sqrt(N) ~ 23 people before its likely that two people share a birthday (as with 23 people in a room you have 253 different pairs of people that could match). (I do know that sqrt(365) != 23: I am using approximate math not focusing on insignificant constant factors. Re-doing the calculation with sqrt(365) ~ 19 people in the room then P(two share birthday) = 19! * comb(365,19)/365**n = 37.9%, which while not strictly 50% still means its fairly likely to happen). Note for the collision birthday problem, you can't have N+1=366 people in a room and there be a chance of not having a collision (ignoring leap days); at best the first 365 people had different birthdays and the last person generated a collision.

The pre-image problem is a very different question, how many people do I need in a room before it is quite likely that someone has one specific birthday (say B=December 18th). In this case, I need roughly N ~ 365 people before its likely to happen. E.g., with 365 people in the room you have a P(somebody has birthday B) = 1 - (1 - 1/365)^(# people) so for # people = 365 you have a 63% chance that someone's birthday will be some fixed date B. In this case, you can easily imagine having any number of people being in a room and there being no one who has a birthday on one specific date. (Say you only invited people into the room if there birthday was not a given date; there's no limit to the number of people you could invite).

When a hash function like MD5/SHA1 is broken for collision attacks that means you can generate a collision in less work than the brute force time of sqrt(N) ~ 2^numbits/2. For MD5 it only takes about ~ 2^24 time to generate a collision; for SHA1 it takes ~ 2^61 time. That means collision attacks on MD5 are extremely easy to make; but practical attacks on SHA1 are still difficult. But collision attacks only matter if you don't care what hash you are trying to match. These collision attacks are quite relevant for some applications like cryptographically signing messages to ensure message integrity, so beware of using MD5/SHA1 in those cases. However, when you have a unique salt, and a specific hash you are trying to match to authenticate, collision attacks do not matter.

Answer 5

See the related Security Meme post

While this may seem very simplistic, the rules hold true - designing crypto algorithms and implementing them correctly/securely is very hard. Even the ones designed by experts and picked at by thousands of people over years have holes discovered in them eventually.

So Do Not Roll Your Own Crypto is good advice for everyone (possible exceptions include notables like Rivest, Adleman, Pornin, Shamir)

Answer 6

OK, fire Dave. At the very least hit him with a very large clue-bat. Open protocols are good because anyone can look and attempt to find vulnerabilities and structural problems, and implement fixes. The visibility improves the protocol. Good security means that everyone can know how the system works and it is still secure.

Answer 7

Convince him with good reasoning. Don't berate him.

You have to think about why we are hashing passwords: The reason is to protect the original password by making the hashing process take a lot of CPU time to execute. Brute force is the way the passwords are typically recovered.

If the attacker is able to steal your password database then they've managed to access your database. If they have access to the database then they probably have access to the code that produces these passwords as well. After reviewing the code they can start brute-forcing the passwords because the hashing algorithm takes too little time to execute.

The idea behind well-known hashing methods and practices like using PBKDF2 with 10,000 iterations is that they take a lot of time to execute on current hardware.

If you have some time on your hands you could also write a brute-forcing program and show him how many attacks per second you can execute with his algorithm vs prevailing standards.

Answer 8

While we can find plenty of flaws with Dave's algorithm, it really isn't horrible because it isn't 100% home brew; he does use hashing protocols that (albeit weak) are based on solid principles. On the other hand, he takes steps that increase complexity for the developer but do little to improve the security of his algorithm.

But the reason I am adding another answer here is to bring up the age-old argument that there is value in obscurity. The primary line of defense should always be strong, widely-accepted hashing algorithms but there is no harm in adding even the tiniest amount of obscurity as an additional layer of defense.

That layer of obscurity should never be considered a defense in itself, simply an additional factor that reduces the chances of compromise. Since a successful attack normally requires a number of factors to fall in place, even small defenses can have a great impact in reducing your overall risk.

I can't tell you how many times I have seen hash dumps that I have tried to crack where I simply get nothing because some Dave somewhere thought of this dumb non-standard algorithm that I can't crack short of doing an intense cryptoanalysis or breaking in to the web servers to get the algorithm. You can't deny that these dumb algorithms are a valid defensive factor.

Now if Dave took his dumb algorithm and added it as a layer to stronger algorithms such as PBKDF2 or bcrypt, then he has obscurity with the backing of solid security practices. Also, obscurity doesn't need to be as complex as Dave's code, all you really need is one bit to be different to be obscure. Remember this is not a defense, just a layer of obscurity.

Better ways to accomplish obscurity are using an HMAC, concatenating a server-side salt constant, and/or using a large, non-standard number of iterations. None of these measures will stand on their own but they certainly do address specific attack vectors that contribute to your overall risk.

tl:dr; security through obscurity is bad, but solid security plus a reasonable amount of obscurity is a valid strategy.

Answer 9

I've written several hashing algorithms. There's nothing wrong with it, if you know what you're doing. In a very slight way, he's right about the fact that tried-and-true algorithms may be a little more vulnerable to attack. So if you can create a good algorithm, you're golden. The only problem is that his totally sucks, for a number of reasons.

1. // timestamp, random # $time = date('mdYHis');

   I don't even feel like I need to explain this one. He was trying to create a seeded "random" number when it doesn't make any sense to do so. The hashes will change day to day. I cannot imagine this working in any reasonable scenario.

2. He says that he want's to stray away from known hashing algorithms, yet he uses MD5 (the known-est of them all, and a fairly weak one too) for the bulk of "his" algorithm. His method is only a very slight derivative of plain ole MD5 hashing. Which brings me to:
3. He's shuffling his MD5'd result. Uh, why? As a fun little activity, try asking him to explain his reasoning there. And I'll give you a tip to help you out: Whatever his answer is -- it's wrong. Shuffling a hashed value is like shuffling a deck of all-blank cards. There is no extra benefit from it whatsoever.

There are more things wrong with it, too, but these are the big ones. Tell him to get back to his usual work, and just use bcrypt (or scrypt, which is probably even a little more secure to do the fact that it's really CPU intensive)