Why are randomly generated passwords often hexadecimal?

Question

Most of the time, when I see a randomly generated password, it's hexadecimal, i.e. only uses the character set [0-9a-f]. Wouldn't it be much more secure if the character set were [0-9a-z], let alone [0-9a-zA-Z]?

Answer 1

Arminius mentioned simplicity of hex, and I think that's something worth expanding on.

Random number generators typically work on bits, so the range of numbers they can generate is a power of two. A range of characters like [0-9a-zA-Z] has 62 characters, which is two shy of a power of two (64), so the computer has to do some conversion between the ranges.

That can be done, but it's easy to get it wrong. The "standard" way is to take the actual number, divide it by the range you want, and take the remainder as your random number. That introduces bias though. For a simple example, say you generate numbers in [0-3] but you want them in [0-2] instead. The [0-3] range would map to the [0-2] range like so:

0 => 0 mod 3 => 0
1 => 1 mod 3 => 1
2 => 2 mod 3 => 2
3 => 3 mod 3 => 0

Note how you can get a 0 in two different ways: 0 and 3 both go to 0. The approach is biased toward generating 0s, which will make your password easier to guess.

The correct way involves computing the right padding to make what you generated fit evenly into the range, which is tricky and could potentially make your password a lot longer depending on how your output range compares to that of the random number generator.

An easier approach is to just use a range that's already a power of two so you can ignore the bias entirely. Most have problems.

• Base-2 (binary) produces extremely long strings.
• Base-4 (quaternary) isn't much better.
• Base-8 (octal) is better but still lengthy.
• Base-16 (hex) is a little long, but reasonable. It also encodes 4 bits per character, which is pretty convenient when computers prefer multiples of 8.
• Base-32 encodes 5 bits per character, which is not convenient when computers prefer multiples of 8.
• Base-64 encodes 6 bits per character, which is still awkward (but slightly less since at least it's an even number).
• Base-96 is popular, but not a power of 2 so it has the same problem as [0-9a-zA-Z].
• Base-128 and above all involve symbols you can't easily type on a typical querty keyboard.

To expand a little on why base-64 is a problem, consider what happens when you try to encode a single byte (8 bits). You can't do it with a single base-64 character since that only gets 6 of the 8 bits. But if you use two characters, you have to figure out how to pad your 8-bit byte to a 12-bit output without introducing bias or suggesting there might be an extra byte.

Hex, in contrast, is almost trivial to encode bytes in. Just look up each byte in a 256 element table to get two characters and spit them out. It gives reasonably short passwords, doesn't use weird symbols, and it's simple to implement. It's the best choice for security conscious password generators.

Perhaps best of all is that most programming languages support hex out of the box. The C library has printf(), which can format as octal, decimal or hex. Likewise, C++ has IO manipulators that use the same formats. There's no built-in support for other bases though (not even base-64), so you have to do it yourself (tricky) or find a library that does it right (violating the NIH rule (and also possibly a lot of work to verify it)). It's a lot easier to just use what's in the standard library, especially when it works.

It's true that you get less security per character with hex than with other encodings, but as Eric Lagergren helpfully pointed out in the comments, hex(random_bytes) is exactly as secure as just random_bytes. It's just longer is all. In fact, all you need is 16 bytes (encoded as 32 hex digits) to have a password so strong that any attacker will burn through all the energy in the solar system before they get a minuscule chance of guessing it. Most websites will happily accept 32 character passwords, so there's no trouble using them.

Answer 2

(Do you have evidence that random passwords are usually hex?)

Here are some possible reasons why you frequently find hexadecimal passwords:

• The passwords you see might be already hashed. Since hash functions return binary strings, you will often see them stored in a hexadecimal form.

• Hashing a random binary sequence (e.g. from /dev/urandom) is a cheap but reasonably safe way to create a random password. You'll occasionally see something like this in the wild:

  $ head -c 100 /dev/urandom | sha256sum | head -c 25
  c9ea6b67af36753a68ef42429
  

• All characters are easy to distinguish. That is, you won't run into problems with confusing 1 and I, O and 0, etc. if you only permit hexadecimals in the first place.

• Randomly generated sequences are usually binary, so displaying them in a hexadecimal form seems logical and efficient.

That said, choosing a random password from a larger range than [0-9a-f] is obviously safer if you don't adjust the length. I usually go with your suggestion of [0-9a-zA-Z] which gives me a broad range and avoids special characters that might cause issues with storing and transmitting.

---

Also, here is what I put in my shell config to quickly generate passwords on Linux:

function spw() {
    cat /dev/urandom | tr -dc a-zA-Z0-9 | head -c ${1:-20}
}

E.g., spw 25 will give me a 25-character passwords of [0-9a-zA-Z].

Answer 3

Overview

While there are many examples of "random" password generator programs available on the Internet, generating randomness can be tricky and many programs do not generate random characters in a way that ensures strong security.

Type and strength of password generated

Random password generators normally output a string of symbols of specified length. These can be individual characters from some character set, syllables designed to form pronounceable passwords, or words from some word list to form a passphrase. The program can be customized to ensure the resulting password complies with the local password policy, say by always producing a mix of letters, numbers and special characters. It should be noted that such policies typically reduce strength slightly below the formula that follows, because symbols are no longer independently produced.

With that in mind lets move on to WHY you only saw a-f and not a-z.

Symbol Sets

The reason you saw a-f instead of a-z, is primarily because of it's Symbol Set. For example:

Hexadecimal Numerals: (0-9, A-F) (e.g. WEP Key)  

As you can see, It can be created for a WEP Key using Hexadecimal Numerals Symbol Set. If you want something like (a-z, A-Z, 0-9) Then I would suggest the Case Sensitive Alphanumeric Symbol Set which is more complex than a Hexadecimal Symbol Set.

Resources

You can read more into generated passwords and symbols sets with the link below:

I'm the link below!

Hopefully this helps you solve your problem.

Answer 4

In addition to programmers being lazy (as stated in Arminuis' answer, generating a hex password is a trivial one-liner even if you have nothing but the shell and standard tools), the arguably "better" encodings that pack more bits into each character have usability issues.

The issue is not so much that converting the numbers is hard or that there are odd numbers of bits in each character which don't easily sum up to a power of two (though base-32 works very well for e.g. 80- or 160-bit numbers).

The conversion code is half a dozen lines of code, and not precisely rocket science (division, modulo, and table lookup). The fact that bits don't sum up is mildly annoying but not really an issue. It only means that the last 8 characters encode slightly fewer than 40 bits (or the last 4 encode slightly less than 24 bits for base-64). While that means there's less entropy in those trailing characters, still the password as such is perfectly random, there's no issue other than maybe being "less elegant than you wish". The fact that you have that = at the end is a non-issue as well, you can just leave it away.

The bigger problem is human interaction. Base-32 and base-64 contain enough symbols to produce (random) meaningful human-readable words or word fragments, some of them being more acceptable and some generating a less favourable reaction.
With hexadecimal, dead, f001 or affe (monkey in German) is pretty much the worst you can get, but in base-32, the substrings fuck or wanker would be entirely feasible, as well as virtually every insult that exists in any language that uses the roman alphabet. There you go, universal, internationalized insult-o-matic for your users.

Also, naive base-32 contains the symbols 0 and O as well as 1 and I which the user may confuse especially when printing or writing down passwords and re-entering them or having to communicate over the phone.
In base-64 you have not just 1 and I but also i and j as well as / to increase your joy. And, of course, telling someone a mixed-case 20-character string over the phone is sheer bliss.

There exist -- at least for base-32 -- several workarounds like the RFC4648 alphabet or Crockford's encoding which try to address the above issues (confusing symbols and insult-o-matic) to different extents.

These are however "clumsy" in a different way. To someone regularly using the system, it soon becomes noticeable that something is just not right, some symbols (notably 0 and 1) seem to never appear. Something must be broken?

Now, hexadecimal is a bit longer, a few more characters to type, but it has none of the above issues, and it is the most trivial to generate.