8

Most of the time, when I see a randomly generated password, it's hexadecimal, i.e. only uses the character set [0-9a-f]. Wouldn't it be much more secure if the character set were [0-9a-z], let alone [0-9a-zA-Z]?

  • 9
    I don't think the premise is correct. If it is randomly generated, then unless the generated is forced to keep to 4bits, it is not reasonable to expect only hex chars. That said, the question is about "what you have observed". It is possible that somehow the sample you've seen could be this way. I've been using randomly generated passwords for several years now, and have never seen this "hex-only" symptom. – Sas3 Aug 6 '17 at 8:46
  • 5
    The simplest answer is that the hex string you see is just a hexadecimal representation of a random binary value. – Craig Aug 6 '17 at 11:00
  • 1
    Where are you seeing them? My password generator has checkboxes and is usually set for upper/lower/number/special. – chrylis -on strike- Aug 6 '17 at 18:58
6

Arminius mentioned simplicity of hex, and I think that's something worth expanding on.

Random number generators typically work on bits, so the range of numbers they can generate is a power of two. A range of characters like [0-9a-zA-Z] has 62 characters, which is two shy of a power of two (64), so the computer has to do some conversion between the ranges.

That can be done, but it's easy to get it wrong. The "standard" way is to take the actual number, divide it by the range you want, and take the remainder as your random number. That introduces bias though. For a simple example, say you generate numbers in [0-3] but you want them in [0-2] instead. The [0-3] range would map to the [0-2] range like so:

0 => 0 mod 3 => 0
1 => 1 mod 3 => 1
2 => 2 mod 3 => 2
3 => 3 mod 3 => 0

Note how you can get a 0 in two different ways: 0 and 3 both go to 0. The approach is biased toward generating 0s, which will make your password easier to guess.

The correct way involves computing the right padding to make what you generated fit evenly into the range, which is tricky and could potentially make your password a lot longer depending on how your output range compares to that of the random number generator.

An easier approach is to just use a range that's already a power of two so you can ignore the bias entirely. Most have problems.

  • Base-2 (binary) produces extremely long strings.
  • Base-4 (quaternary) isn't much better.
  • Base-8 (octal) is better but still lengthy.
  • Base-16 (hex) is a little long, but reasonable. It also encodes 4 bits per character, which is pretty convenient when computers prefer multiples of 8.
  • Base-32 encodes 5 bits per character, which is not convenient when computers prefer multiples of 8.
  • Base-64 encodes 6 bits per character, which is still awkward (but slightly less since at least it's an even number).
  • Base-96 is popular, but not a power of 2 so it has the same problem as [0-9a-zA-Z].
  • Base-128 and above all involve symbols you can't easily type on a typical querty keyboard.

To expand a little on why base-64 is a problem, consider what happens when you try to encode a single byte (8 bits). You can't do it with a single base-64 character since that only gets 6 of the 8 bits. But if you use two characters, you have to figure out how to pad your 8-bit byte to a 12-bit output without introducing bias or suggesting there might be an extra byte.

Hex, in contrast, is almost trivial to encode bytes in. Just look up each byte in a 256 element table to get two characters and spit them out. It gives reasonably short passwords, doesn't use weird symbols, and it's simple to implement. It's the best choice for security conscious password generators.

Perhaps best of all is that most programming languages support hex out of the box. The C library has printf(), which can format as octal, decimal or hex. Likewise, C++ has IO manipulators that use the same formats. There's no built-in support for other bases though (not even base-64), so you have to do it yourself (tricky) or find a library that does it right (violating the NIH rule (and also possibly a lot of work to verify it)). It's a lot easier to just use what's in the standard library, especially when it works.

It's true that you get less security per character with hex than with other encodings, but as Eric Lagergren helpfully pointed out in the comments, hex(random_bytes) is exactly as secure as just random_bytes. It's just longer is all. In fact, all you need is 16 bytes (encoded as 32 hex digits) to have a password so strong that any attacker will burn through all the energy in the solar system before they get a minuscule chance of guessing it. Most websites will happily accept 32 character passwords, so there's no trouble using them.

  • 4
    Because I know this would've tripped me up a couple years ago: "less security per character" doesn't mean hex(random_bytes) is less secure than simply random_bytes. It means that, for example, if you want 32 random bytes, you'll get a 64 char hex string. If you only use 32 of those characters in your password, your password will only be as strong as a password made up of 16 random bytes. – Eric Lagergren Aug 6 '17 at 6:16
  • 1
    Note that modulo has an inherant bias that makes it insufficient for some cryptographic purposes. I could be wrong but I think this is one of those cases. – Pharap Aug 6 '17 at 22:15
  • 1
    @Pharap: This is definitely one of those cases. If your password is biased, the attacker already knows some information about it, and it won't have as many bits of security as it would if it were unbiased. You'll need a longer password to counteract the bias. Figuring out how many more characters you need is hard though. It's a lot simpler if you're unbiased to begin with. – Mirinth Aug 6 '17 at 23:04
  • 1
    I would have thought that 26 letters with bias would still be more secure than 16 characters without though, right? – Shadow Aug 7 '17 at 0:32
  • 1
    @shadow: It depends on how heavy the bias is. If y is only 10^-100 times more likely than the other letters, then the bias is pretty trivial and the 26 letters are probably better. If it's 10^100 times more likely though, chances are most of your passwords are all ys and you may as well not even bother with the random number generator at all. – Mirinth Aug 7 '17 at 0:51
14

(Do you have evidence that random passwords are usually hex?)

Here are some possible reasons why you frequently find hexadecimal passwords:

  • The passwords you see might be already hashed. Since hash functions return binary strings, you will often see them stored in a hexadecimal form.

  • Hashing a random binary sequence (e.g. from /dev/urandom) is a cheap but reasonably safe way to create a random password. You'll occasionally see something like this in the wild:

    $ head -c 100 /dev/urandom | sha256sum | head -c 25
    c9ea6b67af36753a68ef42429
    
  • All characters are easy to distinguish. That is, you won't run into problems with confusing 1 and I, O and 0, etc. if you only permit hexadecimals in the first place.

  • Randomly generated sequences are usually binary, so displaying them in a hexadecimal form seems logical and efficient.

That said, choosing a random password from a larger range than [0-9a-f] is obviously safer if you don't adjust the length. I usually go with your suggestion of [0-9a-zA-Z] which gives me a broad range and avoids special characters that might cause issues with storing and transmitting.


Also, here is what I put in my shell config to quickly generate passwords on Linux:

function spw() {
    cat /dev/urandom | tr -dc a-zA-Z0-9 | head -c ${1:-20}
}

E.g., spw 25 will give me a 25-character passwords of [0-9a-zA-Z].

  • No, I don't have any actual evidence. I see it pretty often, but I couldn't tell you where. The specific one that prompted me to finally look into it was FreePBX. It fills in a random secure password as the default when you create a new user. I'm going to go with the second option being the most likely, as the first isn't the case and the third seems unlikely. Thanks for your response! – Duncan X Simpson Aug 6 '17 at 0:37
  • 3
    @DuncanXSimpson Actually, I missed an obvious one: If you have a sequence of n random bytes, it's simply most logical to display them in hex and call them a password. It would be harder to make a statement about n bytes of entropy if you convert the password to an alphanumeric sequence. – Arminius Aug 6 '17 at 0:55
  • 2
    For those who aren't familiar with Bash, cat /dev/urandom writes out the contents of a special file Linux fills with random numbers, | sends the output to the next command, tr -dc a-zA-Z0-9 throws out everything that isn't in [a-zA-Z0-9], the next | sends the output to the next command again, and head -c ${1:-20} picks out only the first n characters before quitting instead of the infinity it would normally get from /dev/urandom. It looks like it isn't biased either, so good job avoiding that pitfall. +1 – Mirinth Aug 6 '17 at 23:19
2

Overview

While there are many examples of "random" password generator programs available on the Internet, generating randomness can be tricky and many programs do not generate random characters in a way that ensures strong security.

Type and strength of password generated

Random password generators normally output a string of symbols of specified length. These can be individual characters from some character set, syllables designed to form pronounceable passwords, or words from some word list to form a passphrase. The program can be customized to ensure the resulting password complies with the local password policy, say by always producing a mix of letters, numbers and special characters. It should be noted that such policies typically reduce strength slightly below the formula that follows, because symbols are no longer independently produced.

With that in mind lets move on to WHY you only saw a-f and not a-z.

Symbol Sets

The reason you saw a-f instead of a-z, is primarily because of it's Symbol Set. For example:

Hexadecimal Numerals: (0-9, A-F) (e.g. WEP Key)  

As you can see, It can be created for a WEP Key using Hexadecimal Numerals Symbol Set. If you want something like (a-z, A-Z, 0-9) Then I would suggest the Case Sensitive Alphanumeric Symbol Set which is more complex than a Hexadecimal Symbol Set.

Resources

You can read more into generated passwords and symbols sets with the link below:

I'm the link below!

Hopefully this helps you solve your problem.

  • I understand character sets, I was simply wondering why said character set was used so often. – Duncan X Simpson Aug 6 '17 at 5:12
1

In addition to programmers being lazy (as stated in Arminuis' answer, generating a hex password is a trivial one-liner even if you have nothing but the shell and standard tools), the arguably "better" encodings that pack more bits into each character have usability issues.

The issue is not so much that converting the numbers is hard or that there are odd numbers of bits in each character which don't easily sum up to a power of two (though base-32 works very well for e.g. 80- or 160-bit numbers).

The conversion code is half a dozen lines of code, and not precisely rocket science (division, modulo, and table lookup). The fact that bits don't sum up is mildly annoying but not really an issue. It only means that the last 8 characters encode slightly fewer than 40 bits (or the last 4 encode slightly less than 24 bits for base-64). While that means there's less entropy in those trailing characters, still the password as such is perfectly random, there's no issue other than maybe being "less elegant than you wish". The fact that you have that = at the end is a non-issue as well, you can just leave it away.

The bigger problem is human interaction. Base-32 and base-64 contain enough symbols to produce (random) meaningful human-readable words or word fragments, some of them being more acceptable and some generating a less favourable reaction.
With hexadecimal, dead, f001 or affe (monkey in German) is pretty much the worst you can get, but in base-32, the substrings fuck or wanker would be entirely feasible, as well as virtually every insult that exists in any language that uses the roman alphabet. There you go, universal, internationalized insult-o-matic for your users.

Also, naive base-32 contains the symbols 0 and O as well as 1 and I which the user may confuse especially when printing or writing down passwords and re-entering them or having to communicate over the phone.
In base-64 you have not just 1 and I but also i and j as well as / to increase your joy. And, of course, telling someone a mixed-case 20-character string over the phone is sheer bliss.

There exist -- at least for base-32 -- several workarounds like the RFC4648 alphabet or Crockford's encoding which try to address the above issues (confusing symbols and insult-o-matic) to different extents.

These are however "clumsy" in a different way. To someone regularly using the system, it soon becomes noticeable that something is just not right, some symbols (notably 0 and 1) seem to never appear. Something must be broken?

Now, hexadecimal is a bit longer, a few more characters to type, but it has none of the above issues, and it is the most trivial to generate.

  • What's the likelihood of actually obtaining a valid insult in a language the specific user getting the password knows, though? It seems to me that should be very low, along the same lines as generating a result that has a hash collision with the user's previous password. – JAB Aug 6 '17 at 18:48
  • @JAB: The likelihood for a typical English 4-letter insult is 2^-20 in base-32, so about one in one million. Which sounds like not-a-lot, but is enough to have caused real incidents in real applications (I personally know one case where it played into two-factor auth). – Damon Aug 6 '17 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.