2

I know that most hash functions today use Base64 encoding for their functions, resulting in hashes that use a-Z and 0-9, and, sometimes, other special characters. This results in 62-76ish possible values for each character, so if your hash ends up being say, 70 characters long, there are 70^62 possible combinations.

But what if there was a hashing functioned designed to take advantage of UTF-8? From what I understand there are roughly 100,000 possible values for each character. This means, near as I can tell, that there would be 70^100,000 possible values, which is a lot. Seems like you couldn't make a rainbow table of that.

Also, I know that speed is very important in hashing functions. Would this be inherently slower than a Base64 hashing function?

I know that "bits of entropy" have a lot to do with the security of a password. Does this somehow improve those entropied bits? I'm pretty confused about entropy, to be honest.

Note that I'm not talking about combining a existing function with UTF8. I'm wondering if a whole new cryptographic hashing function was made to take advantage of UTF-8's larger character set, would it be better (at least in theory) than existing functions?

From my reading there seems to be some problems with bits floating around and causing confusion. Would it be possible to work around this, or is it the reason why you can't successfully hash UTF-8 for passwords, as the risk of collision increases dramatically and unpredictably?

Improve this question
5
  • 1
    Perhaps this title could be re-worded as something like "Implications of hashing to UTF-8 string" to make the intent more clear that the end result is a UTF-8 string, not the input. Regardless, well written question, and I am interested in the responses that I'm not qualified to make!
    – Quentin Skousen
    Commented Sep 3, 2015 at 20:36
  • 1
    Just a nitpick, but you've got your powers backwards. If you pick 70 characters from a 100,000 character alphabet, the number of combinations is 100,000^70, not 70^100,000.
    – Ilmari Karonen
    Commented Sep 4, 2015 at 1:31
  • Why not use a longer hash? Then you can have, say, 64^200 (200 Base64 symbols) instead of 64^70 (Base64 uses 64 symbols, btw). That's roughly equal to 100000^72 (72 Unicode characters if there are 100000). You don't gain any security in absolute terms, what you can do is represent the hash in less characters (not even bits).
    – Criticizing Israel not allowed
    Commented Sep 4, 2015 at 1:49
  • There's one ugly corner case here: errors in UTF-8 parsing may be much more frequent than in ASCII parsing.
    – Deer Hunter
    Commented Sep 4, 2015 at 8:21
  • This question shows a deep misunderstanding between characters and character encoding. UTF-8 is an encoding in bytes of Unicode code points, where each character takes one or more bytes. (Password) hash functions already operate on bytes, reducing the whole idea to nothing. Base64 is only used to convert the result of the (password) hash to text - if that is required at all.
    – Maarten Bodewes
    Commented Sep 5, 2015 at 14:19

3 Answers 3

Reset to default
7

First, Base64 will use, well, 64 different chars (hence the name) to encode binary data. The almost only set of symbols used is this:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

I am not aware of any implementation that uses a different one.

According to Wikipedia, "A hash function is any function that can be used to map digital data of arbitrary size to digital data of fixed size." Basically, you use a hash function to turn an undetermined sized text into a fixed size number.

The data returned from the hashing function is binary, not text. It seems a text because is more useful (and easy) to encode the result in a text representation than to handle a binary one. So they generally (but not always) encode the result before displaying. Generally the hash function will return hexadecimal encoding, not base64.

On PHP, if you execute this:

$a = sha1('some string', true);

It will return the binary data, not the hexadecimal representation.

On Python, you can use sha.digest() to achieve the same: you will get the binary 20-byte representation, not the encoded value.

When you encode your binary data in base64 (or uuencode, or yEnc), you are not changing the data, only the representation. It's just like the relation between one trillion, two hundred and seventy four million, five hundred and two thousand, nine hundred and fifty tree and 1274502953. The encoding does not change the value.

Using UTF-8, UTF-32, Base64 or ASCII will not change anything.

Improve this answer
7
  • I was pretty confused about how it all works, and you cleared up a lot of things for me. Particularly the wiki comment. And what Base64 encoding actually is, I thought it was more or less ASCII. Thanks!
    – Asa
    Commented Sep 3, 2015 at 21:24
  • 2
    FYI, as far a character variants go, bouncy castle pads with . instead of =. "URL-safe" base64 uses - and _ instead of + and /, though those are less common. And whitespace is frequently added.
    – Paul Draper
    Commented Sep 3, 2015 at 23:10
  • 1
    Re: variant Base64 alphabets, there's always base64url, which uses - and _ instead of + and /.
    – Ilmari Karonen
    Commented Sep 4, 2015 at 1:28
  • 1
    In general the hash function will not return any encoding. Some will encode as hexadecimal.
    – Criticizing Israel not allowed
    Commented Sep 4, 2015 at 1:49
  • 1
    What's "UTF-64"?
    – Maarten Bodewes
    Commented Sep 5, 2015 at 14:21
6

I know that most hash functions today use Base64 encoding for their functions, resulting in hashes that use a-Z and 0-9, and, sometimes, other special characters. This results in 62-76ish possible values for each character, so if your hash ends up being say, 70 characters long, there are 70^62 possible combinations.

Most hashes are expressed using hexadecimal or base64 notation. The hash itself is merely a string of bits, as such is not directly expressible as readable characters.

See also MD5 Hash and Base64 Encoding and MD5 is 128 bits but why is it 32 characters?

But what if there was a hashing functioned designed to take advantage of UTF-8? From what I understand there are roughly 100,000 possible values for each character.

Again, the hash is a string of bits - 128 of them in the case of MD5, 256 in the case of SHA-256. If you were to express those bits using UTF-8, the actual hash complexity is exactly the same, but the number of characters will actually go up because UTF-8 is not an efficient text format.

Also, I know that speed is very important in hashing functions. Would this be inherently slower than a Base64 hashing function?

The hashing function would remain the same; only the translation of binary hash to textual characters would change, which would not significantly impact speed.

I know that "bits of entropy" have a lot to do with the security of a password. Does this somehow improve those entropied bits? I'm pretty confused about entropy, to be honest.

This has nothing to do with that. Entropy has to do with encryption, not hashing, and the character set used to express a hash doesn't actually impact the hashing anyway.

Improve this answer
2

I'm far of being an expert in crypto but I think that if you look at the result of the hash function as a stream of bits it's just the same if you "see" it as a UTF-8 string or Base64. The way you see it may be different but the actual binary value is the same. But this is just an assumption I made

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .