Implications of hashing to UTF-8 in a cryptographic function

Question

I know that most hash functions today use Base64 encoding for their functions, resulting in hashes that use a-Z and 0-9, and, sometimes, other special characters. This results in 62-76ish possible values for each character, so if your hash ends up being say, 70 characters long, there are 70^62 possible combinations.

But what if there was a hashing functioned designed to take advantage of UTF-8? From what I understand there are roughly 100,000 possible values for each character. This means, near as I can tell, that there would be 70^100,000 possible values, which is a lot. Seems like you couldn't make a rainbow table of that.

Also, I know that speed is very important in hashing functions. Would this be inherently slower than a Base64 hashing function?

I know that "bits of entropy" have a lot to do with the security of a password. Does this somehow improve those entropied bits? I'm pretty confused about entropy, to be honest.

Note that I'm not talking about combining a existing function with UTF8. I'm wondering if a whole new cryptographic hashing function was made to take advantage of UTF-8's larger character set, would it be better (at least in theory) than existing functions?

From my reading there seems to be some problems with bits floating around and causing confusion. Would it be possible to work around this, or is it the reason why you can't successfully hash UTF-8 for passwords, as the risk of collision increases dramatically and unpredictably?

Answer 1

First, Base64 will use, well, 64 different chars (hence the name) to encode binary data. The almost only set of symbols used is this:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

I am not aware of any implementation that uses a different one.

According to Wikipedia, "A hash function is any function that can be used to map digital data of arbitrary size to digital data of fixed size." Basically, you use a hash function to turn an undetermined sized text into a fixed size number.

The data returned from the hashing function is binary, not text. It seems a text because is more useful (and easy) to encode the result in a text representation than to handle a binary one. So they generally (but not always) encode the result before displaying. Generally the hash function will return hexadecimal encoding, not base64.

On PHP, if you execute this:

$a = sha1('some string', true);

It will return the binary data, not the hexadecimal representation.

On Python, you can use sha.digest() to achieve the same: you will get the binary 20-byte representation, not the encoded value.

When you encode your binary data in base64 (or uuencode, or yEnc), you are not changing the data, only the representation. It's just like the relation between one trillion, two hundred and seventy four million, five hundred and two thousand, nine hundred and fifty tree and 1274502953. The encoding does not change the value.

Using UTF-8, UTF-32, Base64 or ASCII will not change anything.

Answer 2

I know that most hash functions today use Base64 encoding for their functions, resulting in hashes that use a-Z and 0-9, and, sometimes, other special characters. This results in 62-76ish possible values for each character, so if your hash ends up being say, 70 characters long, there are 70^62 possible combinations.

Most hashes are expressed using hexadecimal or base64 notation. The hash itself is merely a string of bits, as such is not directly expressible as readable characters.

See also MD5 Hash and Base64 Encoding and MD5 is 128 bits but why is it 32 characters?

But what if there was a hashing functioned designed to take advantage of UTF-8? From what I understand there are roughly 100,000 possible values for each character.

Again, the hash is a string of bits - 128 of them in the case of MD5, 256 in the case of SHA-256. If you were to express those bits using UTF-8, the actual hash complexity is exactly the same, but the number of characters will actually go up because UTF-8 is not an efficient text format.

Also, I know that speed is very important in hashing functions. Would this be inherently slower than a Base64 hashing function?

The hashing function would remain the same; only the translation of binary hash to textual characters would change, which would not significantly impact speed.

I know that "bits of entropy" have a lot to do with the security of a password. Does this somehow improve those entropied bits? I'm pretty confused about entropy, to be honest.

This has nothing to do with that. Entropy has to do with encryption, not hashing, and the character set used to express a hash doesn't actually impact the hashing anyway.

Answer 3

I'm far of being an expert in crypto but I think that if you look at the result of the hash function as a stream of bits it's just the same if you "see" it as a UTF-8 string or Base64. The way you see it may be different but the actual binary value is the same. But this is just an assumption I made