13

Is it possible to ensure that a string is hashed always the same way no matter which language you use to do it (Java, vb.net... ) and no matter what operative system you are?

1
  • 1
    Short answer: Yes, except the lower/uppercase may vary. Hashes are (usually) hexadecimal anyway, so they can be treated as case-insensitive. Of course when outputted in another format (like the raw binary data, e.g. 128 'random' bits for MD5), it may be case sensitive. The output will always be the same though.
    – Luc
    Aug 8, 2012 at 21:45

5 Answers 5

31

Hash functions are deterministic: same input yields the same output. Any implementation of a given hash function, regardless of the language it is implemented in, must act the same.

However, note that hash functions take sequences of bits as input. When we "hash a string", we actually convert a sequence of characters into a sequence of bits, and then hash it. There begins the trouble. Consider the string "café": among all the possible conversions to bits, all of the following are common:

63 61 66 e9                             ISO-8859-1 ("latin-1")
63 61 66 ca a9                          UTF-8
63 61 66 65 cc 81                       UTF-8 (NFD)
ef bb bf 63 61 66 ca a9                 UTF-8 (with BOM)
ef bb bf 63 61 66 65 cc 81              UTF-8 (NFD with BOM)
63 00 61 00 66 00 e9 00                 UTF-16 little-endian
00 63 00 61 00 66 00 e9                 UTF-16 big-endian
ff fe 63 00 61 00 66 00 e9 00           UTF-16 little-endian (with BOM)
fe ff 00 63 00 61 00 66 00 e9           UTF-16 big-endian (with BOM)
63 00 61 00 66 00 65 00 01 03           UTF-16 little-endian (NFD)
00 63 00 61 00 66 00 65 03 01           UTF-16 big-endian (NFD)
ff fe 63 00 61 00 66 00 65 00 01 03     UTF-16 little-endian (NFD with BOM)
fe ff 00 63 00 61 00 66 00 65 03 01     UTF-16 big-endian (NFD with BOM)

and all will yield very different hash values when processed with a given hash function. You have to be very precise about what you do when dealing with cryptographic functions; every bit counts.

3
  • 6
    Or even nicer, using .net's ASCII encoding: 63 61 66 3F (no idea how they came up with the ingenious idea to silently replace characters the encoding doesn't support with ? by default) Mar 3, 2013 at 21:11
  • Python (among other high-level languages) handles problem you described very well: there is a strict distinguition between byte and a character. Jan 2, 2015 at 17:49
  • @OskarSkog you are wrong. Python 2 just have implicit conversion from unicode to bytes for ascii characters, which will throw an exception as far as your unicode string contains non-ascii characters. But it's actually a problem of the programmer who passed unicode string to a function that await bytes. Nov 11, 2016 at 3:56
14

I'm not exactly sure what you mean, but yes. The output of a properly written hash function should be the same regardless of language.

The only difference between the hashes of different programming languages libraries and on different platforms will be speed. Although in properly written libraries - the difference will be trivial.

6
  • I was not sure if the hash is different for the same string if you hash it using linux and then with windows
    – eversor
    Aug 8, 2012 at 9:56
  • Yes it will be. The output of SHA1(x) will always be the same, no matter what OS or what library you use.
    – user10211
    Aug 8, 2012 at 9:59
  • 5
    The only possible difference between OSes will come if you're hashing something with new lines in it where it looks the same but isn't (i.e. you're hashing something that the user typed and hence they think it is the same because they pressed the same keys, but it isn't quite the same because Windows uses \r\n and Linux uses \n). Or badly written libraries, of course - I've seen some incorrect MD5s because they didn't zero-pad numbers and ended up with hashes that weren't 32 characters.
    – IBBoard
    Aug 8, 2012 at 10:05
  • 2
    @IBBoard That isn't really a hashing issue. Given x, SHA1(x) will always output the same thing no matter what. Most of the current libraries work fine, i have not come across any that outputs the wrong hash results so far.
    – user10211
    Aug 8, 2012 at 10:30
  • 3
    Agreed on "Given X, SHA1(X) is the same", but I was trying to warn that although "X1" may look like "X2" to make you think that X1==X2, whitespace (especially line endings) may differ. Wrong hashes that I've seen have normally come from people trying to wrap the Java MD5 code, which returns a byte array (IIRC) when they want a text string.
    – IBBoard
    Aug 8, 2012 at 12:31
9

Yes, the exact same "byte sequence" will always yield the exact same digest value regardless of implementation (assuming it's a correct implementation!)

The key word is this is always true for "byte sequence", but not always for "string" as you wrote. Depending on a lot of things, strings can be generated differently on different systems. There is the potential for a lot of white space or line ending differences, or ASCII vs Unicode UTF-16 encoding issues.

Also, be aware that when you display the digest value, you run into similar issues. Different implementations might represent hexadecimal digits with either upper case or lower case values, so a string equality test might fail.

3

It will always be the same if you're not using a salt. If you're using a salt then it will be different if you change the salt.

0

The behaviour that every string in a sha1 hash is the same lies in a common convert format called base64 as a convention (and byte sequence). This is due to encoding a 20-byte hash value into 40 bytes of hex, and then encoding those 40 bytes of hex into 56 bytes (40 / 3 * 4, then rounded up to the nearest 4 bytes) of base64 data.

So the proof is that 2^160 = 16^40 in hexadecimal representation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.