Does the md5 (or any hash for that matter) of a string include the null terminator?

Question

Does the md5 (or any hash for that matter) of a string include the null terminator in its calculations? That is does the hash of a string equal the hash of the sequence of bytes that represents the string minus the null terminator.

Answer 1

I think this depends more on the implementation of the hashing algorithm than their ability to hash a null-terminated string including the null-terminator itself.

A few of such occurances of hash function implementations were observed in TrouSerS (version 0.2 built on Trousers Software Stack 1.1, or earlier) used in Gentoo Linux as one of the crypto libraries in its software stack and in TrustedJava (before jTSS 0.4a), and in IBM Client Security Software, both implementations included null-terminators on password popup dialog entries (null-terminated strings including null-terminators themselves).

Similarly, you can UTF-16 encode plaintext before passing it to the hashing function, causing every other character in the input to be procesed as 0x00 or null character for non-extended characters by the function that processes input as plain ASCII. Depending on the hashing algorithm itself, its intended purpose, and the input length, this could be either beneficial (say, to password hash security, e.g. hashcat has some problems with these), or cause decreased password entropy, if that causes the hashing algorithm to ignore parts of the input (e.g. bcrypt, a cryptographic key stretching hash algorithm implementation based on Blowfish symmetric block cipher, limits plaintext to 72, 55, or 51 characters - depending on who you ask - and ignores the rest).

As for a general advice whether to include null-terminators in inputs to your hashing algorithms, I have none and you should be a bit more specific about the intended use you're hashing data for (are we talking of cryptographic hashing for password storage or HMAC here?), and will vary greatly depending on your specific needs and chosen hashing algorithm.

Answer 2

I'll risk a wild guess : it should not. A string value, and it's null terminator, are two different things.

Null-termination could be something else on another machine / OS / compiler, but the value should stay the same.

So I'll assume that the MD5 is not taking into account the way the string is stored on that particular machine, and therefore only MD5 the string "value" (or content), and not its full machine representation.

(on a slightly related note: see the C-faq for interresting insights on why you can't assume anything about the internal representation. In C, but can be extended to almost everything. There is good examples, in the C-faq, for the null pointer, which can be treated as a '0' in many cases in C programs, but which could be anything internally (see some examples: http://c-faq.com/null/machexamp.html). And here, we talk about a md5 (or other) of a string, so it should be program/OS/whatever dependant, and as the internal representation can change, it has to be on the value of the string and not including it's storage terminator).