Does the md5 (or any hash for that matter) of a string include the null terminator in its calculations? That is does the hash of a string equal the hash of the sequence of bytes that represents the string minus the null terminator.
2 Answers
I think this depends more on the implementation of the hashing algorithm than their ability to hash a null-terminated string including the null-terminator itself.
A few of such occurances of hash function implementations were observed in TrouSerS (version 0.2 built on Trousers Software Stack 1.1, or earlier) used in Gentoo Linux as one of the crypto libraries in its software stack and in TrustedJava (before jTSS 0.4a), and in IBM Client Security Software, both implementations included null-terminators on password popup dialog entries (null-terminated strings including null-terminators themselves).
Similarly, you can UTF-16 encode plaintext before passing it to the hashing function, causing every other character in the input to be procesed as 0x00
or null character for non-extended characters by the function that processes input as plain ASCII. Depending on the hashing algorithm itself, its intended purpose, and the input length, this could be either beneficial (say, to password hash security, e.g. hashcat has some problems with these), or cause decreased password entropy, if that causes the hashing algorithm to ignore parts of the input (e.g. bcrypt, a cryptographic key stretching hash algorithm implementation based on Blowfish symmetric block cipher, limits plaintext to 72, 55, or 51 characters - depending on who you ask - and ignores the rest).
As for a general advice whether to include null-terminators in inputs to your hashing algorithms, I have none and you should be a bit more specific about the intended use you're hashing data for (are we talking of cryptographic hashing for password storage or HMAC here?), and will vary greatly depending on your specific needs and chosen hashing algorithm.
-
1The UTF-8 encoding never uses 0x00 characters. You might be confusing it with UTF-16 and/or UCS-2. (For example, Windows NT uses MD4 hashes of the UTF-16-LE encoded password.)– user1686May 26, 2013 at 9:39
-
@grawity - You are of course correct! I blame it on the rather poor explanation of the hashcat bug report suggesting it and rather late hours (or early, depending on how you look at things LOL). I meant UTF-16 of course, that will cause every other character to be
0x00
(or null character) when parsed as plain ASCII for non-extended characters. Extended characters will of course be represented by non-null bytes as they occupy the whole address space of UTF-16. My bad, thanks for correction! May 26, 2013 at 11:28
I'll risk a wild guess : it should not. A string value, and it's null terminator, are two different things.
Null-termination could be something else on another machine / OS / compiler, but the value should stay the same.
So I'll assume that the MD5 is not taking into account the way the string is stored on that particular machine, and therefore only MD5 the string "value" (or content), and not its full machine representation.
(on a slightly related note: see the C-faq for interresting insights on why you can't assume anything about the internal representation. In C, but can be extended to almost everything. There is good examples, in the C-faq, for the null pointer, which can be treated as a '0' in many cases in C programs, but which could be anything internally (see some examples: http://c-faq.com/null/machexamp.html). And here, we talk about a md5 (or other) of a string, so it should be program/OS/whatever dependant, and as the internal representation can change, it has to be on the value of the string and not including it's storage terminator).
-
1There may not even be a null terminator - A
BSTR
string implemention in COM doesn't have a null terminator. May 30, 2013 at 11:06 -
@user93353: nice example. There are also many other ways of storing strings, which could be used by this or that program (for example: [0x000size][string], etc). It could even be stored in a Huffman tree, etc. May 31, 2013 at 9:09
'\0'
as a valid character - even appearing inside the string - and others not (i.e. it's appearance marks the end of the string).char *
orchar
array denotes the terminator. Even in those languages there are string implementations without a 0 at the end.