Does the md5 (or any hash for that matter) of a string include the null terminator in its calculations? That is does the hash of a string equal the hash of the sequence of bytes that represents the string minus the null terminator.

  • I guess that would depend on the implementation... Besides, an "array of ASCII characters", an "array of bytes" and a "sequence of characters encoded as UTF-8" (for instance) are usually different things, some allowing '\0' as a valid character - even appearing inside the string - and others not (i.e. it's appearance marks the end of the string). – mgibsonbr May 26 '13 at 1:48
  • 1
    Strings don't have to have null terminators. There are some programming languages (C, C++) where some library functions adopt a convention that a 0 in a char * or char array denotes the terminator. Even in those languages there are string implementations without a 0 at the end. – user93353 May 30 '13 at 11:04

I think this depends more on the implementation of the hashing algorithm than their ability to hash a null-terminated string including the null-terminator itself.

A few of such occurances of hash function implementations were observed in TrouSerS (version 0.2 built on Trousers Software Stack 1.1, or earlier) used in Gentoo Linux as one of the crypto libraries in its software stack and in TrustedJava (before jTSS 0.4a), and in IBM Client Security Software, both implementations included null-terminators on password popup dialog entries (null-terminated strings including null-terminators themselves).

Similarly, you can UTF-16 encode plaintext before passing it to the hashing function, causing every other character in the input to be procesed as 0x00 or null character for non-extended characters by the function that processes input as plain ASCII. Depending on the hashing algorithm itself, its intended purpose, and the input length, this could be either beneficial (say, to password hash security, e.g. hashcat has some problems with these), or cause decreased password entropy, if that causes the hashing algorithm to ignore parts of the input (e.g. bcrypt, a cryptographic key stretching hash algorithm implementation based on Blowfish symmetric block cipher, limits plaintext to 72, 55, or 51 characters - depending on who you ask - and ignores the rest).

As for a general advice whether to include null-terminators in inputs to your hashing algorithms, I have none and you should be a bit more specific about the intended use you're hashing data for (are we talking of cryptographic hashing for password storage or HMAC here?), and will vary greatly depending on your specific needs and chosen hashing algorithm.

  • 1
    The UTF-8 encoding never uses 0x00 characters. You might be confusing it with UTF-16 and/or UCS-2. (For example, Windows NT uses MD4 hashes of the UTF-16-LE encoded password.) – grawity May 26 '13 at 9:39
  • @grawity - You are of course correct! I blame it on the rather poor explanation of the hashcat bug report suggesting it and rather late hours (or early, depending on how you look at things LOL). I meant UTF-16 of course, that will cause every other character to be 0x00 (or null character) when parsed as plain ASCII for non-extended characters. Extended characters will of course be represented by non-null bytes as they occupy the whole address space of UTF-16. My bad, thanks for correction! – TildalWave May 26 '13 at 11:28

I'll risk a wild guess : it should not. A string value, and it's null terminator, are two different things.

Null-termination could be something else on another machine / OS / compiler, but the value should stay the same.

So I'll assume that the MD5 is not taking into account the way the string is stored on that particular machine, and therefore only MD5 the string "value" (or content), and not its full machine representation.

(on a slightly related note: see the C-faq for interresting insights on why you can't assume anything about the internal representation. In C, but can be extended to almost everything. There is good examples, in the C-faq, for the null pointer, which can be treated as a '0' in many cases in C programs, but which could be anything internally (see some examples: http://c-faq.com/null/machexamp.html). And here, we talk about a md5 (or other) of a string, so it should be program/OS/whatever dependant, and as the internal representation can change, it has to be on the value of the string and not including it's storage terminator).

  • 1
    There may not even be a null terminator - A BSTR string implemention in COM doesn't have a null terminator. – user93353 May 30 '13 at 11:06
  • @user93353: nice example. There are also many other ways of storing strings, which could be used by this or that program (for example: [0x000size][string], etc). It could even be stored in a Huffman tree, etc. – Olivier Dulac May 31 '13 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.