Is it okay to normalize unicode passwords with NFC/NFD?

Question

I am currently designing a login for a web service. I will use a PBKDF2 implementation for hashing the passwords.

However, I intend to allow unicode for passwords, as I will have international users, which might want to use, for example, cyrillic characters. To avoid any issues with unicode ambiguity, I thought of using the NFC unicode normalization before encoding the password as utf8 and passing it on to the hash.

The question now is: Is that safe or does it introduce any unwanted ambiguity into the password validation? It is clear that "a\u0308" (a + combining diaresis) and "ä" should be the same, but does NFC fold any more differences which users could be relying on?

Edit:

I found that there is a stringprep (RFC 3454) profile called SASLprep (RFC 4013) which is appearantly used for passwords and usernames in some protocols. It specifies to use a KD normalization, which I consider a bad idea. It will fold differences like ² and 2, which are two characters commonly on keybords in the western world at least, which could be used to enrich the password entropy. Unfortunately, no rationale is given for that.

Answer 1

If you treat 2 and ² as the same character, you're essentially removing one character from the character set. That isn't really so bad if it increases usability, especially if that encourages longer passwords.

Say you take a 8-character password, with a drawn randomly from a set of 2000 characters. That gives log₂(2000⁸) ≈ 88 bits of entropy. If you had a 9-character password, drawn from 1000 characters (half as many!), that's log₂(1000⁹) ≈ 90 bits of entropy. In fact:

+-----+---------------------------+
|     |    character set size     |
| len |  500 | 1000 | 2000 | 4000 |
|-----+------+------+------+------|
| 6   |  54  |  60  |  66  |  72  |
| 7   |  63  |  70  |  77  |  84  |
| 8   |  72  |  80  |  88  |  96  |
| 9   |  81  |  90  |  99  |  108 |
| 10  |  90  |  100 |  110 |  120 |
| 11  |  99  |  110 |  121 |  132 |
+---------------------------------+

As you can see, in the normal range of password lengths and Unicode character set sizes, the exact size of the character set isn't that important.

Answer 2

Current guidance is to use NFC normalization but that is only the final step of a multistep process:

As of November 2022, the currently relevant authority from IETF is RFC 8265, “Preparation, Enforcement, and Comparison of Internationalized Strings Representing Usernames and Passwords,” October 2017. This document about usernames and passwords is a special case of the more-general PRECIS specification in the still-authoritative RFC 8264, “PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols,” October 2017.

RFC 8265, § 4.1:

This document specifies that a password is a string of Unicode code points [Unicode] that is conformant to the OpaqueString profile (specified below) of the PRECIS FreeformClass defined in Section 4.3 of [RFC8264] and expressed in a standard Unicode Encoding Form (such as UTF-8 [RFC3629]).

RFC 8265, § 4.2 defines the OpaqueString profile, the enforcement of which requires that the following rules be applied in the following order:

• the string must be prepared to ensure that it consists only of Unicode code point explicitly allowed by the FreeformClass string class defined in RFC 8264, § 4.3. Certain characters are specified as:
  • Valid: traditional letters and number, all printable, non-space code points from the 7-bit ASCII range, space code points, symbol code points, punctuation code points, “[a]ny code point that is decomposed and recomposed into something other than itself under Unicode Normalization Form KC, i.e., the HasCompat (‘Q’) category defined under Section 9.17,” and “[l]etters and digits other than the ‘traditional’ letters and digits allowed in IDNs, i.e., the OtherLetterDigits (‘R’) category defined under Section 9.18.”
  • Invalid: Old Hangul Jamo code points, control code points, and ignorable code points. Further, any currently unassigned code points are considered invalid.
  • “Contextual Rule Required”: a number of code points from an “ Exceptions” category and “joining code points.” (“Contextual Rule Required” means: “Some characteristics of the code point, such as its being invisible in certain contexts or problematic in others, require that it not be used in a string unless specific other code points or properties are present in the string.”)
• Width Mapping Rule: Fullwidth and halfwidth code points MUST NOT be mapped to their decomposition mappings.
• Additional Mapping Rule: Any instances of non-ASCII space MUST be mapped to SPACE (U+0020).
• Unicode Normalization Form C (NFC) MUST be applied to all strings.

I can’t speak for any other programming language, but the Python package precis-i18n implements the PRECIS framework described in RFCs 8264, 8265, 8266.

Here’s an example of how simple it is to enforce the OpaqueString profile on a password string:

# pip install precis-i18n
>>> import precis_i18n
>>> precis_i18n.get_profile('OpaqueString').enforce('😳å∆3⨁ucei=The4e-iy5am=3iemoo')
'😳å∆3⨁ucei=The4e-iy5am=3iemoo'
>>> 

I found Paweł Krawczyk’s “PRECIS, the next step in Unicode validation” a very helpful introduction and source of Python examples.

Answer 3

The question is whether there is an appropriate trade-off between entropy reduction and user experience. I wanted to look at the possible UX benefits of canonical and compatibility normalizations rather than at the exact entropy in my answer:

Evading problems where two possible representations of the same grapheme cluster exist makes a lot of sense to me, since these representations mostly exist for legacy reasons (Unicode’s goal was to have code points for all possible characters from legacy encodings) and look the same (unless there are font rendering issues). Also, as far as I know there is no standard way of choosing which form (single code point or modifiers) of, say, ä I want to input (speaking from my experience with most desktop and mobile operating systems).

Compatibility normalization on the other hand will “merge” characters which are distinctly different, e.g. ſ and s or ² and 2. Choosing between these is often possible, although some symbols might require IME, emoji pickers or symbol pickers. In my opinion there is no usability advantage here, since the user decides which version they want in their password. It can make sense for user names or other identifiers that might be vulnerable to homograph attacks though (to some extent—mixing similar looking alphabets is still possible).

As to why the RFC you mentioned prefers NFKD for passwords (in addition to user names where it might fend off some homograph attacks), I don’t know. But personally I find NFD (or NFC) more reasonable.