1

I've got two byte sequences, and I decode them as UTF-8 (in strict mode to be sure they're valid UTF-8).

Is it correct to assume that, if the bytes are different, the decoded codepoints will also be different?

In other words, is it possible to find 2 valid UTF-8 byte-sequences that decode to the same string?

  • i.e. I checked with the bytes ed a0 bd ed b8 8a (a surrogate pair encoded as UTF-8) and they're correctly identified as invalid... I also need to check how BOM is handled, and longer encodings of a codepoint – Alba Mendez Sep 16 '19 at 21:03
3

In other words, is it possible to find 2 valid UTF-8 sequences that decode to the same text?

Since a valid UTF-8 encoding has to be minimal by definition it is impossible to have two valid but different UTF-8 sequences which result in the same UTF-32 sequence.

But, this does not mean that different UTF-32 sequences are always a different "text" for some definition of text. Unicode has sometimes multiple ways to present the same character - see Wikipedia: Unicode equivalence and Wikipedia: Duplicate characters in Unicode. And even if it is not the same character it might look the same - see Wikipedia: Homoglyph.

3

Combining characters

You could write the character é as both

  • U+00E9 LATIN SMALL LETTER E WITH ACUTE
  • U+0301 COMBINING ACUTE ACCENT + U+0065 LATIN SMALL LETTER E

You could also have several combining characters provided in different order without that affecting the resulting character.

That's why you should do Unicode normalization before attempting to compare two utf-8 byte sequences.

  • +1 Good catch, but those are still different strings (different codepoints), so they are not a problem. I am looking for distinct bytes that represent the same codepoints. – Alba Mendez Sep 17 '19 at 18:38
  • Also, I think you meant U+0301 instead of U+00B4. – Alba Mendez Sep 17 '19 at 18:38
  • Good point. U+00B4 is the standalone acute, in this context it would be U+0301. Edited. – Ángel Sep 17 '19 at 20:37
  • @AlbaMendez: I'm nor sure what should be considered a problem. While this is an interesting question by itself, I think you have an ulterior reason (XY problem) which has not been shared. For instance, having two users, one called AlbaMéndez and the other AlbaMéndez that are actually different would usually be problem. – Ángel Sep 17 '19 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.