I'm reading this report on unicode security and found the following paragraphs confusing:

When converting from a multi-byte encoding, a byte value may not be a valid trailing byte, in a context where it follows a particular leading byte. For example, when converting UTF-8 input, the byte sequence E3 80 22 is malformed because 0x22 is not a valid second trailing byte following the leading byte 0xE3. Some conversion code may report the three-byte sequence E3 80 22 as one illegal sequence and continue converting the rest, while other conversion code may report only the two-byte sequence E3 80 as an illegal sequence and continue converting with the 0x22 byte which is a syntax character in HTML and XML (U+0022 double quote). Implementations that report the 0x22 byte as part of the illegal sequence can be exploited for cross-site-scripting (XSS) attacks.

Therefore, an illegal byte sequence must not include bytes that encode valid characters or are leading bytes for valid characters.

Based on the example described (E3 80 22) as a byte sequence, it is clear that it not valid:

>>> b'\xe3\x80\x22'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: invalid continuation byte

and the question is how a good parser/converter is supposed to manage this type of error.

Probably I'm misunderstanding something, but it says that some may report an error with the whole sequence (E3 80 22), but others may report an error with E3 80 and continue converting the 22 byte as a double quote. However, it says that when the report includes the 22 byte, then this can be exploited in a XSS attack. That's the part that is confusing; I would have thought that it was the second instance the one leading to XSS vulnerabilities. What is the rationale to think it should be the first instance vulnerable to XSS?

An additional question: How is this type of issue exploitable in practice (assuming we are interested in web applications)? Am I supposed to simply use URL encoding or HTML encoding (%E3%80%22 and &#xE3&#x80&#x22, respectively) and hope for the best?


1 Answer 1


If your webpage considers 'E3 80 22' as one sequence then the '22' won't be escaped... if you deliver this page to a browser who considers 'E3 80 22' as 'E3 80' + '22', stripping the illegal sequence results in '22' then you have an '22' in there you don't want and this allows XSS attacks.

  • Thanks, but aren't you saying that detecting E3 80 as illegal and continue parsing 22 is the one case that facilities XSS? The document says that when E3 80 22 is reported as an illegal sequence, then this can be exploited in a XSS.
    – r_31415
    Commented Nov 10, 2017 at 20:56
  • No. Let's say you loop through every character in a string... e3 80 22 counts as one 'illegal' entity so you skip it. If you replace the character 22 with 32 in '22 e3 80 22' you get '32 e3 80 22' which is '(32) (e3 80 22)' but now a DIFFERENT implementation treats this as '(32) (e3 80) (22)'. If the first implementation treated it as '(32) (e3 80) (22)' as well we'd get '(32) (e3 80) (32)' which would be safer.. worst case this gets treated as a space plus illegal (so skipped) which at least isn't an XSS but you loose the last space. I might be wrong but that's how I understand it.
    – mroman
    Commented Nov 10, 2017 at 21:26
  • Of course... browsers should just reject illegal uft8 completely... any program should instead of trying to fix things which often just introduces other problems.
    – mroman
    Commented Nov 10, 2017 at 21:28
  • I understand what you're saying, but you're adding an additional byte which is not considered in the initial example. I think you're using the same logic explained in the question; basically, a parser that outputs (e3 80)(22) is unsafe, but according to the document, the other implementation is considered unsafe (e3 80 22).
    – r_31415
    Commented Nov 11, 2017 at 0:23
  • Maybe I'm misunderstanding your example, but it would be helpful if you could explain it only with three bytes.
    – r_31415
    Commented Nov 11, 2017 at 0:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .