PHP - How to do safe input filtering in the light of multibyte encoding vulnerabilities?

Question

For days now I try to get my head round how to write a secure web application in php, and it turns out to be particularly difficult. The more I read, the more I sink in deep swamps full of vulnerabilities which are not mentioned by kind people like Matt Robinson or Chris Shiflett.

For a few examples, take:

• http://sirdarckcat.blogspot.de/2009/10/couple-of-unicode-issues-on-php-and.html
• https://www.owasp.org/index.php/Double_Encoding
• Also see character consumption as in Rook's answer

In short,I see the following problems:

• When filtering input, it is not very clear how that data will be decoded later, and thus character encoding and escaping systems can bypass the input filtering. (such as double url decoding)
• When escaping output, one uses standard functions such as htmlspecialchars. It is all nice that htmlspecialchars has an encoding parameter, however that doesn't prevent you from sending UTF-16 input to it, and that could probably break the security value of the function.

It seems there is a mbstring module in php, but if it is vaguely as secure as it's documentation is comprehensible, than it 'll probably be useless even if I can figure out how to use it. Just a sample of the docs for illustration:

mbstring.strict_detection boolean

    Enables the strict encoding detection.

Great, that's helpful.

Unfortunately the functions also depend on what you set in the configuration options... There seems to be a function called mb_convert_encoding, but the docs say nothing about the security aspect and it seems you need to know the input encoding (a no go zone for security). There also is mb_check_encoding. Which seems to be meant for the purpuse, but reading the user comments on the docs doesn't exactly inspire confidence.

So the question is, in the light of all this, how do you do secure input filtering? Something like this?

1. mb_convert_encoding to utf-8
2. mb_check_encoding to reject invalid input
3. loop url_decode until the string stops changing
4. Do your normal input filtering with text comparison and regex etc...

edit: note that 3 is problematic because your normal input filtering might introduce entities again that can be url decoded

edit
I found a partial answer here, from Shiflett. It seems that for htmlspecialchars using it's encoding parameter and making sure you set your character encoding header for the browser to the same would avoid that the browser interprets characters differently than htmlspecialchars does. That is all assuming that the input of htmlspecialchars is valid for the given encoding or that for every possible invalid input htmlspecialchars interprets the string in exactly the same way than every browser does. We know that if we find no way of sanitizing our input we cannot assure that the input of htmlspecialchars is validly encoded, because an attacker might cook up a string with invalid encoding. This leads us to the second possibity, that htmlspecialchars will behave identical as the browser for all possible inputs. This is a problem, since we cannot use the same tokenizer for escaping and using, since one happens on the server and one happens in the browser.

This is all similar to what msql_real_escape does for a database, although I think you can properly solve this problem for msql by using prepared statements instead.

A third problematic output is the filesystem when using php to do file uploads or other file system manipulations. About this last one there seems to be very little information available. I don't even know of a specific escape function, let alone one that is robust when it gets crooked input.

Answer 1

In order to mount a proper defense against a threat you need to understand it. Blindly layering encoding schemes is very dangerous as this can introduce a problem when by default this usually isn't an issue. In general security problems related to encoding arise because escape functions can act upon data differently than they are interpreted. But that is not the only concern, it can also confuse the programmer in that they think a string is escaped, when its not.

The first concern is that escape functions can build dangerous strings for the attacker.

mysql_real_escape_string() is a special function that knows what encoding type the database is using, and adjusts its escaping to follow suit. There for there can never be a disconnect between the encoding method the escape function and the interpreter. Simply using addslashes() can be very dangerous for this reason.

As an example lets use 0xbf27, which is a single GBK character. addslashes() doesn't understand GBK, it only understands ASCII. If this string where interpreted as ASCII the first character would be 0xbf which is an unprintable ASCII character lets call ¿. The 2nd ASCII character is 0x27 which is is a single quote '. After addslashes() 0xbf27 becomes 0xbf5c27 or if it where printed in ASCII it would be ¿\'. The problem is that 0xbf5c is a valid GBK character, so addslashes() has turned a multi-byte character into 2 characters, one of them being a single quote. Another way of thinking of this is that the backslash is being consumed by GBK encoding.

Character consumption can be a problem without escaping functions. Lets use SHIFT-JS and HTML as an example. This example was taken from The Tangled Web:

<img src="http://fuzzybunnies.com/0xEO">
...this is still a part of the markup...
 " onerror="alret('this will execute!')"
<div>
...the page continues...

In this case the "> at the end of the img tag are very important control characters that are being consumed. The " is being consumed by the encoding scheme, and then the > and some other text appears to be apart of the HTML src attribute, its not until it hits the " onerror that the attribute is terminated.

... But that's not all. Lets switch gears a bit. What about urldecode()?

<?php
$id=mysql_real_escape_string($_GET['id']);
$id=urldecode($id);
mysql_query("select * from user where id='".$id."'");

Do you see the vulnerability? From my experience most PHP programmers don't see it. The problem is that urldecode() can be used to build a vulnerable string, lets say the attacker provided a string like http://localhost/vuln.php?id=%2527 or sleep(30)--. Well just about every web application platform will automatically run a urldecode over all HTTP inputs. So the contents of $_GET['id'] is actually %27 or sleep(30)-- That is because %25 was decoded into a % (% is hex 25). After mysql_real_escape_string() or even addslashes() the value is still just %27 or sleep(30)-- After running urldecode() again, the contents of $id is now ' or sleep(30)--, which is dangerous.

So as a hacker tip, when I am auditing a code base that uses magic_quotes_gpc or something similar, I grep though the code looking for decode functions, like urldcode(), htmlspecialchars_decode(), base64_decode() and similar functions. Even though, the application runs addslashes() blindly on all input, a decode function will allow the attacker to build his attack string.

So how do you defend your self? Well test your code. Some web application scanners will test for encoding flaws exactly like this. In fact I researched this topic while building a vulnerable scanner. Another point, always sanitize input at the time of use, this avoids the problem of decoding after escaping. You cannot know how it will be used, so encoding/decoding/escaping everything will always be flawed. Used parametrized queries for building sql statements. UTF-8 is a great default in most cases (but not always), and *most* of these problems don't arise with this muli-byte encoding scheme.

Answer 2

No, I don't recommend the approach you mentioned.

First off, let me back up. Conceptually, perhaps the safest approach is to apply a combination of input validation and output escaping. Input validation means you define what the form of expected/good inputs are, and check that the inputs have that form. Output escaping means that you escape outputs, according to the context where they will be consumed. Input validation is generally done to inputs as soon as you receive them; output escaping is generally done at the very end, just before inserting a value into a HTML document or other output.

For general background on the concepts of input validation and output escaping, as well as implementation tips, OWASP has some excellent resources. See also this question: Filter user input before the database or upon display?, for more on the concepts.

To do input validation, I think I would define a whitelist of allowed characters or a regexp that characters the expected input, and check that the input matches this regexp/whitelist. (An appropriate whitelist should deal with issues related to funny input encodings.) In some cases another way to do input validation is by casting the input to a specific type, like (int). The specific validation function will depend upon the type and format of the input; you can't use a one-size-fits-all validator.

To do output escaping, I recommend using a escaping library, such as OWASP ESAPI. You need to understand the different parse contexts where values might appear, and then use the right escaping function for that parse context. For instance, htmlspecialchars() is an appropriate escaping function for dynamic values that will be inserted into a HTML document between tags, but it is not appropriate for a URL (for URLs, you must also check that the protocol is valid). OWASP ESAPI provides a suite of escaping functions for the most common contexts where you might be inserting data into outputs.

Make sure to use prepared statements. Don't build up SQL queries through string concatenation.

That said, PHP's support for security is weak. Many other modern web programming frameworks provide stronger support for security, e.g., by providing a template system with context-sensitive auto-escaping, by providing automatic support for CSRF tokens, secure session management, user logins, a database ORM, and so forth.

See also Security attacks on PHP web applications, What security features should a PHP framework have?, Is strip_tags() horribly unsafe?, What are some good, free tools to run automated security audits for PHP code?, How to perform a security audit for a PHP application?, Why do people say that PHP is inherently insecure?.

Answer 3

I have come to the conclusion to use:

$input = mb_convert_encoding( $input, 'UTF-8' );

In order to sanitize the character encoding before doing any other operations like input validation and output escaping. I'll probably have to look at the source for mb_convert_encoding and it's unit tests to be sure it's safe, but I have the impression that it will always return a valid UTF-8 string.

The rest of my application will use utf-8 throughout which avoids anything being interpreted differently, all the way to the browser.