I agree it is tricky to secure yourself against an attack when your are inviting potential attackers with a way to upload files to your server. The best advice I can give is to use multiple approaches when securing uploaded content.
Ensure that the filename that is submitted is sanitised. For example, you don't want someone submitting '/../../index.html' as a filename, and overwriting your index page.
Filenames should also be checked for a valid mime type and extension (.jpg, .gif), however it is also worth investigating if your language of choice has any potential vulnerabilities when dealing with filenames, for example, when dealing with PHP it is sometimes possible to pass a null byte which may bypass extension restrictions.
By choosing a custom filename yourself for uploaded content, you can mitigate a lot of the risk of malicious filenames. So if you accept a mime type of 'image/jpeg', write this to a file with a filename of '[RANDOMSTRING].jpg'.
As woliveirajr commented, checking magic bytes can also be a good idea, however there would be nothing to stop someone from uploading a php backdoor script containing a few magic bytes, and then proceeding with <%php code.
Once you are happy that you have secured your upload script, check if your web server has a configuration option to disable execution of scripts in your destination directory (such as IIS execute permissions). This would mean that even if someone is able to bypass your upload whitelisting, they would find it difficult to have a script to execute on your server.
Also be aware of attacks not directly related to execution of code on your server. I have seen countless times that file uploads are protected against server side execution, however uploading a filename of '.....' results in a XSS attack.
This is by no means an extensive list, but I hope it gives you an few ideas of the different areas you should be protecting against.