How would you analyze an image to find out if there are scripts or possible malware embedded in it? What tools / methods to use?
Most virus scanners use a signature based detection can find out embedded malware in any files.
However, those malwares are rarely executed if it is a real image (file extension) the image is displayed using a correct program, who does not try to "execute" an image.
I am assuming that you're going to be accepting user-uploaded image files and serving them to other users through a web application. In that case there are two different attack vectors here.
(1) files that can be interpreted as both image and another type.
(1b) files that can be interpreted as plug-in data, most notably Java (GIFAR attack). Just hosting these can get you compromised for XSS because the Same Origin Policy in Java and Flash is based on the location hosting the file, not the including page.
Such files can commonly be valid images of the expected type as well as working as the vulnerable target type.
You can make these attacks difficult by processing submitted images, loading them in and re-saving them. You may be doing this anyway in order to do cropping, resizing or quality-level changes.
It is in principle possible for someone to submit an image that, when compressed by whatever algorithm you're using, pops out an image with
Either way, you should mitigate this attack by serving any user-submitted files from a separate domain that does not have the ability to script or share cookies with the main website domain. (It can be a subdomain as long as there is nothing served on any shared ancestor domain.)
(2) malformed images that exploit bugs in browser image decoders to cause arbitrary code execution. These aren't common today but there have been image decoder bugs in the past and it is possible some more may arise.
This can also be defeated by loading and re-saving the image; even just not getting an error on load is generally a good sign. However, it is conceivable that there could be a bug in your own image decoder, which would then make your server-side vulnerable to exactly the same problems.
So make sure your server-side image processing components up-to-date, and, where you can, limit the number of accepted input formats in order to reduce the attack surface. You may also wish to consider running the image handling in a lower-privilege process (privsep) if this is a likely threat for your app.
Loading an image for processing also carries the risk of server-side denial of service attack through very large images (in size and/or compression bombs). Sanity-check the file size and resolution from image file header before attempting to load.
Depending on what your app is, there may be limited benefit from detecting malformed-image exploits because (apart from their rarity) it's usually easy for an attacker to host them elsewhere and post the image or a link to it in other user-submitted content.
In addition to @bobince's answer, you should also watch out for scripts embedded in image metadata. This is a risk if you are displaying, for example, EXIF data from user-submitted images. (I found a couple of XSS vulnerabilities in a popular image gallery based on this method of injection.)
Many tools/libraries will show you image metadata. Exiv2 is a handy tool for viewing and manipulating EXIF and IPTC image metadata.
In order to prevent Internet Explorer >=8 from MIME-sniffing a response away from the declared content-type, you could add the X-Content-Type-Options header with the value nosniff. See http://blogs.msdn.com/b/ie/archive/2008/09/02/ie8-security-part-vi-beta-2-update.aspx