I wanted to know if its generally possible to inject executable code into files like PDFs or JPEGs etc., or must there be some kind of security hole in the application?
And if so, how would one do that?
I often hear that people get infected by opening PDFs that contain malicious code, that's why I ask.
There must be some security hole in the application.
Think like any very-simple-and-common .txt file: if you open it with an hex viewer, or with a well-designed textpad editor, it should only display the file content, and ok.
Then think about of processing the file, somehow, instead of just showing the contents. For example, reading the file and interpreting it's values. If it isn't done correctly, this could lead to execution of the bytes that are inside the file.
For example: if you have designed your app to load the whole file and show it, but somehow you have a variable inside your program that only holds 256 bytes. This could make you read (and write to memory) more bytes than your app expected. And, imagine, inside your app there would be any command to jump to position NNNN in memory and execute what is there, but since that memory position was written with data your program didn't expect, then you'll execute some code that shouldn't be there, and was loaded from your file...
That was a buffer overflow attack.
The same could happen with pdf, jpg, mp3, etc, if the app didn't load the data correctly.
Another possibility: for any other reason, the app (or some DLL it loads to read your data) executes some part of the data, instead of reading it. If you know what would be the command (or the data) that would trigger this behavior, you put those commands inside the data file (like the pdf file) so that the app executes it.
PDF virus: read this site: http://lwn.net/2001/0809/a/adobe-pdf-vul.php3 to know a bit about one virus that spread using PDF files.
There are two ways for a piece of code to be executed: intentionally and unintentionally.
Intentional execution is when a file is read by an application and the application does something based on whatever the file says. Reading the file is called parsing the file.
Unintentional execution is when the parser reads something it shouldn't, and instead of breaking, it continues executing. This unintentional execution is a vulnerability because if I can get the parser to execute something, I can probably get it to execute something malicious.
To answer your questions:
Is it generally possible? It all depends on the file format and the application that reads it. Some files are designed to allow executable stuff, some aren't. Some applications allow for the code to execute, others don't. If the application doesn't support it, there must be a vulnerability present to execute.
It all depends on the file format, but it's usually by finding a flaw in the file parser logic.
The key problem with pdf's, Word documents etc is that the current standards allow macros and executable code. (In my opinion this is a fatal flaw, but then I like emails to be text only...)
It is that macro execution stage that is usually the target for attack, as it provides a way to run code. The attacker just needs to figure out how to get past the controls present, which may be buffer overflow or other attack mechanism.
I disagree with the answer "There must be some security hole in the application"
It is generally incorrect. Most breaches arise from accessing files (not just providing/having them) and alluding people to believe that they access something different from what they really are, for example, a bigger image while it is executable code or a link with one (known and trusted) site description while it links to another, with malicious intents, etc.
I would recommend to read:
