There must be some security hole in the application.
Think like any very-simple-and-common .txt file: if you open it with an hex viewer, or with a well-designed textpad editor, it should only display the file content, and ok.
Then think about of processing the file, somehow, instead of just showing the contents. For example, reading the file and interpreting it's values. If it isn't done correctly, this could lead to execution of the bytes that are inside the file.
For example: if you have designed your app to load the whole file and show it, but somehow you have a variable inside your program that only holds 256 bytes. This could make you read (and write to memory) more bytes than your app expected. And, imagine, inside your app there would be any command to jump to position NNNN in memory and execute what is there, but since that memory position was written with data your program didn't expect, then you'll execute some code that shouldn't be there, and was loaded from your file...
That was a buffer overflow attack.
The same could happen with pdf, jpg, mp3, etc, if the app didn't load the data correctly.
Another possibility: for any other reason, the app (or some DLL it loads to read your data) executes some part of the data, instead of executing it. If you know what would be the command (or the data) that would trigger this behavior, you put those commands inside the data file (like the pdf file) so that the app executes it.
PDF virus: read this site: http://lwn.net/2001/0809/a/adobe-pdf-vul.php3 to know a bit about one virus that spread using PDF files.