I have a program to check if incoming files are in an expected filetype, e.g. PDF. The program first takes a look at the magic number, and determines the file type out of that. The problem is that I still can inject malicious script-files (for example batch-files on windows) by simply putting the magic number in the beginning, which is later ignored in the command line when executing the file. Is there another way to make sure that the file is (in this example) a true pdf-file?
Ensuring a file is the right file type is challenging, and sometimes impossible.
When possible, use a strict file parser.
For a PDF file, use a pdf library or pdf parser to parse the file and reject the file if any parsing issues arise. This can often have real compatibility consequences, since many PDF clients allow malformed PDF documents.
Save the file in a restricted location with restricted permissions
Be sure to save the file in a location that isn't processed by your web server. Ensure the extension is non executable by your webserver. Allow only read access to the file (explicitly deny execute privileges).
Understand what kinds of attacks are made possible by user uploaded files
In the case of a file being uploaded to a server, if non execute precautions are taken, then a malicious file getting past your file parser is mitigated. However, if the file is made available to other users, you may want to use antivirus or scan for known malicious signatures. This blacklist approach is a last resort, but it is a reasonable measure to take.