For native code binaries, reverse engineering is hard (though not impossible). IDA (Interactive DisAssembler) is the most popular tool for disassembling native binaries back into something that is approximately human-comprehensible, though it's still not trivial to use correctly and may take some time to become proficient. The developer also publishes a decompiler, which tries to turn the binary code into something resembling C source code. Both of these are commercial (and expensive) programs, although you can get an evaluation version of IDA.
Note that even if you had the original source code (which no tool can give you), unless you are extremely adept at source code analysis there's no guarantee you'd be able to determine if the library is malicious or not. The International Obfuscated C Code Contest and especially the Underhanded C Contest are neat repositories of examples showing how hard it is to tell what a C program/library is doing, and how safe it is. That's not to say it's worthless to try, but it's far from guaranteed that you'd succeed (and you don't have the source to this binary anyhow).
There is no practical way to tell, for sure, that a binary is not malicious.
Modern malware is often heavily obfuscated (by hand and/or with a utility), both to deter reverse-engineering and to try to bypass antivirus checks. However, not all such obfuscated code is malicious; things like license-checking and DRM-enforcing libraries are generally very obfuscated as well, as are any binaries that contain highly-sensitive proprietary algorithms that the developer/publisher is trying to keep secret.
Running the binary through VirusTotal is a good idea, but all it can tell you is "would any antivirus software detect this as malicious?" and that's not enough info. Virus scanners can produce both false positives (flagging safe software as malicious) and false negatives (failing to recognize malicious software), and malware developers are well aware of the need to avoid detection by antivirus software.
Another thing to consider is how trustworthy the file's source is. Adobe is notorious for buggy (e.g. Photoshop) and/or insecure (e.g. Flashplayer) software, but as far as I know hasn't ever released anything meeting the conventional definition of "malicious" (some people count all DRM as malicious, but while I see their point I think they are diluting the term and don't personally count DRM as malicious unless it impacts anything other than the software itself, e.g. the Sony BMG rootkits). If the file has a valid digital signature from Adobe (or any other reasonably-trustworthy developer), the odds that it is malicious are much lower. If the signature is missing, invalid, or created by a third party that you don't recognize (check very carefully that the signer is who you think it is, not somebody else with a similar-looking name), then that's reason to be suspicious, especially in the last case (look-alike name trying to pass itself off as somebody else).