You either do it yourself or trust someone else
As with most things in life, you must either do it yourself or trust someone else with it. Here trusting covers both having no malicious intent and being competent enough to properly perform the task.
For example, you could file your taxes yourself or trust a tax adviser to do so (who not only should not attempt to defraud you, but also know how to file the taxes!).
If you are a company, doing it by itself will actually be performed by one or several of your employees, which in turn need to be trusted.
The third party you trust, doesn't need to be a single person, either. It could be the Microsoft Windows Development Team, or the Wordpress core developers.
On source code security, you want the expert not only to be well-meaning, but also knowledgeable to code the program in a secure way / find any potential security issues that may be there.
(plus a few additional border systems when treated as a whole, eg. you want that their code did not get compromised while they uploaded it to the repository, or the email from your employee stating the results getting replaced by a malicious hacker inside your network to say that the application was fine)
You will need to evaluate your options, assess the risk associated with each one, and choose the path that best suits your interests (and budget!).
If I were to check the security of the source code of a blog that used Wordpress, I would generally trust that the original code was fine1¹ and check the differences between the official version and the used one. If the website was compromised, that would make it much easier to find out.
¹ Obviously checking the changelog of later versions if it used an outdated one.
However, if it was developed by the nephew of the owner, I would expect to find lots of vulnerabilities there, and would recommend a thorough checking of everything.
In your case, you should evaluate the risk and cost of developing the equivalent of that library (take into account that the chance of issues in your inhouse product is not zero, either, and it will depend -amongst other things- on the quality of the people involved) versus the risk and cost of auditing and using that library.
Now, there may be attenuating factors that simplify the auditing. For example, if the untrusted code can run in an isolated Virtual Machine, that may be enough to not need further auditing (even here, you are trusting the VM implementation). Or it may be considered sufficient to audit the parts of that program that run as root.
For auditing a library, code analysers can help point out problematic parts (as pointed out), but in order to consider it to be clean I would actually have someone read and understand the code, even if superficially.
For instance, the ability to remove arbitrary files is not malicious per se. You need to understand the program in order to know if it makes sense.
Again, it is a matter of the threats and risks for what you are doing. If you are only concerned with the library exfiltrating data, filtering connections at the firewall could be enough. If you are concerned with the library deleting important files (and for some odd reason you can't deny such permission), you could simply scroll by a bunch of code that only did mathematical computations. If that library computes the parameters for launching a rocket... well, you better make sure those computations are correct, too!
straceall system calls, and see if something fishy is going on, for instance, the application attempting to stat files it doesn't need to care about.