if I have a developer who wants to install a new software and he ask me to check this software security before installing it what shall I do
You could be asking one of many questions here.
If you are asking "How do I know that the software I have here is what the vendor made - could it have been altered before I got it?" then you will want to compare the hash of what you have to what the vendor has published, assuming they have. (Almost everyone does nowadays.)
If you are asking "How can I tell if the vendor has put malware or back doors into this software?" then the key thing to consider is how much you trust the vendor. (Although that's not a guarantee. Big companies have accidentally gotten a virus on their gold masters before now, and the bigger you are, the more likely a government is to lean on you to put a back door in.) Of course you've scanned it with a couple of AV scanners, like you do with anything you put on your network. Talk to other users and see if they have any issues. If it's open source, you can read the source before compiling, although that is a huge amount of work. Lastly, you can run it in a sandbox for a while, monitoring what it does, especially it's network activity to see if it phones home.
If you are asking "How can I tell if it has programming mistakes that make it vulnerable to attack?" then again there are various approaches. As before, consider the reputation of the vendor; talk to other users; and if it's open source and you have the resources, review the source. Check it's bug tracking system if this is public, and it's historical release notes, to see how well security issues are addressed. Think about what it does and what threats you face, and identify other security controls you can put around it.
(Liked the Graham Hill approach to answer):
One more question/answer you could think about:
- If you are asking "What I want to protect if the software isn't trustworthy?" then you might consider that you could install the software in a virtual machine that doesn't have connection with your network, so that a vulnerability wouldn't be able to get data from other computers, for example. And even if the user puts some "secret" data in the program, it won't be able to send it out.