I am new to computer security. I was wondering what an experienced information security user would do when inspecting a file for potential download. Obviously, it would be very difficult to look through all of the files that download and all of the source code (the package is open-source). How does one know what to download and what not to download?
Deep down you don't , from a security perspective i have more software running on my phone than I could effectively ever review in my lifetime. My desktop machine would be 100,000 worse than that.
so you have to trust where you get the software from, you can check to see if it has been tampered with. But that's still no guarantee ( for example a few years ago a tiny tiny change to debian caused a massive reduction in entropy of encrypted communications, to the point where data could be guessed , or proftpd ( maybe vsftpd I don't remember which ) had a malicious commit from a contributor that got missed allowing root level access when a user logged in with a hard coded username and password )
When you are downloading software from some site on the Internet, you have to see on what criteria, if any, you can establish trust and then determine if the basis for establishing trust is valid.
First, consider if you trust the developer. Is the developer some anonymous person, or is it a company or person with a real name, real contact information, etc. It's less likely that someone is going to purposely create malicious software if they are leaving a trail back to him/herself; of course the website could be stealing someone's identity. Does the author have a reputation, provide sources, publish release notes, ever provide updates, admit when security issues were found in the past?
Next, you can look at the source of the download. Is it from the author/publisher's website, or is it a third party website (like download.com) a torrent, etc. A third party may modify the program or package it with spyware or other junk. Is the website secure, are there reasonable assurances of that the file has integrity relative to what the author compiled and/or uploaded?
On the individual file itself, author's often publish a md5/sha1 hash, which you can use to compare against the file you have downloaded to ensure its valid. This is less robust then usually advertised, because a site could be hacked. More reliable would be if the author cryptographically signed the executable or zip file, this provides reasonable assurance of integrity and authenticity. Unlike publishing the hash on some website, the data for authenticity is part of the file you are downloading.
If you want to be careful, you can always run the file in a sanbox or a virtual machine to isolate it from the rest of your system and data. This gives you the opportunity to use it for a while, see what types of network requests are being made, run anti-virus on it. After evaluating in isolation, you can load it into your main system if this provides some tangible advantage.
You can solicit reviews or advice from other uses or people on the Internet, but this often provides false assurance: unless they are security and software experts, they can only provide anecdotal feedback from their experience, limited by their abilities to perform such an assessment, I'm sure you know someone who operates a computer loaded with malware, but doesn't seem to get there is a problem or anything wrong.
Ultimately, you can never be sure, and new vulnerabilities may be found in the future. "Security" can often better be though of as a process of risk management. In general, you have to establish some level or trust and then determine what additional steps you can do to reduce the likelihood of attack/compromise in case your trust was improperly established or there were criteria you did not or were not technically skilled enough to consider.
Assume that the download is malicious every time. Make sure you have antivirus software to scan it for threats every time, and realize that there may be nothing you can do to stop a threat from compromising your browser. Understand that even reputable sources of software can be compromised, and that you do not necessarily have to click "download" to download. Be very careful what you download, and do try to use trusted suppliers of software, and media, but watch out for problems with them too. Keep your computer secure, so that even if it is compromised, you can recover quickly. Never put yourself in place where you might hesitate to wipe your machine and start over.
What's worse than getting a virus, is to know you have a virus and not be able to anything about it.