In this question I will use the example of Linux OS, but I am also interested in general situation.
Linux OS distributions are famous for being open source and thus rather secure. But how can you be sure that the Linux that you set on your PC, with the installer you have downloaded, is coded with exactly the same code as is available in the source for everyone to read ? And same for other OS projects ?
When you download just the binary, then you can't. You have to trust the distributor of the binary that it is indeed compiled from the published sourcecode.
When you consider the developer trustworthy but you aren't sure you can trust a download mirror, then note that many developers publish hashsums of their binaries, so when you download the binary from an untrusted mirror, you can validate it by checking the hash. This, however, assumes 1. that the binary is the one compiled by the developer and not an independent build by the distributor, and 2. that the developer isn't the one who wants to screw people by releasing a binary compiled from a different sourcecode than what he is publishing.
When you want to make sure that the binary you get is really compiled from the source, you have to get the sourcecode and compile it yourself. This might take quite some time with more complex projects.
But it's questionable if it is really worth it from a security point of view, because just because the sourcecode is public doesn't mean that it is free of backdoors. There is the underhanded C code contest with many creative examples of how to hide malicious sourcecode in plain view. When you want a more real-world example: The heartbleed bug was a vulnerability which existed for years in a very widely used piece of open source software and nobody noticed. There is no reason to believe that it was an intentional backdoor, but when it would have been intentional, the fact that it is in public soucecode and not smuggled into a binary would give the one who planted it plausible deniability.
There is a concept of deterministic builds where multiple parties can compile the exact, byte-for-byte identical binaries. One tool that I know of that does this is Gitian. Gitian allows multiple parties to compile the exact same binary and sign it. This gives you more confidence that the binary you will be downloading is a legitimate one since there is no single point of failure anymore.
Bitcoin and Tor are two notable projects that are using a deterministic build process.
