When you download just the binary, then you can't. You have to trust the distributor of the binary that it is indeed compiled from the published sourcecode.
When you consider the developer trustworthy but you aren't sure you can trust a download mirror, then note that many developers publish hashsums of their binaries, so when you download the binary from an untrusted mirror, you can validate it by checking the hash. This, however, assumes 1. that the binary is the one compiled by the developer and not an independent build by the distributor, and 2. that the developer isn't the one who wants to screw people by releasing a binary compiled from a different sourcecode than what he is publishing.
When you want to make sure that the binary you get is really compiled from the source, you have to get the sourcecode and compile it yourself. This might take quite some time with more complex projects.
But it's questionable if it is really worth it from a security point of view, because just because the sourcecode is public doesn't mean that it is free of backdoors. There is the underhanded C code contest with many creative examples of how to hide malicious sourcecode in plain view. When you want a more real-world example: The heartbleed bug was a vulnerability which existed for years in a very widely used piece of open source software and nobody noticed. There is no reason to believe that it was an intentional backdoor, but when it would have been intentional, the fact that it is in public soucecode and not smuggled into a binary would give the one who planted it plausible deniability.