Yes, if one blindly downloads and compiles source code, that code could contain an exploit that, if run, could harm one's system. What's more, the resulting binary may not need to be explicitly run. In his 1984 classic Reflections On Trusting Trust, Ken Thompson demonstrated how one might go about creating C code that, when compiled, exploited the compiler and backdoored the system upon which that compiler resides.
There are some ways to defend against this. In 2006, Bruce Schneier blogged a pretty good breakdown of a paper by David A. Wheeler on defending against Thompson's specific example. The paper itself is still paywalled as of this time, to the best of my knowledge.
The Wheeler paper is very interesting, but it is focused on the bowels of compiler design, and this question seems more focused on end-user precautions than compiler design or even systems programming. There are generally two ways we understand the risk involved with compiling a specific piece of code:
- Authenticating the code as a true, untampered-with piece of code written by someone whom we have chosen to trust.
- Closely examining the content of the code itself, and thoroughly understanding what it does.
The second case--a thorough code audit--is a huge, long, resource-intensive task. It almost never really happens for codebases of nontrivial size, because it is simply too costly. Much more often, we are looking at the first case: trusting the coder, and validating that the code hasn't been tampered with between the coder and the consumer.
In 2015, I did an article for LinuxJournal on how code typically gets from the developer to the Linux user. Chain of Custody has since been republished by my employer outside the LinuxJournal paywall. It's heavily focused on the path that goes through package management, but if you read carefully, you'll realize that the parts applicable to a package maintainer compiling code obtained somewhere on the internet are also applicable to an end user doing so.
Of course, in the end, as others have pointed out, these integrity checks only do one any good if the developers' infrastructure hasn't been compromised, if the developer was coding well, and so on.