It's a hardware implementation that hasn't been tested formally, and it's proprietary. The potential worry is that Intel could have backdoored the implementation at the NSA's demand.
The current way of mixing the rdrand output into the Linux kernel PRNG is that it's xor'ed into the pool, which mathematically means that there's no possible way for a weak output from the rdrand implementation to weaken the overall pool - it will either strengthen it or do nothing to the security.
However, the real risk is that the xor instruction is backdoored in a way that detects for the use of rdrand in a special scenario, then produces a different output when xor is called, causing only the purposefully weakened rdrand output to be placed into the pool.
Feasible? Yes. Plausible? Given recent revelations, maybe. If it is backdoored, is Linus complicit in it? Your guess is as good as mine.
Also, there's a great paper [PDF] on hiding hardware backdoors at transistor level in CPUs.
Edit, Feb 2019. User Luc commented below that things have changed since this answer was originally written:
As of Linux 4.19, the kernel trusts RDRAND to seed its CSPRNG fully, unless one passes the random.trust_cpu=0 flag on boot (or sets it compile time). This should not be an issue if this is not your first boot, but newly installed systems or newly created VMs might have a predictable startup seed file (or no seed file at all), so for those systems this is relevant to gather good entropy.
RNG deterministic mode
discovered in i.blackhat.com/asia-19/Thu-March-28/… be related?without anyone noticing is difficult
. Not that much difficult if you activate the backdoor mode only for specific requested targets.