It depends on your threat model. A TPM has multiple purposes, but the most common purpose is measured boot. That is, a TPM will be able to detect an evil maid attack or even a modified BIOS. If your threat model includes an adversary which is able to modify firmware or software on your computer, a TPM can provide tamper-evidence to ensure that it will not go undetected.
Remote attestation is not something you will likely need to use. It is however not at all unsafe. There are no privacy issues due to a TPM's ability to sign things anonymously using DAA, or Direct Anonymous Attestation. And there is no issue with access control because a TPM has no ability to control the OS or any software running on it, since it is in an unprivileged position on the LPC bus. So let's go even further and assume the TPM is not only useless, but downright malicious. What could it do then? Well, nothing really. It lacks the ability to send the so-called LDRQ# signal which is necessary to perform a DMA attack. The only thing it could do is say "everything is OK" when in reality the firmware has been tampered with. In other words, the worst a malicious TPM could do is pretend it doesn't exist, making a malicious TPM no worse than no TPM.
It is completely possible to safely remove the TPM from the motherboard. There is nothing that requires it be there. If it is not present, you will simply not be able to verify a chain of trust to be sure that firmware has not been tampered with. Note however that many modern CPUs have an integrated TPM, but it can be easily disabled, with the same results as removing the physical one.
See this post for an explanation of how a TPM works and how it can detect tampering.