It depends on your threat model. A TPM has multiple purposes, but the most common purpose is measured boot. That is, a TPM will verify the integrity of the BIOS, option ROMs, bootloader, and other sensitive boot components so that it is able to detect an evil maid attack or modified firmware. If your threat model includes an adversary which is able to modify firmware or software on your computer, a TPM can provide tamper-evidence to ensure that it will not go undetected.
So how does a TPM work? It's actually pretty simple when you get down to it. The TPM measures the hashes of various firmware components* and stores the hashes in registers called PCRs. If the hashes all match a known value, the TPM will unseal, allowing itself to be used to decrypt arbitrary data. What data it decrypts is up to you. Most commonly, it is part of the disk encryption key. Unless every piece of firmware and boot software has the correct hash, the TPM will not unseal and the encryption key will not be revealed. TPMs can be used for a lot more, but the idea is the same.
* Technically, the TPM is passive and cannot actively read firmware, bootloaders, or other data. Instead, a read-only component of the BIOS called the CRTM sends a hash of the BIOS to the TPM, starting the chain of trust. This component is read-only to ensure that a modified BIOS cannot lie to the TPM about its hash.
So is TPM worth it or is it just an unnecessary potential point of failure? Would my security and privacy be safer if I didn't use a computer with TPM at all? Full disk encryption with VeraCrypt sounds safe enough even for the most illegal use cases (NSA-proofed).
Remote attestation is not something you will likely need to use. It is however not at all unsafe. All it does is allow a remote device to prove to the appraiser that the firmware and software it is running matches a known-good hash. It does not allow remotely controlling the machine. In fact, it is up to the OS to do the remote connections and send the data to the TPM. The TPM itself isn't even aware that it is being used for remote attestation. In fact, remote doesn't even have to mean over a network. There are very clever implementations that use a TPM to remotely attest the computer's state to a secure USB device! There are no privacy issues with a TPM's unique private key either due to a TPM's ability to sign things anonymously using DAA, or Direct Anonymous Attestation.
Let's go even further and assume the TPM is not only useless, but downright malicious. What could it do then? Well, nothing really. It lacks the ability to send the so-called LDRQ# signal over the LPC bus which is necessary to perform a DMA attack. The only thing it could do is say "everything is OK" when in reality the firmware has been tampered with. In other words, the worst a malicious TPM could do is pretend it doesn't exist, making a malicious TPM no worse than no TPM.
It is completely possible to safely remove the TPM from the motherboard. There is nothing that requires it be there. If it is not present, you will simply not be able to verify a chain of trust to be sure that firmware has not been tampered with. Note however that many modern CPUs have an integrated TPM, but it can be easily disabled, with the same results as removing the physical one.
In the section on the bottom, it mentions some criticisms of TPM such as remote validation of software - manufacturer, not the user decides what can be run on the computer. This sounds scary.
The worry is that, in the future, manufacturers might use the TPM to prevent you from making modifications to your system. By default, TPMs will obey only its owner. If you tell a TPM that the current state of the system is known-good, it will always check to make sure the system is in that state. If an evil manufacturer sets the TPM to believe that a known-good state is one where malicious DRM and other rights-restricting software is enabled, then we have a problem. For current TPMs, it's entirely up to you to decide what software you want to run!
Another criticism is that it may be used to prove to remote websites that you are running the software you want them to run, or that you are using a device which is not fully under your control. The TPM can prove to the remote server that your system's firmware has not been tampered with, and if your system's firmware is designed to restrict your rights, then the TPM is proving that your rights are sufficiently curtailed and that you are allowed to watch that latest DRM-ridden video you wanted to see. Thankfully, TPMs are not currently being used to do this, but the technology is there.
The overreaching issue is that a TPM can prove both to you locally, and to a remote server (with the OS handling the networking, of course) that your computer is in the correct state. What counts as "correct" hinges on whoever owns the TPM. If you own the TPM, then "correct" means without bootkits or other tampering. If some company owns the TPM, it means that the system's anti-piracy and DRM features are fully functional. For the TPMs in PCs you can buy today, you are the owner.
Also, VeraCrypt doesn't support TPM which raises some concerns. If they don't trust it, why should I?
VeraCrypt actually has added beta support, if I recall correctly, although they have not yet edit their documentation. They originally were resistant because the original TrueCrypt authors did not understand the TPM. Its purpose is not to assist with disk encryption, but to verify that the firmware and important boot software (including the VeraCrypt bootloader!) have not been tampered with.