How do you know a computer is not compromised when you first get it? How do you know that manufacturers have not intentionally built vulnerabilities into the system?
You don't. Some vendors do indeed ship backdoors with their products, and many computers come with "crapware" pre-installed as a source of revenue for the manufacturers. Even apps that don't contain a backdoor can cause other damage (e.g. Browser toolbars that track browsing).
Same concerns apply with hardware, especially in networking equipment.
What you can do:
You don't. Most of the alleged backdoors have been software problems (Google for _NSAKEY, or Digital Encryption Standard and NSA interference, or Huawei and back doors), but a hardware backdoor isn't out of the question. The issue of concealing "intellectual property" or other concerns in proprietary software or hardware make this problem worse.
If all hardware were open, and you could get schematics and trace them out, you could ferret out hardware compromises. Via Diverse Double-Compiling, you could detect software compromises, provided the software was open source.
It would be pretty costly to do hardware and software verification. Since "security" is an economic good, with a price and a value, we all decide what value trusting hardware or trusting software gives us, then see if the price is less than the value. For some, the value isn't so great. Microsoft has let the government of China look at Windows source code. Presumably, it was worth it to the People's Republic to pay Microsoft for that privilege, and to pay their security experts to go through the code.