It doesn't matter that much
Hardware backdoors are expensive. Very expensive.
You need to influence complex supply chains, and the people responsible for them.
Each time a backdoor is used it risks being discovered and published, becoming useless over time, and bringing down costly reputations of people and companies. For each of these times you need to account the benefit brought to the attacker vs the cost of the attack.
Hardware is usually the most secure part of systems. The security track record of software you typically have on your computer is frankly terrible. Even if you are an above-average user, there's good chances you'll have at least one critical vulnerability in your computer software within a year.
Also, there is no way to have absolute trust in anything.
As the old saying goes: the only secure computer is the one that is turned off! And disconnected from any power sources.
But that doesn't really answer your question - How can you trust it (somewhat)? There's actually a number of practical things that you can do as an interested user or developer.
Detecting hardware backdoors
In most realistic attack scenarios, a hardware backdoor will have to escalate to a intrusion on the OS level or communicate over a distance (network, radio, ...). You can put systems in place to detect these.
Intrusion detection systems can help you detect backdoors. This can be done both within the potentially compromised system and at the network level. Obviously you can only trust the former so far (read on for mitigations to that problem).
Anti-virus and rootkit detection might offer some very baseline of protection if they're heuristic-based, but most likely would be evaded by an attacker that has a hardware backdoor if they're well known.
A SDR receiver let's you monitor the airwaves for anomalies if you have reason to suspect them. Easier said than done, but within the realm of possibility in a controlled environment.
The RTL SDR costs around $25
It's worth noting that a sophisticated attacker will likely keep the detectable time window very small, so detection alone might not help you a lot.
Using open source hardware is the easiest thing you can do. There's a number of vendors selling completely open systems, including BIOS and peripherals firmware, or disabling Intel's Management Engine, for example.
A well configured hardware firewall can prevent undesirable communication with the outside world.
Of course, the firewall itself can be compromised. Stacking several firewalls/IDS with different hardware, software and OS makes things more expensive for your attacker (and you!)
If your attacker is very sophisticated they might decide to build a mesh network to bypass your firewall entirely, so it might be a good idea to consider what other things your computer can communicate with on the same network.
Unplugging the network might not be a bad idea, actually. While you are at it, do the same to any plugged USB cables that might have embedded radio transmitters, including your keyboard. Or maybe just put your computer into a large safe, that could be safer and almost as convenient.
If you really have enough time money and willpower to do the whole computer-inside-safe thing, you're a real pain. Your attacker will avoid the network altogether. They will transmit data from your computer using sounds you can't hear, heating up your CPU or transforming your motherboard wires into a GSM antenna
Perhaps you should consider using a typewriter instead? :)