I am not an expert of hardware manufactoring but here are my 2 cents on the topic.
Step one: set up a root PKI
On your server, you should create a custom Certification Authority (see my comment) that will be used later to sign all certificates.
Configure your server to trust only that CA
Step two: you need an extra manufacturing step to enroll the devices
You should switch your device(s) on a first time at manufacturing time, at some stage. At that stage, before the device is shipped to the final user, have the firmware generate a custom private key, which will be unique for every device, and have that keypair signed with the CA certificate.
In order to enroll devices securely, i.e. make sure that no final user alters the firmware before the first boot, this process must be done at your physical facility. Either at the place the chip is manufactured, or at an intermediate stage of delivery chain but within your (office?) boundaries.
Step three: your devices will authenticate using a TLS signed CA certificate
One of the greatest features of TLS mutual authentication is that the server must not know in advance what client certificates are authorized, but they can be just signed by the same CA.
Hardware-based key management, e.g. a TPM device, may be helpful a lot. a TPM device can use the Endorsement Key feature to help secure the firmware.
The EK is issued by the manufacturer. I may assume that you are the manufacturer. While not a complete answer, TPM could be a good starting point to investigate.