Design review/criticism of secure master-workers system

Question

I am designing a system which will have one main process/service (master) which can potentially offload part of it's work to N number of external remote processes (workers). I want the worker deployment to be simple, yet the the master-workers communication to be secure, with authentication and authorization on both sides. This might operate in isolated environment, so dependency on any external services isn't permitted.

Below is the design I came up with and I would like to understand whether it's sound or not. I am looking for criticism of the design both for reasons why it's not secure or why it's overcomplicated. I am not security expert, so I might be missing some major point.

1. The user installs the worker using public/generic repo.
2. In post-install init the worker performs this:
   1. Generates it's own unique ID. The ID will be used as base of the CA, server and client DNs in the certs below.
   2. Generates self-signed CA certificate.
   3. Generates server key-pair and certificate signed by CA from above.
   4. Generates client key-pair and certificate signed by CA from above.
   5. Places the CA certificate, client certificate, client key and it's ID into base-64 encoded connection token.
   6. The private key belonging to the CA is discarded.
3. The worker starts it's service on defined port, serving the server certificate, requiring client certs for connection, using the CA-certificate to verify the client certs and checking client DNs.
4. The user transfers the connection token to the master, where he initiates the worker registration. He provides the worker hostname/ip and the connection token.
5. The master connects to the worker using the client cert from the token and using the CA cert from the token to verify the server identity. The actual worker hostname/ip is not used, instead it verifies the alternative identity derived from the worker ID to match the one provided in the server's cert.

Providing that the connection token distribution is trusted (e.g. the installing user does not provide spoofed token when registering worker), this should ensure that the worker only accepts connections from the master to which the token was presented. At the same time, the master can be sure it's connecting to the worker who initially generated the token.

There is no cert-renewal process build in. The workers are supposed to be state-less and disposable, so if there is a need for regular cert rotation, it could simply be achieved by killing the worker and re-installing with new identity. I could imagine that the worker might not drop it's CA private key in (2.6) and do some cert-reissuing logic, but I am not sure that would improve security while definitely increasing complexity.