is there any way to both sides share a secret key and then generates key pairs using it
You can derive further keys from an initial shared key, there are various key derivation mechanisms that would allow you to do this, but you still have the problem of distributing the initial key. Anyone who intercepts this key could replicate the derivation process and derive the same subsequent keys, so you still need to securely transfer this initial key and you have the same problem you started with. This applies to your suggested solution as well; wrapping the keys with a shared secret is a sound idea but you still need a way of securely sharing that secret.
Key distribution is probably the single most difficult part of implementing a cryptographic system. Distributing symmetric/secret keys, as would be needed in the solution you mention, is especially difficult, which is why they are often negotiated wrapped under asymmetric keys, such as is the case for SSL and similar, as the encryption is done using the public half of the receivers key which is not confidential, but you then still have the issue of distributing and verifying the public keys you are using to encrypt the symmetric keys, which is often solved by implementing an entire PKI (public key infrastructure) system, which includes CAs (certificate authorities).
You do mention using a CA, but what you are describing does not seem to be the manner in which a CA usually operates, which would usually involve the client generating their public/private key pair then sending the public half to the CA to be signed and receiving a certificate in return. These certificates can be verified by other clients due to them including the root CA certificate in their store of trusted root certificates. You've described in the comments why you're doing it differently but I'm still not sure I really follow your reasoning...
It also surprises me that the overhead of a simple symmetric encryption is too high for you. Have you actually tested the performance of this and deemed the performance hit to be unacceptable? This smells a little like premature optimisation to me, as symmetric encryption is not particularly computationally expensive (unlike asymmetric, which is why asymmetric is usually only used to negotiate a shared symmetric key which is used for any subsequent data encryption). If the overhead of a single symmetric key wrapping is too intensive then I would be surprised if you have the computational power to use encryption at all, and possibly you should rethink your approach to what it is you are attempting to do.
