You don't need to keep the client's private key on your CA/OpenVPN server, and therefore you should not.
Note that you understood correctly how asymmetric cryptography works.
I believe that your server acts both as a CA and an OpenVPN server, you should not1. Easy-RSA
is an utility used to create and manage a Certificate Authority, which can be used to either:
- Generate the server/clients private keys and signed certificates;
- Sign the certificates if the server/clients generated their private keys and certificates themselves.
At no point the CA needs to get ahold of and to store the private keys, they just happen to have them if you used Easy-RSA
to generate them. This means that they should be discarded after being securely sent to the server/clients.
In your tutorial, they decide to keep the private keys on the CA. I believe for convenience purposes, but I would advise against it as it adds an unnecessary risk: an illegitimate third party could get access to that directory and all the private keys. Note that if that third party compromises your whole CA you will have a separate2 but still big issue.
The private key is the technical 'proof' that someone is behind a certificate and the onus is on that person to secure them. Otherwise, you decrease the trust you can put into your PKI.
1: you should separate the 2 roles on different servers and keep the CA offline unless when necessary. It should not be as exposed as an OpenVPN server is.
2: if your CA is compromised, you would not be able to trust any certificates delivered. However, all your clients previous communications are still safe as long as their private keys are kept secret.