You don't need to keep the client's private key on your CA/OpenVPN server, and therefore you should not.
Note that you understood correctly how asymmetric cryptography works, except that in practice we do not use that process to encrypt all the data sent between A and B. It's way too slow compared to symmetric algorithms. We use it at the beginning of a session so A and B can share a secret key that will then be used with a symmetric algorithm.
I believe that your server acts both as a CA and an OpenVPN server, you should not1. I will separate the 2 in this answer and focus on the CA role as it is the one that get to store the files in your example, but the reasoning applies even more to the OpenVPN server.
Easy-RSA is an utility used to create and manage a Certificate Authority, which can be used to either:
- Generate the server/clients private keys and signed certificates;
- Sign the certificates if the server/clients generated their private keys and certificates themselves.
At no point the CA needs to get ahold of and to store the private keys, they just happen to have them if you used
Easy-RSA to generate them. This means that they should be discarded after being securely sent to the server/clients. The OpenVPN documentation confirms it. Regarding the certificate, they may be needed for revocation, but
Easy-RSA has a copy elsewhere I believe.
In your tutorial, they decide to keep the private keys on the CA. I believe for convenience purposes, but I would advise against it as it adds an unnecessary risk: an illegitimate third party could get access to that directory and all the private keys. Note that if that third party compromises your whole CA you will have a separate2 but still big issue.
The private key is the technical 'proof' that someone is behind a certificate and the onus is on that person to secure them. Otherwise, you decrease the trust you can put into your PKI.
1: you should separate the 2 roles on different servers and keep the CA offline unless when necessary. It should not be as exposed as an OpenVPN server is.
2: if your CA is compromised, you would not be able to trust any certificates delivered. However, all your clients previous communications are still safe as long as their private keys are kept secret.