I already know the basic concept like public key and private key, the message encrypted by public key can be decrypted by private key, and vice versa, but I just can't understand how client know the requested server is the right one and I want to understand the detail of the process.
The client compares the server hostname you type into the URL against the subject name in the certificate. If they match, then it's the right server. If they don't match, then it's the wrong server. That simple.
See: How does SSL work?