Your concern is misplaced though everything you wrote was technically correct. The tricky part of understanding PKI is seeing how each type of protection is gained.
If a sender wants to ensure that a message can only be read by the intended recipient, the sender uses the recipient's public key. In this way, no one else can decrypt the message since the complimentary private key is secret and the sender has gained confidentiality.
If the sender can also provide the recipient with confidence that the message was created by the specific sender (non-repudiation) and has not been tampered with since the sender created it (integrity) by making a hash of the message, encrypting the hash with the sender's private key and attaching the encrypted hash to the encrypted message. Since the decryption of the hash can be performed with the sender's public key, the recipient can decrypt it and confirm the hash against the message. The encrypted hash is called a digital signature since it ties sender to message.
[NOTE: The message and the hash are not encrypted with the same or even related keys. The message is encrypted with the recipient's public key while the hash is encrypted with the sender's private key.]
Both integrity and non-repudiation can be strengthened by using a time-stamp server. This is a lesser know but very valuable step in which the sender sends the hash to a trusted server (typically a third-party commercial service) which replies with the hash and a time-stamp encrypted with its private key. The sender then encrypts that response and sends it as his signature.
With all those steps, the recipient knows:
- No one but the sender has seen the contents;
- Only the sender could have sent the message;
- The message is unchanged since the sender signed it
- When the sender signed it.
All of this assumes that both recipient and sender have maintained the confidentiality of their private keys. With this assumption, the recipient can not deny having sent the message and, if time stamped, having sent it at the time indicated by the time stamp.