For #1 the usual answer would be to the store the key in an HSM (hardware security module). This is the only truly secure way to store a private key, as any keys simply stored on the hard drive will be compromised if your machine is compromised.
Question #2 is what is usually solved by a PKI (public key infrastructure). Others have said that there is no need to securely transfer a public key, but this is not entirely true; if someone were to perform a man-in-the-middle attack and insert their public key instead, then they would be able to decrypt all future communication. The key itself is not sensitive, and doesn't need to be secured, but you do need some way of ensuring that it is authentic. Confidentiality is not required but authenticity is.
When you use a PKI, the public key is signed by another key higher up the hierarchy to produce a certificate, and this signature verifies that the key is authentic. Of course, then you have the issue of distributing the public key of the next key up the hierarchy and so on. Generally this is solved by distributing the root public key along with your application, as a trusted root CA (certificate authority), then this can be used to verify the authenticity of any certificate chains with this CA at their root.
Question #3 is again something that can be achieved if you are using an HSM to secure your keys.