Everywhere I go, people tell me the private keys need to be kept away, possibly even locked into a safe. Then how come Firefox and other keystores expect you to give them a private key?
Firefox is a client, it shouldn't need this right?
The private key is need by Firefox for client authentication, i.e. when the client demonstrates to the server ownership of a certificate. By definition, this requires knowledge of the private key on the client side. This does not happen often, because most Web servers authenticate clients with passwords, not with certificates.
Of course, Firefox has no need of the private key corresponding to a server certificate, and in practice won't have access to that private key.
Generally speaking, the less a private key travel, the happier we are. However, when it becomes necessary, for some reason, to transport a private key and its accompanying certificate, then we prefer to do it with some extra security to maintain confidentiality; this often uses the PKCS#12 format, also known as "PFX" or ".p12". Such a file is an archive which can contain just about anything, but in particular certificates and private keys, and (that's the point) it includes password-based encryption: parts of the archive contents (e.g. the private key) can be encrypted with a key derived from the password chosen to lock the archive.