In Windows, you can have private keys "by themselves". Programmatically, you use CryptAquireContext()
to access a key "by name". The CryptoAPI contains many functions which allow you to import and use keys, independently of certificates.
However, there is no existing graphical interface or file format for handling private keys, and applications do not use keys by name. They use certificates. Certificates, in Windows, are stored "elsewhere", but each certificate in the "My" store can optionally contain a link to a corresponding private key (the link would really be a CSP name, and name of a container within that CSP). This maps to what is expected in various protocols. For instance, in SSL, when the server requests a client authentication with a private key, it actually asks for a certificate: the client must present a certificate, and then, only then, demonstrate that it also has access to the corresponding private key.
Thus, in practice, certificates and keys "live together" and keys are reached only through certificates. A certificate and its private key travel together, and this means a PKCS#12 file (aka "PFX").