Every Private Key has a corresponding Public Key. The public key is mathematically derived from the private key. These two keys, together called a "key pair", can be used for two purposes: Encryption and Signing. For the purposes of certificates, signing is far more relevant.
A certificate is basically just a public key, which has been signed by someone else's private key. This forms the basis of public key infrastructure (PKI), which is explained in the articles linked in the question.
How do Certificates and Private Keys relate?
A certificate is just a "fancy" public key, which is related to a private key. You can do the same thing with a certificate as you can do with a public key.
If Bob gets the certificate of Alice, he can encrypt a message for Alice. Likewise, if Alice publishes some data and signs it with her private key, Bob can use Alice's certificate to see if it is really from Alice.
What are all those different file types?
.pem is a de-facto file format called Privacy-Enhanced Mail. A PEM file can contain a lot of different things, such as certificates, private keys, public keys and lots of other things. A file being in PEM format says nothing about the content, just like something being Base64-encoded says nothing about the content.
.cer: This is another pseudo-format that is commonly used to store certificates. These can either be in the PEM or in the DER format.
.pfx: These are interchangable file extensions for the PKCS#12 format. Technically, PKCS#12 is the successor to Microsoft's PFX format, but they have become interchangable. PKCS#12 files are archives for cryptographic material. Again, what kind of material this contains is completely up to the user.
.p12 can all be used to store certificates, public keys and primary keys. From a purely technical standpoint, you can not tell what the semantic content of any of these files is just by their file extension. If you ever get confused, don't worry - you're not alone.
However, there are some common conventions that are being followed.
.pfx files are usually used to store a certificate together with the private key that corresponds to this certificate.
.crt files usually contain single certificates without any related private key material.
.pem files are wildcards. They can contain anything, and it's not uncommon to see them used for all different kinds of purposes. Luckily, they are all plain text, and are prefixed in a human-readable way, such as
Why would an application not handle a
.crt file if it wants a client certificate?
A certificate is just a public key, and thus by definition public. A client certificate is no different - just a public key by a person, machine or other "client", that is signed by some authority.
An application that wants a client certificate usually wants to use that certificate for something, such as to authenticate the client to a server. In order to do that, one needs the certificate and the corresponding private key.
So an application should really write "certificate plus private key", because the certificate alone is not enough to prove one's identity. It's actually the private key that does it.