A certificate contains the public key. The certificate is that which you show to other people; for SSL client authentication, the client shows his certificate to the server when the server asks for it. A certificate needs not be hidden in any way; it is inherently public data.
The private key is the sensitive part. The private key is used to compute a digital signature, which can be verified by using the public key (from the certificate). Within SSL, when the server asks for a client certificate, the client shows his certificate (a
Certificate message) and also computes and sends a signature (a
CertificateVerify message). The data that is signed is a compound of previous messages in the SSL handshake, so it includes the "server random", which serves as a kind of "challenge". From the server point of view, the client just demonstrated his mastery of the private key by being able to generate a signature on a piece of data which includes some bytes chosen by the server.
The certificate is also a signed object; it was signed by the Certification Authority who created the certificate. By verifying this signature, the server can make sure that the contents of the certificate are "correct". This is the other piece of the authentication: though the signature by the client showed that the client indeed controls the private key corresponding to the public key which is in the certificate, the server must still make sure that the public key in the certificate is indeed the public key owned by the alleged user identity. The certificate is a vessel to convey such name-key bindings.
Your certificate is public, so it cannot be "stolen". There is no problem in other people obtaining a copy of your certificate. After all, you are prone to show it to any server who asks for it.
Your private key is sensitive. If some steals your private key, then he can impersonate you when talking to all the servers which have been configured to accept your certificate. Under normal conditions, no entity whatsoever learns your private key; the CA does not have it, the SSL server does not see it. Your private key is stored in the entrails of your computer, or possibly in a smart card. In the latter case, the private key never leaves the card; the card computes the signature itself.