With client-certificate authentication, the secret (the private key) never leaves the client and doesn't go to the server. Whether you trust the server or not (you should check that first anyway, though), your private key will not be leaked. This is an advantage over traditional form-based or HTTP Basic authentication.
You can even use some hardware cryptographic tokens/smartcards designed in such a way that the private key never leaves the token (the signatures involved in the TLS handshake happen on-board).
If you use client-certificates within the context of a Public Key Infrastructure (most likely), you can also enjoy the benefits offered by the PKI. This is mostly useful for large structures, but you can:
Recognise the identity of users you had never registered before.
This is what Certification Authorities are for. If you trust the CA, you trust the certificate it issues. If unknown users logs on without a pre-existing account onto your system, and if they present certificates that you trust, you will be able to recognise their identities, as vouched for by the CA. You may or may not want to get further details from the user, but the main identity will have been asserted with the CA and will have left an administrative trail there.
The same identity asserted by the certificate may be used across multiple independent websites (provided they trust the same CA), possibly used as a form of Single-Sign On.
A compromised certificate may be revoked by the CA directly.
Via the CA, you decouple the problem of proving the identity of the user from the provision of the service itself. Other methods like OpenID also achieve this goal, but hardly provide the same level of assurance as what CAs can do (provided CAs do their job correctly). The level of assurance will vary depending on the quality of the CA.
An advantage with this is that you can re-issue a new certificate to the same user with different keys (if the previous certificate has expired or if the private key was compromised) and keep the same identifier (Subject) across all systems that trust this CA.
(You can also use client-certificates outside the context of a PKI, but you have to define your own trust rules, which can be quite tedious.)
Client certificates can also be used independently of the protocol running on top of SSL/TLS. You may even use them for S/MIME for example, if applicable.
Another advantage is that, because the authentication doesn't happen at the HTTP level, it's effectively stateless, if things like the REST architectural style matter to you.
Some web services may also use this for message-level security, thereby potentially leaving an audit-trail for non-repudiation of certain messages if necessary.
The main downside is that you'll need to educate your users to the basic concepts of public key cryptography. This is particularly important if you're not using hardware tokens (but just keeping the certificates and private key within the user's software).
Unlike passwords, users won't remember the private key/cert. They'll need to use a machine where the certificate has been installed (or where there is a suitable card reader for hardware solutions). To cut corners, some may be tempted not to take care of their private keys as carefully as they should (they are normally password-protected themselves).
When explaining, the notion of "certificate" can be confusing. Even experts shorten sentences sometimes. If you say "use your certificate to log on", what you actually mean is "use the private key and your certificate to log on": the private key is implied in this expression. In contrast, if someone tells you "send me your certificate", you shouldn't use your private key then. If you look around for documentation, you'll find a number of references to PKCS#12 files (
.pfx) and PEM certificate files (
.crt, typically). Typically, the former contains the private key whereas the second doesn't (although it may also do). All these notions will confuse users unless they know what they're doing.
Browser's user interfaces for client-certificates are generally quite poor. It's quite difficult from a UI perspective to log out of a web-site where you've authenticated with a client-certificate for example (like HTTP Basic). (It makes CSRF protection even more important.) If your clients are "machines" and not users via a browser, this is not so much of a problem.
In terms of infrastructure, if you're not willing to use the services of a commercial CA for your client certificates, you will have to deploy your own CA. Note that the CA used for authenticating the clients can be independent of the CA used for authenticating the server. You can run a website with a certificate issued by a well-known CA but have it trust client-certificates from your own CA. There exist various tools for CA management, including open source ones. Some can even do in-browser key generation, so that the private key is ready in the browser (the downside with that is that the user must re-use the same browser to import the certificate once it's issued).
Configuration of the servers require a certain understanding of what the certificates (CA certificates, server certificate), but it's not actually that complicated. Most servers support this one way or another.
Client-certificates only provide you with authentication. You may still need to get further attributes (e.g. from LDAP or a database against the certificates' subjects). You will certainly need to have an authorisation logic on top of this, as it would be for any other authentication system. It's typical to map a Subject DN to a local identifier in your system.
(There are more advanced usages where you can delegate authentication using proxy certificates, or pass authorisation tokens via attribute certificates, but these are more unusual and are not accepted by all software stacks.)