In TLS there cannot be an encrypted record before the first handshake is completed; the first encrypted record sent by either the client or the server is a
Finished message. If the message is encrypted, then it is meant to be decrypted on the other side; since the symmetric encryption keys are derived from the "master secret" which itself comes from the agreed-upon key exchange mechanism, the receiving end (here, the server) cannot logically decrypt a record before the completion of the key exchange. Thus, from the client, it would make no sense to send an encrypted alert before the
ClientKeyExchange (that is, not only doing encryption before the
Finished messages would violate the standard, but doing so before the
ClientKeyExchange would also break the Laws of Nature).
This leads to two theories:
Perhaps the "encrypted" alert message is not encrypted. An alert record consists in a sequence of "alerts", each of them consisting in a couple of bytes: first byte specifies the alert level (1 = warning, 2 = fatal), second byte documents the nature of the alert. See the standard for details.
Maybe your network trace conflates the traffic from several TLS connections, and the "encrypted alert" pertains to another, older handshake (it could be the
close_notify alert that marks the end of the previous connection).
In any case, if the server requests a client certificate (with a
CertificateRequest message), and the client has no suitable certificate, then the client behaviour should be to send an empty
Certificate message (before its
ClientKeyExchange). This is what is supposed to happen with TLS-1.0 and later versions; in SSL-3.0, a client with no suitable certificate would omit the
Certificate message altogether, and instead send an alert message of level "warning" and type "no_certificate" (numerical value: 41). If your trace shows no
Certificate message from the client, then this would mean that you are using SSL-3.0, which, as of August 2015, is weird (clients and servers should know TLS-1.0 by now). This should be verified by looking at the contents of the
ServerHello messages. An SSL-3.0 handshake would explain the "alert" message, but it would certainly not be encrypted.
(SSL-3.0 and TLS-1.0 are very similar, but have a few differences, one of them being the client behaviour when having been requested a certificate but being unable to provide one.)
If you are really using SSL-3.0, then, first, you should not, because SSL-3.0 has an irremediable weakness (called POODLE). However, even if you update to a better versions (one of the TLS versions), you would probably have the same problem, which is that the client does not send a certificate.
Common causes for that are:
The client does not have, indeed, such a certificate. Maybe the certificate is there but has expired (or the client may believe it so, because of a poorly set internal clock). Maybe the client has the certificate but lacks the secret key (e.g. in a Windows world, access rights on the private key are off).
The client has a certificate but it is not appropriate for the server. The
CertificateRequest from the server includes the list of names of root CA that the server will use to validate the client certificate. The client will send its certificate only if that certificates can be validated relatively to one of these CA. If the list of root CA is misconfigured on the server, then the client may simply abstain from sending a certificate.
Similarly, the client may believe that its certificate is not appropriate for the server. This is a rare case but it happens in some scenarios involving migrations to a new root CA: given a PKI with a root CA and an intermediate CA, one may create a new root CA (new name, new key) that issues a new certificate for the intermediate CA (but the intermediate CA keeps its name and key). This kind of migration allows existing certificates to be validated against both the old and the new root, which is a good thing for a smooth transition. However, some client software (yeah, Windows, I am looking at you) operates under the belief that a given certificates relates to a single root CA, always. Thus, in such a scenario, if the server wants to use the new root but the client believes its certificate to come from the old root, the client will not send it, even though it would have worked fine on the server side.