Such a message is sent by one party (e.g. the server) to the other party (e.g. the client) to signify that it got a problem with the cryptographic elements sent by that party. Nominally, this occurs in the following conditions:
This alert is returned if a record is received with an incorrect
MAC. This alert also MUST be returned if an alert is sent because
a TLSCiphertext decrypted in an invalid way: either it wasn't an
even multiple of the block length, or its padding values, when
checked, weren't correct. This message is always fatal and should
never be observed in communication between proper implementations
(except when messages were corrupted in the network).
In the SSL/TLS handshake, the first encrypted message sent by any party is the
Finished handshake message which precedes the application data. When the crypto went wrong, this will show up at that point, with the
It must be noted that when the asymmetric key exchange fails, e.g. if the server tried to decrypt the RSA-encrypted "pre-master secret" sent by the client but did not find a properly encrypted RSA message, then most modern server implementations will keep on with a random pre-master secret. This is a defence against Bleichenbacher attacks, which try to get extra information on the server private key by trying to guess whether the decryption failed at the RSA stage, or later. To counter these attacks, servers "delay" errors until the
Finished messages, and will try real hard not to explain the exact error cause. So, while
bad_record_mac is nominally a problem with the integrity check layer, such errors will tend to appear for any cryptographic-related issue.
A plausible cause would be the following: the client is using a purported server public key which does not match the actual server private key. Normally, the client will extract the server public key from the server certificate, which the server sends to the client during the handshake. The server also owns a private key, which is mathematically linked with the public key. If the server is configured to use the wrong file, then you could obtain the symptoms you observe.
I suggest that you use a network monitoring tool (such as Wireshark) to observe the handshake messages: you should see the
Certificate message from the server, containing the server certificate chain; for the first certificate of that chain, you can extract the server public key, that the client will use. On the server, try to print out the server private key details (whether this is easy or not depends on the involved software and actual key storage), to see if they match.
Other possible causes involve buggy implementations of some cryptographic algorithms (on the client and/or the server). Pinpointing these errors could be hard.
To get more debug details, you may also try to connect to the server with the
openssl command-line tool:
openssl s_client -connect theservername:443 -msg -debug
Also try to play with some of the options to select the protocol version (
-tls1...) and the supported cipher suites (
-ciphers). Ultimately, you could even recompile your own version of OpenSSL with custom debug code inserted in it (to print out intermediate values and so on). OpenSSL is opensource, so this is technically doable, if you have some C programming knowledge.
Some mastery of the details of the SSL protocol will help. See this answer as a starting point.