I work as a tester at a VoIP software company. We've been running into a problem with using TLS to communicate with certain phones. The phones are from different manufacturers but both exhibit the same problem. To summarize as succinctly as I can, the phone is rejecting the certificate provided during SSL Handshake with "Unknown CA".
The phones use versions of OpenSSL in their firmware, but I do not know which version.
I have verified in every possible way that the Trusted CA certificate provided to the phone during provisioning and the certificate provided to the phone during the handshake are the same and match. Everything SHOULD be golden, however it isn't.
What I've found is that on one system where everything is functioning properly, the CA cert provided has a 10 digit serial number and on the system where everything is not, the CA cert has a 12 digit serial number.
It seems like such an arbitrary thing to be causing the rejection of the cert, but that is literally the ONLY difference I could find when comparing the working and not working systems and their certificates.
Does this sound like something anyone has heard of before? Am I going crazy? Any and all help appreciated.
--Edit-- Our PBX uses a self-signed CA Cert by default. It then uses that Cert to Sign a Line Cert (Server Cert) and also signs Certs used by the Phones (Client Cert). What appears to be happening is that when new Server Certs are generated, they're doing so w/ a 10-digit serial.
I have the ability to use previously generated certificates (generated using the same methodology) that have 12-digit serials. When using the 12-digit Server Cert to complete the TLS Handshake, the phones accept the 12-digit CA cert. When using the 10-Digit Server Cert, the phones reject the 12-digit CA as "Unknown CA". This seems... completely arbitrary and wrong but it's what I've got.
The new 10-digit certs are being generated using OpenSSL 1.0.1i. The 12-digit certs were being generated using an older version (0.9.7e I think). I doubt that matters much. One phone manufacturer uses OpenSSL 0.9.7e and the other uses OpenSSL 1.0.1c-fips