SSL fingerprint inconsistency: what does it mean?

Question

I apologize if this is not the best place to ask my question on the Stack Exchange network, I couldn't figure out where to get enough attention and be relevant.

Facebook provides a SSL host, it can be accessed by https://www.facebook.com. Basically, as I understand it, changes in the certificate should only occur when the old certificate is to be removed. However, Perspectives (an addon for Firefox, checking fingerprints from many servers) finds inconsistencies in the fingerprints of Facebook. I am concerned and I would like your insights about this matter.

EDIT: explanation of the screenshot

Notary and current key column lists all the servers that tried to access www.facebook.com, and what fingerprint they found in the host's certificate. The Key history shows key changes over a period of 30 days. We can therefore conjecture that:

• Globally, certificate changes occur
• One certificate (blue) seems to be the main certificate
• The others replace it sometimes and are rather suspicious

Why would Facebook change their certificate in this manner? Is it a common/safe practice?

I understand that PKI with Certification authorities are not very secure, I know that it's easy for some people (either rich, powerful or knowledgeable) to get a certificate for whatever domain they wish. But before going all paranoid, I wish to know if there's a logical explanation that has nothing to do with security.

As a comparison here is the notary results for https://encrypted.google.com:

Certificate changes occur regularly, however the browser's key can be seen only once on a 30 days period. I start to think that nothing is really fishy, but these practices seem to undermine perspective's usefulness and requires users to trust corporations. They should at least provide some kind of explanation for this and why not, a list of their certificate fingerprints.

Bounty: I'd like so see some references about such a system, surely it is possible to find a specification or document recommending such a configuration with multiples certificates within a same regional cluster as it seems to be the case for Facebook. Also, if other credible theories exist please share them.

Established facts

Thanks to answers and comments in this thread, we could figure out:

• Facebook indeed has different certificates
• Even behind a single IP (load balancer)
• Revoking a certificate and replacing it would be easier this way

BUT

• Does this improve security?
• Keeping track of which server has which certificate looks tedious, is this method really worthwhile? It's not even scalable since certificates are different within a single regional cluster
• How about just keeping spare certificates in case of a leak? Keeping them versus deploying them all seems better from a security point of view: if one leaked, why not the others?

EDIT: accepted answer

Refer to the accepted answer and comments, I see things more clearly. It helped me understand the reasons behind such a setup, real life deployments may not be as simple as I originally thought. I find it sad that perspective addon is almost useless in such occasions though, but I can't expect Facebook to care about this, it's meaningless for almost everyone. Thank you all for your input.

Answer 1

This probably depends on dns resolution + caching that might happen also at the notary servers. If you resolve www.facebook.com you usually get one IP address. But this IP changes over time and may lead you to somewhere different.

I just did a quick check and indeed the www.facebook.com IP address I was given changed, and also the certificate information was different.

Over the course of a few minutes, I received 3 different IP addresses for www.facebook.com: 69.171.229.16, 69.171.234.80, 69.171.228.40

Getting the SSL certificate for each of those IPs produced different certificates. However the certificates weren't consistent even for the same IP address it seems. Over those few minutes I received at least two certificates:

-----BEGIN CERTIFICATE-----
MIIDzDCCAzWgAwIBAgIQPAjP7r6f68QrsT7gPWIL3zANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0x
MTExMTcwMDAwMDBaFw0xMjA3MTMyMzU5NTlaMGoxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xFzAVBgNVBAoTDkZh
Y2Vib29rLCBJbmMuMRkwFwYDVQQDExB3d3cuZmFjZWJvb2suY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC4e9C0eD3zy0YR829bH7fd3PGGDp/3Rd7UR65Q
+jcsRoLZaer9k9tPEOd5ZmWR1MTzwVEmZ94fhoWf219K2Nx/v7fQaWYh5U0DETUo
bkDfR4zBAe+oMuFDIGrhEkUEdlWUOcpvScrtzRjRLyikTc4twjRlpB5RdnGGFopw
CRTNMwIDAQABo4IBIDCCARwwCQYDVR0TBAIwADBEBgNVHSAEPTA7MDkGC2CGSAGG
+EUBBxcDMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9y
cGEwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL1NWUkludGwtY3JsLnZlcmlzaWdu
LmNvbS9TVlJJbnRsLmNybDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
CwYDVR0PBAQDAgWgMCkGA1UdEQQiMCCCEHd3dy5mYWNlYm9vay5jb22CDGZhY2Vi
b29rLmNvbTA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LnZlcmlzaWduLmNvbTANBgkqhkiG9w0BAQUFAAOBgQANiGfuAUQqkUZiD2cozCmb
7+e6vK5yzc/3o/zAd+QBydo7dkPo/t8nIHUfGxcAKUICjAzH/j3FDykK3bupFvBW
D4GoU6qVvNFgH8ucYWMNbxsknN/s3lcFryIGZYFcsYuPEB/rbBEEseKTilUSR7vt
pUEsLwSGdWwTNtKgy/COcQ==
-----END CERTIFICATE-----

and

-----BEGIN CERTIFICATE-----
MIIDnzCCAwigAwIBAgIQCSGX4cDpzQPaNSQ2VhCGgTANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0x
MTA3MTQwMDAwMDBaFw0xMjA3MTMyMzU5NTlaMGoxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xFzAVBgNVBAoTDkZh
Y2Vib29rLCBJbmMuMRkwFwYDVQQDExB3d3cuZmFjZWJvb2suY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC4e9C0eD3zy0YR829bH7fd3PGGDp/3Rd7UR65Q
+jcsRoLZaer9k9tPEOd5ZmWR1MTzwVEmZ94fhoWf219K2Nx/v7fQaWYh5U0DETUo
bkDfR4zBAe+oMuFDIGrhEkUEdlWUOcpvScrtzRjRLyikTc4twjRlpB5RdnGGFopw
CRTNMwIDAQABo4H0MIHxMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEFBQcwAYYYaHR0
cDovL29jc3AudmVyaXNpZ24uY29tMAkGA1UdEwQCMAAwRAYDVR0gBD0wOzA5Bgtg
hkgBhvhFAQcXAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy52ZXJpc2lnbi5j
b20vcnBhMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9TVlJJbnRsLWNybC52ZXJp
c2lnbi5jb20vU1ZSSW50bC5jcmwwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMAsGA1UdDwQEAwIFoDANBgkqhkiG9w0BAQUFAAOBgQAwUGokl1919Kf6YPDK
uoF+k7CFm+j2YtZqw4ilP9166qkM8IDWyYa87MB084WXSyfS1VnT5FSwzjP37+Qx
gjRaROuWGxfY25KebCQpoBW2PJp3S1JmqHHyxjk4mzr+tzWK0Qn+tlBUy9igtkIh
VybjO+AxBZve1qyJIsVraz8wrw==
-----END CERTIFICATE-----

Facebook might have some priority-list of which IP address the DNS issues and then the SSL endpoint you hit (via their load balancers) when you access any one of the facebook IP addresses. Their infrastructure setup might 'prefer' some endpoints over others (maybe they have stronger hardware on one endpoint over another), which is why you'd mostly see one fingerprint, but occasionally another.

UPDATE:

To answer your additional questions more specifically:

Does this improve security?

I think it does, but primarily if you consider availability rather than confidentiality or integrity when you say 'security'. If one certificate expires/needs to be revoked/needs replacing for any other reason, you still have another one fully-operational, live and ready.

Keeping track of which server has which certificate looks tedious, is this method really worthwhile? It's not even scalable since certificates are different within a single regional cluster

It might be a bit tedious, but I'm sure it's only a fraction of the tedious tasks facebook have to deal with in the grand scheme of things. Is it worth it? I think so, and I'd say facebook do it for a reason (As I mentioned above, primarily availability reasons). I would imagine it's not running on a server, but some beefy dedicated SSL accelerator/endpoint cluster of some sort. Those will handle SSL traffic and then reverse-proxy communication to some front-end web servers (which wouldn't need to worry about certificates).

How about just keeping spare certificates in case of a leak? Keeping them versus deploying them all seems better from a security point of view: if one leaked, why not the others?

Yes, keeping some spare certificates offline is a very good idea, and without knowing I would bet Facebook do that too. Having multiple online certificates is not mutually exclusive to having a few offline too. You are assuming that they deploy all of them, which might not be the case. Also making the logical connection that if one leaked so would the others requires more facts. If they have good security in place to keep things segregated, then one compromise doesn't immediately link to another. They could, for example, use different HSMs from different vendors on different server OS, protected by a separate set of firewalls and managed by a completely independent teams.

Answer 2

Facebook may have deployed multiple SSL certificates for the same endpoint. This is often done for security reasons (a compromised private key only affects a limited number of hosts) or for manageability reasons (I can stagger the validity periods so they don't all need to be replaced at the same time). Different certificates means different serial numbers. As a result, the fingerprints will be different.

It is also possible that the contents of the certificate are different. For example, they could have different subjectAltName extension values to reference a given server by several DNS aliases.