Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

I apologize if this is not the best place to ask my question on the stackexchange network, I couldn't figure out where to get enough attention and be relevant.

Facebook provides a SSL host, it can be accessed by https://www.facebook.com. Basically, as I understand it, changes in the certificate should only occur when the old certificate is to be removed. However, perspective (an addon for firefox, checking fingerprints from many servers) find inconsistencies in the fingerprints of facebook. I am concerned and I would like your insights about this matter.

Perspective screenshot Feb, 23th

EDIT: explanation of the screenshot

Notary and current key column lists all the servers that tried to access www.facebook.com, and what fingerprint they found in the host's certificate. The Key history shows key changes over a period of 30 days. We can therefore conjecture that:

  • Globally, certificate changes occur
  • One certificate (blue) seems to be the main certificate
  • The others replace it sometimes and are rather suspicious

Why would facebook change their certificate in this manner? Is it a common/safe practice?

I understand that PKI with Certification authorities are not very secure, I know that it's easy for some people (either rich, powerful or knowledgeble) to get a certificate for whatever domain they wish. But before going all paranoid, I wish to know if there's a logical explanation that has nothing to do with security.

As a comparison here is the notary results for https://encrypted.google.com:

encrypted.google.com

Certificate changes occur regularly, however the browser's key can be seen only once on a 30 days period. I start to think that nothing is really fishy, but these practices seem to undermine perspective's usefulness and requires users to trust corporations. They should at least provide some kind of explanation for this and why not, a list of their certificate fingerprints.

Bounty: I'd like so see some references about such a system, surely it is possible to find a specification or document recommending such a configuration with multiples certificates within a same regional cluster as it seems to be the case for facebook. Also, if other credible theories exist please share them.

Established facts

Thanks to answers and comments in this thread, we could figure out:

  • Facebook indeed has different certificates
  • Even behind a single IP (load balancer)
  • Revoking a certificate and replacing it would be easier this way

BUT

  • Does this improve security?
  • Keeping track of which server has which certificate looks tedious, is this method really worthwhile? It's not even scalable since certificates are different within a single regional cluster
  • How about just keeping spare certificates in case of a leak? Keeping them versus deploying them all seems better from a security point of view: if one leaked, why not the others?

EDIT: accepted answer

Refer to the accepted answer and comments, I see things more clearly. It helped me understand the reasons behind such a setup, real life deployments may not be as simple as I originally thought. I find it sad that perspective addon is almost useless in such occasions though, but I can't expect facebook to care about this, it's meaningless for almost everyone. Thank you all for your input.

share|improve this question

migrated from stackoverflow.com Feb 23 '12 at 17:10

This question came from our site for professional and enthusiast programmers.

    
I guess Facebook servers present different certificates for whatever reason. –  Eugene Mayevski 'EldoS Corp Feb 22 '12 at 12:30
    
Is there a valid reason to do that? It's important to know. Or should we just trust? Besides, what if facebook doesn't know about this and someone else is performing MITMs attacks? I want to get to the bottom of this. –  Aki Feb 22 '12 at 12:44
    
I think management and security purposes. Eg. servers in different locations could have different certificates to prevent leaking of the private key (one compromised certificate could do lots of harm, but if it's the only one for the whole ecosystem, chances for it to leak are much higher and the damage is much higher too) –  Eugene Mayevski 'EldoS Corp Feb 22 '12 at 16:48
3  
+1 Seems peculiar to me too. If it's legit, I want to know why I haven't heard of this technique / what the benefits are. Good question (although possibly more suited to the security SE) –  Basic Feb 22 '12 at 20:18
2  
If this is valid and expected behavior, doesn't it entirely undercut the perspectives/convergence approach of basing trust on consistency? –  Will M Feb 26 '12 at 17:37

2 Answers 2

up vote 7 down vote accepted
+50

This probably depends on dns resolution + caching that might happen also at the notary servers. If you resolve www.facebook.com you usually get one IP address. But this IP changes over time and may lead you to somewhere different.

I just did a quick check and indeed the www.facebook.com IP address I was given changed, and also the certificate information was different.

Over the course of a few minutes, I received 3 different IP addresses for www.facebook.com: 69.171.229.16, 69.171.234.80, 69.171.228.40

Getting the SSL certificate for each of those IPs produced different certificates. However the certificates weren't consistent even for the same IP address it seems. Over those few minutes I received at least two certificates:

-----BEGIN CERTIFICATE-----
MIIDzDCCAzWgAwIBAgIQPAjP7r6f68QrsT7gPWIL3zANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0x
MTExMTcwMDAwMDBaFw0xMjA3MTMyMzU5NTlaMGoxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xFzAVBgNVBAoTDkZh
Y2Vib29rLCBJbmMuMRkwFwYDVQQDExB3d3cuZmFjZWJvb2suY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC4e9C0eD3zy0YR829bH7fd3PGGDp/3Rd7UR65Q
+jcsRoLZaer9k9tPEOd5ZmWR1MTzwVEmZ94fhoWf219K2Nx/v7fQaWYh5U0DETUo
bkDfR4zBAe+oMuFDIGrhEkUEdlWUOcpvScrtzRjRLyikTc4twjRlpB5RdnGGFopw
CRTNMwIDAQABo4IBIDCCARwwCQYDVR0TBAIwADBEBgNVHSAEPTA7MDkGC2CGSAGG
+EUBBxcDMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9y
cGEwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL1NWUkludGwtY3JsLnZlcmlzaWdu
LmNvbS9TVlJJbnRsLmNybDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
CwYDVR0PBAQDAgWgMCkGA1UdEQQiMCCCEHd3dy5mYWNlYm9vay5jb22CDGZhY2Vi
b29rLmNvbTA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LnZlcmlzaWduLmNvbTANBgkqhkiG9w0BAQUFAAOBgQANiGfuAUQqkUZiD2cozCmb
7+e6vK5yzc/3o/zAd+QBydo7dkPo/t8nIHUfGxcAKUICjAzH/j3FDykK3bupFvBW
D4GoU6qVvNFgH8ucYWMNbxsknN/s3lcFryIGZYFcsYuPEB/rbBEEseKTilUSR7vt
pUEsLwSGdWwTNtKgy/COcQ==
-----END CERTIFICATE-----

and

-----BEGIN CERTIFICATE-----
MIIDnzCCAwigAwIBAgIQCSGX4cDpzQPaNSQ2VhCGgTANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0x
MTA3MTQwMDAwMDBaFw0xMjA3MTMyMzU5NTlaMGoxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xFzAVBgNVBAoTDkZh
Y2Vib29rLCBJbmMuMRkwFwYDVQQDExB3d3cuZmFjZWJvb2suY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC4e9C0eD3zy0YR829bH7fd3PGGDp/3Rd7UR65Q
+jcsRoLZaer9k9tPEOd5ZmWR1MTzwVEmZ94fhoWf219K2Nx/v7fQaWYh5U0DETUo
bkDfR4zBAe+oMuFDIGrhEkUEdlWUOcpvScrtzRjRLyikTc4twjRlpB5RdnGGFopw
CRTNMwIDAQABo4H0MIHxMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEFBQcwAYYYaHR0
cDovL29jc3AudmVyaXNpZ24uY29tMAkGA1UdEwQCMAAwRAYDVR0gBD0wOzA5Bgtg
hkgBhvhFAQcXAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy52ZXJpc2lnbi5j
b20vcnBhMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9TVlJJbnRsLWNybC52ZXJp
c2lnbi5jb20vU1ZSSW50bC5jcmwwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMAsGA1UdDwQEAwIFoDANBgkqhkiG9w0BAQUFAAOBgQAwUGokl1919Kf6YPDK
uoF+k7CFm+j2YtZqw4ilP9166qkM8IDWyYa87MB084WXSyfS1VnT5FSwzjP37+Qx
gjRaROuWGxfY25KebCQpoBW2PJp3S1JmqHHyxjk4mzr+tzWK0Qn+tlBUy9igtkIh
VybjO+AxBZve1qyJIsVraz8wrw==
-----END CERTIFICATE-----

Facebook might have some priority-list of which IP address the DNS issues and then the SSL endpoint you hit (via their load balancers) when you access any one of the facebook IP addresses. Their infrastructure setup might 'prefer' some endpoints over others (maybe they have stronger hardware on one endpoint over another), which is why you'd mostly see one fingerprint, but occasionally another.

UPDATE:

To answer your additional questions more specifically:

Does this improve security?

I think it does, but primarily if you consider availability rather than confidentiality or integrity when you say 'security'. If one certificate expires/needs to be revoked/needs replacing for any other reason, you still have another one fully-operational, live and ready.

Keeping track of which server has which certificate looks tedious, is this method really worthwhile? It's not even scalable since certificates are different within a single regional cluster

It might be a bit tedious, but I'm sure it's only a fraction of the tedious tasks facebook have to deal with in the grand scheme of things. Is it worth it? I think so, and I'd say facebook do it for a reason (As I mentioned above, primarily availability reasons). I would imagine it's not running on a server, but some beefy dedicated SSL accelerator/endpoint cluster of some sort. Those will handle SSL traffic and then reverse-proxy communication to some front-end web servers (which wouldn't need to worry about certificates).

How about just keeping spare certificates in case of a leak? Keeping them versus deploying them all seems better from a security point of view: if one leaked, why not the others?

Yes, keeping some spare certificates offline is a very good idea, and without knowing I would bet Facebook do that too. Having multiple online certificates is not mutually exclusive to having a few offline too. You are assuming that they deploy all of them, which might not be the case. Also making the logical connection that if one leaked so would the others requires more facts. If they have good security in place to keep things segregated, then one compromise doesn't immediately link to another. They could, for example, use different HSMs from different vendors on different server OS, protected by a separate set of firewalls and managed by a completely independent teams.

share|improve this answer
    
This looks very correct. I edited my question over time to ask what potential benefits such a set up can bring from a security point of view. Do you know any paper or public recommendation that talks about multiple certificates? Surely if it's a good practice there should be some kind of paper/presentation about it. –  Aki Feb 26 '12 at 8:39
    
In my mind, this is simply common-sense and avoiding single-point-of-failure. However, key management guidelines usually cover aspects of key rotation, revocation, and replacement of compromised keys. Similar aspects apply to PKI environments. I'd need to check again, but if I'm not mistaken FB even uses different Certification Authority for their certificates. That makes sense so even if the CA itself gets compromised, they don't rely on a single authority. –  Yoav Aner Feb 26 '12 at 11:19
    
I understand. I just wonder, if there is a failure concerning one cert, MITM attacks are possible on everyone visiting facebook, so it doesn't improve security in this case, right? In either case, you have to wait for the revocation and change certificates. If one cert leaked, why not the others? Since it's the same setup, it's very likely. A set up where spare cert are kept hidden is more secure. It's only in the best case scenario that multiple cert are useful (isolated failure), and certs have to be changed, changing 10 or 200 hosts' cert shouldn't take a lot of time. Does this make sense? –  Aki Feb 26 '12 at 12:09
    
It does make sense, but I think you're jumping to conclusions. Having multiple online certificates doesn't mean you don't also have multiple offline certificates, ready to be installed. Also, I wouldn't necessarily say that if one certificate is compromised so are the chances of another. If you keep your systems separated sufficiently, they should also be independently protected (with different layers etc). –  Yoav Aner Feb 26 '12 at 12:14
1  
Glad I could help clarify. This is not based on any familiarity with the facebook infrastructure, but I do think it makes at least some sense. –  Yoav Aner Feb 26 '12 at 16:52

Facebook may have deployed multiple SSL certificates for the same endpoint. This is often done for security reasons (a compromised private key only affects a limited number of hosts) or for manageability reasons (I can stagger the validity periods so they don't all need to be replaced at the same time). Different certificates means different serial numbers. As a result, the fingerprints will be different.

It is also possible that the contents of the certificate are different. For example, they could have different subjectAltName extension values to reference a given server by several DNS aliases.

share|improve this answer
1  
Could be, but do you know why the blue cert on the graph is sometimes replaced by others? It doesn't make sense. I would like to be sure about this. I think this deserve a real analysis of the problem by a security expert, the cert changes don't occur on a regular basis. My wildest guest would be the entry for facebook in their DNS caches expires, they ask for the new address and it's a different server with a different cert. But it doesn't seem to happen regularly again... if I am right it's very strange that we don't observe a pattern, cached entries should expire regularly... –  Aki Feb 23 '12 at 10:04
    
It could be any number of things, really. Hitting different data centers, load balancers, etc. The reality is you're hitting a different physical endpoint, which is why you're seeing a different certificate come back. –  Shadowman Feb 23 '12 at 14:12
1  
How does this improve security? If you have time to revoke a leaked certificate, you have time to buy another one too. Your theory is however right. I'd like to hear other opinions or ideas so I'll let this question open for a few days. Thank you. –  Aki Feb 23 '12 at 14:51
    
Revoking a certificate can be a very quick process. Issuing a new one is usually much more lengthy. You'd want to make sure you have 'spare' certificates in advance, in case you suspect a compromise on any of your existing certificates. –  Yoav Aner Feb 26 '12 at 8:26
    
@YoavAner: As I understand (tools.ietf.org/html/rfc3280#section-3.3), each CA maintains a public repository with a CA revocation list, basically listing each revoked certificate by serial number. Then each browser uses the OCS protocol to retrieve it (tools.ietf.org/html/rfc2560). So it's fast (1-2hours), I agree. But how about keeping the spare certificates and not deploy them randomly inside a cluster? This way you can be sure that they didn't leak. If one cert has leaked, why not the others? How facebook's methods does actually improve security? –  Aki Feb 26 '12 at 8:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.