After some back and forth with OP in the comments, it turns out that the source of the problem is that OP’s web server is serving a certificate chain containing one or more expired certificates. The reason that the problem only manifests itself on rare occasions is that OP’s leaf certificate is issued by Let’s Encrypt (LE), and LE cross-signs its intermediate and root certificates. Most modern browsers are able to create a valid certificate chain to a trusted root certificate using the cross-signed certificates.
It took me a while to get my head around the concept of cross-signing, and how modern browsers are able to use cross-signed certificates to compensate for expired certificates in the chain served by the server - so I thought it would be constructive to summarize the comments into an answer, and add some explanation as to how all of this works under the hood.
The certificate chain that OP’s site serves is:
opsite.com → R3 (exp 9/29/2021) → DST Root CA X3 (exp 9/30/2021)
Notice that the intermediate certificate (R3) and the root certificate (DST DST Root CA X3) are both expired. LE started warning its users about the expiring root certificate well in advance, wherein it reads, ‘What should you do? For most people, nothing at all!’.
Indeed, rather than failing to connect to OP’s site because of the expired certificates, most browsers will build this chain instead (as OP shows in his question):
opsite.com → R3 (9/15/2025) → ISRG Root X1 (exp 6/4/2035)
So, the first takeaway from this is that the trust chain that the browser builds (and shows in the TLS connection information) is not necessarily the same as the one that the site serves.
But more importantly, how are browsers able to compensate for the expired certificates in the certificate chain that the site serves, with the site admin doing ‘nothing at all’ - and build this valid trust chain of unexpired certificates? This where cross-signing comes into play.
LE has two different intermediate certificates named R3. Let’s call them R3 version 1 and R3 version 2:
First, notice what is the same about the two certificates: They both have the same subject name, and they both have the same public key. Now, notice the differences: R3 version 1 is signed by DST Root CA X3, whereas R3 version 2 is signed by ISRG Root X1; and R3 version 1 is expired whereas R3 version 2 does not expire until 2025. Of course, the fingerprints are different as well, because they have different issuers and different expirations. (LE refers to R3 version 1 as the ‘cross-signed certificate’).
So, this is what modern browsers do when they connect to OP’s site:
- Download the certificate chain from OP’s site.
- The leaf certificate is not expired, and the issuer is R3, so look for a certificate in the certificate chain with the subject ‘R3’.
- Found one in the certificate chain from the site (R3 version 1). But, it’s expired, so we can’t use it. Is there another certificate with the subject ‘R3’ somewhere that I can use? Lo and behold, there is! (R3 version 2). And it doesn’t expire until 2025, so let’s see if we can use it.
- Is the signature on the leaf certificate by R3 version 2 is valid? Well, if the signature on the leaf certificate by R3 version 1 was valid, then the signature on the leaf certificate by R3 version 2 will be valid as well, because R3 version 1 and R3 version 2 both have the same public key.
- Continue validating the chain until we come to a trusted root certificate (in this case, ISRG Root X1).
So, the certificate chain built by the browser (using R3 version 2 instead of R3 version 1) is completely kosher.
But, wait a second… Back up to step 3. Where did R3 version 2 come from?
Modern browsers maintain a cache of intermediate certificates. If R3 version 2 is stored in the cache, the browser will find it there. X.509 also has the AIA extension, which is used to specify a URL from where the issuing certificate can be downloaded. Being that OP’s leaf certificate was issued by LE, this field was populated with a URL to a server in LE’s domain where the intermediate certificate can be downloaded, and LE now has R3 version 2 at this URL.
So, why didn’t it work in the case that OP mentioned in his question?
It’s hard to say without more information, but there are a number of possibilities. It could have been that this was an older browser that doesn’t perform the steps above to handle cross-signed certificates. Or, it could have been that for whatever reason, this browser was not able to find R3 version 2. Or, perhaps this browser is behind on updates, and does not have ISRG Root X1 in its trust store.
Final note:
This shows how cross-signing can be used to compensate for expired certificates in the certificate chain served by a site. But, cross-signing serves a more fundamental purpose – it allows for a browser to create multiple certificate chains to trusted roots. Consider a case where a browser connected to OP’s site before Sep 2021. There would have been two chains to a trusted root that the browser could have built – one to DST Root CA X3 and another to ISRG Root X1. In other words, the browser would only have needed either one of those root certificates installed, in order to build a valid trust chain. So, cross-signing provides CA’s with a way to phase-in new root certificates. The certificate chain served by the server can chain-up to the CA’s new root certificate, but if a client does not yet have the new root certificate installed, it can still chain to an older (more widely distributed) one, by way of a cross-signed certificate.
One intermediary:
