If the certificate is expired (say, after two years), this means that a potential attacker had at least two years of time available to brute-force or otherwise attack the key. OK, this means more risk than when the cert is only a few months old, but with good keys there should still be virtually no sudden risk increase at the moment of expiry. Nevertheless, this aspect should not be discarded. (Also, old certs may have been produced in the good old days of less secure signatures)
But more importantly, when a key that is still active is compromised, this fact may be advertised by revocation (e.g., CRL or OSCP). If a key gets compromised after its expiry, this is not done (simply because it would be a waste of resources to track those cases when the certs are invalid anyway by expiry).
In fact, it may be the case that the owner did renew the certificate, but what you are looking at is a fake site created by a MITM.
So, yes, you should be sufficiently worried (and worried as hell if that is any site of importance such as a banking site or a shop) and maybe inform the owner.