When I have a website run on https and there is no certificate, there is wrong certificate or out of date like here does encryption still work or not?
Yes , the encryption still works fine .
It is the trust validation of the certificate that is failing.
Try capturing a wireshark dump and check the requests and see for yourself.
If there is no certificate, there is also no SSL and no encryption. It is not possible to set up a web site for SSL without a certificate.
If a browser doesn't trust a certificate (and there are many reasons why this could be the case) the encryption itself still works.
There is still encryption but its susceptible to MiTM attacks. An MiTM attacker can replace your certificate with his own, thus effectively rendering any encryption used useless.