Yes, sites with invalid certificates do expose you to more risk than regular HTTP sites with no certificate, though only slightly more.
In terms of the security of the connection itself your intuition is correct; a site with an invalid certificate is no worse than a plain HTTP site. In fact, in the (rather uncommon) case where an attacker is capable of passively observing your connection but not of modifying it, an HTTPS site with an invalid certificate is actually just as good as one with a valid certificate. Against active man-in-the-middle attackers however, HTTPS connections using an invalid certificate aren't much better than plain HTTP.
The security of the connection itself isn't the only consideration here though. There are other important differences between visiting a plaintext HTTP site and an HTTPS site with an invalid certificate which may make the latter scenario somewhat more risky.
For one, sites which use HTTPS connections may be more likely to transfer sensitive information over that connection. Cookies marked with the "Secure" flag, for example, will not be transmitted over unencrypted HTTP connections, but will be transmitted over HTTPS connections with an invalid certificate if you bypass the security warning. This means that if you logged into a site previously when it was using a valid certificate, then came back to it later and it's using an invalid certificate, bypassing the security warning could instantly give an attacker access to your account.
Additionally, some of the more powerful features of the web are, for security reasons, only available on sites which use HTTPS. If you bypass the security warning for an HTTPS site with an invalid certificate, you could be giving an attacker access to these features.
For example an attacker might install a Service Worker for the site you're accessing, allowing them to control your future communications with that site even if you later go back to accessing the site over a secure connection with a valid certificate. Or if you previously granted that site access to the Geolocation API, an attacker would be able to use that access to monitor your GPS location. For a full list of APIs that are only accessible on HTTPS pages, see the Features restricted to secure contexts page on MDN.
Overall though, these additional risks are relatively minor compared to the risks that using an insecure connection in the first place already exposes you to. You should exercise caution when browsing plain HTTP sites, just as you would when accessing a site after bypassing a security warning, as both of those scenarios are quite risky.
