> Other than the cost, are there any downsides to HTTPS?

Yes, HTTPS traffic cannot be cached by third party proxies. If your content is highly cacheable (very likely if your site is mostly static content) and you have lots of users that have slower internet connection (e.g. most people in developing countries rely solely on slow, congested mobile network), caching proxies set up by their ISPs or mobile providers or corporate/university network cannot return a locally cached result. Instead they all have to go all the way to hit your server even when the same content has been requested thousands of times around the neighborhood.

This can be mitigated a little if these users install their ISPs proxy certificate, which allows the ISP to essentially do an MITM with the user's permission, but this will significantly compromise the security of sites that actually do need the security of HTTPS.