HTTPS gets you *confidentiality* (encryption), *authentication* (identity), and *integrity* (tamper-evident connections). 

You don't care about so much about the first one in your case, but you *should* care about the second two. The "identity" part implicitly protects you from certain DNS attacks, but there is the chicken-and-egg problem (hence [HSTS](http://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security)).
There's also the "green URL bar" and all that nonsense, but that's mostly a side-show.

HTTPS has at least the following drawbacks:
 
 * it's slower on both sides (but not by much, and possibly hidden by bandwidth constraints)
 * it's got higher latency (more round trips, but not many more), this can be minimised with [SPDY](http://www.chromium.org/spdy/spdy-whitepaper)
 * it complicates legitimate analysis and troubleshooting (e.g. packet capture)
 * it *may* prevent some users from accessing your site (corporate policy, content scanning etc.)
 * the cert and CA chain can really add to connection overheads (there are potential workarounds like [this](https://datatracker.ietf.org/doc/html/draft-ietf-tls-cached-info-16))
 * you have do deal with a CA, and have an administrative process to deal with renewals (sadly more difficult in practice than it sounds)
 * more protocols means more configuration, more software, and a greater [attack surface](http://en.wikipedia.org/wiki/Attack_surface)

(A less tangible drawback is getting on board the whole PKI/CA train, but let's not go there today...)