Based upon your password habits, I don't think it will make a substantial difference.
If you use a long, strong, random password, then it will be infeasible for anyone to crack your password through online guessing (i.e., automated attacks, where the attacker repeatedly connects to Google and tries one guess at your password at a time). This is true even if Google doesn't use a CAPTCHA for the IMAP and even if Google doesn't do any rate-limiting after several erroneous password guesses (which they might). Breaking your password in this way would simply take too long. Consider, for example, a 8-character random password made of alphanumerics. There are 62^8 possible passwords. That's about 10^14. So, to find your password, an attacker would have to try logging into Gmail 10^14 times. At a rate of 1000 login attempts/second, that'll take 10^11 seconds = 3,000 years. Therefore, it's not worth worrying about it.
The weakest link in your security is likely to be something else -- e.g., the security of the client device you use.
In other words, the difficulty of guessing your password is not the weakest link in the chain, so (by Amdahl's law) it's not worth optimizing it: making password guessing harder won't make much of a change to the cost of the cheapest attack, and won't make much of a change to your overall risk.
P.S. I do recommend that you make sure to always use SSL or TLS, to prevent folks from eavesdropping on your password. For instance, use IMAPS (IMAP over SSL/TLS) to connect to Gmail. If your server happens to support both, IMAPS is probably superior to STARTTLS, because STARTTLS will fall back to unencrypted cleartext communications if it cannot negotiate TLS, whereas IMAPS won't.