I would still advise you to replace your card to be on the safe side, but if it was MY card, I wouldn't worry about it.
You obviously shouldn't increase the risk to your credit card unnecessarily, but with all that being said, do you replace your card every time you pay in a restaurant and the waiter goes away with your card and brings it back?
When you consider the added risk to your card in this particular case you described, I think it's worth considering other risks to your card during its usage (usually over a span of a few years). Other scenarios of card usage typically include:
- Restaurants/Bars - have you never paid with a credit card before? How easy it is to write down or memorize your card details
- Call Centres - have you never given your card details over the phone?
- Shops (do you ever notice how many shops have CCTV cameras?)
- Community centres / gyms where you are a member
- Of course so many websites that you don't have any way to know who has access to those details
There are a lot of places where someone can get both your card number and CVV + expiry date and in many of those situations they might already know something about you like your full name and address and maybe even your date of birth.
So comparing those risks, which pretty much anybody with a credit card has to take, to this one email (and good overview from @Rakkhi about what it means to be able to grab this email): I think the other leak scenarios are far more likely than from Gmail or from someone sniffing the network.