Regardless of how careful I am with my account, is it not only as secure as the recovery email registered with it?
Largely, yes. Typically, in a good password recovery scheme, the recovery token is restricted in validity. It is 1) a one-time-use-only token and 2) expires over a specified time. This should reduce the attack window to the bad guy being the first to use the reset link and is able to do so within time period X.
That said, I have never personally tested the validity length of a Google recovery token, but I'm sure its not valid over a great length of time.