Regardless of how careful I am with my account, is it not only as secure as the recovery email registered with it?
Yes, or your recovery phone #. There was a story recently about somebody convincing verizon to switch their phone number to a hacker's phone. They used that to compromise the gmail address.
Fortunately, it's not always obvious what the recovery email or phone # is.
Largely, yes. Typically, in a good password recovery scheme, the recovery token is restricted in validity. It is 1) a one-time-use-only token and 2) expires over a specified time. This should reduce the attack window to the bad guy being the first to use the reset link and is able to do so within time period X.
That said, I have never personally tested the validity length of a Google recovery token, but I'm sure its not valid over a great length of time.