You are not using SMS anymore, so SMS can not be hijacked. If you worry for an recovery access, you should not worry.
You say you have backup codes and you even have the original QR Code which in fact contains the unencrypted secret key, that is used to calculate the TOTP values according to RFC 6238. You (or the attacker) can take this QR-Code at any time and create a 1:1 copy of the authentication device smartphone app. You probalby never will lock yourself out - if this is what you worry about.
So - you are safe.
But are you secure? You may want to read my previous blog post.
You can also activate U2F as @parth-maniar stated. But you should know, that U2F uses a preinstalled attestation certificate with a unique serial number. (Well, this is a feature of x509 certificates.)
The key pair you are registering with google is derivated from the private key belonging to this certificate.
(Please put your tin foil hat on, now)
Thus a possible attack vector for an Intelligence Service might look like this: The attestation certificate is passed to the service (Google) when registering. The Intel Service might ask Google for the serial number of your certificate. (which Google would not give to them!).
Then the Intel Service might go to Yubico and ask Yubico for the private key belonging to the certificate with your serial nubmer. (which Yubico does not have and would not give them!)
Now the Intel Service has your private key. Using this private key the attacker can immediately login to all the accounts where you registered the U2F device. The nice thing is, that the Intel Service can attack a "weak" service provider, where you registered to link your name to the serial number. And then they can simple login with your account to the "strong" service provider, who otherwise would not give them your login or your data.
Imho U2F is not the salvation it claims to be.
(tin foil hat off)
;-)
Correction on Feb 25th, 2017
(unfortunately the markdown does not support strike out of words)
I obviously was mislead by some other blog post and a FIDO spec not being precise about implementation roughly 3-4 years ago. The U2F devices contain an attestation certificate, but these usually are not unique for each device. The attestation certificate is just for verifying to the service, that this is a device of a certain type. If you buy a bunch of U2F devices from one vendor they probably all have the same certificate and of course corresponding keys. (I checked this myself these days)
Another customer would probably get the same cert (and keys) as you.
Still, a vendor could create individual attestation certificates for each key or for each customer buying a bunch of tokens.
However, a vendor probably will change the attestation certificate, if they release a new device version. This is because the service should now, if an old device tries to register or a newer device, because it might deny the registration of the older device type.
Well, you still have to trust the vendor to handle the attestation certificate correctly, because this is in fact information that is sent to each service, when registering.
The most known U2F device vendor thus uses an additional master key, which is bound to the attestation certificate! My previous statement was wrong!
Thus the Intelligence Service in fact usually can not identify your individual device by the attestation certificate. The IS would have to contact the vendor in advance to ask them to put individual attestation certificates on it.
I am sorry for the misunderstanding or having caused any big panic! Only mean to cause mid-sized panic! ;-)
...or the normal way of do-not-switch-of-your-brain-when-someone-claims-his-solution-is-totally-secure.
