Yup, this is the standard problem with increased security; whether it be email 2FA, a crazy alarm system on your house, a passport, whatever. If you say "Don't believe that it's me unless they have the thing", and then you lose the thing, you're in trouble.
SMS OTP is actually one of the easier losses to recover from because you can always get a new SIM card with the same number.
But when setting up OTP apps, or Yubikey / FIDO2 USB tokens, or client certificates / PGP keys / smart cards, etc, it's always a good idea to configure more than one 2FA method. For this exact reason I have two yubikeys; one is on my keychain, and one is in a safe place in my house.