The obvious problem with this is the same problem with just using a password: as soon as someone has that, they have all the access they need. Assuming you'd use your phone, then if your phone was stolen, then that person can log in.
If you feel that the risks of using this method is acceptable, then it's acceptable. But you have to understand the weaknesses of the approach and accept them as part of the deal.
Let's put it another way in an extreme case: is it "safe" (you haven't defined that) to use a password of "123456"? Sure, it does keep people out of your account, until someone tries to guess your password and tries this very, very common password. Your account is "safe" from people who will not try to guess your password (but unsafe against those who do).
Is using a super-complex random password of 30 characters "safe"? Sure, it is now safe against those who would try to guess your password, but unsafe against those who would try a keylogger, or a password database that stores everything in plaintext, or a super-computer dedicated to cracking your password.
Each layer of complexity increases "safety" and secures your account against certain groups of threats. By only using 1 factor, you reduce the complexity and reduce the number of groups that you are securing against, but perhaps that groups that remain are sufficient for your needs.