I have done some thinking on this topic and I would like to present my ideas. Feel free to draw your own conclusions.
The secret used for the TOTP algorithm is a 80-bit randomly generated key encoded with base32. This probably more secure than the passwords most people use.
The secret key is usually presented in the form of a QR-code to be scanned by the Authenticator app. Alternatively, some services also provide the base32 encoded secret key for manual input.
In an Android device, the secret is stored unencrypted in an SQLi database file. The SQLi file is located at
/data/data/com.google.android.apps.authenticator2/databases/databases. However, this file isn't accessible to any application other than the Authenticator app itself. Obviously this does not apply to a rooted device. I do not have a rooted iOS device so I am unable to comment on the specifics of the iOS version of the Google Authenticator app.
In my opinion, there are some benefits and drawbacks to replacing passwords with a TOTP authentication solution.
Assuming that the Android device in question is not rooted, no other application besides the Authenticator app can access the secret stored in the database. This removes a very large risk of malware hijacking the secret used for authentication. Any keyloggers/screenloggers will only be able to grab the temporary password that expires in 60 seconds. This is a very small window unless you are the focus of a very targeted attack.
Using TOTP removes the possibility of an attacker performing an online brute-force attack against the service. The window of opportunity is simply too short.
To some people, this is a good tradeoff between security and convenience. Most people I know carry a smartphone around 24/7. Authentication is a simple matter of accessing the app and keying in 6 digits. To some people, this might be easier than having to remember relatively strong passwords or carrying around a password database.
A database compromise means having to reset the secret. This can be really troublesome for the users.
A user losing his/her smartphone means losing access to the service.
This is quite interesting to think about. However, in the real world you should obviously use regular passwords and one-time passwords together for 2FA.