I don't think it's as bad an idea as you assume. Think about what threats you're trying to guard against by using TOTP, and think about what attack scenarios are made easier (or not) if storing the TOTP seed in your password safe.
With the seed in your password database, TOTP will still guard against:
- Password breaches of the website where you hold your account
- Keyloggers on untrusted computers (on which I assume you will not actually run your password manager, but rather read a password off a trusted device and type it in manually)
- Network captures on open WiFi networks, etc.
- Phishing/social engineering of you, the user
Things which are NOT guarded against include:
- Social engineering of customer service/tech support of the website owner
- A stolen password database and master password
The first case cannot be helped regardless of where your TOTP seed is stored. The second case deserves some additional consideration. Specifically, how can your database and master password be breached? Either:
- The attacker obtains a copy of your database from cloud storage, or a discarded backup, and cracks your master password. A strong master password (80+ bits of entropy) and secure implementation of a KDF on the part of the password manager developers should completely prevent this attack.
- The attacker puts malware on your machine which is specifically designed to target your password manager. But consider: if you have such targeted malware on your machine, why not write malware to hijack a browser session? Or force-install a browser extension to steal all the desired information? Or install a custom certificate to allow a MITM attack? A browser extension could even display pages making it look like you were simply logging into the site again, whereas in reality you were providing the credentials needed to disable 2FA entirely!
Now, some caveats: there is ready-made malware (or at least proof-of-concept code) out there which can already target specific password managers. I'm not aware of any ready-made malware that interactively takes over an account to disable 2FA.
But the point is, the only scenario where the TOTP seed causes problems for you, is a scenario that would also cause problems even if your TOTP seed was stored elsewhere. With that in mind, I've not seen any convincing arguments against keeping the TOTP code in my password manager, especially for password managers that can use the TOTP seed to generate the codes for you in place of Authenticator, Authy, etc.