The problem with OTP authentication, is that human interaction is involved. You are supposed to first type your password, they type the second factor (or let the usb thing simulate a keyboard and type it for you). That is fine for web form authentication, where passwords are expected coming from a keyboard - in fact it even forces password managers to be able to simulate keystrokes.
But if you want to use IMAP or SMTP, the client application must first acquire the credentials and then send them to the server by the protocol. And the protocol generally has no provision for a second factor.
That is the reason why services that allow other accesses that web via a browser, have to find another way. The common one (application password) normally uses a long random password that cannot be easily guessed nor remembered by a normal human being.
You are right in thinking that this is exactly what can be done with a good password manager: you can ask it to generate a 32 characters random password that nobody (not even you) will ever remember.
But we could just stop a little and think about the rationale for the second factor. A human typed password is generally a weak password. If you can remember it, someone else could also. I am not speaking only of the reversed user name, or the first name of the children but even what is generally seen as good password (the first letters of a rather long sentence for example). A good password cannot be guessed, but it can be remembered much more easily that a truely random generated one!
TL/DR: when using application password you are indeed lowering the security provided by 2FA. It is up to you to know whether it is acceptable of not for the threats you want to address. You should simply ensure that the app password has an acceptable length and change it from time to time.