You've hit on a bit of a sore point; using smart devices as something-you-have type credentials has some drawbacks.
As you point out, the smart device often plays an active part in the authentication process in addition to being the second factor credential. This is not so much a problem as it might first seem. The purpose of using your smart device as a something-you-have is so that the attacker must both steal the device and simultaneously have access to your password in order to gain access to your account. This is still true, even when the device is doing both parts of the authentication, but you're right to question it - particularly if you store the password on the device. In this case your dual-factor authentication becomes a single factor - the device.
The other reason smart devices make poor credentials is that they have a software stack, and they typically run third-party applications. In other words, they have malware. A smart device infected with malware is equivalent to a stolen smart device in terms of using it as a credential.
The malware can instigate the login, wait for and capture the incoming SMS/notification, and read out the second-factor code to complete the login. In the most extreme cases, this will happen without the user's knowledge.
So in summary, don't worry too much - the fact that you are aware of this issue is great, just make sure that the phone does not store the password or provide autologin - locking the app is not enough, as an attacker can still read off the stored password from the phone's hdd directly.
And when any real security is required, use hardware tokens that cannot be subverted, such as TOTP/HOTP tokens. That said, using the smart-device is much much better than having no second-factor at all.