Authentication relies on the following: (source)
- Something you know
- Something you have
- Something you are
A good reason to use TOTP is to increase security by using multiple factors from the list above. For example, if your password becomes known to someone else, they would still need access to your TOTP device to authenticate.
However, that does not mean that TOTP devices are invulnerable to attack.
RFC 6238 states that:
The keys MAY be stored in a tamper-resistant device and SHOULD be protected against unauthorized access and usage.
This means that TOTP implementations are not required to protect the shared secret to be RFC compliant. However, you might consider looking for a TOTP application that requires a password to decrypt the shared secrets. This provides some additional protection, but if your phone's security is compromised by an exploit or malicious physical access, keylogger attacks become possible.
You also asked how many keys an attacker would need to see to discover your shared secret. From RFC 4226, Appendix A.4.3:
No matter what strategy the adversary uses, and even if it sees, and tries to exploit, the authenticators from authentication attempts of the user, its success probability will not be above that of the brute force attack -- this is true as long as the number of authentications it observes is not incredibly large.
So, in general, a brute force attack is the best that could occur. However, if the adversary can generate the codes at will (and for some reason they do not already have the shared secret), this does not hold. More information about what influences this attack are available in the RFC.