The TOTP specification points, for the security analysis, to HOTP. HOTP uses a counter, shared by both parties, and "resynchronized" every time a successful authentication occurs; TOTP replaces that counter with knowledge of the current time, which is also a shared value. As such, almost all the security analysis of HOTP applies to TOTP.
The security analysis of HOTP uses the formal tools and notations of cryptography, but boils down to the following assumption:
Let T denotes the time to perform one computation of H. Then if A is
any adversary with running time at most t and making at most p oracle
Adv(A) <= (t/T)/2^k + p^2/2^n
In practice, this assumption means that H is very secure as PRF.
The important word here is PRF: it means that the "H" function (HMAC/SHA-1) behaves, in the view of the attacker (who does not know the key) like a function chosen "at random" among all possible functions. We give the attacker access to an "oracle", which is a machine which knows the key, and computes HMAC/SHA-1 over inputs chosen by the attacker; the attacker can send p such queries. The expression above means that even under these conditions, the probability that the attacker successfully predicts, after all his queries, the output for one extra input (that he did not send to the oracle), is still very low (the "advantage" is what extra success probability the attacker has over pure luck).
Right now, it seems that HMAC/SHA-1 behaves as a PRF. Nobody has found any indication to the contrary. This holds even in the face of the known structural weaknesses which seem to allow (theoretically) some computing SHA-1 collisions.
From this assumption, the security proof of HOTP (as exposed in RFC 4226) establishes the following result:
Suppose m = 10^Digit < 2^31, and let (q,r) = IntDiv(2^31,m). Let B
be any adversary attacking HOTP using v verification oracle queries,
a <= 2^c - s authenticator oracle queries, and running time t. Let T
denote the time to perform one computation of H. If Assumption 1 is
Adv(B) <= sv * (q + 1)/2^31 + (t/T)/2^k + ((sv + a)^2)/2^n
In practice, the (t/T)2^-k + ((sv + a)^2)2^-n term is much smaller
than the sv(q + 1)/2^n term, so that the above says that for all
practical purposes the success rate of an adversary attacking HOTP is
sv(q + 1)/2^n, just as for HOTP-IDEAL, meaning the HOTP algorithm is
in practice essentially as good as its idealized counterpart.
This last paragraph is what you are after: under the assumption that HMAC/SHA-1 is a PRF, it is proven that giving access to a lot of previous one-time passwords does not given any advantage to the attacker for guessing the next one, at least no more that what he would get out of an "ideal" version of HOTP in which the one-time passwords are obtained through pure unpredictable magic.
Summary: No need to worry. Resistance of HOTP (and TOTP) to the situation where many previous one-time passwords have been recorded is part of the security model of HOTP, and it has been specifically shielded against such an occurrence. The shield here relies on an assumption of security on HMAC/SHA-1, which, while not proven, is about as good as these things can get (notably because it is quite possible that the fact that a function family is a PRF might be mathematically impossible to prove; see this for more on that subject).