The algorithm is weak, because it assumes that local timestamp cannot be manipulated, but it can under the right conditions.
Being predictable is one of the worst mistakes that can be made when security is a true concern. Counting on facts that you cannot verify is also naive design.
One great weakness of TOTP is that the combination of secret key and timestamp generates always exactly the same token result. Allowing internal clock manipulation weakens the entire concept, possibly leading security problems.
Secret Seed + Time = Token
ntpd daemon allows clock manipulation. It accepts anything coming from the upstream NTP service. So you could give a false timestamp and ntpdate would accept it without concerns.
Is a common pratice to use ntpdate as a replacement for ntpd(both of them are vulnerable to MITM attacks). ntpdate hostname should be avoided.
NTP vulnerabilities: https://www.cvedetails.com/vulnerability-list/vendor_id-2153/NTP.html