It is a common misconception that the TOTP algorithm is anyhow involved with Google or the other way round. TOTP is based on the HOTP algorithm, that was published in 2005 in RFC 4226.
The TOTP algorithm replaces the counter of the HOTP algorithm with a 30 or 60 seconds time slice. It is defined in RFC 6238.
It is to be noted, that the idea of the xOTP algorithms was based long before there was the first smartphone around. No Google employee was involved in the specification but rather several people from token hardware vendors.
Google simply adopted the TOTP algorithm for android devices, for which it was not designed in the first place!
What happens if for some reason a cell phones clock / calendar is off by a significant amount of time? Does the TOTP (Time-based OTP) algorithm generate an invalid token?
No, not necessarily! Chapter 6 of the RFC recommends a resynchronization, i.e. the clock in the authentication device may drift and the authentication server should remember the clock drift. So over time the authentication device can have a signigicant offset, if the user uses the authentication device regularily. My own project the privacyIDEA authentication system takes care of such a clock drift.
Also, do time zones play a role in the token being correct or does both the client and the server talk to a Network Time Protocol server to ensure that everything is synced up?
As Ángel pointed out, there is no need to think of timezones, since the time slices are used based on the unix system time.
And as RFC 6238 was defined for hardware tokens there is also no recommendation to use NTP, since the hardware token does not have an internet connection but only a battery driven local quartz clock.
Of course, if you implement your server, you should connect your server to ntp. And if you are running a smartphone app as authenticator, this should also use ntp.
So in other words if we wanted to bullet proof the design of the server to be sensitive to the above scenario how would we go about doing this? For instance suppose there was a 5 seconds differential between server time and client time and my grand mother was typing the the code on her phone, and just taking a long time. How would be assess the amount of buffer time the user has to type the code and whether it's correct or not.
This is defined in the RFC. 5 seconds does not matter, since it falls in the same timeslice. Your authentication can check a few time slices before the current time and after the current time and thus know, if the clock (or the user) is a bit slow or fast.