If the Raspberry Pi needs access to your google drive to store your video feed then, well, that means that your Raspberry Pi needs access to your google drive and therefore anyone who gains access to your Raspberry Pi may gain access to your google drive. It's not possible to both allow your application to directly upload to your drive and also not allow an attacker to upload to your drive if they gain control of your application.
However, all is not lost. Many modern app ecosystems (including google) are designed for exactly this sort of thing. However it helps to understand how authentication works and what exactly you are doing.
OAuth Client ID
pydrive doesn't have you create access credentials for directly accessing google drive. Instead, it has you create an OAuth Client ID. In essence, this means that you are registering an "app" with Google (although not in the sense of a literal app on the Play Store). This Client ID does not actually give
pydrive access to your account. Instead it simply registers you as the owner of this "app".
To actually access your account,
pydrive directs you through a login process locally where
pydrive will request access to your account from google. This happens through a web browser (which is why
pydrive mentions a webserver).
Google will then ask if you want to give
pydrive access to your account, and when you approve, Google will give
pydrive a separate set of access credentials that give it actual access to your drive.
From then on your
pydrive application should show up in your Google account under the list of apps with access to your account. You should be able to see that on this page, presumably under whatever application name you provided to Google when you created the OAuth Client ID.
In the event that your pi gets stolen, it would theoretically be possible for an attacker to dump the drive contents and find the spot where
pydrive stored the actual account credentials it uses, and then use those to gain access to your account (although with whatever privileges you gave to
pydrive - not your full google account).
Fortunately if that were to happen the solution would be simple. Login to your Google account in any browser, go to that list of apps with access to your account, and logout
pydrive. Even with the OAuth Client ID still available, an attacker won't be able to get back into your account without first login into Google you with your email and password, which they obviously don't.
You would also be able to go back to the APIs section of Google, find the OAuth Client ID you generated, and revoke it as well. At that point in time the attacker will have nothing.
So is there a window of time when an attacker with a stolen Raspberry Pi might be able to access the Google Drive credentials and cause problems? Yes. However, this window of time can be very short in practice, because presumably you will know if your Pi is stolen and will be able to immediately log out the Pi remotely, preventing any further access.