If someone has the key and a valid referer, they can use the API key from any client as long as the set referer header, correct?
So is it worthwhile to use this restriction?
All security measures are trade offs. Is the cost of the control worth more or less than the value of what is being protected, and how much more work does it force the attacker to do.
A lock on your luggage may stop a casual thief from opening your bag to grab something, but it won't stop someone stealing your bag. But the cost is low, so it is a still a useful security control.
So, just because a malicious agent could copy the referrer header, they may not have the time or inclination to do so. Think of a valid referrer header like a concert wristband. Can you forge them, or sneak around the guard checking wristbands? Sure you can, but it doesn't make the band useless.
It raises the cost for the attacker.
Likewise, your standard, non-modified, well-behaved application will tell the truth about the last page it came from, and then will correctly be denied API access. Maybe somebody trying to abuse your service doesn't know you are checking referrer headers on the server, and will give up.
So, in answer to OP's question "worthwhile" is a value judgement only you can make. How much effort does it take you to check referrer? Probably not much. How valuable is protecting use of your API key? Probably more. Is that amount of effort on your part worth the speedbump you'll place in the attacker's way?
That last one is your call.
What is the point of restricting a google API key by HTTP referer?
A web application cannot easily forge its referrer when issuing requests. Therefore, restricting allowed referrers to a specific list prevents other websites from using your API key for their own web services and thereby tapping your API quota. It's a useful feature you should take advantage of.
However, an application that is capable of setting its own referrer headers will obviously not be affected by this restriction. It is only meant to protect web applications.
If you're allowing anyone to use your client API key, fundamentally there's no way to prevent misuse. This just makes it annoying enough that it might deter some people. Doesn't hurt to enable it, but don't assume it'll stop anyone who's determined. It's kinda like anti-modding measures in online video games.
Opinion: I've seen lots of places, even Google Cloud Platform, not be clear enough about what this HTTP referrer restriction guarantees: not very much.
The "referer" is set by the browser to contain the address of the calling site, so if you restrict access to the google api by only allowing your web site address, you are preventing other people from using your key to make api calls from their sites, because the referer would not match.
Someone that "has the valid referer" can not normally use it on their own site because the referer header can not be modified programmatically in the browser. This prevents people from simply stealing your key and adding it to their HTML.
But the referer can surely be changed on the server, where you're free to change anything in the request. So a hacker could set up some kind of proxy that acts as a "man in the middle" receiving all api requests from his web site and forwarding them to google after changing the referer to what is allowed by the stolen key.
I'm sure that google is aware of this and has put some other countermeasures in place, like checking how many calls come from the same ip address. But then, the hacker could use hundreds of cheap proxies to distribute the calls and bypass the check.
So in conclusion the referer header is a clever and easy way for protecting your api key as long as nobody chooses to spend some effort into bypassing this protection. And with the recent surge in api cost, I see that very likely to happen. You only have to hope they don't steal yours.