Google uses the API key to identify who is sending those requests. Google will impose a quota. As you've correctly identified, if Google receives too many requests using that API key, they'll blacklist the API key. If that happens, your own site/app will stop working.
Therefore, there are risks to leaving the API key in the clear on your page. These risks are somewhat modest: someone who gains the API key doesn't get root on your systems, but they might end up causing your app to stop working.
If this risk bothers you, then don't put the API key on your web page. Instead, have the client issue an AJAX request to your server, and then have the server issue the request to Google and return the results back to the client. This way, the Google API key is never exposed to the client.
The Referer header check prevents someone else from copying your API key and putting it in their own web page and using Javascript to query Google. However, it does not prevent them from copying your API key and using it in requests they initiate from their own machines. (If they use curl or some other similar library on their own machine, they can forge the Referer header and make it say whatever they want. Therefore, the Referer check does not help prevent this scenario. It only prevents others from embedding the API key in their web page and using it in Javascript that runs in the browser of folks who visit their web page.)