Google Authenticator supports the TOTP and HOTP algorithms.
In both algorithms, a secret needs to be shared between the server and the client to successfully generate the one time passwords. I suggest you read the respective RFCs of the algorithms (or my answer here) to understand it better.
Yes, if the attacker manages to get hold of the shared secret the algorithm is defeated. However, this isn't very practical. The secret is only displayed in the form of a QR-code once when setting up the algorithm. Presumably you will be sitting in front of the screen when this happens and will take notice if someone suspicious walks up and starts scanning the QR code with his phone. There are plenty of physical measures you can employ if this is the case.
Once the secret is shared, it is stored on the server and the client (in this case your phone). At least in the case of Android, the secret is stored unencrypted in a SQLi database file. However, this file can only be accessed by the Authethenticator app unless your device is rooted so it isn't easy to get your hands on it. It should be just as difficult if not harder on an iOS device. You have much bigger worries if an attacker manages to compromise the server.
So yes, the algorithm is defeated if the secret is leaked. But the secret leaking is not an easy thing to happen. In any case, it's TWO-factor authentication. You still have the other factor to protect you.