If your PSK is strong (e.g. 128-bit of entropy) you don't have to worry, as attacking it with brute force or a dictionary attacks won't be feasible.
However, weak PSKs can be attacked by a rogue server.
For IKEv2 (RFC 7296, section 2.15), the PSK is not mixed into the key material used to encrypt the
IKE_AUTH request and the
AUTH payload provided by the client, which incorporates the PSK via a PRF, is sent before the server is authenticated (that's different for username/password-based EAP authentication, where the server is first authenticated with a certificate). A rogue server can, therefore, attack weak PSKs relatively easily after getting that AUTH payload. The request is encrypted, though, so a passive listener can't attack it.
With IKEv1 (RFC 2409, section 5.4) in Main Mode, the PSK is additionally mixed into the key material used to encrypt the third request by the client. But a rogue server can still attack weak PSKs by trying to decrypt the third request with different PSKs until the decrypted data can be parsed as IKE message and the
HASH_I payload can be verified with the same PSK (something similar has sometimes to be done by legitimate servers that have multiple PSKs available for the same client IP). As with IKEv2, a passive listener can't attack the PSK due to the encryption.
Now, for IKEv1 in Aggressive Mode, the server has to provide
HASH_R, which incorporates the PSK, first to prove it knows the PSK, so for your scenario it's safer, as the client won't continue if the rogue server responds with an incorrect hash. However, in general, it is not safe at all, because the hashes are exchanged without encryption and can, therefore, be attacked by a passive listener.