The pre-shared key (PSK) is never directly used to encrypt anything. The PSK is used solely for authentication. By proving to the other side that you know the PSK, the other side knows that you are a trusted peer because only trusted peers should be aware of that PSK. Thus the PSK is actually rather like a real key, it unlocks the door so you can step inside.
The PSK serves the same purpose as a WiFi access password. A WiFi access password is not used to encrypt any data send over the air, you just need to know the WiFi access password as otherwise the access point (AP) won't even let you join a WiFi network. Once joined, a session key is negotiated with your device and this key is used to encrypt payload traffic. It works the same way for IKE. Every time you connect, a unique session key is exchanged and that key will be used to encrypt payload traffic.
Of course, when you connect to a VPN gateway, not only you need to prove to the gateway that you know the PSK, the gateway needs to prove the same thing to you as otherwise how can you know you are talking to a trusted gateway? Thus unlike a login with username and password, you cannot just send the PSK to the gateway for confirmation as then the gateway will of course know the PSK. So the trick is to prove to the other side that you know a key without actually sending that key.
In case of IKE that achieved in two steps: First a keyed hash (HMAC) is calculated using the PSK as the key and a value consisting out of two random nonces, one chosen by your side and one chosen by the other side. It's important that both sides provide a random value so in case the random generator of one side is poor or has been corrupted, the result is still random (random + not-random = random); you never know if you can trust the randomness of the other side but that's okay as long as you can trust your own one. The outcome is then mixed with data from the key exchange and some other protocol fields, another keyed hash is calculated of that and this keyed hash is send to the other side for verification. Verification is possible as the other side will know all the values used in calculation when it receives the hash and only if both sides use the same PSK in that calculation (which itself was never exchanged), their results will match.
The other side will then basically do the same thing, yet some of the values in the second step are used in different order, resulting in a different final hash value. This hash is also send to the other side for verification. The reason for different order is that it would be pointless to require both sides to calculate the same hash, the other side could just send back what it has received, that would prove nothing. The reason for adding randomness is that otherwise one could just capture that hash once and reuse it every time it is needed without having to know the PSK either.
As for the difference between main mode and aggressive mode:
In main mode, the key exchange that results in the session keys used for encryption, takes place before the hashes are exchanged. However all messages are encrypted once the key exchange has finished, thus the hashes are encrypted as well when exchanged. In aggressive mode the hashes are already exchanged while the key exchange is still in progress and thus one of the two hashes is exchanged unencrypted (the other one is encrypted but that plays no role).
How would that be less secure? Well, an attacker who can capture all traffic exchanged can also capture the unencrypted hash, as well as all the data required to calculate it, except for the PSK itself. With that information he can run an offline brute force attack trying to guess the PSK. Of course, if the PSK is a 16 character pseudo-random string of letters and digits, such an attack won't succeed in this century but people tend to use ultra secure PSKs like "1234" or "secret!!!" or common English words that won't stand a dictionary attack.
However, even if you know the PSK, this won't help you to decrypt captured traffic of a previous IKE session. As I explained before, that traffic is encrypted with a session key generated by a key exchange (like a DH key exchange for example) and the whole idea of a key exchange is that even if an attacker can read all data exchanged, he cannot get aware of the final key. Thus knowing the PSK only enables you to do two things:
Connect to the VPN yourself, unless there is an additional user authentication that will prevent that.
Perform a man-in-the-middle attack on an IKE session, in case an attacker cannot just capture traffic but also modify traffic along the way. With such an attack he would become aware of the session key, so he could read everything exchanged and that way also capture user login data when they get transmitted (which can be avoided using a challenge-response scheme or one-time-passwords).
Cracking the encryption of an IKE session means cracking the session key. If you can crack the session key, you can decrypt all captured traffic of an IKE session, yet this won't help you to get the PSK. If you want to get the PSK, even if you cracked a main mode session, you still have to run a brute force attack on the exchanged hashes.
