"the only thing that is important to get in order to decrypt an encrypted file": I think this is where you go wrong in your reflection, if I can say so.
The only thing which is important is not for you to decrypt your encrypted file, but it is to prevent other to decrypt it by guessing your password.
Therefore, a good key derivation function will make it computationally hard and slow to derive a key from your password so relatively few different key candidates can be tested by an attacker over time. The answer you link mentions that GPG implements this by using a large number of iterations to derive the actual key from the password.
If the key derivation function does not take this precaution, then an attacker will be able to test a very high number of password candidates very quickly (the answer you link states "several dozens of millions of potential passwords per second (hundreds of millions will be achievable with a GPU)", at this speed one can expect to cover all 8 alphanumeric characters strings in easily less than an hour, so the password does not even need to be a guessable dictionary word to be crackable).