Yes. Because of known-plaintext-attacks, a adversiary that can find out the plaintext, could use that to find out the hash to decrypt ciphertexts that the adversiary does not have the plaintext to.
AES is specifically built to prevent a attacker from finding out the key, even if he knows both the plaintext and ciphertext.
If you want to gain performance and still use XOR, it would be suitable to add random charachters to the password, and then include these chars with the message.
Eg, to encrypt M, you generate a random nonce N.
Then you use:
PBKDF2( Password + N) xor M = C
Send NC to recipient
NC is decrypted by picking out the nonce, and then the recipient supplies the pre-shared password, so
PBKDF2( Password + N) xor C = M.
To find out the key for a message encrypted with XOR + PBKDF2(Password + A ), and have a plaintext and ciphertext belongning to PBKDF2( Password + N ), he would still have to crack the PBKDF2 by using plain bruteforce.