TL;DR: Encryption can be harder to crack because you may not know the plaintext to verify that you found the right encryption key and because encryption may require you to convert the key with similarly expensive algorithms as used for authentication plus the actual encryption itself.
Whether encrypted data or passwords are harder to derive depends on what was encrypted with which algorithm and how passwords are stored for authentication.
Passwords are usually not stored as readable plaintext for authentication but rather converted to hashes or digests with one-way cryptographic hash functions so that you cannot infer the password from it.
Ideally, the only way to get the original password is to brute-force it by hashing all possible or likely (maybe dictionary- or rule-based) passwords and comparing them against the given hash.
Also, it is good if it also requires lots of resources (computing time and/or memory) to get the password hash from the plaintext so that guessing becomes infeasible.
That said, encryption itself consists of a key derivation step and a reversible encryption step.
The key derivation transforms a user-defined key into a key that is used for encryption. The key may be denoted as a password.
In encryption context however, passwords are often termed passphrases because it is more likely that you use not just a single word but some longer text to encrypt while authentication passwords tend to be shorter for historical and practical reasons -- you use them much more frequently in most cases, and older UNIX or Linux systems and even Amazon, for example, restricted passwords to 8 characters some years ago.
Alternatively, you will very often find that key is a more frequently-used term for the secret used to encrypt data because it is more agnostic about the secret's structure.
In the simplest case, the key is used as-is to encrypt data (think of Vigenère encryption).
For better security however, passwords are often converted to hashes just like for authentication.
This is called key derivation.
Key derivation functions can also be used to generate password hashes for authentication.
Another reason for not using passwords directly for encryption is that many modern encryption algorithms require keys of fixed length, usually because they are block-based (the key does not necessarily have to match the block size, though, e.g. AES-256 which uses 128-bit blocks with 256-bit keys).
Many encryption algorithms do not use the user-specified key or anything derived from it for encryption: They derive a "key encryption key" from the user secret which is used to encrypt a data key which is actually used to encrypt data.
The data key should ideally be a random data sequence not relatable to the user, system, plain-text data, or anything, which means you cannot infer the user secret at this point.
The data key may be encrypted with slow asymmetric crypto-algorithms such as RSA.
Combining asymmetric encryption with symmetric algorithms is called hybrid encryption or a hybrid cryptosystem and may allow anyone to encrypt data while restricting decryption to a small set of users.
Another benefit of using this indirect approach is that you can encrypt the data key with multiple user keys so that the encrypted data can be made accessible to all those users, and you can just change a user key without needing to re-encrypt the actual data.
In the end, encryption produces ciphertext that is ideally looking like fully random data with maximal entropy and does not give any hints about either the original user-defined key or the plaintext.
This implies that there are as many plausible plaintexts that could be encrypted to produce the same ciphertext, so unless you know what was encrypted, you cannot verify that you found the correct key & plaintext combination when you try to crack the encryption.
This may however be thwarted by checksums, digests and signatures of the plaintext that allow you to check if your guess was correct.
So, unless the encryption algorithm is bad or the key derivation is significantly inferior to the one used for the password authentication you compare against, or you know something about the encryption key or the plaintext, encryption is much much harder to crack as there are more steps involved and a bazillion plausible inputs.