give the attacker an advantage
Depends. What's the attacker trying to do?
Given that you call one of the pieces of data involved “password”, I assume that the goal of the attacker is to find this password, and your goal is to keep it secret.
It's not clear what you mean by
AES.Encrypt(bytesToEncrypt, password). I assume that you mean that you're using one of the standard block cipher modes of operation using AES as the block cipher, with
bytesToEncrypt being the key and
password being the message to encrypt, or perhaps the other way round. Below I'll write
AES.Encrypt(key, message) where
key is the encryption key and
message is the string to encrypt, since that's the order of arguments used in virtually all APIs out there.
If you reveal
SHA256.Hash(password), then the attacker can find the password by calculting
SHA256.Hash(p) for many values of
p until they find one with a matching hash¹. They may either do the calculations themselves, or leverage calculations done by others. This could be devastating if the hash is something like b9f195c5cc7ef6afadbfbc42892ad47d3b24c6bc94bb510c4564a90a14e8b799, less to if it's something like 3d5cfec95acbabf275d544e138fdfa02ad3945adc681dfa0cec75a127a9ff6aa, but a realistic threat for any password that's memorable.
If you reveal
AES.Encrypt(key, password) to the attacker, then the attacker can know two things:
- They will know the exact or approximate length of the password, depending on which encryption mode you used.
- If the attacker also manages to obtain
bytesToEncrypt, then they will know the password.
If you wanted to calculate
AES.Encrypt(password, message), then you'd run into the problem that this doesn't actually make sense. An AES key must be a string of either 16 bytes, 24 bytes or 32 bytes. No other string length is possible: the algorithm just isn't defined for other string lengths. If you use a password that has the requisite byte length, that's a very bad idea because keys are supposed to be generated randomly. If you use a password which consists of printable characters and has patterns that make it memorable, you aren't using AES the way it was designed. You open the way to related-key attacks: it's possible to learn things from relationships between keys, such as “this key consists only of ASCII characters” (i.e. the most significant bit of each byte is 0). AES has known weaknesses against related-key attacks, which is not a problem in practice because no sane protocol involves related keys. Using a password as a key is not a sane protocol, though. In addition, you're obviously revealing something about the length of the password, and you're exposing it to the same kind of brute-force attack as the SHA-256 case if
message is known.
Finding the password through its hash and finding the password through its encryption are pretty much independent. AES and SHA-256 are unrelated designs. Knowing the encryption does reveal the size and therefore makes the search for a password with a matching hash easier, but only marginally so. Knowing the hash doesn't help breaking the encryption, short of actually finding the password.
Revealing either the encryption or the SHA-256 hash of a password is a terrible idea, since each of them has glaring weaknesses (one breaks down if the key leaks, the other is vulnerable to brute force attacks). Passwords should only be stored in the form of a slow, salted hash, and the hash should be kept secret for good measure.
¹ In theory they wouldn't know whether what they found is the actual password or some different string with the same hash. However, there is no known way to find two strings with the same SHA-256 hash, and it would take a brute-force search running on all currently existing computers approximately the lifetime of the universe to have a non-negligible chance of finding one.