Testing for ECB / CBC / OFB / CTR mode is fairly straightforward. It's also straightforward to see if the mode has authenticated encryption like EAX / GCM (though it isn't straightforward to see which mode of AE).
- ECB: have a file with many similar blocks in the plaintext. Do you see identical blocks in the ciphertext? (Yes this is similar to Dan's answer, last part of Adnan's answer, but included for completeness).
- CBC: Encrypt a file, and then flip a bit in the IV (typically in the first block of the ciphertext); say you flipped the fifth bit of the first block. Do you see an identical flipped bit in the first block of the plaintext after decrypting? Similarly, flip a random bit later in the file say the 8th bit in the 10th block of the ciphertext. When you decrypt do you see the 10th block as gibberish, and then the 8th bit of the 11th block flipped? Again, consult the modes of operation diagram for CBC decryption and the reason should be clear.
- OFB/CTR: Test if you have a stream cipher. Without touching the IV / Random Nonce (typically the first block), if you alter a bit later in the ciphertext, do you see same bit flipped in the decrypted plaintext? If so you have a stream cipher; an important difference from CBC is in this case there's no gibberish block. I agree with CodesInChaos's comment that there's little security difference between OFB/CTR. If you have the encryption key1 call it K, you could differentiate between OFB / CTR. E.g., let
c be the first and second block of ciphertext and
p be the first block of plaintext. Calculate
D(K, c XOR p) and
D(K, c XOR p), where
D(K, c) is the decryption of block
c with key
K. If they are consecutive numbers, you have CTR mode. If
D(K, c XOR p) = c XOR p then you have OFB mode.
- EAX/GCM - Both are authenticated encryption -- they contain a message authentication code (MAC) and the file will not decrypt (even to gibberish) if the file is altered as the MAC will not be valid with overwhelming probability.
Granted, a lot of this assumes things like you know the block size, but that should be easy to figure out. E.g., encrypt a very small (1 byte) message. Then assuming there's no MAC involved, you'll likely create a two block message (IV + one-block of padded ciphertext -- though CTR/OFB may not pad); then encrypt a longer message. From trial and error, you should be able to find out the block size and IV size.
EDIT: I definitely agree with the excellent point Thomas Pornin brought up. This sort of reverse engineering could help uncover the block cipher, but doesn't really test security. There are many backdoors that could have been covertly put in that you can't detect without painfully reverse engineering, by stepping through the assembly. For example, you could believe you have an Authenticated Encryption block cipher that can only be decrypted with your secret key constructed off your passphrase. But really the file is encrypted with a random key, that is itself encrypted at the beginning of the file using both your passphrase, and then once with their backdoor key. Thus, they would be able to decrypt any file you use. Or maybe the scheme does this same scheme, e.g., at the beginning of the file store the per-file-key to be decrypted using your secret key, but there's only about a billion or so keys (that they know all of them). Or maybe its encrypted with only your key, but the key derivation function only uses the first ~30 bits of the hash of your passphrase as your key. Hence, it could be very quickly brute forced with a billion attempts. Similarly, it doesn't have to be a deliberate backdoor; it could just be improper subtle implementation flaw allowing for side-channel ataacks.
1 This was not assumed for other parts -- there are a lot of ways you could know to enter a passphrase to the decryption system, but not know the key derivation function (or encryption function) used internally to generated the key.