Write a script that tries decryption with
openssl using all of the block cipher modes. I would consider using
openssl in a BASH script or importing
subprocess) and using try/except blocks in python (or the equivalent in whatever language you are most comfortable in) to get the right mode quickly.
Write a detection oracle.
If you have not read through the Matasano Crypto Challenges before, I would recommend starting at Set 1 and reading about implementing and detecting ECB. Set 2 goes into implementing/detecting CBC and also creating a CBC/ECB detection oracle. Once you understand how the block cipher modes work, it becomes easier to understand how to differentiate them from eachother.
Unfortunately this is where my knowledge currently gets hazy on the subject, so if an ECB/CBC detection oracle does not identify the ciphertext, I would suggest you take a look at this good answer to a similar question. The user explains what you would need to look for in order to identify OFC/CTR and EAX/GCM as well, which you could then use to create a more complete detection oracle.
I have been trying to tackle the problem of identifying block cipher modes of encryption, given ciphertext-only, over the last week. The problem with the answer I referenced (in relation to this goal) is that it assumes knowledge of the key/IV to be able to analyze error propagation. I have read through every cryptography book I own, searched the interwebs, and have asked a question specific to this on crypto.SE.
From this research so far, I have tentatively come to the conclusion that:
- ECB can be easily and consistently identified, given a ciphertext of a large enough size.
- Stream Ciphers (CFB, OFB, CTR) can easily be identified if the ciphertext is not an appropriate block length, however I have yet to find a way to differentiate stream ciphers from each other given ciphertext-only.
With this information, a ciphertext of block length (AES=128) can be assumed with ~99.22% certainty (1/128 chance of stream ciphertext being multiple of blocklen = ~0.78 chance) to be a block cipher. From here, the aforementioned detection oracle will be able to detect ECB if it is used. If ECB is not detected, you are left with CBC and PCBC as the two main possibilities. This is the best detection analysis I have come up with so far.