Information Security Stack Exchange is a question and answer site for information security professionals. It only takes a minute to sign up.
Sign up to join this community
For school, I have to do an exercise in which I have to decrypt files by brute force attack.
There are a lot of different files in different file formats. The files have been encrypted using XOR or the caesar algorithm.
I know how to try every possible key to decrypt the files but, how can I know if the file is being decrypted with the right key or not?
You really can't, if you're just encrypting / decrypting text.
If you know that the encrypted string is "kdo" and the encryption method is a Caesar shift, the plaintext could just as easily be "IBM" as "HAL". You'd have to have some idea of what the plaintext "looks like". For instance, if you know the plaintext is the name of a Stanley Kubrick character, you'd have a decent idea of which one it should be.
If you have a longer string, it's much easier to narrow things down. A large text file has much fewer intelligible results than the three-character example above. But you'll still need to determine whether it's decrypted yourself.
On the other hand, if you're decrypting an entire file in some specific format (.docx, etc), you can be reasonably sure the file is decrypted if the parsing program (Word, etc) can read it.
You absolutely can tell with varying degrees of certainty if a file, or even string, was successfully decrypted. Most of the challenges at cryptopals depend on it. I have begun to make a tool for ciphertext bruteforce and analysis that automates this very task. You can find it here if you want to take a look.
(it needs a lot of cleaning up, don't judge me)
My goal originally in this project was to improve my efficiency in CTF crypto challenges with a simple brute-force tool, but I'm starting to work on implementing a lot more analysis. As it stands, it can bruteforce all caesar, single-byte XOR, atbash, and a few encodings, with repeating-key XOR developed but not integrated yet.
The way it works now
The one thing that makes this process finicky is how the thresholds for English detection must be adjusted depending on ciphetext length. It defaults to requiring 60% of the cleartext to be words and 75% of the cleartext to be letters to register a match. This setting rarely gives false positives, and even less frequently false negatives, on medium to long length cleartexts (anything over a few strings). When used on short length ciphertexts however, some false positives will pop up and many false negatives will get by. In testing, I have had to lower the thresholds by 30% or more to detect a match on some short strings, and in the process generate many more false positives that I have to sift through to find the real match.
I strongly recommend working through the cryptopals challenges from the beginning if you are interesting in learning more about making oracles and breaking crypto. It starts easy and progresses into real-world attacks, like making a Bleichenbacher Oracle, part of what makes the DROWN attack work.
you need to make a module that detects English and apply it to the result of every decryption attempt. or just fork mine and make it better.
in cases where the cleartext is not going to be English or another language, some more advanced analysis is required.
If you have some idea what the cleartext is, you can use that knowledge guess when you've might have cracked the ciphertext.
If you think that the cleartext is english, for instance, start looking for english words in your decrypt attempt.
If you think the cleartext is a zip file, zip files have a signature at the beginning of the file. Look for that signature.
If you think the cleartext is an email, look for telltale email headers.
In general, you could try to look for the "information content" of the decrypt attempt. Plaintext normally has a lower information content than ciphertext, though this isn't true for a simple caesar cipher.
But you need to start with some inking of what the cleartext might contain, even if (as above) it's merely "a lower information content score than the ciphertext".
The other answers here are excellent for the general case. For a class assignment, I suspect that the professors want to make it very easy to tell when you have the file decrypted.
For example, are the files you're decrypting text files that show a message when they're correctly decrypted? If so, you can take a dictionary in your language and check wether a large proportion of the words in the message are in the dictionary. If you're using English, I would guess a threshold of 50% of words being in an English dictionary as a good number to use - an incorrectly decrypted file will be almost completely gibberish, but you don't want to ignore the correct plaintext if it has a few words, like proper nouns or technical jargon, that don't happen to be in your dictionary.
If the encrypted file is a text you need to check if the phrases inside the file have a sense or not, but if you need to discover something that it's not a word or a phrase (es. password or random letters) you must try all different possibilities. For example, I have a login password (random letters) encrypted with Caesar cipher. I must try, in the worst case, all 26 possibilities (assuming we are using lower case international alphabet) to check if the password is correct.
Many file formats start with a magic number (a byte sequence) that you can look for. Wikipedia has a list.
Other answers have covered most aspects, but another point is that, if you have access to the API that created the encrypted files, you could encrypt your own know file and brute force it (here you have something against to compare).
Once you get the key use it on the other files.
