Let's say I have a file containing a random bunch of bits and then I encrypt it using some modern algorithm (Blowfish, AES, or whatever). If someone captures the file and mounts a brute force attack on it, how will they know they've successfully decrypted it? It would be obvious if the original input was meaningful, but in this case the original input was just a bunch of random bytes. How would they know, if at all, that this was the CORRECT bunch of random bytes?
Generally people do not encrypt random garbage. Assuming they did, a ciphertext only attack would be impossible.
Consider, however, that common cryptographic schemes add to the message non random data, such as padding data and message authentication codes. In some schemes, these can be used to check whether a guessed key is correct.
In general way they can't break the encryption (if you encrypt randomfile (for example secret key) or even compress file) but if sniffer have some other access to sender or receiver system they can do some other type of attack, and can break random file encryption. you read more there:
If the cleartext just random then they won't know. However, most file types have some sort of recognizable structure. Passing blob A through algorithm B with key C will yield some output. If that output has structure to it then you've got a winner.
They don't. TrueCrypt takes advantage of this very fact to offer plausible deniability through 'hidden volumes': http://www.truecrypt.org/docs/?s=hidden-volume
Your question appears twofold to me. First of all, a good encryption algorithm is indistinguishable from randomness. So, nobody will even distinguish your cipher text from random data. Furthermore a falsely decrypted cipher text will also look random. Thus, you will not be able to find the random data originally fed into the cipher with an exhaustive key search.
Secondly the point is that people usually don't feed random data into the cipher. Current encryption standards do not only cover the algorithm but also the data used. People usually pre- and suffix your data always with a certain padding and certain metadata (encryption schemes have a fixed lengh. you should somehow specify how long your text was). Knowing this standard bits will obviously make the decision between the right and the wrong key easier :) While padding serves mostly practical purposes, it may also enhance or even ensure security features (cf. chosen plain-text attack on textbook RSA).
If the file is unicode/ansi/etc, you can make some algorithm to parse something like the 200 first character of a file and see if there are more latin characters than other characters.
I remember I was quite annoyed when I tried the XOR brute force attack on some simple euler-project exercise, but it was easy and I just had to search for common english words.
I read somewhere that in encryption software, the implementation is very much important, sometimes more important than the alorithms. When I read that, I still wonder if one is talking about obvious subjects like pseudo random generators, or rather less obvious details like how to hide the format of the file you encrypt.
For example, if you encrypt a file, if it's a PNG file or GIF file, make sure to remove the magic number/string those file formats contains, and if it is a text file, don't use an ASCII table: use you own character table, for example just put all the latin character at 0, numbers at 245-255, and so on. you could also permutation, or rot13, or else.
Algorithms such as AES or Blowfish/TwoFish are "mathematically" secure, because it has been proven no attacks OTHER THAN BRUTEFORCE has been tested as efficient enough: you can only decipher the text by finding the actual key.
But those algorithms are only theorically efficient, you MUST implement them considering other practice factors such as file size, compression use, text encoding etc.
For example, know that it would be just plain stupid to store the filename in plain text next to your crypted file.
Besides the theoretically ignorant approach by some "pros" in the field, there's a rather simple answer to your question: by comparing the "brute-force decryption results" with obvious stuff for categorization purposes.
Let me give you practical examples: text files will most probably contain "stop words" ( http://en.wikipedia.org/wiki/Stop_words ). Find more than one stop-word after decryption and you've most probably decoded some text successfully.
Media files like images and most other filetypes (audio, video, etc.) all have specific headers, making identification of the kind of data rather easy. Think you've decrypted a jpeg because you detected a look-alike header? Check the file format. Looks like it's correct? Then you've most probably successfully decrypted a jpeg media file.
I could provide books of examples... but I be you know what I mean by now, don't you? ;)
The rest is up to a little "human verification" so you know when to stop brute-forcing whatever you're trying to decrypt.
Is it practical? No. But it can be done when coded correctly and it does make the job easier. I've seen it in action several times; not only in a corporate environment.