0

I've heard of tools that could be used to graph entropy of a file. Is there a graphical Linux program that I could use for this job that would let me conveniently explore which blocks of a file have certain entropy patterns that could suggest compressed or encrypted data?

1
  • What do you mean by 'entropy'? Have you heard of REMnux? It's a Linux distro tailored for Malware analysis. It allows you to look at how files are made up, but I'm not sure if that's exactly what you're after? zeltser.com/remnux
    – TimC
    Commented Jul 23, 2014 at 9:18

1 Answer 1

1

There is no such thing as "entropy" of a file. The entropy is a characteristic of a process -- for instance, of a process that generates a file. The entropy does not describe what the file contents are; it describes what the file contents could have been. As such, any tool which purports to identify "entropy patterns" is, at best, loosing extremely poor terminology.

That being said, one can still do some heuristics. It so happens that most "real-life data" exhibits a lot of redundancy in its structure; this is what data compression is about. So you may work on the following assumption: data which can be compressed by a non-trivial amount is not data which has already been compressed or encrypted. Indeed:

  • Compression algorithms strive at identifying and removing redundancies. The result should not be susceptible to further compression; otherwise, this means that the compression algorithm does a poor job.

  • Encrypted data is supposed to be indistinguishable from randomness (if the encryption algorithm is any good); thus, it has extremely low probability of being compressible.

So you could extract chunks from your file, and compress them (e.g. with gzip), to see if they can be substantially reduced. For instance, this command extracts a 2000-byte chunk from file data.bin (at offset 15000), compresses it, and reports the resulting length:

dd if=data.bin bs=1 skip=15000 count=2000 2>/dev/null | gzip | wc -c

If the resulting value is substantially smaller than 2000, then the chunk, most probably, contains data which is not encrypted or already compressed. For instance, if I use as data file the /usr/bin/gcc-4.8 file (an executable full of code, but not compressed) from my Linux laptop, I get an output length of 1491 bytes for that chunk, which is indeed quite smaller than 2000. If I try on /dev/urandom, I get 2023 bytes.

(These figures get printed on my screen as sequences of characters, that I process with my eyes and brain. This, in my view, qualifies as "graphical".)

Running the command in a loop which extracts various file chunks and compresses them all is left as an exercise to the reader (it is not hard, and if you cannot do that, then chances are that you won't be able to do much with the files anyway; practical data processing requires some programming skills).

Caveat: gzip, like most general-purpose compressors, works on a per-byte basis, and tries to find repeated patterns and biases in usage of byte values. This is known to work poorly on analog data which has been digitized, e.g. audio samples. Compression ratio offered by gzip on WAV files is thus often poor, while the WAV file is still "not compressed" (this page claims to have measured a compressed length of about 92% of the original, though it will depend on the WAV file characteristics).

So don't imagine that the "compress by gzip" test is the ultimate detection method for compressed/encrypted data. Still, that can give good results, provided that the compressed/encrypted chunks you are trying to locate are long enough (you won't find 50-byte chunks that way).

1
  • I didn't mention that I consider compressing with gzip as a good way to detect if the file is already compressed. Still, thanks for the writeup, though it doesn't really answer my question.
    – d33tah
    Commented Jul 23, 2014 at 13:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .