If we want both encryption and compression during transmission then what will be the most preferable order.
- Encrypt then compress
- Compress then encrypt
You should compress before encrypting.
Encryption turns your data into high-entropy data, usually indistinguishable from a random stream. Compression relies on patterns in order to gain any size reduction. Since encryption destroys such patterns, the compression algorithm would be unable to give you much (if any) reduction in size if you apply it to encrypted data.
Compression before encryption also slightly increases your practical resistance against differential cryptanalysis (and certain other attacks) if the attacker can only control the uncompressed plaintext, since the resulting output may be difficult to deduce.
|show 3 more comments|
If you compress after encryption and the compression does any good (i.e. it really reduces the length by a non-negligible amount) then you can ditch the encryption, it is awfully weak. Encrypted text ought to be indistinguishable from randomness; even badly encrypted data cannot usually be compressed.
Therefore, compress before encryption. This is why protocols which deal with encryption usually include some support for compression, e.g. OpenPGP (section 5.6) and SSL/TLS. In some scenarios, compression can leak information about confidential data (because compression reduces length depending on the data, and encrypted length more or less matches plaintext length); this is the idea behind the new CRIME attack on SSL/TLS.
Fringe exception: if you encrypt a message with OpenPGP and then "ACSII armor" the result, i.e. encode it in Base64, then this encoding enlarges the data by 34%: 3 bytes become 4 characters (plus the odd newline). Compression with DEFLATE will be effective at cancelling this enlargement (thanks to Huffman codes). That's a case of usefulness of compression after encryption -- but, really, that's more compression over Base64, rather than compression over encryption.
I would recommend to first compress the data and than encrypt it.