How to find out if a connection encrypted?

Question

I'm sniffing a connection that uses ICEP (ZeroC ICE Protocol), and its packet bodies compressed using BZ2 (GNU BZip 2) algorithm.

I want to find out whether packets encrypted, or not. ICEP can use OpenSSL to encrypt the traffic.

Additional Info

It's a local network, and I use WireShark to analyze the traffic. The client is a .Net application on windows, and Server is a Python application on a Gnu/Linux machine. If they use any encryption, it is possibly the ADH (Anonymous Diffie-Hellman cipher) cipher.

I also tried to use the command # openssl s_client, but it says:

CONNECTED(00000003) 3894:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:

Answer 1

Watch the stream and look for negotiation using defined encryption protocols. If you know the data is compressed with bzip2, look for the strings 0x314159265359 and 0x177245385090. Unless headers are totally stripped out, they'll appear once for every block. You can take a guess at whether data is encrypted by following the stream and checking for entropy. The more entropy per bit, the more likely you're seeing encryption. This unfortunately applies to compression as well.

I would say that you can discern known encrypted, or known unencrypted. Differentiating encryption or compression would take a while and involve more complex code without header information for magic strings (like above) to give it away.