How do you analyze an unknown network protocol?

Question

Of course Wireshark can be used to sniff traffic - you can inspect each packet or chain of packets.

But how would you do it smarter? Which tools do you use to identify behaviors or to automate the process?

My goal is to identify the different functions, opcodes and everything I can learn about the protocol and software (I have both the client and the server).

Answer 1

There's no way to easily automate this in any useful way, because protocols are designed by humans. As such, they don't really follow any set pattern or rules. You're going to have to put the brain-work in yourself to dissect them.

However, there are some tricks:

• Use Wireshark to separate out individual conversations and identify the high-level communication patterns.
• Use a "smart" hex editor to dissect the packets. Hex Workshop is a good choice (though it's not free) since it allows you to define C-style structs on data.
• Use a diff tool to identify changes between packets.
• Use IDA (or another static analysis tool) to identify where network APIs (e.g. Winsock) are used to send / receive data, and work out what they're doing.
• Use ImmunityDebugger, OllyDbg, WinDbg or another debugger to trace execution in a live environment. This is especially useful when sending test packets to a client or server app, because you can see where validation tests pass / fail and why. Using distinctive values in these packets allows you to see what types of data flow to different parts of the code, allowing you to identify their function.

In the end, you still have to be a decent reverse engineer to work with this. If you've never attempted to do something like this, I suggest you try it with a simple openly documented protocol and work backwards - try to figure out how it works, then compare your observations with the documentation.