What are the best tools available in order to reverse engineer a communication protocol?
Assuming it is communicating over a network:
- Wireshark and/or Tcpdump to capture and analyse data.
- Hex Editor to define headers, footers and other data structures within the packet streams. A lot of the traffic will most likely not be human readable characters.
- TCPReplay to replay the traffic, allowing you to modify and re-test.
- WireEdit, allows you to modify PCAP (capture files) and save them.
- Reverse engineering tools for the communicating application such as OllyDBG, Immunity Debugger, GDB.
- A text editor to edit your systems host file. Useful to intercept and reply to communication destined to remote servers. Play back the traffic you want and see how the protocol behaves.