Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

Of course Wireshark can be used to sniff traffic - you can inspect each packet or chain of packets.

But how would you do it smarter? Which tools do you use to identify behaviors or to automate the process?

My goal is to identify the different functions, opcodes and everything I can learn about the protocol and software (I have both the client and the server).

share|improve this question
add comment

1 Answer

up vote 9 down vote accepted

There's no way to easily automate this in any useful way, because protocols are designed by humans. As such, they don't really follow any set pattern or rules. You're going to have to put the brain-work in yourself to dissect them.

However, there are some tricks:

  • Use Wireshark to separate out individual conversations and identify the high-level communication patterns.
  • Use a "smart" hex editor to dissect the packets. Hex Workshop is a good choice (though it's not free) since it allows you to define C-style structs on data.
  • Use a diff tool to identify changes between packets.
  • Use IDA (or another static analysis tool) to identify where network APIs (e.g. Winsock) are used to send / receive data, and work out what they're doing.
  • Use ImmunityDebugger, OllyDbg, WinDbg or another debugger to trace execution in a live environment. This is especially useful when sending test packets to a client or server app, because you can see where validation tests pass / fail and why. Using distinctive values in these packets allows you to see what types of data flow to different parts of the code, allowing you to identify their function.

In the end, you still have to be a decent reverse engineer to work with this. If you've never attempted to do something like this, I suggest you try it with a simple openly documented protocol and work backwards - try to figure out how it works, then compare your observations with the documentation.

share|improve this answer
    
"they don't really follow any set pattern or rules" that a machine can understand. –  Zippoxer Jul 18 '12 at 14:03
    
@Zippoxer Or that a human can understand. There's no "general pattern" that fits all protocols and packet formats ever designed. A human can understand each protocol at a time, but never understand all protocols in general. It's a subtle abstraction, but a big difference. –  Polynomial Jul 18 '12 at 14:07
    
That's right, there's no such pattern, but according to what the program does, human can assume how it does that. For example, if I'd try to analyze an MMO's network protocol, I already know what I'm looking for: login message (must have username & password), walk message (must have relative/absolute target location), talk message (must have text), etc. –  Zippoxer Jul 18 '12 at 14:21
1  
Sure, but that's a miniscule part of the protocol. How is the data segmented? What about movement packets? How does the server respond if you send a packet with invalid parameters? A human can deduce these things, but there's no fixed model regardless. I'm not disagreeing with you in the sense that deduction is possible, but "ability for a human to deduce how a protocol works" is certainly not a pattern of how all protocols work. –  Polynomial Jul 18 '12 at 14:33
    
That was my point. I just didn't explain in my first comment. Recently I comment in short since I know that short comments have higher chance to get massively upvoted :P –  Zippoxer Jul 18 '12 at 14:47
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.