Web Application Firewalls relies on negative and positive security traffic rules to protect web application from being exploit. My question is that is their any other web traffic modelling technique that can identify the unintended violation in the workflow of web application
Here are some research papers that tackle this sort of problem and might interest you:
Ripley: automatically securing web 2.0 applications through replicated execution. K. Vikram, Abhishek Prateek, Benjamin Livshits. ACM CCS 2009.
Eliminating navigation errors in web applications via model checking and runtime enforcement of navigation state machines. Sylvain Hall, Taylor Ettema, Chris Bunch, and Tevfik Bultan. ASE 2010.
Using Static Analysis for Ajax Intrusion Detection. Arjun Guha, Shriram Krishnamurthi, and Trevor Jim. WWW 2009.
Enforcing Request Integrity in Web Applications. Karthick Jayaraman, Grzegorz Lewandowski, Paul G. Talaga and Steve J. Chapin. Data and Applications Security 2010.
Modeling user interactions for (fun and) profit: preventing request forgery attacks on web applications. Karthick Jayaraman, Grzegorz Lewandowski, Paul G. Talaga, Steve J. Chapin. PLoP 2009.
The Ripley paper runs a replica of the client-side code on the server, to verify that the sequence of requests from the client could have arisen from a legal interaction with the system.
The WWW 2009 paper derives a finite-state automaton that captures all possible legal sequences of requests that could arise (while interacting with the web application through its intended interface), and then uses this to build a runtime monitor that can detect attacks by detecting deviations from this finite-state automaton. Several other works also take a similar approach of attempting to somehow obtain a finite-state automaton that characterizes all legal sequences of requests, and then enforcing the requirement that requests must be consistent with this finite-state automaton.