I've been playing with mod_security for some time and I'd like somehow to use it in my master thesis. I don't know exactly how this work could be more interesting than describing what mod_security does, what kind of web attacks can be prevented using it, what are other functions of it (like analysing if the requests are really HTTP, analysing XML). I'd rather not focus on this particular tool, but create some system that would secure for example some specific kinds of websites. What could make it more reaserch-like? Any hints please?
Maybe look into creating or applying advanced analytics to web application activity to profile users for the purpose of identifying potential bad activity. Build on top of what ModSecurity outputs and test algorithms/models that make it easier for a security operations team to quickly understand what activity a particular user is doing and why it may be hostile (talking beyond reporting simple atomic events).
Do an amazon search for “machine learning security” or “data mining security” and find books, for example, like Statistical Techniques for Network Security: Modern Statistically-Based Intrusion Detection and Protection http://www.amazon.com/gp/product/159904708X/ref=wms_ohs_product and research which techniques described in those books may advance web application security. There seems to be a near infinite way to mix and match algorithms - your research could result in new ways to combine existing algorithms to produce better results.
Reading Robert Hansen’s blog entry on web log forensics may spawn additional ideas: http://ha.ckers.org/blog/20100613/web-server-log-forensics-app-wanted/
If you've not looked at it already the OWASP Modsecurity Core Ruleset Project, has some interesting work being done on Modsecurity.
Maybe, instead of only looking at what it does and what it prevents, consider taking a look at what it does not do, and what it cannot prevent.
From that point, you might want to consider generically looking at all web application firewalls, and what is possible (and what is not) to block using current technologies, maybe broken down according to the different classes of engines.
E.g. ModSecurity is purely syntactical, as opposed to e.g. Imperva which is behavioral (or at least claims to be so). Each type of engine could hypothetically detect and block different types of attacks. And, is completely impotent in regards to others.
Start by reading through the academic literature in this area, to understand the work that has been done so far and where the state-of-the-art stands. You might start by looking through the proceedings of RAID, DIMVA, and Usenix Security to find papers that sound relevant to your interest. Read those papers, the papers they cite (if they sound relevant), and the papers that cite them (if they sound relevant; Google Scholar and Citeseer are your friend, to help you find what cites what). That should give you a good sense for where the gaps in the state-of-the-art are.
Also, I recommend that you talk this over with your research advisor. It is your advisor's job to help you select a master's project that will advance the state of the art and that is in your rough area of interest.