I am working on a scenario if we have two or more web application hosted on a same server. Let says Web application X is vulnerable so how can i be sure that it will not effect the other web application Y. In short my question how can I isolate web application hosted on the same server ? What are the techniques that need to be in configured on the web server ?
If you have multiple websites on the same server then depending on the issue, a security problem with one site may well affect another.
Mitigating this essentially means increasing the separation between the two applications. You could do this by using somthing like a chroot jail or if that's not enough isolation you could look at using VM guests on the same host to provide more isolation.
There are various levels of protection you can use. The most basic would be to configure them as separate sites running as different users and only allow each site's user access to the assets it needs. At a basic level, this should prevent one site from being able to access the other.
There are still some attacks that may be able to escape this though (if for example they are able to escalate privileges). To offer further protection, you can do something like running VMs for each of the sites that will further sandbox each site. There are still some theoretical ways to escape a VM, but you'd be very well protected at that point but also would have far higher resource demands (since two additional kernels would have to be running).
There are also a few other levels you could do in-between that attempt to do a sandbox isolation without actually doing a full VM, but I don't know much about those techniques or their effectiveness. Personally, for all the sites that I have run, simply properly configuring users and only allowing each site access to it's own assets (and keeping the server patched) has been sufficient for my needs.