It seems that its like a black point in my mind that I think about
browser when I research information security. Its clear that its the
browser applies security rules while we navigate websites. In any
event that if browser failed to apply cookie rules or origin policy;
how we can detect that?
You cant really tell if the browser fails at applying a policy unless the browser itself tells you. If the browser has not been tampered with when you start using it, you can use some third party addons for the respective browsers to give you a fine grained control over these kind of policies.
Firefox has addons like Cookie Monster and Self-Destructing Cookies to make it easier to manage cookies and apply policies on them. I am not familiar with Chrome or IE so I cannot give any example for them.
There are also blockers like uBlock Origin and uMatrix which provide loggers that you can use to analyze what the browser is doing, as well as see what is blocked and what is allowed.
Now, if someone downloaded malicious browser (infected version of
genuine browsers or any other browser that originally designed to
steal user data)
how we can validate trustworthy of the browser and proof that all
security rules applied correctly?
There are checksums available for firefox here you can compare them with the file you download to make sure nothing has been altered from the version published. Its important to note that checksums assume there is no MITM when you are downloading/viewing them. Obviously if someone has a MITM they can just alter the file before sending it to you.
Again, these browsers don't offer any real way to audit them, you need to use some third party tools for that.
Dose anyone validated Mozilla, Internet Explorer or Google Chrome?
Firefox and Chrome (Chromium) are open source projects so they have bug trackers that are publicly visible as well as the ability for anyone to download the source code. The problem is that these projects are so large, its difficult for a single person to audit the code.
If you are worried about exploits on the browser making your data/actions vulnerable then the best you can do is find some addons that help you track the browser, run in a sandbox environment and use programs like apparmor or selinux to prevent the browser from doing things you don't want it to do.