Do the browser posses an Antivirus, which scans JavaScript code and compare it to malicious codes?
No
What is doing the XSS-Protection header behind the scenes?
Nothing or not much. The feature was either neither implemented or was removed from most browsers. There was never a standard implementation of the feature, so whoever it implemented it did so with their own heuristics.
When actually implemented and enabled it is mostly about using heuristics to detect common reflected XSS attacks. This is done by checking if parts of the URL are reflected in dangerous places in the HTML where they could be part of script execution. In some cases such heuristics could be hijacked to make XSS attacks possible were none was before.
See X-XSS-Protection at MDN for more.
Are there any other layers of security against XSS ?
The best layer is done by the developer using Content-Security-Policy (CSP). CSP can be done to forbid inline script and restrict from which sources script can be loaded and even the external script content itself, which is useful when included script from third parties outside of your own control. This way CSP eliminates the major vectors for XSS: An attacker cannot cause the execution of their own script, even if they manage to place the script into the page, try to include the script from some external site or manage to compromise a third party script source. If you really need to use inline script in your code, then CSP can be used to restrict its usage to sane places.
Note that CSP needs to be explicitly enabled, since it is not enabled with a restrictive safe default for compatibility reasons. It is unfortunately easy to accidentally disable simply by having a typo in the field name or forgetting to put it in the right place. Browsers will not warn you if there is no CSP, so better have some automated tests which really make sure that it is still enabled after you made some changes to your site.
