I am scanning browser extensions of Chrome and Firefox for vulnerabilities. Is there any tool which would help me do this?
There is one for Chrome, released mid-last year, called XSS ChEF. Based on some earlier work finding vulnerabilities in Chrome extensions. Unfortunately it doesn't seem to be about finding new vulnerabilities but rather an easier way of exploiting/demonstrating them once you have found one.
I haven't used it myself but based on the way it works (it's just a web server), I imagine it's possible to use it to exploit Firefox extensions too or at least to extend it to be able to access Firefox internal state once you have found a vulnerable extension.
BeEF might be a good place to start looking too.
One similar tool is Qualys BrowserCheck but it announces that checks for plugin vulnerabilities and not for extension vulnerabilities. I have tried it and just detects updates for plugins in my case. Remember that, as @Ladadadada has commented, browser plugins and extensions are different elements.
Qualys BrowserCheck will perform a security analysis of your browser and its plugins to identify any security issues.
2020 was a bad year for Chrome extension trustworthiness, but it also revealed some of the malicious techniques that are being used in the wild. Most common being loading and executing dynamic scripts or conditionally executing obfuscated code when certain conditions are met.
It’s very unlikely that you would find out if the extension is malicious just by performing a static analysis. Otherwise, Chrome Web Store would have flagged the extension at submission time. I would argue that only a security expert-led security audit can truly determine if the extension is secure.