How can I check if an extension/addon consist of malware code? Is it easy to detect if it collects sensitive data (like spyware), logging keystrokes and so on? Can AV recognize malware extension as "bad"? Can an extension exploit browser vulnerabilities?
I'll answer these separately:
Usually browser addons exist as a file archive somewhere, within your browser's application files. You can find out where by doing a quick Google search. Once you've found the file, upload it to somewhere like VirusTotal. You could also extract the files (most plugins are just renamed .zip or .tar files) and manually dig through their code. Certain browsers also load DLLs as "browser helper objects" (BHOs), which can contain malicious code. These can be found and scanned in the same way, and Autoruns will tell you which BHOs are loaded into IE.
Depends on the malware. Keyloggers usually hook APIs related to processing keystrokes, which most AV software can detect or prevent. There are other methods though, so it's not possible to catch all logging mechanisms. Spyware almost always phones home at some point, so you can keep an eye out for unusual network connections in your firewall logs.
Most can, yes. Some will only process executables (exe, dll, sys, etc.) on standard scans, but you can fix that by running a deep or full scan. On-access scanners should pick out known malware in any file the moment it's accessed.
Yes, but sometimes it's not even necessary. In older (and crappier) browsers, a plugin is already native executable code running at the same privileges as the browser. Modern browsers tend to implement their plugins in a sandboxed scripting environment. There may be vulnerabilities in those scripting engines that allow the malware plugin to escape and escalate.