In Firefox or Chrome, is it possible for a web page script to compromise a user installed extension?
Absolutely, extensions are getting their fair share of vulnerabilities. What bugs you need to look out for depends on how the extension interacts with web content. E.g., the LastPass addon for Firefox was shown to be vulnerable multiple times. Due to one of the flaws it was possible for a prepared website to exfiltrate the stored account credentials belonging to different domains by confusing the extension's URL parser.
What measures can be taken by extension developers to reduce the attack vector?
Just like with regular web applications, you need to validate user input, prevent injections, verify origins, etc. - you can't give a complete list here. Since browser extensions are often implemented using web technologies (JS, HTML, ...), most of the typical web application vulnerabilities apply. Also note that extensions often operate with higher privileges than web scripts which means that an XSS flaw in a Firefox extension can potentially compromise the host.