Short answer, yes.
One reason is that you're increasing your attack surface without maintaining part of it. Another is that adversaries continue to try decades-old exploits for exactly that reason. Firefox isn't a firewall or IDS/IPS designed to protect the soft insides of your system. Rich and interactive, it's designed for a pleasant experience, with sandboxing and other protections bolted on, relying on things it doesn't control like: games, plugins, the network layer (HTTP2 will blur that a little), feature development, and interactions with the OS (filled with IE libraries), etc.
@pdubs' advice to remove IE works from that perspective. Exploits often will just detect a Mozilla browser, drop a 0-day through Java/Flash/etc, then fork off with an old exploit you should have patched.
Removing IE may not remove libraries Windows needs to render HTML; default browser settings don't affect the whole-system's rendering engine. Firefox will still have all its features, and while Windows warns about untrusted downloads and privilege escalation, it's not foolproof. I've seen system compromises where IE and Firefox were configured to use system proxy settings (which is not Firefox's default setting, thanks @void_in). The problem probably came from a download that compromised one of Windows or IE, added malware to IE (if IE were hardened it should have blocked that), and then proxied both browsers under adversarial control.
Here are some things IE's Enhanced Security Configuration changes, because IE is so deeply involved with the OS:
- Local Intranet is restricted, because it can share your credentials, affecting access to UNC shares (network and local). Standard Windows management tools are added to this zone because the browser changes actually restrict Windows management.
- Trusted Zone is changed, affecting all applications, not just IE. Windows Update and Error Reporting are added to this zone, because now they also need additional permissions.
- ActiveX is disabled. It blocks running programs from the web, Windows component installation, etc. This is such an important area that Microsoft additionally has "killbits" to disable specific components.
- Scripting (not just one type) is disabled; rich Windows applications that need web scripting might not function.
- In anticipation of future threats, changes include disabling 3rd-party plugins (the same type of thing left enabled in Firefox), installing IE and Web components on-demand, disabling the just-in-time compiler / virtual machine, media content playback, music, animations, video clips, certificate revocation, enables program identity verification, disables saving secured temporary data and clears temporary files on exit.
It reduces functionality that affects the Web, Web-based applications, local network, help, support, and general user assistance, because security is more important than being a browser.
And that's just halfway down the page.
Windows Firewall can give you an idea how IE Zones interact within Windows and your network. The number of settings might help see why you want to use some lockdown tools, but realistically...
Your Joe Sixpack may be frustrated with enhanced security while still engaging in risky behavior (like an unverified codec). Defense In Depth (OWASP) here might consist of normal patching and mitigations specific to the deployment. You might keep a standard configuration, adjusting for changes, and then reimage and patch back up occasionally. The system's going to break anyway (The Inevitability of Failure...).