Lately I've been reading about things like BadUSB and RubberDucky which are essentially USB sticks that tell the computer they are a keyboard. Once they are plugged in they "type in" whatever commands they were told to execute. My question is, why are keyboards automatically trusted in almost every OS? For example, if an OS detects a new keyboard plugged in, why not pop up a password prompt and disallow that keyboard from doing anything until it enters the password? It doesn't seem like this would create a ton of usability issues. Is there a reason why this or another protection measure isn't used?
The trust model for a device you plug in to your computer is just inherently difficult. The USB standard was created to allow literally anyone to create a USB device. Security wasn't a factor. Even if it was, where do you place the trust? The movie industry tried this model with HDMI, and it's essentially failed miserably. You can't simultaneously give someone a device that does something, and prevent them from understanding how to do the same thing.
Your example proposes to put the trust in the user. The most obvious problem is nobody wants to type in passwords just to use a keyboard. Barring that, would it really solve anything?
The user already trusts the device, otherwise they wouldn't be plugging it into their computer. Since trust has already been established, why wouldn't they simply do whatever is required to get it to work?
For a start, keyboards tend to be trusted from a lot earlier in the boot process than the OS - if you have a BIOS password, or a Bitlocker key, you'll enter that before the OS has loaded, using the keyboard. In fact, a particularly malicious keyboard could do pretty much anything to prevent the OS from loading, up to, and probably including, pretending to be a bootable drive, and starting up a rootkit before letting the OS start.
You could also extend the same rules to mice (they could click on a predefined set of points to open the virtual keyboard, then type whatever they like).
Alternatively, you could decide that you will only use devices you trust, and accept the slim risk of bad things happening.
The answer is Usability
How should the user give consent that the mouse/keyboard is trusted? With the keyboard/mouse which could be malicious? How do you handle the case when one has to swap/replace the keyboard? Especially in server-scenario you have multiple keyboards/mouse stored somewhere else and you grap the next best when you need physical access to the server. You will not remember which keyboard belonged to which server after months/years and the keyboard might even get destroyed. How to use the replacement keyboard? Give your consent with the unknown keyboard? How to do this with the first keyboard? Let's say you try a new PC of a friend and use your own keyboard and then give it to your friend. How should he give consent to his keyboard? Edit: You could ask for a password before first use but see my one but last paragraph.
So basically the unsolved question is: How can the computer establish a trusted/secure connection to the user which cannot be faked/circumvented by other hardware/software/bad guy/... in an easy usable way?
Rule 3 of the 10 Immutable laws of computer security is: "If a bad guy has unrestricted physical access to your computer, it's not your computer anymore." If you put in a BadUSB from a bad guy you are the minion of the bad guy and give him physical access to it by proxy. Notice that there are similar worse attacks than bad USB. For example putting a device from a bad guy into a FireWire or other DMA interface lets him read/write any memory and run any code and even circumwent lock screens of Windows/Linux/Mac. So best never put an untrusted device into your computer.
Edit: Because of this rule and such attacks were not thought of when the standard was thought of (physical security was less important at that time except in cases where physical access was restricted anyway), something like came never part of the standard. There were already many easier possible attacks with physical access so it was not worth to consider such a small edge case.It would have massivly increased the complexity of the system, especially if the autorisation has to be shared between multiple operating systems and the BIOS ("Press F10 for BIOS") and how many to store. The next problem arises when deciding where to display the password, especially if multiple monitors are detected like a defective laptop screen. All this also would have had negative impact on the acceptance from users and an easier to use standard might have become the standard instead. Since the devices are produced by economic working companies increased complexity (=cost) and lower acceptance (less pieces sold) this slim edge case would not have been important at that time.
There is specialized software on the market which let's you define trusted USB-devices for corporate high security environments but because of the points I mentioned it is not in broad use.
The question always seems to be between security and convienience. With the HID attacks the balance seems to be in conveniences favour due to the physical access needed for these attacks. Obviously this could be implemented but there doesnt seem to be a need to do this at the moment, why add extra code and issues if the threat is minimal at the time.
The OS knows nothing about the world outside of itself. It is naturally designed to trust hardware, because it has no way of verifying if the hardware really does exist. In fact, if you were to compare the concept of an OS running on hardware to the movie The Matrix, you'd pretty much be spot on. The OS is simply a collection of bytes that are eventually processed by the hardware. It may be running on a piece of real hardware, virtualized with other OSes that are equally unaware of each other, or even physically distributed across multiple units of hardware that act as a cohesive whole. The only real requirement is that the hardware acts in a way that is consistent with how the OS believes it should behave.
At the end of the day, the OS cannot exist without the hardware, and is utterly dependent upon the hardware to tell the truth. While some progress has been made towards making more secure systems (e.g. when they started restricted how PCI buses can use DMA), those are still mostly hardware solutions to security. The OS can refuse input from USB devices, but it can't reliably determine what a USB device is by examining it, because the device can lie. It can identify itself as whatever it wants to, and the OS can't do anything about it. Any verification of hardware would have to come from more hardware. All the OS knows is that it's receiving a signal on a known bus that conforms to a known protocol. You could easily emulate that using any other sort of software running in a hypervisor, for example; the OS can't tell the difference.
We can certainly do things to make things harder for malicious software, such as requiring some type of encryption chip with asynchronous keys, perhaps using a blacklist/whitelist key system, which would protect against casual acts of maliciousness, but that would only hinder development and raise the cost of new hardware, frustrate consumers, and lock out competition that is unable to get on to the whitelist or is even actively blocked by a blacklist. There is no perfect solution to the problem, and any reasonable solution would need to be done at the hardware level, since the OS can't readily determine if the hardware is what it says it is.
My motto of software is 3-D :
Every component MUST do what it's supposed to do and nothing extra/else, because you don't expect your fridge to open your beer. OS must provide a unified environment and API consistence. Including for the actual protection from bad usb, but the protection itself must be a module, an extension based on commonly-accepted API's. That's it - it's an architectural question.