Can any application install a kernel driver? if not, then how Windows decides who can and who cannot?
If AVs can install kernel drivers, considering they are an application with an installer just like the rest of apps, then what stops malware from installing a kernel driver and bypassing the protection of the AV?
On Windows your require local admin permissions to install drivers. I.e. either running as administrator or the user allowing via a UAC prompt.
If malware has that then yes it could install its own. This is why generally you should use a low privilege account and only offer admin passwords in prompts that are expected.
Antivirus core runs with system permissions. That is the highest possible. But even at this point windows will not allow you to install any driver. Antivirus companies sign their drivers with trusted certificates then send those drivers for further testing and double signing at Microsoft. Only after Microsoft put their signature on the driver it can be run in windows.
Windows protects the OS by requiring a digital signature for the following drivers:
- Kernel-mode device drivers
- User-mode device drivers
- Drivers that stream protected content.
The digital signature is an Authenticode certificate used together with Microsoft's cross-certificate for cross-signing. This ensures that (a) the driver has not been modified/corrupted (the signature), (b) the driver is authorised and trusted (to some degree) by Microsoft (the cross-signing).
You can read a ton of useful info about driver signing here.