As you said, most drivers run in kernel mode, so they have access to all the interesting stuff and can easily hide from debuggers.
There are more reasons which makes drivers an interesting place:
There is a huge amount of device drivers, and vendors provide new versions of drivers that differ from the version included in the operating system. So unlike core parts of the operating system, it is not easy to manage a database with hashes of known good files.
There are standard tools and documentation for developing drivers. So this is a lot easier than directly patching kernel memory or replacing the boot loader.
Direct memory access in a device driver is not suspicious.
Some drivers are loaded early during the boot process, so they can apply their magic before a virus scanner is loaded. So the virus scanner only sees, what the rootkit wants it to see.
One particular driver type that is interesting to hide in for a rootkit is a file system driver. The rootkit can easily filter the results of directory listings and hide itself from the user and other system tools.
Similarly, hiding in a network driver can filter some connections and remove them from the list of open connections.
at driver level, applications can have full access to the system's hardware. That includes filter drivers that inserts between the HW driver and the OS in a totally transparent manner, making it almost impossible for other applications to detect their presence.
That can be applied to filesystem driver, network driver, input devices (keyboards..).
driver level applications can also be made invisible to process monitoring making them really hard to detect and get rid off.