I mostly found examples the other way around: compromising the OS e.g. through a compromised BIOS or Intel ME. But are there known examples where stuff runs underneath the kernel and can be used to protect it or the OS in general? Regarding ring -1/ hypervisor mode I found some examples but could not really find anything else. I'm not thinking about SecureBoot et al but more about exploit protection of a running kernel.
Yes, with varying degrees of success. Of these, only the first two are commonly used in production:
- RKP (Real-time Kernel Protection) from Samsung Knox, which blocks unauthorized access or modification to kernel code by userspace, and monitors sensitive credentials in the kernel.
- KDP, a Microsoft technology using a set of hypervisor-based features called VBS.
- Various experimental hypervisors based on BitVisor, such as SecVisor, with varying goals.
- Cappsule by Quarkslab, which runs a CoW copy of the kernel for each spawned process.
- McAfee DeepSAFE, a hypervisor that provides a tamper-resistant view of system events.
- Co-processor-based integrity protection such as the experimental Copilot.
- VBH, an experimental and out-of-tree Linux hardening patch similar to Microsoft's KDP.
I believe there was also some short-lived custom Intel ME module ("ring -3") which was designed for similar purposes, but I forget the name. SMM ("ring -2") could also theoretically be used for integrity, depending on what events can trigger a SMI (System Management Interrupt). Note that these are not "true" protection rings, as I explained in greater detail in another answer.
You're very vague. Narrow it down: which OS? Which processor protections?
If you're speaking Linux, you can use the kernel to subvert the kernel (insmod a kernel module).
Windows is a little less straightforward depending on platform (x64 requires signed drivers, etc.).
Depending on the hardware, Rowhammer, etc. could be used.
Windows: from low level up, most AVs drop drivers to perform I/O Request Packet (IRP) hooking, rewrite/install or hook their own System Service Dispatch Table (SSDT), manipulate the Intterupt Descriptor Table (IDT), or even manipulate all processes' Import Address Tables (IATs), etc.:
You can even view this with the gmer AV tool: http://www.gmer.net/
Linux: I'm not too familiar with RE on Linux, only OSX. However, there may be some merit in a formally verified kernel, e.g. SeL4: https://wiki.sel4.systems/FrequentlyAskedQuestions
I don't think it's possible to protect any layer from the layer bellow it, no matter how much you spend protecting it.
Take the Playstation 3 platform, for example. There were several layers of protection, from the boot ROM inside the chip to the code signing, everything was meant to resist attacks from hardware AND software. It took a while, but hackers started to break the security of the PS3 by using a memory corruption glitch.
The kernel can have routines to detect unauthorized changed memory, but an attacker can patch the routines to always return false. Functions can be signed, but the signature can be forged. Digital signatures depending on public key signatures can only protect themselves until the signatures are changed.