Let's say I have a linux guest running in xen and I want xen to check the integrity of the guest kernel so that I know there aren't any rootkits, or similar, active.
Is there a way to accomplish this in with xen or other hypervisors?
That exists handily? I'm not aware. However, within the memory space that the VM allocates, the kernel is in predictable location. One could write code which reads the memory and compares the structure to what is expected.
If I were implementing such a creature, I'd focus on following the system APIs and ensuring that they are appropriate. One likely challenge is that different kernel versions will have changes in different areas. You may have to do mapping on a kernel-by-kernel basis.
You may be able to run chrootkit externally to the VM by exporting your filesystems. I've never tried such a thing, but I bet it would make an excellent research project.
EDIT: or read your disk images direct live and use known good hash comparisons from outside the vM. Then your VM continues running, but you have the benefit of the "LiveCD" confidence. There, now I have answers ordered from most esoteric to readily available.
xm dump-core --> xen memory dump
searching for active processes and open files.
foremost for searching files
BlockWatch monitor's guest OS's by inspecting memory snapshots.
It uses snapshots because they can typically be converted into a common format (MINIDUMP), this is the case for Hyper-V and VMWare.
BlockWatch also has python scripting to automate snapshot/export/memory-scanning/cleanup. The memory validation is done with cryptographically secure hash (Tiger192). Currently it validates Windows 32 and 64 bit OS's.
In the time since this question was asked, a few have been released. Of those, only one is in common use, which is RKP from Samsung Knox. It is a hypervisor-based solution that verifies the integrity of the running kernel. It operates by detecting modification to kernel structures and monitoring credentials.
The Linux kernel is in the process of implementing ROE for KVM on x86 systems:
ROE is a hypercall that enables host operating system to restrict guest's access to its own memory. This will provide a hardening mechanism that can be used to stop rootkits from manipulating kernel static data structures and code. Once a memory region is protected the guest kernel can't even request undoing the protection.