Let's say I have a linux guest running in xen and I want xen to check the integrity of the guest kernel so that I know there aren't any rootkits, or similar, active.
Is there a way to accomplish this in with xen or other hypervisors?
That exists handily? I'm not aware. However, within the memory space that the VM allocates, the kernel is in predictable location. One could write code which reads the memory and compares the structure to what is expected.
If I were implementing such a creature, I'd focus on following the system APIs and ensuring that they are appropriate. One likely challenge is that different kernel versions will have changes in different areas. You may have to do mapping on a kernel-by-kernel basis.
You may be able to run chrootkit externally to the VM by exporting your filesystems. I've never tried such a thing, but I bet it would make an excellent research project.
EDIT: or read your disk images direct live and use known good hash comparisons from outside the vM. Then your VM continues running, but you have the benefit of the "LiveCD" confidence. There, now I have answers ordered from most esoteric to readily available.
xm dump-core --> xen memory dump
foremost for searching files
BlockWatch monitor's guest OS's by inspecting memory snapshots.
It uses snapshots because they can typically be converted into a common format (MINIDUMP), this is the case for Hyper-V and VMWare.
BlockWatch also has python scripting to automate snapshot/export/memory-scanning/cleanup. The memory validation is done with cryptographically secure hash (Tiger192). Currently it validates Windows 32 and 64 bit OS's.