I cannot differentiate between Instrumentation and Virtual machine introspection. Since the goal of Virtual Machine introspection is to examine the properties of a target through virtual machine monitor then what is the purpose and use of instrumentation technique.
The terminology is slippery and not completely stabilized; however, we can say that, usually, "code instrumentation" relates to modifications of some target code so that monitoring tools (or other kind of automatic management tools) can be applied on it; this does not necessarily involve a virtual machine. With a virtual machine, on the other hand, the code to inspect and monitor is unmodified and unaware of the fact that it is monitored.
For instance, if you compile C code with GCC, you can "profile" it (performance monitoring at the granularity of individual functions) with the
-pg command-line flag: this flag instructs GCC to add extra instructions in the binary which record function entry and exit, for each function. This is instrumentation. It does not need any kind of virtual machine.
To make a metaphor, virtual machine introspection is like a zoologist crouched on a rocky hilltop, with powerful binoculars, observing from afar a lion stalking a herd of zebras. Instrumentation is more like the lion being sedated, strapped on an examination table, and the zoologist forcing an endoscope in its throat. Not the same activity, not the same goals, not the same kind of harvested data; and instrumentation is a bit more "invasive".
The lion simile is a great one. An interesting point that should be included is that introspecting on a VM is much more difficult / target specific than instrumentation. As tom-leek says above, instrumentation adds to the code under inspection where introspection does not. This is a big problem for introspection.
The introspection APIs for a hypervisor like Xen don't do a whole lot more than allow one VM to inspect / map memory that belongs to another VM. Without any instrumentation to assist the introspecting program it must be able to deduce for itself what is in the memory it is observing. This would be like the zoologist in tom-leeks example having to determine what a lion and a zebra are, maybe never having seen either before (not a zoologist then?). This problem has been termed the "semantic gap".
Thus the value in an introspection engine is its ability to determine what it's looking at. Typically it will have to know something about the structure of the binary program it's watching. Most systems use a system.map file to get data about the Linux kernel and then select specific data structures (process list etc) to monitor. For Windows it's a completely different story and the companies that do introspection on Windows typically consider their methods to be extremely valuable intellectual property.
There was a great company called Virtuata that did introspection on the Windows kernel using Xen. Unfortunately they were acquired by Cisco last year and have since disappeared: http://www.cisco.com/web/about/ac49/ac0/ac1/ac259/virtuata.html
Good reading on this topic:
- Linux Kernel Integrity Monitor (LKIM) - http://www.techrepublic.com/resource-library/whitepapers/linux-kernel-integrity-measurement-using-contextual-inspection/
- Bromium has a marketing blog which plays up the limitations of introspection (aka the semantic gap). It's a good read but keep in mind their goals and the product they sell - https://web.archive.org/web/20150612190727/http://blogs.bromium.com:80/2012/10/01/mind-the-gap-the-limitations-of-vm-introspection/
- There are also a number of papers in which academics attempt to "bridge" the semantic gap with automated methods. I don't know many of them but Virtuoso is a good read - http://www.cc.gatech.edu/~brendan/virtuoso.pdf