Y'know I've heard a lot about how virtualization will just fix things. It annoys me a tad - because it's just using the word "virtual" to say "oh it's not real so you can do no damage". Extending from Thomas's answer, then:
The x86 architecture already features an application isolation mechanism called privilege levels which determine what applications can access. No doubt you've heard of ring 0 and ring 3? Well, ring 0 is allowed to modify virtual memory, configure the processor etc. Ring 3 can't. However - yes, you've guessed it, there are two other rings, ring 1 and ring 2, and they remain unused for most operating systems, including Windows.
What does this get us?
Every ring 3 application, hereafter called userland, can do absolutely nothing much useful on its own. It needs the operating system to talk to hardware, to save files, to access the internet and so on.
As such, every executing application in userland must pass through the kernel on its way to greater glory. This allows the kernel to implement access control, to decide what applications can or cannot do.
Applications in userland are also isolated from each other - except by asking the kernel for connections to other applications. Again, this allows for access control.
What we have, as is then, is an isolation strategy for applications. It's simple: they each believe they have their own address space, they each can only persuade the kernel to do things if they have permission to do so.
How does virtualization of the hardware variety work in a security context?
Essentially by multiplexing the processor. Is that even a word? Probably not. Ok, basically, you set up a number of virtual CPUs by setting up their respective tables in the processor, then you press go. You get to map memory just as you do for applications, so you can map new, blank space - or existing space.
The long and the short of this is that it is possible to map an existing operating system into a new virtual machine, You can then do the Linux of equivalent
MAP_PRIVATE; that is, these pages are copy on write. If anything writes back to this VM, you duplicate it, hold it whilst working and discard it when done.
What does this mean? Well, your userland application might trigger lots of other things to happen, such as royally destroying the system. But that's ok, since all this happened inside a copy on write simulated version of the OS - so you've just installed a fancy rootkit in something that's... whoops, sorry chaps, closed the browser and the vm went with it.
Right, great, but I've decided I actually want this file to persist because I like this pictcha of a kitty. So how does your fangle dangle vm thing handle that?
Fundamentally, it comes down to this: somewhere, some part of the system must be trusted, and some applications must be authorised to talk to it. It (the trusted part) must make some access control decisions and decide what to allow and what to deny.
Wait, doesn't the kernel already do that?
Yep. fundamentally, you still have the problem of access control.
Access control... problem?
One of the major complaints against systems such as SELinux is their complexity. These systems offer rules and configurations that control the implementation details of the OS to a fine grained level, far beyond what traditional DAC-like systems do. And this is complicated - especially in terms of knowing what should and should not be allowed. This problem extends to the simpler DAC/RBAC model - it's really very difficult to ensure all permissions everywhere reflect what you intended to allow and do not cause a security breach.
The other issue with access control is privilege escalation. In short, this occurs when something you have trusted can be exploited to do arbitrary things based on untrusted input: typically launching a process as an administrative role. If you require trust of this component, and yet this component leaves you vulnerable, you have potentially introduced an exploit that even with a virtualization communication layer thing will not help, since this component is trusted in all it does.
Ok, so aside from getting the security policy/access to the securo-visor thingy right, what else might be a problem?
- Hypervisors don't currently nest, at least not in a way that gives you acceptable performance. VMware's hypervisor will nest, but it's about the only one that will (at the time of writing and that I know of). Essentially, this means no virtualisation for you, cause bromie stole your vmm.
- In practise, large complex pieces of third party software can't be operated day to day in sandbox throw away when done mode, because users expect their cookies to persist. They expect their history to persist. They want to save files. AND THEY WANT TO BE SAFE TOO, DAMMIT! This means some form of integration with the third party software, e.g. the browser - and this means knowing about it and keeping up. In practise, end users don't like waiting too long for the latest and greatest, especially if the delay reason is just to keep them safe (cause hey, who wants that anyway). Thomas has essentially said the same thing here, I'm just repeating it.
- There is a lot of software to customise like this. That, or the system will be mostly unusable for anything interesting.
- Some stuff just doesn't lend itself well to this type of thing. Imagine how much pain would be involved in isolating say Visual Studio. Don't, and it's another attack vector. Do, and you've got to make damn certain you can still debug everything VS could...
- You still don't prevent social engineering, XSS etc (again, Thomas).
- We already use virtualization of the address space with access control for our standard security model. It would actually be possible to produce COW snapshots of an OS using existing, unvirtualised OSes, but performance means nobody would, in practise, do this.
- CPU virtualization doesn't aid security. It's the memory mapping OS clone technique which does. Fundamentally, it's a von neumann architecture. Memory is king.
- Access control and what we trust (and how much) remain problems.
- It's not massively user friendly. This is a perennial enemy to security, but in this case the trade off is likely to be huge to make the system work.
Will micro-virtualisation affect security in any way?
No. Access control is still a hard problem regardless of how you enforce the boundaries.
- Ring 1 and Ring 2 usage would make an awful lot of sense. Not being able to alter critical structures, or run privileged commands, would prevent many rootkit techniques from working well. Unfortunately, x86 and a handful of other architectures were the only ones providing these four modes - ARM processors, for example, only offer supervisor and user modes (ring 0/ring 3 in x86 terms). So operating systems that strive for portability can't make use of the other rings.
- The memory mapping technique to run an OS inside a hypervisor is the technique behind blue pill. Specifically, I am referring to on the fly remapping the running OS to a VM. You can also just boot the hypervisor first. SubVirt did this at the time.
- SELinux and the like fall down when they have to start protecting things like X. Which, urgh, runs as root and lets applications communicate with each other and is therefore a gaping security issue potentially. X.org has published XACE which is to X as LSM is to Linux; that is, it lets plugin writers build a security framework. Again, however, you have the problem of complexity. This is a good example of where a security policy cannot be applied to a large working application clearly.
- If you put the word "cyber" in front of it, it actually makes it the solution to life, the universe and everything.