In security discussions, the topic of sandboxing often comes up. What is application sandboxing? How does it work, and what security vulnerabilities does it prevent?
A "sandbox" is a play area for young children: it is supposed to be safe for them (they cannot hurt themselves) and safe from them (it is sand, they cannot break it). In the context of IT security, "sandboxing" means isolating some piece of software in such a way that whatever it does, it will not spread havoc elsewhere.
A common Unix way of sandboxing is the
chroot command (which uses the
chroot() system call). A process launched with this command sees only a subdirectory (what it sees as the root of the filesystem, the
'/', is that subdirectory). The FreeBSD system has an enhanced version. More modern and thorough solutions use virtual machines (as in VMWare). This kind of sandboxing is meant for damage containment: you put a given process in a jail because in case it gets hacked through, say, a buffer overflow, the attacker is still in the jail and will not be able to impact other applications running on the same system.
Another kind of sandboxing is the one which lies at the core of programming languages like Java or C# (confusingly also called a "virtual machine"): the application code runs on a virtual CPU in which every single opcode is checked for violations (buffer overflows and underflows, type mismatches...). The structure of the languages and VM have been designed so that these checks are not too expensive. There again, this is damage containment, but with a finer granularity.
Sandboxing does not remove vulnerabilities; it just reduces the consequences of those vulnerabilities (e.g transforming a remote shell into a remote crash). A buffer overflow is a bug, regardless of whether a VM traps it or not.
Ok, so firstly, how do applications run?
On a standard operating system of the kind you're likely to be using right now, applications are launched by users. When an application is launched it is loaded into memory and given time to run on the cpu.
In order to get things done, rather than controlling the computer hardware directly, the application makes requests of the operating system. Developers call these system calls, or API calls. Essentially, everything a program does is either an operation on its own memory, or a system call to do something else.
Now, obviously, we can't just load applications and let them do anything, because otherwise users could do anything. So, the OS you're using right now has a concept of permissions. When a system call is made, the OS looks at the application context - who is running it? - and the permissions on the resource in question. I say resource rather than file because
regedt32 will let you set permissions on registry keys and only root can open sockets below port 1024 on Linux.
So what can an application do? Anything you can do. Clearly, this is okay for say Windows explorer, but let's say you download another program, such as a pdf to sound converter (just a random suggestion). You're not sure if you trust "Ninefingerz-so-l33t-man", the developer who implemented it, but if you run it, it has access to everything you do.
Application sandboxing is about preventing that on some level. Tools like sandboxie or the SELinux sandbox hook those operating system calls and control at a more fine grained level what that program can do. For example, you might allow the program to edit anything under
C:\Users\Ninefingers\temporaryisolateddata but everywhere else is read only. You might prevent it from making outbound connections on sockets, or reading the registry at all.
Of course, that is not the only way (hooking the kernel) to implement a sandbox. Google Chrome's sandbox on windows is implemented entirely by the program - the master google chrome process controls what the child processes can access, in a nutshell. This sandbox works by exploiting existing security controls in Windows, only allowing the child process to communicate back to the target process when it wants to take an action it would normally achieve via a system call. This way, the target process does the filtering of requests.
At its core, application sandboxing is about additional restrictions to prevent potentially malicious processes from hurting your system. It is an extension to the existing permissions model.
Note that Mandatory Access Control, a feature available in some operating systems, should, if implemented properly, sandbox everything according to who the user is and what the process is.
Sandboxing is a mechanism for information flow control. By running some code in a sandbox, you gain additional control about the inputs and outputs of the program. That means you can deny/mediate access to external data or other ressources in case the sandboxed program is compromised or itself malicious.
Often, a complex execution environment is included in the sandbox, for example Java and Java libraries in case of JVM-based sandboxes. Or you could run an application inside a dedicated VM, making that a sandbox for you with a complete legacy OS as execution environment.
To my knowledge, the state-of-the-art sandbox technology for x86 execution is described in Google's Java Native Client: http://research.google.com/pubs/pub34913.html
The concept of information flow control is essential to security design. The whole thing about privileges and MAC in linux is all about information flow control. Consequently, DJB notes in his paper "Some thoughts on security after ten years of qmail 1.0" that information flow and interfaces are substantial and "least privilege" is nothing but a distraction. However, he did not use explicit sandboxes such as a complete JVM, he used standard OS functionality to split qmail into several small programs with explicit, well-defined interfaces. And in fact, sandboxes are by design not much more than a replication/fine-tuning/improvement of what the OS (should) give you.