I am thinking about the design for a sandbox, which is able to execute arbitrary unmodified, untrusted binaries. The goal is to implement an application

Unlike something like e.g. Google Chrome's sandbox it has to provide additional isolation that the security mechanisms baked into Windows can not provide. However, it should also be able to run a large number of sandboxes on one physical machine, so an full blown VM is out of the question.

It is supposed to work like this: Any system call made by the untrusted code should first be intercepted by the sandbox code, so that it can be blocked or modified before it is passed to the system. The sandbox should additionally be able to hook high-level APIs if that is convenient. The untrusted code should not be a able to temper with the sandboxes code (including it's imported libraries) and data (it's stack and heap). However, the sandbox code should have full access to the sandboxed code's address space.

Given these constraints it would also prefer if the sandbox would involve as little as possible kernel mode code (because of security and testability concerns), it would be as simple as possible (so no stunts like binary translation) and the sandboxed code would run reasonably fast.

The sandbox would run on my own server, so tempering with the underling OS is acceptable and portability between different versions of Windows is not a hard requirement. The target architecture is x86 - 64 bit support is not important, so use of segmentation is allowed. However, it should be able to run inside a VM; so if possible no hardware virtualization (VT-x or it's AMD equivalent).

How would you implement a sandbox like this? Has a sandbox-design like this been documented? Are there any important points that I overlooked? Given these requirements you might already have a reasonable idea, what kind of design I had in mind, however I am open to any idea.

EDIT: The goal is to provide an Application virtualization layer (see http://en.wikipedia.org/wiki/Application_virtualization), which the virtualized application can not break out of.

  • 1
    On Windows, the normal way is to hook APIs at the user level, using code caves and redirects.
    – Polynomial
    May 15, 2013 at 12:48
  • 4
    While I'm personally intrigued by the question I must ask you to be more precise in what you're asking. It's a complex issue and certainly nothing that could be reasonably crammed into one answer. Demonstrate that you've done your homework and looked into what's actually out there. May 15, 2013 at 13:58
  • 1
    are you talking about something like that? sandboxie.com
    – user26063
    May 15, 2013 at 20:21
  • 1
    @Christoffer Ideally I was hoping for an answer among the lines of: "Have a look at X-sandbox. It loads itself as a dll into the target process, it then uses a kernel mode driver to hook the system call handler and bounce them back into usermode, then it uses Y-method to prevent the target process from tempering with its code. Here is a paper that goes into excruciating detail."
    – Maarten
    May 15, 2013 at 21:11
  • 1
    @Bahaa I don't know. Sandboxie's design is not documented as far as I know.
    – Maarten
    May 15, 2013 at 21:17

3 Answers 3


I agree with the answer that suggested the overall design of determining what is, and is not, a good system call may be difficult. Indeed, in isolation, a single system call might not be enough information.

In terms of the way Windows works, there are actually two levels of operation:

  • NT-level calls. These are the typical system calls you'd expect via sysenter to code in ntdll.dll i.e. Kernel mode routines.
  • The Win32 subsystem level. These are the functions implemented in advapi32.dll, kernel32.dll, user32.dll and others. Some of these act directly, others may call the above functions themselves depending on what is needed.

In addition to this, as a broad survey there are two abstraction layers on top of the Win32 subsystem:

  • COM and friends, also known as COM, COM+, DCOM, ActiveX, SuperDuper (okay, joking) and others.
  • .Net and friends. .net executables are modified PE binaries that load in mscoree.dll amongst other things and may use APIs in other .net binaries, or Win32, or via COM interfaces.

Now, for hooking techniques, you have:

  • Kernel. These days, ISR Hooking is tantamount to stability suicide unless you want to reverse engineer PatchGuard and disable it, or are permanently running your OS under a debugger. However, there are options for filtering implemented in Windows, provided by Filter Drivers. This is typically how an antivirus product might scan network activity and implement a firewall.
  • User. You have a few options here:
    • Patching the binary on launch. @Poly talked about code caves and redirects - given a target binary, one option is to overwrite code in its functions, or the functions of interesting APIs to redirect to your code. Typically, you don't have enough space to patch a jump to any old place in the address space, so you need a "code cave" near to the function to put your call/full size jump.
    • Alternatives to this include IAT Hooking, a way to modify a binary's import table to pull in your functions rather than the original target.
    • Another alternative is to use DLL redirection. Under Linux, you'd know this as LD_PRELOAD, - in plain English, you'd load your DLL of the same name and with matching symbols before the desired API dll, and steal the calls that way. It's even a feature of MSVC that you can produce redirected link functions in DLLs rather than having to implement your own stubs for things you don't care about!
    • You can patch COM calls by overwriting the relevant entry in the vtable of the resultant interface class.
    • There are other techniques, e.g. DLL Injection etc.

So if you are looking to implement a sandbox meeting your criteria, then there are a few considerations to take into account:

  • First up, where and what you hook. If you hook all of kernel32.dll it is still feasible that API calls may get around that if they do not call these functions. I don't know any COM or .net code like this, but I'd bet there is some.
  • You need to be careful with Kernel patching, as a result of PatchGuard. The filter manager provides most of what you could ever want to do antivirus (and therefore probably sandbox) wise, however, if you're going outside the filter manager you're going into unstable land pretty quickly. There may be limits to what you can do.
  • Then, whatever hooks you place in userland are almost definitely detectable by the sandbox assuming you are not using binary translation. For example, let's pick on patching CreateFile. Untrusted code could simply read the memory of this function and check that the prologue is as expected. Likewise, there are ways to detect injected dlls, IAT hooks etc.
  • The same applies to sandboxing kernel mode code. If you let the untrusted code install drivers, you are again on a level playing field with it.

This kind of work has been done - Sandboxie is one product that attempts to implement a sandbox at this kind of level, although it likely has more kernel level stuff than you'd necessarily like. I've only briefly looked at Sandboxie back when I knew less about this sort of thing, but I suspect a core component is kernel level because almost any amount of user mode hooking can be undone if you know to expect it. The only defence is to have a greater level of permission.

You've mentioned not having binary translation - I've not covered the various debugging APIs in Windows for this reason. Using these, you could essentially run a program as if you were stepping through it, and analyse any calls or jumps before they were made. However, this is basically binary translation, and debugging affects performance in any case. Not only that, but if I were writing a program to resist your sandbox, there are a number of known tricks for detecting debuggers (one being you can't debug a debugger, so if you relaunch yourself and try to debug that new process and can't, you're being debugged. This definitely applies to ptrace) which would render the program unusable in the sandbox.

On a positive note, most non-malware products would likely play well with whichever set of techniques you used, since they're probably not trying to resist running under these environments. This can be quite a useful fact.

Last but not least, I apologise for the excessive use of bullet points! And, my list of techniques for hooking calls is not exhaustive - there are definitely others.

  • Like I said, I don't mind running in 32 bit. As far as I know PatchGuard is only an issue on x64.
    – Maarten
    May 19, 2013 at 7:14
  • However, any user level hooking that could be circumvented using a direct system call would be unexceptable. This just leaves OS provided mechanisms, binary translation and some kind of messing with the OS in kernel mode (hooking or hardware virtualization).
    – Maarten
    May 19, 2013 at 7:23
  • @Maarten that's correct, patchguard is only a 64-bit thing. It has also been circumvented in almost all of its implementations, but you'd need someone more skilled than I am to do it :) But yes, you can hook ISRs on 32-bit. I'd be tempted to hook on several levels, because a COM call for example might make more sense filter-wise than a series of syscalls. Just work on the basis the userland hooks might not be reliable. The other issue with any hook is speed, of course - you have to be careful not to slow things down too much with your checking stage.
    – user2213
    May 19, 2013 at 10:03

I just found Featherweight Virtual Machine, which is an open source sandbox implementation for windows:



Also interesting is Dune:


It implements a proof-of-concept sandbox like I described under linux using VT-x (which unfortunately means it won't work under most virtual machines).

  • Just for your info, in the future it might. VMWare currently let you emulate VT-x, so you can run their ESX product on workstation. It's slow as hell, but conceivable you could do it. This will probably improve in the future...
    – user2213
    May 16, 2013 at 12:46

Your plan is based on the flawed assumption that you can identify "bad" behavior by some kind of screening of system calls. You can't. Windows is replete with bugs, hidden features, and unexposed dependencies. As soon as you pass control to some other piece of code you didn't provide, you have absolutely no guarantee that it will do only things you approve of.

  • 2
    My plan is not what it you think it is. Maybe I should edit the question to describe more specifically what I am actually trying to do. Meanwhile next time please post a remark like this as a comment, because it does not answer the question at all.
    – Maarten
    May 15, 2013 at 20:13
  • the question is "how would you..." I wouldn't.
    – ddyer
    May 15, 2013 at 20:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .