You should not envision things as a "black list" of things to trap. Black lists don't work. At least, they don't work well. Instead of trying to work out a list of "forbidden system calls", you should instead create a list of "definitely harmless system calls" which you explicitly allow.

What you need is a sandbox. The Chromium Web browser (the open-source side of Chrome) contains such a sandbox which works on Windows. That page contains a rather detailed architecture of the sandbox system. I don't know to what extent this sandbox mechanism could be extracted from the Chromium source code and used on its own.

If you want something more immediately usable, then you may want to investigate virtual machines. In that model, you give a complete OS and machine to the client code, and you control things "from the outside": the VM talks to the external world through a network interface, and the host system (your server) controls exactly which packets make it through and which do not. For instance, if you want the client code to access some local files from your server, then you just export (as a "network share") the specific directory to the VM. For a low-resource guest VM, I suggest NetBSD, which is notorious for requiring very little RAM.

Edit: regardless of the technology you will use (sandbox, VM...), running potentially hostile code on your systems can be dangerous. Indeed, all the code snippets running concurrently on a given machine necessarily share some resources, i.e. L1 caches, jump prediction caches... which can be used to gain some information on other code snippets, be they other users of your system, or code from the host itself. This has been demonstrated as part of attacks to retrieve nothing less than an AES encryption key -- i.e. the Secret of Secrets in a cryptographically powered system. See for instance this answer. Though the demonstration were done on encryption systems, this really impacts anything which you do on a computer with confidential data.

Thus, if you have confidential data on your server, or if some of your users could find advantageous to spy on other users, then... don't do it.

To mitigate such cross-sandbox eavesdropping, you would have to arrange for the sandbox code not to be able to communicate with the external world (except to return a "result" at the very end) and to deactivate facilities for precise measurement of elapsed time (namely the rdtsc opcode on x86 -- it can be disabled with a flag in register CR4, but I don't know if usual virtualization solutions offer a simple way to enforce this deactivation).

You should not envision things as a "black list" of things to trap. Black lists don't work. At least, they don't work well. Instead of trying to work out a list of "forbidden system calls", you should instead create a list of "definitely harmless system calls" which you explicitly allow.

What you need is a sandbox. The Chromium Web browser (the open-source side of Chrome) contains such a sandbox which works on Windows. That page contains a rather detailed architecture of the sandbox system. I don't know to what extent this sandbox mechanism could be extracted from the Chromium source code and used on its own.

If you want something more immediately usable, then you may want to investigate virtual machines. In that model, you give a complete OS and machine to the client code, and you control things "from the outside": the VM talks to the external world through a network interface, and the host system (your server) controls exactly which packets make it through and which do not. For instance, if you want the client code to access some local files from your server, then you just export (as a "network share") the specific directory to the VM. For a low-resource guest VM, I suggest NetBSD, which is notorious for requiring very little RAM.

Edit: regardless of the technology you will use (sandbox, VM...), running potentially hostile code on your systems can be dangerous. Indeed, all the code snippets running concurrently on a given machine necessarily share some resources, i.e. L1 caches, jump prediction caches... which can be used to gain some information on other code snippets, be they other users of your system, or code from the host itself. This has been demonstrated as part of attacks to retrieve nothing less than an AES encryption key -- i.e. the Secret of Secrets in a cryptographically powered system. See for instance this answer. Though the demonstration were done on encryption systems, this really impacts anything which you do on a computer with confidential data.

Thus, if you have confidential data on your server, or if some of your users could find advantageous to spy on other users, then... don't do it.

To mitigate such cross-sandbox eavesdropping, you would have to arrange for the sandbox code not to be able to communicate with the external world (except to return a "result" at the very end) and to deactivate facilities for precise measurement of elapsed time (namely the rdtsc opcode on x86 -- it can be disabled with a flag in register CR4, but I don't know if usual virtualization solutions offer a simple way to enforce this deactivation).

You should not envision things as a "black list" of things to trap. Black lists don't work. At least, they don't work well. Instead of trying to work out a list of "forbidden system calls", you should instead create a list of "definitely harmless system calls" which you explicitly allow.

What you need is a sandbox. The Chromium Web browser (the open-source side of Chrome) contains such a sandbox which works on Windows. That page contains a rather detailed architecture of the sandbox system. I don't know to what extent this sandbox mechanism could be extracted from the Chromium source code and used on its own.

If you want something more immediately usable, then you may want to investigate virtual machines. In that model, you give a complete OS and machine to the client code, and you control things "from the outside": the VM talks to the external world through a network interface, and the host system (your server) controls exactly which packets make it through and which do not. For instance, if you want the client code to access some local files from your server, then you just export (as a "network share") the specific directory to the VM. For a low-resource guest VM, I suggest NetBSD, which is notorious for requiring very little RAM.

You should not envision things as a "black list" of things to trap. Black lists don't work. At least, they don't work well. Instead of trying to work out a list of "forbidden system calls", you should instead create a list of "definitely harmless system calls" which you explicitly allow.

What you need is a sandbox. The Chromium Web browser (the open-source side of Chrome) contains such a sandbox which works on Windows. That page contains a rather detailed architecture of the sandbox system. I don't know to what extent this sandbox mechanism could be extracted from the Chromium source code and used on its own.

If you want something more immediately usable, then you may want to investigate virtual machines. In that model, you give a complete OS and machine to the client code, and you control things "from the outside": the VM talks to the external world through a network interface, and the host system (your server) controls exactly which packets make it through and which do not. For instance, if you want the client code to access some local files from your server, then you just export (as a "network share") the specific directory to the VM. For a low-resource guest VM, I suggest NetBSD, which is notorious for requiring very little RAM.

Edit: regardless of the technology you will use (sandbox, VM...), running potentially hostile code on your systems can be dangerous. Indeed, all the code snippets running concurrently on a given machine necessarily share some resources, i.e. L1 caches, jump prediction caches... which can be used to gain some information on other code snippets, be they other users of your system, or code from the host itself. This has been demonstrated as part of attacks to retrieve nothing less than an AES encryption key -- i.e. the Secret of Secrets in a cryptographically powered system. See for instance this answer. Though the demonstration were done on encryption systems, this really impacts anything which you do on a computer with confidential data.

Thus, if you have confidential data on your server, or if some of your users could find advantageous to spy on other users, then... don't do it.

To mitigate such cross-sandbox eavesdropping, you would have to arrange for the sandbox code not to be able to communicate with the external world (except to return a "result" at the very end) and to deactivate facilities for precise measurement of elapsed time (namely the rdtsc opcode on x86 -- it can be disabled with a flag in register CR4, but I don't know if usual virtualization solutions offer a simple way to enforce this deactivation).