You should not envision things as a "black list" of things to trap. Black lists don't work. At least, they don't work well. Instead of trying to work out a list of "forbidden system calls", you should instead create a list of "definitely harmless system calls" which you explicitly allow.
What you need is a sandbox. The Chromium Web browser (the open-source side of Chrome) contains such a sandbox which works on Windows. That page contains a rather detailed architecture of the sandbox system. I don't know to what extent this sandbox mechanism could be extracted from the Chromium source code and used on its own.
If you want something more immediately usable, then you may want to investigate virtual machines. In that model, you give a complete OS and machine to the client code, and you control things "from the outside": the VM talks to the external world through a network interface, and the host system (your server) controls exactly which packets make it through and which do not. For instance, if you want the client code to access some local files from your server, then you just export (as a "network share") the specific directory to the VM. For a low-resource guest VM, I suggest NetBSD, which is notorious for requiring very little RAM.