I want to create a simple videogame engine that can run untrusted custom code in order to add functionality to the game. I thought a about using an akka like concurrent actor system, so that the code can be confined to actors and keep the game interface as small as possible. Now I need a solution to that allows people to download and run user generated code (mods) on their machines. The language doesn't even matter I just want a few ideas for running such code on my own machine.
EDIT: The question originally asked about running untrusted code in general, not about the specific scenario laid out above. For something like that, you usually want a language that provides an API sandbox (I think Lua can be used for this?) which provides enough functionality for game logic or mods, but doesn't allow open access to the file system, creating other processes, etc.
You could also sandbox the whole game (with an OS-level process privilege sandbox, such as by developing it as a modern Windows "app" or using the MacOS sandbox), with a special location (readable by the sandbox) where the user can drop mods, etc. That way, even if the game gets compromised by a malicious mod or similar, it cannot impact the OS much (if at all), although it could still attack the game and anything it has access to. Games generally don't need a lot of OS permissions - maybe network connectivity, somewhere to put save files, all stuff that the OS-provided sandboxes handle easily - and are actually kind of an ideal use case for these app sandboxes.
Generically speaking, the term you're looking for is "sandboxing" (as in, a place this kiddies can make a mess without affecting anything else). Sandboxing is a kind of hard problem but it's also very useful, so there are a few different places it's commonly used.
Process-based sandboxes are pretty common these days, and all modern OSes have [at least some] support for them. The Windows, Mac, iOS, and Android app stores all provide sandboxed apps. Linux provides sandboxing functionality that is used by things like Docker (and Chrome on Linux). FreeBSD (and derivatives) have "Jails", etc. There's a bunch of ways to do it. A relatively-simple one can be created just using user permissions and ACLs; you create a new user account for the sandbox, give it access to nothing at all by default (which is tricky, since normally there's a lot of stuff that's world-readable, at least), and then grant that user account access to the things the sandboxed code is allowed to touch. A process launched as that user will have very limited access to the system, until/unless they find a way out.
Unfortunately, creating a secure sandbox tends to be somewhat platform-specific, and tricky for each individual platform. I've personally reviewed, and found breaches for, sandboxes used by multiple Big Software Companies' products (you've heard of them, might even have them open right now). The app store sandboxes model - which give the developer fairly little control over what can be done, in exchange for the OS handling all the sandbox creation and enforcement - is appealing and if you're writing for Mac or recent Windows I recommend considering it.
Another kind of sandbox, available on any modern desktop OS but pretty expensive to run, is a virtual machine (VM) sandbox. Using any major VM platform (VMWare, VirtualBox, Hyper-V, whatever), you can create a VM that has little or no access to the host OS. This is the standard way cloud computing providers work; from Amazon's perspective your tiny EC2 instance is running untrusted code, yet has to share hardware with other mutually-untrusted users to be cost-effective, and VMs are used to do this. It's also a way to run potentially-malicious code, because the host OS can watch what the VM does but the VM cannot control the host.
Securing mods is hard. You have some apparently conflicting requirements:
- High performance, both in terms of running fast and communicating efficiently with the base game (and/or other mods). This generally implies machine code, running with the executable, e.g. as a DLL.
- Preventing the code from accessing any dangerous functionality, like reading arbitrary files off the disk, directly capturing keyboard input, etc. which implies some form of sandboxing, which kills cross-communication performance.
You basically have two options which will work: Sandboxing the entire game, and putting mods in the same sandbox with it, and picking something that can be trivially prevented from doing bad things in the first place.
Sandbox it all
This actually isn't a half-bad idea. You can look into how things like browsers sandbox themselves by denying themselves permission to do certain things while remaining efficient and still making use of GPU acceleration (which is critical in most game development). Docker might also be a good source of inspiration, as they manage to severely limit what containers can do while maintaining good performance. Unfortunately, it's also an incredibly broad topic, which I know very little about.
You could also look into running the entire thing in a VM of some kind, but while that would likely be more secure, it'd also likely be much slower.
If you're developing a browser game, by the way, you probably won't need sandboxing of any kind -- unless you're concerned about the mods manipulating the actual game code in unintended ways, then you can count on the browser's sandbox to keep them secure.
Prevent it all
This is the solution most modding engines go with, if they do anything at all. It's easier than it sounds -- while you can't easily restrict raw bytecode to a subset of functionality, you can quite easily use a scripting language, limit that language's functionality to just what a reasonable mod would need, and compile this known-good code into bytecode to run efficiently. Then you have two threat angles to cover:
Some flaw in your compiler which lets them put in code you didn't want them to have.
In general, this can be mostly avoided by using an off-the-shelf compiler. There are a ton of options for Lua and Python which you could look into, and probably more for other languages, too. There might be flaws in those compilers, but much fewer than you'd introduce writing your own, and where they exist you'd have a whole community to draw on to fix them.
A function you do intentionally expose leaking information or functionality.
For example, if you want mods to be able to add new assets, your first thought might be to give them a
loadAsset(path, type)function. However, they might call
loadAsset("/etc/passwd", "text")-- not great. Instead, provide a
loadAsset(name), and then let them give each of their custom assets names. Then you don't need to expose any filesystem access, and they still get control over when things are loaded, which can really help performance.
You'll need to go through every function you expose with a fine-toothed comb to ensure that it's doing only what you want, but that's still less effort than trying to statically verify that arbitrary x86 bytecode isn't misbehaving.