I want to create a simple videogame engine that can run untrusted custom code in order to add functionality to the game. I thought a about using an akka like concurrent actor system, so that the code can be confined to actors and keep the game interface as small as possible. Now I need a solution to that allows people to download and run user generated code (mods) on their machines. The language doesn't even matter I just want a few ideas for running such code on my own machine.

  • 1
    This sounds pretty broad. What kind of untrusted code? What kind of machine? Who are the users? What purpose does this serve? Chances are, for whatever you're doing, there's an off-the-shelf solution that you can either use (if you need to meet a real-world need) or take inspiration from (if you're doing this to learn how similar things work)
    – Nic
    Jul 9 '19 at 18:41
  • Mostly I wan't people to extend functionality in a videogame which can be downloaded from the internet. But I have not made concrete plans yet on how to make that happen Jul 9 '19 at 18:53
  • Oh, hm, that's a very different problem from just safely running random user code. See, just taking input and output like from a console isn't going to be remotely fast enough for game mods to work. Have you considered implementing a scripting language of some kind (Lua is popular, but not your only option) and restricting the capabilities of the scripting language instead? That'd be much easier to lock down than letting people run truly arbitrary code (any functionality you don't want them to have, just... don't give it to them).
    – Nic
    Jul 9 '19 at 18:58
  • I have been told not to do that since scripting languages are orders of maginitudes slower than native code. But thats what I was thinking aswell. I was thinking about using something like webassembly or a sandboxed JVM that uses an Akka type actor system to talk to other game objects. But I haven't found my resources on that and its still bytecode, so I don't know ho w safe it is Jul 9 '19 at 19:07
  • Whoever told you scripting languages are too slow for games is flat-out wrong. Plenty of games are built in or with scripting languages (e.g. Far Cry's modding engine was Lua, Don't Starve is mostly Lua, Factorio has a Lua modding API...) and if you're really concerned about speed, you can always JIT-compile when the mod is loaded so you get basically native performance. If this is a browser game, virtually anything would work, as the browser automatically sandboxes for you, but if it's a desktop game... good luck with that. Java sandboxing will be equally slow and difficult to secure.
    – Nic
    Jul 9 '19 at 19:11

EDIT: The question originally asked about running untrusted code in general, not about the specific scenario laid out above. For something like that, you usually want a language that provides an API sandbox (I think Lua can be used for this?) which provides enough functionality for game logic or mods, but doesn't allow open access to the file system, creating other processes, etc.

You could also sandbox the whole game (with an OS-level process privilege sandbox, such as by developing it as a modern Windows "app" or using the MacOS sandbox), with a special location (readable by the sandbox) where the user can drop mods, etc. That way, even if the game gets compromised by a malicious mod or similar, it cannot impact the OS much (if at all), although it could still attack the game and anything it has access to. Games generally don't need a lot of OS permissions - maybe network connectivity, somewhere to put save files, all stuff that the OS-provided sandboxes handle easily - and are actually kind of an ideal use case for these app sandboxes.

Generically speaking, the term you're looking for is "sandboxing" (as in, a place this kiddies can make a mess without affecting anything else). Sandboxing is a kind of hard problem but it's also very useful, so there are a few different places it's commonly used.

For example, all modern web browsers have not just one but two layers of sandbox. The old one is the JavaScript sandbox: people run untrusted JS code on their machine all the time, usually safely. This is implemented as an API sandbox; the JS that you can run on a browser simply has no functions you can call, and no way to define functions, that do particularly malicious things (like read or write arbitrary files, open arbitrary network connections, etc; everything is safe, restricted to a safe subset, or unavailable). However, JS is complicated and JS runtimes are prone to security bugs, so modern browsers additionally run their rendering and JS engine in a sandbox too. This second sandbox is implemented as a process privilege sandbox; the browser process that creates the window and such spins up a bunch of child processes, each with extremely restricted permissions, and uses them to process all untrusted code, communicating with them through very carefully secured and minimal inter-process communication channels. That way, even if malicious script finds a way out of the API sandbox and can call arbitrary C functions, most of the interesting ones (like file reads) will fail because the OS says the process doesn't have permission to do that.

Process-based sandboxes are pretty common these days, and all modern OSes have [at least some] support for them. The Windows, Mac, iOS, and Android app stores all provide sandboxed apps. Linux provides sandboxing functionality that is used by things like Docker (and Chrome on Linux). FreeBSD (and derivatives) have "Jails", etc. There's a bunch of ways to do it. A relatively-simple one can be created just using user permissions and ACLs; you create a new user account for the sandbox, give it access to nothing at all by default (which is tricky, since normally there's a lot of stuff that's world-readable, at least), and then grant that user account access to the things the sandboxed code is allowed to touch. A process launched as that user will have very limited access to the system, until/unless they find a way out.

Unfortunately, creating a secure sandbox tends to be somewhat platform-specific, and tricky for each individual platform. I've personally reviewed, and found breaches for, sandboxes used by multiple Big Software Companies' products (you've heard of them, might even have them open right now). The app store sandboxes model - which give the developer fairly little control over what can be done, in exchange for the OS handling all the sandbox creation and enforcement - is appealing and if you're writing for Mac or recent Windows I recommend considering it.

Another kind of sandbox, available on any modern desktop OS but pretty expensive to run, is a virtual machine (VM) sandbox. Using any major VM platform (VMWare, VirtualBox, Hyper-V, whatever), you can create a VM that has little or no access to the host OS. This is the standard way cloud computing providers work; from Amazon's perspective your tiny EC2 instance is running untrusted code, yet has to share hardware with other mutually-untrusted users to be cost-effective, and VMs are used to do this. It's also a way to run potentially-malicious code, because the host OS can watch what the VM does but the VM cannot control the host.

  • I actually just wanted to a post that has a list of options for sandboxing, half out of curiosity half to apply to my personal game project. And so far both of you have been very helpful in finding a place to start learning about it. Jul 9 '19 at 19:56
  • 1
    Added an edit pertaining to the edited question and comments.
    – CBHacking
    Jul 9 '19 at 19:58

Securing mods is hard. You have some apparently conflicting requirements:

  1. High performance, both in terms of running fast and communicating efficiently with the base game (and/or other mods). This generally implies machine code, running with the executable, e.g. as a DLL.
  2. Preventing the code from accessing any dangerous functionality, like reading arbitrary files off the disk, directly capturing keyboard input, etc. which implies some form of sandboxing, which kills cross-communication performance.

You basically have two options which will work: Sandboxing the entire game, and putting mods in the same sandbox with it, and picking something that can be trivially prevented from doing bad things in the first place.

Sandbox it all

This actually isn't a half-bad idea. You can look into how things like browsers sandbox themselves by denying themselves permission to do certain things while remaining efficient and still making use of GPU acceleration (which is critical in most game development). Docker might also be a good source of inspiration, as they manage to severely limit what containers can do while maintaining good performance. Unfortunately, it's also an incredibly broad topic, which I know very little about.

You could also look into running the entire thing in a VM of some kind, but while that would likely be more secure, it'd also likely be much slower.

If you're developing a browser game, by the way, you probably won't need sandboxing of any kind -- unless you're concerned about the mods manipulating the actual game code in unintended ways, then you can count on the browser's sandbox to keep them secure.

Prevent it all

This is the solution most modding engines go with, if they do anything at all. It's easier than it sounds -- while you can't easily restrict raw bytecode to a subset of functionality, you can quite easily use a scripting language, limit that language's functionality to just what a reasonable mod would need, and compile this known-good code into bytecode to run efficiently. Then you have two threat angles to cover:

  1. Some flaw in your compiler which lets them put in code you didn't want them to have.

    In general, this can be mostly avoided by using an off-the-shelf compiler. There are a ton of options for Lua and Python which you could look into, and probably more for other languages, too. There might be flaws in those compilers, but much fewer than you'd introduce writing your own, and where they exist you'd have a whole community to draw on to fix them.

  2. A function you do intentionally expose leaking information or functionality.

    For example, if you want mods to be able to add new assets, your first thought might be to give them a loadAsset(path, type) function. However, they might call loadAsset("/etc/passwd", "text") -- not great. Instead, provide a loadAsset(name), and then let them give each of their custom assets names. Then you don't need to expose any filesystem access, and they still get control over when things are loaded, which can really help performance.

    You'll need to go through every function you expose with a fine-toothed comb to ensure that it's doing only what you want, but that's still less effort than trying to statically verify that arbitrary x86 bytecode isn't misbehaving.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.