Is there a way to safely run untrusted code on a local machine for a videogame

Question

I want to create a simple videogame engine that can run untrusted custom code in order to add functionality to the game. I thought a about using an akka like concurrent actor system, so that the code can be confined to actors and keep the game interface as small as possible. Now I need a solution to that allows people to download and run user generated code (mods) on their machines. The language doesn't even matter I just want a few ideas for running such code on my own machine.

Answer 1

EDIT: The question originally asked about running untrusted code in general, not about the specific scenario laid out above. For something like that, you usually want a language that provides an API sandbox (I think Lua can be used for this?) which provides enough functionality for game logic or mods, but doesn't allow open access to the file system, creating other processes, etc.

You could also sandbox the whole game (with an OS-level process privilege sandbox, such as by developing it as a modern Windows "app" or using the MacOS sandbox), with a special location (readable by the sandbox) where the user can drop mods, etc. That way, even if the game gets compromised by a malicious mod or similar, it cannot impact the OS much (if at all), although it could still attack the game and anything it has access to. Games generally don't need a lot of OS permissions - maybe network connectivity, somewhere to put save files, all stuff that the OS-provided sandboxes handle easily - and are actually kind of an ideal use case for these app sandboxes.

---

Generically speaking, the term you're looking for is "sandboxing" (as in, a place this kiddies can make a mess without affecting anything else). Sandboxing is a kind of hard problem but it's also very useful, so there are a few different places it's commonly used.

For example, all modern web browsers have not just one but two layers of sandbox. The old one is the JavaScript sandbox: people run untrusted JS code on their machine all the time, usually safely. This is implemented as an API sandbox; the JS that you can run on a browser simply has no functions you can call, and no way to define functions, that do particularly malicious things (like read or write arbitrary files, open arbitrary network connections, etc; everything is safe, restricted to a safe subset, or unavailable). However, JS is complicated and JS runtimes are prone to security bugs, so modern browsers additionally run their rendering and JS engine in a sandbox too. This second sandbox is implemented as a process privilege sandbox; the browser process that creates the window and such spins up a bunch of child processes, each with extremely restricted permissions, and uses them to process all untrusted code, communicating with them through very carefully secured and minimal inter-process communication channels. That way, even if malicious script finds a way out of the API sandbox and can call arbitrary C functions, most of the interesting ones (like file reads) will fail because the OS says the process doesn't have permission to do that.

Process-based sandboxes are pretty common these days, and all modern OSes have [at least some] support for them. The Windows, Mac, iOS, and Android app stores all provide sandboxed apps. Linux provides sandboxing functionality that is used by things like Docker (and Chrome on Linux). FreeBSD (and derivatives) have "Jails", etc. There's a bunch of ways to do it. A relatively-simple one can be created just using user permissions and ACLs; you create a new user account for the sandbox, give it access to nothing at all by default (which is tricky, since normally there's a lot of stuff that's world-readable, at least), and then grant that user account access to the things the sandboxed code is allowed to touch. A process launched as that user will have very limited access to the system, until/unless they find a way out.

Unfortunately, creating a secure sandbox tends to be somewhat platform-specific, and tricky for each individual platform. I've personally reviewed, and found breaches for, sandboxes used by multiple Big Software Companies' products (you've heard of them, might even have them open right now). The app store sandboxes model - which give the developer fairly little control over what can be done, in exchange for the OS handling all the sandbox creation and enforcement - is appealing and if you're writing for Mac or recent Windows I recommend considering it.

Another kind of sandbox, available on any modern desktop OS but pretty expensive to run, is a virtual machine (VM) sandbox. Using any major VM platform (VMWare, VirtualBox, Hyper-V, whatever), you can create a VM that has little or no access to the host OS. This is the standard way cloud computing providers work; from Amazon's perspective your tiny EC2 instance is running untrusted code, yet has to share hardware with other mutually-untrusted users to be cost-effective, and VMs are used to do this. It's also a way to run potentially-malicious code, because the host OS can watch what the VM does but the VM cannot control the host.

Answer 2

Securing mods is hard. You have some apparently conflicting requirements:

1. High performance, both in terms of running fast and communicating efficiently with the base game (and/or other mods). This generally implies machine code, running with the executable, e.g. as a DLL.
2. Preventing the code from accessing any dangerous functionality, like reading arbitrary files off the disk, directly capturing keyboard input, etc. which implies some form of sandboxing, which kills cross-communication performance.

You basically have two options which will work: Sandboxing the entire game, and putting mods in the same sandbox with it, or picking something that can be trivially prevented from doing bad things in the first place.

Also, note that these answers apply to sandboxing the game from the host system. Sandboxing mods to prevent cheating, "unfairness", or even just accidental conflicts is an in-depth API design problem that no one online could possibly have the context required to solve for you.

Sandbox it all

This actually isn't a half-bad idea. You can look into how things like browsers sandbox themselves by denying themselves permission to do certain things while remaining efficient and still making use of GPU acceleration (which is critical in most game development). Docker and its ilk are also a good source of inspiration, since they severely limit what containers can do while maintaining good performance.

Unfortunately, that's an enormous topic. Entire textbooks can -- and probably have -- been written on it. Some searchable keywords are "kernel namespaces", "chroot jail", and "principle of least privilege", but if you go this route, expect to spend a significant portion of your dev effort on it.

You could also look into running the entire thing in a VM of some kind, but while that would likely be more secure, it'd also likely be much slower. This is at least testable: Pop open VirtualBox or VMWare, try playing your game in it, and see if you can tune the settings so it runs smoothly. Now try doing that on your minimum-spec target.

If you're developing a browser game, by the way, you probably won't need sandboxing of any kind -- you can count on the browser's sandbox to keep them secure.

Prevent it all

This is the solution most modding engines go with, if they do anything at all. It's easier than it sounds -- while you can't easily restrict raw bytecode to a subset of functionality, you can quite easily use a scripting language, limit that language's functionality to just what a reasonable mod would need, and expose it to modders. Then you have two threat angles to cover:

1. Some flaw in your language which lets them put in code you didn't want them to have.

   In general, this can be mostly avoided by using an off-the-shelf language. There might be flaws in those interpreters, but far fewer than you'd introduce writing your own, and where they exist you'd have a whole community to draw on to fix them.

2. A function you do intentionally expose leaking information or functionality.

   For example, if you want mods to be able to add new assets, your first thought might be to give them a loadAsset(path, type) function. However, they might call loadAsset("/etc/passwd", "text") -- not great. Instead, provide a loadAsset(name), and then let them give each of their custom assets names. Then you don't need to expose any filesystem access, and they still get control over when things are loaded, which can really help performance.

   You'll need to go through every function you expose with a fine-toothed comb to ensure that it's doing only what you want, but that's still less effort than trying to statically verify that arbitrary x86 bytecode isn't misbehaving.

Some common examples of embeddable scripting languages are:

• Lua, which tends to be the go-to pick, but isn't a very friendly programming environment for the modder
• Python, which is harder to embed well, but is much friendlier to modders
• WASM/WASI, which is currently very hard to use because it's very new, but will absolutely be worth keeping an eye on.

Answer 3

I'm not really an expert on anything in computer science, so I'll let the "Oracle® VM VirtualBox® User Manual, Version 6.1.36" do the talking, but I think what you might want (to sandbox with the minimum effect on performance) is hardware virtualization.

https://www.virtualbox.org/manual/ch10.html#hwvirt

(Note that there's also a separate "Oracle® VM VirtualBox® Programming Guide and Reference".)

This works simply for a "sandbox it all approach", but that has some drawbacks for your problem, since it means bad or malicious mods can still wreck the rest of the player's copy of the game software, even if it stops it from wrecking or infecting the rest of their computer.

I suppose you could make copies of the game to test out new mods. You could also copy save data to a location outside the virtual machine. (Save data, possibly including custom levels, configurations/settings, ... = things players would be sadder about losing that than just the application, which they could just reinstall at any point.)

Implemented another way, the application might be permanently stored outside of the virtual machine where the game is played: The outside part of the game could start and run its own virtual machine somehow whenever you start the game, extract whatever save data it needs out of the virtual machine, either whenever you stop the virtual machine or at regular intervals, likely having to interrupt the virtual machine for non-disruptive amounts of time, and then completely reverting the virtual machine to some standard state after each use. This way no code in the virtual machine is responsible for actually saving any data, and in fact nothing in the virtual machine can make any permanent change on the system except in terms of what save data the outside application extracts, and that save data should be given no opportunity to be executed as code or anything, except MAYBE withing the virtual machine after it is restarted with the new save data inserted into it. (Ideally, new save data would not automatically overwrite old save data.)

Another thing that occurs to me is that a virtual machine used to run just a single application doesn't really have to have a conventional operating system at all, if you are comfortable writing software that includes its own bootloader and operating system specific to that application and including all of the dependencies it needs on the virtual machine. (The Host OS or something will handle allowing processes outside the virtual machine run simultaneously.) That might speed it up a little and save a significant amount of memory, since it wouldn't need all the bloat that normal operating systems have to deal with all of the many things they might need to handle.