I would be allowing a user to upload Python scripts and would execute them on my server. The base application is in Java and I am using ProcessBuilder to get Python to run from my Java application. How could I go about testing for security threats from the end user, when they upload their script to run on my server.
Sorry to say, but this seems like a really questionable thing to do. Period.
There's a famous blog from Microsoft called "Ten Immutable Laws Of Security". Law #1 is:
Law #1: If a bad guy can persuade you to run his program on your computer, it's not solely your computer anymore.
There's a reason it's number 1; creating a list of "bad things" to block is a losing game: for a full-featured and complex language like python, there will always be more creative ways to do bad things that aren't in your list.
I would encourage you to consider why you need to allow users to upload scripts, and whether you can accomplish the same functionality through a set of UIs, REST APIs, or some other standard method where the security is well understood.
If you absolutely need to allow scripts (python or any other language), then I would figure out the minimal amount of python syntax that users need to accomplish whatever it is your site does and reject any script that uses something not on that list. Do your research about every item on that list and whether it could be used to make a system call or otherwise change files on your filesystem. If that list includes
import then you know you're sunk right off the bat because then you're faced with either trying to enumerate everything on your
sys.path and block the "bad things", or trying to enumerate which modules users need, which methods within those modules they need, and manually verifying that nothing bad could possibly be done with them. Even assuming you have good well-researched lists, building the regular expressions to properly enforce the list is probably a difficult task in-and-of itself.
The other approach is to assume the server is totally compromised all the time, don't use it for anything else, keep it in its own isolation network far away from any other servers you care about, firewall off any outbound traffic, and tear it down and stand up a new one every now and then.
The idea of letting users upload and run scripts on your server seems a bit like letting random teenagers experiment with fireworks in your living room.
As Mike notes, this is generally a really bad idea. However, if you're set on doing it what you might do is create a separate VM that is responsible for running the untrusted Python code and then put restrictions on the resources that the VM can access - and they pray that no one uploads something that can circumvent those restrictions.
Trying to scan the Python code for issues, which is what you seem to have in mind, is not a reliable way of determining if it has security issues.
To get an idea of the kind of peril you're thinking of taking on check out the following links: