Why do some Java APIs bypass standard SecurityManager checks?

Question

In Java, normally permission checks are handled by the SecurityManager. To prevent untrusted code from invoking privileged code and exploiting some bug in the privileged code, SecurityManager checks the entire call stack; if any of the callers in the stack trace are unprivileged, by default the request is denied. At least, that's how the standard SecurityManager checks work.

However, a few special Java APIs follow different rules. They bypass the standard SecurityManager checks, and substitute a weaker check. In particular, they check only the immediate caller, not the entire call stack. (See Guideline 9-8 of the Java Secure Coding Guidelines for details. The special APIs include, for example, Class.forName(), Class.getMethod(), and more.)

Why? Why do these special APIs bypass the standard checks and substitute a weaker check? And, why is this safe? In other words, why is it sufficient for them to check only the immediate caller? Doesn't this re-introduce all of the risks that the standard SecurityManager checks were designed to defend against?

I first learned of this when reading an analysis of the recent Java zero-day exploit (CVE-2012-4681). That analysis deconstructs how the exploit works. Among other things, the attack involves taking advantage of the weaker checking done by these special APIs. In particular, the malicious Java code manages to get a reference to a trusted system class (through a separate bug), then it fools that trusted system class into invoking one of these special APIs. The resulting permission check looks only at its immediate caller, sees that the immediate caller is trusted, and allows the operation -- even though the operation was originally initiated by untrusted code. Thus the weaker checks don't stop the attack, but as far as I can see, this attack would have been prevented by using the standard SecurityManager checks (since the caller's caller is untrusted). In other words, this recent attack looks like an example of why the weaker check is risky.

However, I know the Java designers are smart folks. I suspect the Java designers must have considered these issues and had some good reason to bypass the standard checks and substitute the weaker checks for these special APIs -- or, at least, thought they had a good reason why this was safe. So, maybe I'm missing something.

Can anyone shed any light on this? Did the Java designers screw up with these special APIs, or were there valid reasons for substituting the weaker checks?

Edit 9/1: I'm not asking about how the exploit works; I think I understand how the exploit works. I'm also not asking why, in this particular example, the trusted code that invoked these special APIs was buggy. Rather, I'm asking why the special APIs -- like Class.forName(), Class.getMethod(), and so on -- are specified and implemented to use the non-standard weaker permission check (only look at the immediate caller) instead of the standard SecurityManager permission check (look at the entire call stack). This design decision (of using weaker permission checks for those special APIs) allowed the recent vulnerability, so it would be easy to criticize the design decision. However, I imagine there might have been some good reasons for doing things this way, and I'm wondering what those might be.

Answer 1

(This is only a general comment on the "why", not on the specific attack you are alluding to.)

Unfortunately, the Java designers found that it was highly possible to paint yourself into a corner, structurally speaking. For instance, there are classes in java.io and in java.net, which are both involved in doing I/O. Let's assume that a given JVM has special OS-interaction native code in java.io.FileDescriptor, which allows it to do send() and receive() system calls (that's not the case of the Sun/Oracle JVM, but it could happen in another JVM, and indeed it does in at least the one I once wrote). To enforce the sandbox semantics, these methods are not public, of course.

The implementation of java.net.Socket would like, quite naturally, to use these methods. But, as per the naming conventions, Socket is a class which is located in the java.net package, not java.io. How can it access the non-public methods of java.io.FileDescriptor, considering that it must do so even if invoked from non-trusted code (non-trusted code can open sockets, albeit not to every destination) ? There are mainly two ways:

• Add some "bridge" native methods in java.net.Socket, which forward the calls to the methods in java.io.FileDescriptor. Native code sneers at packages and visibility; native code can do everything.

• Allow the code from java.net to do some reflection to access non-public methods in other packages. Code which can do that can do about everything, because it could modify the data structures used by SecurityManager itself (hey, I remember that you pointed that to me yourself, maybe ten years ago, in a Usenet discussion where we were both using our real names). So, unbounded reflection must not be granted to everybody, but, in the situation I describe here, it must be granted to code in a specific system package (java.net) even though that code is called from an untrusted applet.

The second method requires the weakening of the security model. Actually, it is quite generic: if the untrusted applet must do anything useful at some point, it must be able to access the system (if only to display the result of computations or send it back to the server), which needs a kind of gate. Native code is such a gate. The security weakening model is another kind of gate, which has the added benefit of staying in the "pure Java" world (I can understand that in the JVM maintenance team, native code is probably frowned upon because it makes things more expensive). The bad side is that by weakening the security model, the attack surface is greatly enlarged: now, all the Java code in the packages-with-privileges has become critical. To reuse @Hendrik's analogy, it is about giving a root setuid bit to a substantial chunk of code (from the java.* packages).

In particular, the way that Java weakened the security model is to designate some special APIs -- like Class.findClass(), Class.newInstance(), Class.getMethod(), and other reflection-related APIs -- as using weaker permission checks. In the example above, this allows trusted system code in java.net.Socket to use these special APIs to get a reference to a non-public method in java.io.FileDescriptor and invoke it reflectively.

• For instance, the java.net.Socket code can use Class.getMethod() to get a reference to the non-public method in java.io.FileDescriptor (this is allowed since the immediate caller of Class.getMethod() is java.net.Socket(), which is trusted code) and then invoke it.

  Notice that this implementation strategy relies upon Class.getMethod() to use the weaker permission checks. It just wouldn't work, if Class.getMethod() used the standard permission checks. If untrusted code invokes java.net.Socket, and then the java.net.Socket code calls Class.getMethod(), the standard permission checks would reject this call because there is untrusted code somewhere on the call stack, and the java.net.Socket stuff wouldn't work right (in non-attack scenarios). In contrast, the weaker permission checks used in the weakened security model do allow it.

So, the weaker permission checks help the Java designers get themselves out of the corner they painted themselves into.

Supposedly, with a "perfect" structure of the system code and naming conventions, a handful of native methods would be sufficient, but the presence of specific is-immediate-caller-from-a-trusted-package calls in the JVM system library code shows that the structure of said code is not perfect.

Answer 2

Summary

Some trusted methods need more permissions internally to fulfill their tasks. But if those methods have bugs, they may allow an untrusted attacker to execute undesired actions at an increased privilege level.

Background

We have a very similar situation at the operating system level. On Unix there are setuid/setgid flags and the sudo command. They allow an unprivileged user to execute tasks, which require a higher level of privileges internally.

For example a normal user is not allowed to modify /etc/shadow. But we want users to be able to change their password. Therefore the passwd command is flagged as trusted (setuid root) and allowed to modify that file. Of course it has to perform its own security checks.

The same situation applies to the Java sandbox. For example unprivileged code is not allowed to dynamically invoke methods. But parts of the system need to do that internally.

Just like the passwd command allows normal users to change their own password only, the MethodFinder.findMethod() is supposed to only allow trusted code to call arbitrary methods.

So far, everything is fine.

Class loading exploit

ClassFinder.findClass() is such a trusted method. It loads additional classes with the the privileges of the calling code. Just like passwd lets you change your own password.

But in case of an error, it tried to recover by loading the class with full permission. If we think about passwd, a similar bug would cause passwd to change the password of root instead of the current user.

The operating systems will allow passwd to change the root password, just like Class.forName() allows ClassFinder.findClass() to load classes in the wrong security context.

Method invocation exploit

This one is a bit more complex. The trusted method is Method.invoke() here.

But there was a second trusted method involved called MethodFinder.findMethod(). To stay with the operating system analogy, think of it as a shell script that is run with root privileges via sudo.

This method/program does not validate it's parameters, it just passed them on to passwd. Now passwd is invoked in a trusted context. Therefore it will happily change the password of anyone.