The JAVA software provided by Oracle is a security mess. But what exactly, in that software package is the most vulnerable. If someone install it and disables the Java plugin in the browser, will he still be vulnerable?
The most problematic part with Java is that it offers a way to sandbox code for restricted execution but that this protection could often be bypassed in the past. Thus the main problem is the use of Java in scenarios where this sandbox gets used, which is mainly in the browser. When used instead with standalone applications and not applets the security of Java is similar to that of other programming languages which mostly don't even offer such sandbox facility.
If you make sure that Java is not usable as a plugin by any browser on your system it is as safe to use as other languages, i.e. similar to applications written in C/C++/C#/whatever these standalone Java applications will have full access to everything your user account has. This means you have to still watch out where you get your application from and what it does, but you don't have to take special precautions just because this is a Java application.
For further information you might look at the discussion Why do I hear about so many Java insecurities? Are other languages more secure?.