Can I 100% rely on $_SERVER[] to be a safe source of data that I do not need to sanitized like I do $_GET[] and $_POST[]?
Your question immediately indicates failure. All sources of input must be sanitized. Input is not just considered channels that the user can directly control, but all sources of data outside of your application.
Think of it this way, your application has 2 ways of getting data: information that's hard coded in your application, and input. Even if it's generated by another program on the same system, it's still input to your program.
The common idiom Filter-In, Escape-Out applies not just to user input, but to anything that enters and leaves your application.
So if it's in $_SERVER, it MUST be filtered/sanitized. You should never rely upon anything that is not hard-coded in your application.
Why is this important? Let's imagine that you filter all input from the user, but then trust data that comes from your database. If I can exploit a hole in your filtering, I can inject data that then becomes trusted. This can result in second-order XSS or SQLi. But if you filter everything that comes into your application, when it comes in, regardless of where it comes from, then you'd be safe!
So no, you can never 100% rely upon anything in $_SERVER, $_GET, $_POST, $_COOKIE, $_REQUEST, $_ENV, $argc, $argv, from your database, from the filesystem (aside from your version-controlled code), etc...