Probably not applicable to newer stuff, but historically I think it was big dish and complex custom protocols rather than encryption that "protected" satellite communications. Consider the case of an older satellite that NASA allowed amateurs to try and connect to, and the obstacles they faced and overcame:
How to talk to a 36-year-old Space Probe
As the primary link is now offline, some relevant snippets:
Actually, one of the most difficult parts of this project was sifting
through hundreds and hundreds of pages of specifications from NASA.
...
Now that we had a system that was capable of transmitting correct
commands to ISEE-3, we had a little bit of guess work to do. There
were a number uncertainties about the behavior of the space craft
receiver. Ambiguous specifications on items like receive lock range,
phase modulation index, and mark/space tones, left us with some
uncertainties to deal with. Also, a rough link budget showed that we
should have about 5 dB of margin under ideal circumstances. If we had
any significant pointing errors, or there were other factors we hadn’t
accounted for, there was a decent chance that we were operation near
the best sensitivity of the space craft receiver. This meant we could
have a marginal link with frequent bit errors.