With regards to encryption and preventing a 3rd party from viewing my emails (and ignoring validity/signing), is encrypting an email via PGP or S/MIME useful when I control my server and my recipient controls their server?
In practice: No, but you need to make sure many other elements of the network aren't fooling you. I will concentrate on the transmission here. End-to-End encryption gives to extra security on top of other elements, described here. If some elements fail, GPG still protects the content of your mails.
First, you want to make sure, you can contact the right recipient’s server. To do that, your sending mail servers does a DNS-Lookup for the MX-Record of the domain. Then the resulting FQDN has to be resolved to get an IP. This is the first possible attack vector, as the used DNS-Server must be trustworthy and not controlled by the attackers. Additionally, the DNS can also be fooled. You may have a look on DNSSEC or DNScurve, both want to ensure, the answer from a DNS is authoritative, as the DNS protocol is completely unencrypted and tampering is very easy.
A possible solution is to overwrite the domain -> MX and A lookup on your server, by specifying a custom transport in your mailserver explicitly ("use this server to deliver mails to that domain").
The next possible problem is, you can't be sure you're talking with the real other side, even with SMTPS (SMTP over TLS) or SMTP+STARTTLS, as usually server certificated are not checked. Thus, the sending server must check the other's server certificate, and also the other way round: The receiving server has to check the client certificate of the sending server, because otherwise the receiver can't be sure, the trusted server sent the mail. Also make sure the connection can't be downgraded to plain SMTP.
As CA's and the PKI-System also have their problems, certificate pinning is appropriate. Others already brought up the issue of backups.