Performance was once the biggie, but there's also the impact on low-level access and operations.
A good aspect to explore might be DRM. DRM is bad (or good) because it makes it hard to access your data in ways that you desire to. Almost always, there are very good reasons that you might want to access information in a non-standard or out-of-band way that the encryption system does not permit. For debugging, information transfer, monitoring, administration, maintenance...
As I understand it, the introduction of DRM into Windows added a significant burden for system designers, who, in order to get the "compatible" badge, had to create non-tappable video paths (that is, no test points or exposed wire, no data tracks running along the surface of the PCB, no exposed data pins on chips: how did they debug the signals in these devices?); and reading hardware that could not easily be reprogrammed (so no upgradable firmware, no replaceable proms...).
The requirement of full end-to-end security is that even when someone has access to your machine, they cannot access the data you send through those channels. Unfortunately, the MS approach was to protect Hollywood data (video bus), but leave user-data (cables to hard drive, keyboard, mouse, etc) in the clear, unprotected. Thus, DRM is useless for protecting user-data.
Another good thing to explore is failure modes. Prior to encryption, anything that went wrong with my hard drive (board failure, corrupt sectors, etc) I felt confident that I could resolve and repair - and time and again, I had recovered my data. But after encryption, when my boot sector (which held the encryption key) was overwritten on my encrypted drive, I lost decades of work and it cannot ever be recovered. Unlike any normal drive corruption, you cannot use any recovery tool known to man to recover data from damaged keys.
Offline and out-of-band tools cannot work on an encrypted datastream. You cannot perform partial offline backups of the partitions: you can only copy the entire encrypted partition, or mount it to copy folders. You cannot fsck an unmounted encrypted disk, view it with hex editors, virus scan it, run a meta-analysis to see how many png headers are on the drive, etc: you must mount it.
You cannot packetshape encrypted transmissions, inspecting packets to ensure that the most important traffic makes it through while unimportant emails and stuff get pushed to the back of the queue. You cannot filter encrypted emails for spam, viruses, or other unwanted content.
Can you debug an encryption application? If yes, then what's the point in encrypting, if another app can just hook it? If no, then how can you detect and fix bugs in it?
So: in sum, I'd argue that over and above the effect on server load, the effect on administrative tasks is greater.
There's also a false sense of security. If you have encryption on your system, you feel safe. Nobody can steal your data. Except, to use the data, you must mount it... so the data is available in the clear while it is mounted, available to read or send over any internet connection by any malware your system picks up. It's only actually protected when it's off! A server that's on 24/7 and has its encrypted partition constantly mounted, has gained exactly zero benefit from this protection.
All this said, I am an absolutely vehement proponent of encryption. I feel it should be built into everything, and should default to on for all forms of communication, even in the absence of confirmed identity.