Is using IPTables to only allow traffic between two servers considered an acceptable practice? For example TCP traffic on a defined port. I realize this is referred to as an ACL but how secure is this when an encrypted transport is not available? If it is not recommended, why?
It depends what you are trying to protect against, if this traffic goes over the internet, technically people would still be able to sniff or modify your traffic if they were able to put themselves between your two machines. IPTables is a firewall but it does not offer means of encrypting the traffic. Therefore you need to depend on other protocols like encrypted VPNs or IPsec.
Practically, it's relatively, commonly deployed on Linux machines unless you have dedicated firewalls. For instance I have implemented it to only allow SSH to be accessible from a certain amount of public and local IPs.
Network security is best performed in layers. You're right in that this works similar to an ACL.
If you don't control all the devices between those servers, remember that this does not protect against eavesdropping, MITM, or spoofing attacks. For that, you'll still need crypto. If your server hardware doesn't support that, you may want to consider VPN appliances.
That said, an ACL will limit the number of devices that can reach AND respond to your service to just the devices between your two servers. IOW, from every skiddie on the internet to only elite hackers, ISPs, and the government.
It's not ideal, but better than leaving it wide open.