What is the best protocol for client-server communication? TLS or IPsec? I have a public channel where users communicate with each other using a digital certificate like X.509.
It depends on your use case which is not really clear from your question.
What is the best protocol for client-server communication?
If you only care about securing data transfer between two applications (i.e. typical client and server scenario) SSL/TLS like used in the browsers is probably better because only changes on the applications are necessary.
IPSec on the other hand is more suitable for connecting networks with each other, i.e. VPN. In this case not only the traffic between two applications need to be secured but every traffic between two computers, independent of the application. This requires deeper changes on the systems which usually require administrative rights and thus is too complex if you just want two applications communicate securely with each other. You could also use SSL based VPN solutions instead of IPSec to create a VPN between various systems and it is more a question of usability and coexistence with other technologies (like traversing firewalls or NAT) if you choose IPSec or SSL VPN if you need network level instead of application level secure connectivity. Both offer adequate security if properly set up.
L2TP/IPsec is a good choice if OpenVPN isn't supported by your device and security is top priority. OpenVPN is the recommended protocol for desktops including Windows, Mac OS X and Linux.
Check that link, there is more information there.