I would like to design a client-server application where the server is placed on Internet. I assume that I could set up the client-server connection using VPN (is it using IPSec?) or using a SSL connection (possibly https). What are the differences between VPN/IPsec and SSL/https for securing a client server connection over Internet?
VPN means "Virtual Private Network". It is a generic concept which designates a part of a bigger network (e.g. the Internet at large) which is logically isolated from the bigger network through non-hardware means (that's what "virtual" means): it is not that we are using distinct cables and switches; rather, isolation is performed through use of cryptography.
SSL (now known as TLS) is a technology which takes a bidirectional transport medium and provides a secured bidirectional medium. It requires the underlying transport medium to be "mostly reliable" (when not attacked, data bytes are transferred in due order, with no loss and no repetition). SSL provides confidentiality, integrity (active alterations are reliably detected), and some authentication (usually server authentication, possibly mutual client-server authentication if using certificates on both sides).
So VPN and SSL are not from the same level. A VPN implementation requires some cryptography at some point. Some VPN implementations actually use SSL, resulting in a layered system: the VPN transfers IP packets (of the virtual network) by serializing them on a SSL connection, which itself uses TCP as a transport medium, which is built over IP packets (on the physical unprotected network). IPsec is another technology which is more deeply integrated in the packets, which suppresses some of those layers, and is thus a bit more efficient (less bandwidth overhead). On the other hand, IPsec must be managed quite deep within the operating system network code, while a SSL-based VPN only needs some way to hijack incoming and outgoing traffic; the rest can be down in user-level software.
As I understand your question, you have an application where some machines must communicate over the Internet. You have some security requirements, and are think about either using SSL (over TCP over IP) or possibly possibly HTTPS (which is HTTP-over-SSL-over-TCP-over-IP), or setting up a VPN between client and server and using "plain" TCP in that private network (the point of the VPN is that is gives you a secure network where you need not worry anymore about confidentiality). With SSL, your connection code must be aware of the security; from a programming point of view, you do not open a SSL connection as if it was "just a socket". Some libraries make it relatively simple, but still, you must manage security at application level. A VPN, on the other hand, is configured at operating system level, so the security is not between your application on the client and your application on the server, but between the client operating system and the server operating system: that's not the same security model, although in many situations the difference turns out not to be relevant.
In practice, a VPN means that some configuration step is needed on the client operating system. It is quite invasive. Using two VPN-based applications on the same client may be problematic (security-wise, because the client then acts as a bridge which links together two VPN which should nominally be isolated from each other, and also in practice, because of collisions in address space). If the client is a customer, having him configure a VPN properly looks like an impossible task. However, a VPN means that applications need not be aware of security, so this makes it much easier to integrate third-party software within your application.
Both have security issues if not configured correctly. But first lets start with some definitions:
Cisco have a good definition of a VPN:
As for SSL:
In relation to your question, the main difference is that SSL often makes use of the browser to encrypt data between end user and the server, and is commonly used for areas of websites that require the protection of confidentiality and integrity of the data.
VPN/IPSEC requires specific VPN Client software and is generally for providing remote access to systems or networks. Also there is the option to go for L2TP or L2F instead of IPSEC.
However, SSL VPNs are becoming more prevalent as a means to provide access to networks / systems via the web browser. This approach has many benefits as it uses the common web browser to enable the secure connection. The granularity of this approach is also a good way to control accesses to specific applications.
As for security issues -
Are you looking at these options to create a secure VPN? SSL is generally easier to deploy and better supported for a desktop-to-network type of VPN, such as when an employee at home is connecting to the corporate network. If you're doing a more complex deployment, such as a network-to-network encrypted VPN (e.g., between two different organizations), then IPSEC will provide better control and more customization options.
There is a decent white paper on the topic by Juniper Networks, although it might be skewed to the strengths of their products.
Some very good answers here, I won't repeat what was already said.
As such, IPsec is usually more fit for a controlled network, and less so over the wild wild unknown Internet. See some more info at this other question: "IPsec (Internet Protocol Security) facts".
Thus, getting back to your actual question, in almost all cases where you're putting the server on the Internet, you wouldnt expect your users to connect using a VPN. (Exceptions exist, of course.)
Well, the difference is kind of like the difference between a circle and a square (both are shapes, but differ greatly). They both secure communications, but do it at different levels and in different ways. IPSEC is wireline encryption and authorization whereas SSL is application-specific.
IPSEC has access control whereas SSL does not.
Can you be more specific with what you are trying to figure out?
This could be a very long answer, but I'll try the short one.
When you use https, your browser (acts as a SSL client) will only encrypt this connection to the webserver.
When you use VPN, you need a special client and establish a tunnel between the client and the server. Then you can configure which traffic goes through the tunnel. This can be everything or just your http traffic.
When you only want to set up a client/server application which can communicate with http, the easiest solution should be the https traffic, when it needs to be encrypted. It is much more complicated to setup a VPN and maintain it.
This depends on your threat model, on the nature of the client server protocol you need, and on your customers.
Is this intended for unsophisticated end-users? Then use SSL - at this point VPN complexity will just turn off a lot of potential users.
Does the server ever need to asynchronously notify the client of something? Then HTTPS may not be what you want (though it can be made to do so).
How big a risk is phishing? If it would be easy for attackers to lure folks to them as a MITM, SSL is probably better since it authenticates each server to the client. A typical VPN, once set up, doesn't help the user avoid an attacker who has gotten into other hosts on the VPN. This would probably not be a huge risk but again it depends on what you're doing.
If you're deploying this in the cloud (both client and server), then you may get a VPN of sorts almost for free which may address some very casual threats.