Why do most VPN protocols work on the Transport Layer and not on the Network Layer instead?

Question

In both the TCP/IP model and the OSI model, IPSEC offers encryption on the Network Layer (IP), while other VPN protocols such as OpenVPN offer it on the Transport Layer (TCP).

Due to encapsulation obviously more information is encrypted on the lower Network Layer (IPSEC) then on the higher Transport Layer (TCP).

Why do most VPN protocols work on the Transport Layer and not on the Network Layer instead?

Answer 1

I think you are confusing what the VPN is using to transport the encrypted data vs. what the VPN provides. You look at the first while you should instead look at the second.

A VPN provides a virtual network, i.e. usually at layer 3 (there are also layer 2 VPN). This is true for IPSec, OpenVPN, Wireguard, ... . This can be seen in that it provides virtual network interfaces (tunnel interfaces) which have their own IP addresses and that the traffic gets simply routed trough these interfaces in order to get encrypted between the tunnel endpoints. So each of these layer 3 VPN is an IP encapsulation.

For deciding which layer the VPN is does not matter how the VPN protocol then transports the encapsulated and encrypted traffic. This can be IP (as in usual IPSec), UDP (as in wireguard, OpenVPN and UDP encapsulated IPSec), it can be TCP (another option offered by OpenVPN), one can even use layer 7 for example for sending the traffic through Websockets in order to bypass restrictive Firewalls. No matter what it is used - from the perspective of the tunnel endpoint they get still a network interface at layer 3 with its own IP address.

Answer 2

They do.

Why do most VPN protocols work on the Transport Layer and not on the Network Layer instead?

Assuming you are referring to your VPN as an application on your phone/laptop—all your information in the packet(s) you send are encrypted at the datalink layer.

Router wireless receiver/transmitter (WRT) works specifically at the network layer.

Answer 3

Why do most VPN protocols work on the Transport Layer and not on the Network Layer instead?

Most VPN/tunnel protocols transport (encapsulate, protect) network layer (IP) or sometimes link layer (Ethernet, PPP) protocols. The main exception is IPsec in transport mode which encapsulates (protects) the transport layers.

However, many VPN/tunnel protocols work on top of a transport layer (UDP, TCP) or some higher (application) layers (TLS tunnels) and not directly on top of the network layer (IP). Some reasons might include:

• this approach is friendlier with NATs and firewalls;
• this makes it is easier to run several instances of the same service on different ports;
• adding a new protocol directly on top of IP requires the allocation of a a protocol number whereas you can always run your transport-based service using some ephemeral port (WireGuard does no have any reserved port).

Let's look at ESP (IPSec) transport mode:

[...        ]
[TCP,UDP,...]
[ESP        ]
[IP         ]
ESP transport

It works on top of the network layer (IP) and protects the transport layers (TCP, UDP, etc.). The source and destination IP addresses are not protected but an MITM does not know which transport protocol is used and, which ports are used.

Now let's look at ESP (IPSec) tunnel mode:

[...        ]
[TCP,UDP,...]
[IP         ] <- inner IP
[ESP        ]
[IP         ] <- outer IP
ESP tunnel

It works on top of the network layer (IP) and protects a tunneled network layer (IP). The outer IP header contains the IP addresses of the tunnel endpoints: this information is available to a MITM. However, the outer packet which contains the address and source IP addresses of the peers is protected.

IPSec ESP can work on top of UDP as well:

                                  [...        ]
[...        ]                     [TCP,UDP,...]
[TCP,UDP,...] <- inner transports [IP         ] <- inner IP
[ESP        ]                     [ESP        ]
[UDP        ] <- outer transport  [UDP        ]
[IP         ]                     [IP         ] <- outer IP
IPsec ESP/UDP                     IPsec ESP/UDP
transport                         tunnel

In transport mode, it works on top of a transport layer (UDP) and protects the transport layers. Not that the ports of the outer transport (UDP) layer does not need to match with the ports of the inner transport layers.

In tunnel mode, it works on top the of a transport layer (UDP) and protects the network layer (IP).

OpenVPN, WireGuard (and the OpenSSH tunnel mode) are similar in this regard:

[...        ]           [...        ]
[TCP,UDP,...]           [TCP,UDP,...]
[IP         |TLS    ]   [IP         ]  [IP  ]          <- inner IP
[OpenVPN            ]   [WireGuard  ]  [SSH ]
[TCP or UDP         ]   [UDP        ]  [TCP ]
[IP                 ]   [IP         ]  [IP  ]          <- outer IP
OpenVPN                 WireGuard      OpenSSH tunnel 
(TUN mode)                             (TUN mode)

These ones work on top of a transport layer (TDP or UDP) and protect the network layer (IP).

For completeness, OpenVPN (and OpenSSH) can transport a Ethernet frames (link layer) as well:

[...     ]
[Ethernet|TLS] [Ethernet]
[OpenVPN     ] [SSH     ]
[TCP or UDP  ] [TCP     ]
[IP          ] [IP      ]
OpenVPN        OpenSSH
(TAP mode)     (TAP mode)

Examples of TLS tunnels include MS-SSTP and FortiSSL:

[...        ]
[TCP,UDP,...] [...        ]
[IP         ] [TCP,UDP,...]
[PPP        ] [IP         ]
[SSTP       ] [PPP        ]
[(HTTP)     ] [(HTTP)     ]
[TLS        ] [TLS        ]
[TCP        ] [TCP        ]
[IP         ] [IP         ]
MS-SSTP

The tunnel works on top of TLS/HTTP(S) and transport PPP (link layer) (which itself probably always transport only IP traffic):

Summary

Tunnel type	Works on top of	Transports/encapsulates
ESP transport mode	IP	TCP, UDP, etc.
ESP tunnel mode	IP	IP
ESP/UDP transport mode	UDP	TCP, UDP, etc.
ESP/UDP tunnel mode	UDP	IP
OpenVPN TUN	UDP or TCP	IP
OpenSSH TUN	TCP	IP
OpenVPN TAP	UDP or TCP	Ethernet
OpenSSH TAP	TCP	Ethernet
Wireguard	UDP	IP
MS-SSTP	(HTTP)/TLS/TCP	IP/PPP
FortiSSL	(HTTP)/TLS/TCP	IP/PPP