- Link Layer
- Internet (IP) Layer
- Transport Layer
TLS operates between the Transport layer and the Application Layer (kind of). Really it just wraps Application Layer traffic in encryption during transport.
The TLS Key Exchange happens in the in between layers. Here it's not really Transport Layer because things like port numbers, and sequences numbers are already in place at the Transport Layer. It's only sending data to establish encryption protocols so that it can wrap the Application layer.
IPSec sort of does the same thing, only in an earlier layer. IP Security protects everything above the IP layer. Transport Security protects everything above the Transport Layer. The key exchanges for both are in a weird place because they just need to send data to establish the secure layer. A lot like how ICMP is considered an IP protocol, but it still contains data after the IP layer. Does this make it Transport Layer? No.
The OSI model has a bit more granularity.
- Physical Layer
- Data Link Layer
- Network Layer (IP)
- Transport Layer (TCP)
- Session Layer (TLS)
- Presentation Layer
- Application Layer (HTTP)
TLS establishes an encrypted session. In the OSI model this is where TLS operates. It sets up its session, and adds a layer of encryption for the Application Layer (HTTP).
A related answer.
Parsing out TLS
In the Transmission Control Protocol Header there is a Data Offset field. This field represents the size of the TCP header in 32-bit words. The minimum size for this value is 5 (20 bytes). This is at a fixed offset from the beginning of the TCP header. Using this value you'll know the size of the TCP, and can use that to calculate the beginning of the TLS portion.
A TLS Record will always start with a content type, and then the SSL version. There are only 5 different content types, and 4 different versions to worry about. So using the first 3 bytes of the data following the TCP header you should be able to determine if SSL/TLS is being used.
I would run Wireshark on your machine (non-work), and filter on "ssl". Then just browse to https://www.google.com. You'll be able to see the entire handshake. All the layers are broken out for you. It'll give a good representation of how the layers are constructed with TLS, and you'll be able to see each of the TLS Records.