In a recent pentest (Java thick client), it was discovered by the pentesters that TLS version 1.3 is not supported. I'm investigating why it is not supported, so I was looking in the Java code for where TLS 1.3 is defined. However, someone told me that the TLS version is related to OpenSSL on the server itself, so it is not related to the Java code. But in Java code they only exclude old versions of TLS like 1.0 and 1.1 but they don't define which version of TLS. Is that true?
is tls version related to server?
Yes, it is related to the server as well as the client. The client makes the first move in establishing the TLS connection.
If the client is trying to use TLS1.3, but the server does not support TLS 1.3 it will inform that client of this during the TLS handshake and the connection will proceed via TLS 1.2.
but in java code they only exclude old versions of tls like 1.0 and 1.1 but they don't define which version of tls, is that true?
There's lots of different "java code" out there. We don't know what code you are referring to.
On the client side, when you setup the TLS (or "SSL") socket, you can likely pass parameters, or if not you accept the default. You may need to figure out what version of Java you are using and whether that version even supports TLS1.3. If you are using Java 11 you have TLS1.3 support as do some versions of Java 8, but not Java 7.
The TLS connection starts with the ClientHello. If you need to treat the client as a "blackbox" you can tell if the client is attempting to use TLS1.2 or TLS1.3 by looking at the ClientHello, e.g., in Wireshark. Unfortunately, both TLS1.2 and TLS1.3 use the same "version number" (0x0303 indicating TLS1.2) for compatibility reasons. So, you have to look at the Supported Version Extension to see whether 0x0304 (TLS1.3) is listed. But, Wireshark will do this analysis for you.
In addition, you can use Wireshark to observer a connection between a modern client and your web server. You will likely see that the client tries a TLS 1.3 handshake, but the server forces TLS 1.2. Wireshark will label some packets of the TCP stream of the connection with TLS1.2 labels rather than TLS1.3 labels. Another way to see if TLS 1.2 is in use it to see if you can observe the Server Certificate during the handshake. If TLS 1.2 is in use you will be able to see the plaintext certificate, if TLS 1.3 is in use you will not be able to see the plaintext certificate.
In the above answer, I have been discussing client and server as if these are the only two computers involved in the data transmission process.
A commentator has requested a few words about TLS termination endpoints. This is a device that might, for example, sit in between a client and an HTTP server and perform decryption of HTTPS traffic before passing it on to the HTTP server.
The commentator suggests that if OP's Java server application is not a TLS termination point then the TLS configuration in Java on the server side is irrelevant. In this case OP should look at configuration of the TLS termination endpoint rather than the server application.
In general, there may be lots of different devices in between the client and the server. For example, a TLS termination device, or a load balancer, or a MITM proxy (used, e.g., by corporations to inspect work computer traffic) not to mention lots of switches and routers.
Devices that manipulate TLS such as TLS termination devices or TLS-MITM proxy devices can also cause TLS downgrade (e.g., from TLS1.3 to TLS1.2) and so if both the client and the server ostensibly want to use TLS1.3, but TLS 1.2 is still in use, the culprit might be one of these devices.
You first have to understand the TLS-Handshake.
Then you understand that both sides have an affect on which TLS Version is used for the connection.
The Client send the server a List of which TLS Versions it supports.
The Server decides which TLS Version will be used for the connection. It chooses the newest Version of the Clients List, which the Server itself supports.
In most cases the Server will support not only TLSv1.3 to support clients that aren't able to use TLSv1.3.
Server Supports Client Supports TLSv1.3 TLSv1.2 TLSv1.2 TLSv1.0 The Server selects TLSv1.2
Server Supports Client Supports TLSv1.3 TLSv1.2 TLSv1.0 A connection between the Client and the Server is not possible.
Server Supports Client Supports TLSv1.3 TLSv1.3 TLSv1.2 TLSv1.2 The Server selects TLSv1.2
Conclusion: You can influence the used TLS Version on both sides. But the Server is the decision maker. A malicious person could trick the server to use a lower Version of TLS by pretend it only supports TLSv1.0 even if it is capable of using a higher version. For this reason it is important to always stay up to date and disable older TLS Version on the Server side from time to time.
At this point of time (May 2022) it is pretty normal to have Servers support TLSv1.2 and TLSv1.3 side by side.