I was running MQTT broker in my pc and tried to connect to it via mqtt client in same PC. using x509 cert for mutual tls. SSL Handshake was failing with RST from server. This was happening only in my PC. When I tried to do the same setup in VM running in my PC, it works. Not sure what's going wrong. Can someone please help me troubleshoot this.
Wireshark logs:
192.168.1.12 192.168.1.12 TCP 56 56793 → 8883 [SYN] Seq=0 Win=8192 Len=0 MSS=65495 WS=1 SACK_PERM
192.168.1.12 192.168.1.12 TCP 56 8883 → 56793 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=65495 WS=256 SACK_PERM
192.168.1.12 192.168.1.12 TCP 44 56793 → 8883 [ACK] Seq=1 Ack=1 Win=8192 Len=0
192.168.1.12 192.168.1.12 TLSv1.2 197 Client Hello
192.168.1.12 192.168.1.12 TCP 44 8883 → 56793 [ACK] Seq=1 Ack=154 Win=2161152 Len=0
192.168.1.12 192.168.1.12 TCP 44 8883 → 56793 [RST, ACK] Seq=1 Ack=154 Win=0 Len=0
Client hello (from failure PC):
Transport Layer Security
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 148
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 144
Version: TLS 1.2 (0x0303)
Random: 6415564f8d4243a378676945ae90bcdd3865458b456e3c2b82d1e3fc46832daa
Session ID Length: 0
Cipher Suites Length: 42
Cipher Suites (21 suites)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
Compression Methods Length: 1
Compression Methods (1 method)
Extensions Length: 61
Extension: supported_groups (len=8)
Extension: ec_point_formats (len=2)
Extension: signature_algorithms (len=26)
Extension: session_ticket (len=0)
Extension: extended_master_secret (len=0)
Extension: renegotiation_info (len=1)
[JA3 Fullstring: 771,49196-49195-49200-49199-49188-49187-49192-49191-49162-49161-49172-49171-157-156-61-60-53-47-56-158-159,10-11-13-35-23-65281,29-23-24,0]
[JA3: 1a2e744230d93b103604fc70cf0ba509]
Client hello (from success PC):
Transport Layer Security
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 148
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 144
Version: TLS 1.2 (0x0303)
Random: 64154e9a612aca60229078178df4ce178d9eabaea00dded4ccc99dc6fbc8108a
Session ID Length: 0
Cipher Suites Length: 42
Cipher Suites (21 suites)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Compression Methods Length: 1
Compression Methods (1 method)
Extensions Length: 61
Extension: supported_groups (len=8)
Extension: ec_point_formats (len=2)
Extension: signature_algorithms (len=26)
Extension: session_ticket (len=0)
Extension: extended_master_secret (len=0)
Extension: renegotiation_info (len=1)
[JA3 Fullstring: 771,49196-49195-49200-49199-159-158-49188-49187-49192-49191-49162-49161-49172-49171-157-156-61-60-53-47-10,10-11-13-35-23-65281,29-23-24,0]
[JA3: c12f54a3f91dc7bafd92cb59fe009a35]