I am trying to get my head around the setup above. A simple netcat session between two hosts and a pentesting machine is trying to get that established session to be terminated by only knowing source and destination IP:port combination. Shouldn't be sending a FIN packet with source IP:sport to destination IP:dport be sufficient to tear down this connection? I am using hping3 to craft packets but netcat session doesn't drop. Can you point me to some IEEE document where protection mechanism is explained or if not then do you think it is a application dependent protection?
To successfully close a connection the FIN packet must be acknowledged in both directions. One side of the connection sends a FIN packet, to which the other side must return a FIN+ACK, and then finally the original side must send back another ACK. You must spoof both parts of this handshake in order to be successful.
To perform this attack you need to know the source IP, destination IP, source port, destination port, and the TCP sequence and acknolwedgement numbers. These numbers are randomly selected 32-bit numbers generated as part of the TCP handshake, incremented by the length of each packet sent/received. Their primary use is to ensure strong packet ordering, but their values are also useful to protect against spoofing attacks. Packets with invalid sequence or acknowledgement numbers (i.e. values outside the transmit or receive window) are ignored. You would generally obtain these numbers by packet sniffing if you're on the same network as them. You can also try to exhaustively guess the number, but this is noisy and slow; an exhaustive brute-force of the sequence and acknowledgement numbers requires a minimum of 4 billion packets.
A more common attack is a TCP reset attack, which is easier since an RST packet can be sent without requiring two-way closure and you only need the correct sequence number since the ACK flag is not set.
You are probably messing up when the sequence number of the FIN doesn't match up with the next expected sequence number. As to my knowledge, non-consecutive sequence numbers not within the transmission window are discarded.