TL;DR - There was a Network Load Balancer (NLB) in the private subnet in front of the host in the private network. That NLB had a public IP.
When there is a Public IP associated to the network interface, traffic from the client public IP (source) will reach the ENI provided the security group and ACL allows the traffic to come in, regardless of whether there is IGW as default route or not.
Because an AWS NLB does not use security groups and no NACL was in place, routable traffic was able to reach the private subnet. The security group on the host rejected the traffic and that's what was showing up in the Flow Logs.
Response traffic would not make it out of the private subnet because the route tables are followed. Route tables aren't used to control inbound traffic but do control responses.
Can you clarify why an NLB in a private subnet even with a publicly routable IP address can receive traffic when it's subnet does not route through the Internet Gateway?
From a port scanners perspective they would have received no response from the NLB.
SYN from the client will reach the NLB, but no SYN-ACK would be sent to the client as now route table would be evaluated for response traffic and as there is no route to IGW the response traffic would be dropped.
To summarize, NLB or your backend target are not responding to internet traffic. Traffic would be accepted by Public IP on private subnet regardless of the route table default route.
Reference: AWS Support
