So last semester I studied a bit about network hardening and one of the things that we did was write IP table rules to ensure that only a whitelisted set of services are able to talk to each other between machines.
But what this essentially means is that any two services running on specific ports are able to talk to each other. It doesn't quite bind the service with the port and I guess depends on the user to ensure that it is only the intended services that run on those ports.
Is there some way to ensure maybe have the service associated with the IP table rules? Or something other than IP tables that does this for you?
I am sure somebody somewhere would have exploited IP table based hardening.
EDIT To be clear, my goal is:
I want to have an application level restriction instead of port level. I want to restrict communication such that only specific services can talk to each other instead of a port based system like IP tables. For e.g. I just want MySQL services running on two hosts to talk to each other instead of saying only port 5000 should be used for communication and depending on the user to ensure that no other service is using port 5000
I am not asking for any product recommendations. This is for me to learn about security. I am guessing that you can do a signature based filtering of packets but it may not work on encrypted packets.