The short answer
Usually you will only need connection tracking for outbound connections. If any local device makes a connection to the Internet, the firewall records that this specific IP and port tried to make a connection to the other IP and port.
Thus when the answer from the Internet arrives, the firewall knows to let it pass, because it has seen an outgoing packet before.
If there was no connection tracking in place, you would have to place explicit rules as to where from or where to a packet may travel (for example, allow all packets from google.com port 80). It's easy to see that this would make firewall rules quite complicated.
The long answer
There are only subtle differences between the two solutions if used for inbound connections. One difference is on which system the CPU load of checking packets occurs. If you have a dedicated computer/router/firewall (called firewall from here on) doing this, with connection tracking enabled, this device will check at least the plausibility of a packet. This will free the application server (the server getting the packets) from doing so.
The plausibility refers to that the packet comes from the expected host and the expected port in the case of UDP, and additionally on the sequence number in the case of TCP.
Usually the validation can only happen inside the application layer, since the validity can only be assured based upon the protocol in the application itself.
There are three consequences from these assumptions:
- The CPU and network load is moved towards the "edge" of the network in the case of state tracking. That means the (usually) first device on your network will perform some basic inspection of the packet, before forwarding it into the local network, thus keeping your internal network clean and the load on the firewall.
- The application server will only need to worry about whether the packet is valid, i.e. if the data inside the packet conforms to the expected protocol.
- This might open up the possibility to overtake an UDP connection, depending on the protocol. If the application has no (or no reliable) way to authenticate the sender of a packet, it would accept the correct payload from any IP address. For example in a data transmission it would be possible to send garbage to the receiver, if the UDP packet passes validation.
Nevertheless common practice (unless in high-security environments) is to use (general) connection tracking for outgoing connections only, as to not having to explicitly allow every Internet server to answer to a request from the local network. For the case of incoming connections, you usually really just allow the (defined) port to be reachable (-A INPUT -p tcp --dport 80 -j ACCEPT). This would apply to new connections, but would also apply to existing connections, avoiding the overhead, to keep track of the connection. Similarly you would only forward packets to a specific destination port to an application server inside your local network.
application-specific connection tracking
There is one case I did not mention yet and that is application-specific connection tracking. For a few protocols (ftp, irc, tftp, amanda, sip and possibly others, see http://www.iptables.info/en/connection-state.html#COMPLEXPROTOCOLS for example) there may be special iptables modules to allow advanced tracking of the connection. For example in FTP, you would open a connection from the client to the server on port 21, called the control connnection, which should be opened in the firewall by normal means. Inside this channel, the client and server then define a second connection where the raw file data would be sent. Usually this connection is established from the server to the client (in ACTIVE mode). This local and remote port are seemingly random, so they cannot be defined in a static firewall configuration file. If there was a firewall with iptables on the client's network, it would deny this connection, because there is no former association. With application-specific connection tracking the module would look "into" the packet content to see whether such an agreement has been made and the firewall would allow a new incoming connection on the agreed port, even if there was no connection on it before.