I have read the Wikipedia article on firewalls but don't understand firewall security and how an attacker from the outside can bypass the firewall to hack target system.
We all know it happens but what methods make it possible to do that?
Firewalls aren't "bypassed" in the sense Hollywood would have you believe. They operate by checking incoming and outgoing traffic against a set of rules. These rules might be based on metadata (e.g. port number, IP address, protocol type, etc) or real data, i.e. the payload of the packet.
Modern firewalls are usually comprised of the following rule sets:
Bypassing a firewall isn't really something that can be done. All traffic that goes through it is filtered according to the configured rules. However, a firewall only does what it is told - a misconfigured or out of date firewall might allow an attack through.
Ways I can think of to get round a firewall:
The easiest way to get around a firewall is what is known as 'client-side' attacks. If a computer on the protected side of the firewall makes a valid connection to an attacker, there is nothing to trigger a typical firewall rule. For example, if a firewalled computer makes an HTTP connection on port 80 to a website designed to exploit browser (or Java) vulnerabilities, there is little for the firewall to recognize as malicious: web traffic over a web port.
Once a foothold is gained within the network, the attacker can set up encrypted tunnels that pass through the firewall on allowed ports, which is another kind of 'bypass'.
On the topic of direct firewall attacks, tools exist to map out how a firewall is configured for various ports. With this information, traffic can be configured to pass through the firewall. At the simplest level, fragmenting packets can be effective in not triggering various firewall and IPS rulesets because each packet does not contain enough data. The firewall has to be configured to store the entire fragmented packet set before inspection.
The answer really depends on your definition of "bypassing".
The most important factor in ensuring a firewall provides maximum protection is to ensure it is configured appropriately. A firewall is a dumb device in the sense that you must configure what you'd like it to allow through/block. A poorly configured firewall will leave gaping holes in your attack surface. If an attacker gets in, it's not the firewall's fault; it was just doing what it was told. One could argue that the firewall hasn't technically been "bypassed" because it was never told to restrict the relevant traffic in the first place.
Depending on the feature-set of the firewall, it will only allow you to restrict access in certain ways. Although some penetration techniques might try to exploit a vulnerability or weakness in the firewall's software - which I guess you could class as "bypassing" - the majority of techniques are focused on exploiting poorly configured firewalls (see point above), or systems that are behind the firewall. As an example, if you have a poorly configured SSH server behind the firewall, then it's not the firewall's fault that the attacker was able to authenticate as root with "password" as a password. The firewall was configured to only allow access via port 22 (SSH), so it's done its job. Again, one could rightly argue that the firewall hasn't been bypassed in this situation, but someone's still got into your network.
Some firewalls offer more advanced features such as intrusion prevention and application layer filtering. IPS firewalls make an attempt to understand the content of the traffic that's flowing and block some common methods of exploiting weaknesses in systems hosted behind it. Again, this relies on careful configuration to be effective. If you haven't enabled the correct IPS protections, then it's not the firewall's fault if someone successfully exploits that vulnerability. Some penetration techniques exist which try to slip traffic past these protections in a form that doesn't trigger the block, but still exploits the weakness. It's a cat-and-mouse game similar to anti-virus. I guess you could call these "bypassing" the firewall.
In short, a firewall is only as good as the admin who is configuring it, and it can only be expected to restrict traffic based on its capabilities. It's no substitute for hardening the systems behind it, which is where most attacks will focus.
Firewalls are core elements in network security. However, managing firewall rules, especially for enterprise networks is a complex and error-prone task. Firewall filtering rules have to be carefully written and organized in order to correctly implement the security policy. In addition, inserting or modifying a filtering rule requires thorough analysis of the relationship between this rule and all other rules in order to determine the proper order of this rule and commit the updates. Identifying the anomalies in Firewall rule configurations is a much heated research topic and there is a lot of research on it some of which i find interesting is.
The goal of the attacker is to expoilt these anomalies in firewall configurations and it is done through firewall fingerprinting in which he send benign packet to guess firewall rules and find loopholes in them .To prevent such sort of exploitation most firewall are deploy behind IPS in a pattern call DMZ where IPS tries to prevent firewall fingerprinting through heuristics or statistical measurement (entropy) i.e. port scanning .