Comprehensive audit of firewall - is Internet truly not accessible?

Question

We have many Linux servers in several co-locations over the world. By design, these machines generally should not be able to access the Internet. Depending on location and role, some sites may have very limited Internet access (e.g. default deny with one or two sites white listed). So depending on the circumstances, the no-Internet policy is configured either by routing rules at the switch level, or by a filter device (e.g. Untangle).

One team is responsible for configuring the network per policy, and a separate team (me) is responsible for periodic auditing to make sure things are actually working as intended.

The question, therefore, is how to comprehensively check that the Internet cannot be reached? The naive method is to simply "ping google.com". But what if Internet is available, but DNS is simply not working? OK, so ping a known public IP, e.g. "ping 8.8.4.4". But what if ICMP is filtered, but e.g. TCP connections are still allowed? What if TCP and UDP are filtered, but raw IP is not? What if the network team left themselves a little hole (not necessarily by malicious intent, maybe honest mistake, maybe bug in firewall or switch)?

Obviously, ping is a nice easy check when you want Internet access, but it's woefully insufficient for ensuring that Internet is completely blocked.

The ideal approach would be to try every possible combination of

• Port
• Public IP
• Protocol

But that's not practical. But I'm looking for a reasonable compromise between that and a naive ping test.

My thought was to try to use nmap to look for any open ports on public IPs that should generally be up---public DNS servers. So, for example (8.8.4.4 is a public DNS server):

# nmap -v -v -v -T3 -sS 8.8.4.4

Starting Nmap 6.40 ( http://nmap.org ) at 2014-03-18 10:24 CDT
Initiating Ping Scan at 10:24
Scanning 8.8.4.4 [4 ports]
Completed Ping Scan at 10:24, 0.01s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 10:24
Completed Parallel DNS resolution of 1 host. at 10:24, 13.00s elapsed
DNS resolution of 1 IPs took 13.00s. Mode: Async [#: 2, OK: 0, NX: 0, DR: 1, SF: 0, TR: 4, CN: 0]
Initiating SYN Stealth Scan at 10:24
Scanning 8.8.4.4 [1000 ports]
Completed SYN Stealth Scan at 10:24, 0.07s elapsed (1000 total ports)
Nmap scan report for 8.8.4.4
Host is up (0.0015s latency).
All 1000 scanned ports on 8.8.4.4 are closed

Read data files from: /usr/local/encap/nmap-6.40/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 13.14 seconds
           Raw packets sent: 1004 (44.152KB) | Rcvd: 1001 (40.040KB)

This was run from a server where the Internet access block is believed to be working correctly. Yet why does it say "Host is up"? How does it know? A simple ping fails, and I am unable to use that server with dig.

I was hoping to get away with a simple script that looked for "Host is up" to flag Internet access, but it looks like that would generate too many false positives.

Anyone have any better suggestions or approaches to this problem?

Answer 1

From what you're describing, I'd say that the only way to achieve your goal is a credentialed audit of the server and the firewall, as opposed to network scanning from the server.

The reason for this is say that outbound traffic is only allowed to a single high port on a single host, there's no way that your port scanning approach will find that.

I'd recommend doing a manual review of the firewall ruleset to make sure that the rules work as intended. Depending on what's used/how familiar you are with that kind of thing, you can either do it manually or use something like nipper to help out.

To explain what you're seeing on the port scan.

All the ports are responding as closed rather than filtered. That's somewhat unusual for a firewalled connection as most firewalls will drop blocked traffic rather than explicitly reject it.

However that explains why nmap thinks that the host is up. Nmap uses a couple of mechanisms to determine that a host is live and one of them is a TCP ping (look at the -sP section of this page). If it gets a response to that it assumes that the host is up. As your firewall appears to be responding with RST/ACK (the closed message) that would cause nmap to think that everything is up, whether it is or not.

Answer 2

In a previous life we used a tool called Firemon to do this. Firemon does not do physical "real/live" host based tests such as the ones you are describing above, but rather, parses the firewall rules and then determines what ports may be open.

One of the reports we were able to run would be to determine what ports/protocols are allowed between two hosts. You could even expand that to compare ports/protocols between two networks.

Of course this would only give you valid results if you know there is no way for your server traffic to bypass your firewall. This might require a review of the network architecture and then some basic traffic test to determine path of packets etc.

Putting a few of those tests together you could probably achieve what you are looking to do without scanning out to the internet. It isn't a physical real test, but would give you some level of confidence. (ie: You are not testing that the firewall is not leaky). You could then combine this with logging from your routers and switches and watch lists in a SIEM type solution. IE: alert if host from ShouldNotHitInternetList communicates with a non-local ip.

You can start putting together a series of controls that will give you the confidence you need.

Answer 3

So, a SYN scan is not going to show you any UDP based connections, opened ports, etc., nor will it do anything against say ICMP tunneling. But let's get back to basics here. You have a goal, to prevent connections, or rather audit those connections. For that, you should just audit the FW rules itself for starters. E.g.

iptables -L -n 

Which will show you what rules are currently running on a machine. If it were me, I would diagram it all out (Visio). What servers perform what tasks, and what firewall rules are currently implemented on those servers. Do they match what you need/want to do/is being done?

Me: I'd install remote syslog (syslog-ng) with ALL packets/connections being logged, to which I can put into something like Splunk for analysis and alerting. Perhaps even throwing in AlienVault to alert me to anomalies, etc.. I would not bother really scanning once I audited the fw rules, my goal would be to know/understand what is coming in, and leaving my servers. For that, I would have logfile analysis (remote syslog) and preferably an SIEM.

Answer 4

If it were me, then my first line of defence for a co-located server would be on the server itself. I don't have access to all the other devices involved in moving data around the data centre. I don't have time to test every change applied in the data centre (even if the operators told me about it).

What you describe can be implemented in a few stateless iptables rules - which offers a proof-by-induction that the system will behave as defined. I would be tempted to go one furhter and make sure there is no default route configured on the devices - only for explicitly defined end points.