How do I protect an application from malformed packets?

Question

I'm running an application behind a Linux firewall, and I'm afraid that it might be vulnerable to malformed packets (eg. a webserver that is vulnerable to malformed HTTP requests). How can I filter the packets so that they are conformant to some format I define (eg. a formal grammar)?

Answer 1

How can I filter the packets so that they are conformant to some format I define (eg. a formal grammar)?

As far as I know there is no general purpose filter software you could use to normalize the traffic based on a formal grammar. For specific protocols like HTTP you might try to use IPS (or inline IDS) like Snort, Suricata, Bro or commercial firewalls but in my experience they are very tolerant to protocol errors. For HTTP you might also application level gateways (i.e. reverse proxy) which in my experience are a bit more strict and at least normalize the traffic, i.e. implicitly remove some malformed data. A special type of these HTTP gateways are web application firewalls (WAF) which are usually more strict than normal HTTP proxies.

But none of this solutions filters everything, i.e. your application should be robust enough so that malformed data will not cause security problems. If this is not the case I would not make the application accessible from the internet.