I am building a service that I can implement equally well with either TCP or UDP. If I use TCP, I expose vulnerability to SYN flood and other attacks on TCP protocol. If I use UDP, it will be harder to block UDP flood upstream without also blocking my legitimate traffic. Is there a clear choice here? I am inclined toward UDP because bandwidth exhaustion attacks impact others on the network so I expect will become increasingly difficult, e.g., with less tolerance for address spoofing and fewer services available to amplify traffic.
When choosing TCP and UDP, you need to look beyond simply security. Without knowing the details of what you're trying to design it's impossible to make a recommendation of which protocol to use.
Consider these basic facts.
TCP requires a 3 way handshake to setup the connection, preventing many spoofing attacks based on IP address.
TCP can use an SSL connection if you ever needed to add encryption.
UDP requires less resources on the server end to keep track of
The choice of UDP vs TCP depends on your use case and of the kind of DDos. For simple bandwidth-eating DDos it does not matter much because if all bandwidth is used by the attack there will be no more traffic for your application, no matter if UDP or TCP based. UDP might have a slight advantage in this case because UDP by itself does not need multiple packets for a connection establishment (but your application protocol might need it).
If you care more about attacks eating resources on the system (like SYN flooding or simply keeping lots of connections open) UDP might look better at first since it does not use as much resources in the system as TCP does. But if your application needs some idea of connection, of reliability and ordered data transfer you end up implementing all the nice things TCP already has within your application. And there they will probably take up more resources than the optimized implementations within the OS kernel, which means that you end up being more affected by DDos than if you would simply use TCP.
Also, by using established technologies instead of doing you own application level network handling you can use existing Anti-DDos solutions. With your own special protocol instead you would need to invent your own special Anti-DDos solution too.
I am building a service that I can implement equally well with either TCP or UDP.
If you can implement the same service with the same reliability in UDP and TCP and your UDP implementation would use less system resources than the TCP implementation (i.e. kernel and user space resources combined) then you might go for UDP because using less resources is a good think when confronted with DDos.
I think you are overthinking. You are screwed when it comes to DDOS, either you pick UDP or TCP. If you have no anti-DDOS appliance or service stopping the flood before it reaches your service, there is nothing you can do aside from dropping packets locally. You did not specify the requirements of your application which leads me to provide a generic answer:
- If your application requires no long communications and you are willing to lose packets use UDP. Client sends request, you respond, and you are done(e.g. chargen). Less overhead added by unnecessary handshakes.
- If you need to exchange session information and need reliability, go for TCP (e.g. HTTP).
There is no one size fits all. Even the solutions i gave you are not generic enough: you could use UDP to communicate using HTTP (why not?).
What i would suggest is pick your poison: TCP or UDP and then use a firewall to limit the amount of syn requests using limit: http://www.cyberciti.biz/tips/howto-limit-linux-syn-attacks.html.
There is a technique called SYN cookies to prevent this kind of issues that you can enable in Linux:http://www.cyberciti.biz/faq/enable-tcp-syn-cookie-protection/
Still, as i said, if your you have no external appliance sinking the DDOS traffic (blackhole) or your ISP/service provider preventing the flood, don't even bother about protecting the application/machine on your own. You will probably protect your application from crashing if you drop packets but you won't be able to serve any requests, which is the essence of the DDOS (Distributed Denial of Service).
Stay safe ;)