What is the high level view of categorizing the packet as HTTP (or any "x" protocol)?
Standard port numbers are as close as you get to an an official classification of application-layer protocols. But in practice, services often don't adhere to their default ports and there is simply no universal way to detect which layer-7 protocol is being used. Therefore, deep packet inspection relies in large part on regular expression heuristics or just basic string comparison (as in the referenced code from nDPI).
In most cases there is just no better way of detecting a protocol than by looking for typical patterns in the (preferably initial) packets - like a
GET request line with subsequent headers to identify an HTTP connection.
Consequently, you're right that it's easy to evade these checks by obscuring the protocol (often done by BitTorrent applications). The discontinued Linux Layer 7 filter project has a FAQ that addresses these concerns:
Q: Isn't this just another pointless step in an arms race in which protocols will continually adapt to evade classification?
A: You can look at it like that, but essentially no. Internet standards like HTTP will not do this. Only P2P programs and the like will. This means that, at worst, what you will need to do is to shape the "unknown" classification down, leaving the most bandwidth for known protocols like HTTP, SMTP, and so on.
Also have a look at the l7protocols wiki which documents various methods of protocol identification, most of them regex-based.