I'm looking into security software and it seems like the Next Big Thing is behavioral antivirus. I'd like to try possibly making my own rinky-dink "next-gen behavioral AV" so I can gain a better understanding of its inner workings and so I can see just how close I can come to established products in terms of false negative and false positive rates. While I understand I'll probably never achieve the results the big players can due to a lack of access to the kind of data and workforce and expertise that they have, I'd like to at least see how close I can get.
I understand the gist of how behavioral AV works (AV monitors the operating system's API and installs a filter driver to monitor filesystem changes), but I'm interested in finding out what exactly are the types of things that these antiviruses look for. There are a few things that come to mind -- process injection, connections to servers with poor or nonexistent reputations, software setting itself up to run at startup, software that interacts with a bunch of files in the C:/System directory, etc -- but I'd like to see if anyone has any other ideas on what I should try looking for.
So far, I've found this:
But it only looks into a few actions. Does anyone have any info on what else I should include in my kludgy, not-so-enterprise AV?