I'm looking into security software and it seems like the Next Big Thing is behavioral antivirus. I'd like to try possibly making my own rinky-dink "next-gen behavioral AV" so I can gain a better understanding of its inner workings and so I can see just how close I can come to established products in terms of false negative and false positive rates. While I understand I'll probably never achieve the results the big players can due to a lack of access to the kind of data and workforce and expertise that they have, I'd like to at least see how close I can get.

I understand the gist of how behavioral AV works (AV monitors the operating system's API and installs a filter driver to monitor filesystem changes), but I'm interested in finding out what exactly are the types of things that these antiviruses look for. There are a few things that come to mind -- process injection, connections to servers with poor or nonexistent reputations, software setting itself up to run at startup, software that interacts with a bunch of files in the C:/System directory, etc -- but I'd like to see if anyone has any other ideas on what I should try looking for.

So far, I've found this:


But it only looks into a few actions. Does anyone have any info on what else I should include in my kludgy, not-so-enterprise AV?

  • There was a recent talk at DEF CON 26 about the instruction emulator in Windows Defender.
    – forest
    Commented Sep 29, 2018 at 2:30

1 Answer 1


I will suggest you to look to https://www.clamav.net/, is an open source implementation, so you can check and see how works an AV in general. Of course, the commercial AVs will tell you how good they are, but basically all AVs found a similar approach, rules, syscall interception, behavior of open files and so on.

  • 1
    I've looked through ClamAV to a degree. The problem is that it seems to have a detection rate poorer than many of the other AV engines in a lot of virustotal scans I've seen, so I was interested in seeing how other AV providers do things differently. Is it just an issue of them having more money/a larger team/more data to do regression tests on to look for malicious and non malicious behavior or features, or is it something else? Commented Aug 29, 2018 at 19:47
  • A big part is surely like you mentioned access to the amount of ressources. Another part is also that since the exact workings of ClamAV are public knowledge, malware authors can use that knowledge and trim their malware so that doesn't get detected.
    – H. Idden
    Commented Dec 27, 2018 at 23:19
  • 1
    @H.Idden That's not a big part at all. It's quite easy to understand how even closed source AV heuristics work. The reason ClamAV has inferior heuristic detection is simply because it does not have as many full-time developers as commercial antivirus. This is largely because AV companies are threat analysis companies. Their goal is to sell information on threats to major companies, not to protect you from malware. However, getting everyone to run AV helps them track threats because they can see into everyone's computer. Much of this money goes to improving AV. ClamAV lacks this income.
    – forest
    Commented Dec 28, 2018 at 2:05
  • @forest interesting information. I was never tracked AV companies that closely. What I mentioned is mainly based on information in the area of anti-RE and anti-AV packers in the time before and at the beginning of cloud-lookup.
    – H. Idden
    Commented Dec 28, 2018 at 12:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .