There are common ways malware works - currently quite a few are using powershell for example to call out download the payload and execute. This is normally trigger from say Word.exe calling cmd.exe has then called powershell.exe with arguments.
So detecting if a process for cmd.exe has been created by word is an indicator of potential malware activity and has a child process of powershell.exe.
Now there are quite a few common attacks that have indicators such as these listed by mitre:
Given known attack types you can detect this behaviour and respond. If you start grouping the different types of chained attacks you would expect you can determine if to a certain degree if something is malacious.
A good example of a tool that monitors for malicious behaviour is sysmon
Sysmon can be configured to only look for certain types of behavior and writes to the windows event log.
Obviously if a new attack category with new indicators this becomes a problem.
So now you also begin to determine what is normal behaviour on a system using some ai techniques so if something out of the normal occurs it can be flagged and other indicators looked at.