In their research-paper they introduce a context-aware approach to
document instrumentation and runtime behavior monitoring.
The following quotes and figure give insight in how their developed detection system approached malicious PDF detection.
Our system consists of two major components, front-end and back-end,
working in two phases. In Phase-I, the front- end component statically
parses the document, analyzes the structure, and ﬁnally instruments
instrumented document is opened, the back-end component detects
execution and conﬁnes malicious attempts.
Phase-I Static Analysis and Instrumentation
For suspicious PDF, the front-end
ﬁrst parses the document structure and then decompresses the objects
and streams. A set of static features are extracted in this process.
When a document has been decompressed, the front-end will instrument
the document is encrypted using an owner’s password, i.e., a mode of
PDF in which the document is readable but non-modiﬁable, we need to
remove the owner’s password. With the help of PDF password recovery
tools like , this can be done easily and very fast.
Phase-II Static Runtime Detection
The back-end component works in two steps, runtime monitoring and
runtime detection. When an instrumented PDF is loaded, the context
monitoring code inside will cooperate with our runtime monitor, which
tries to collect evidence of potential infection attempts. When
runtime detector will compute a malscore. If the malscore exceeds a
predeﬁned threshold, the document will be classiﬁed as malicious.
The original URL to this research-paper (PDF) is https://cs.gmu.edu/~astavrou/research/Daiping_dsn14.pdf, also referenced from in the beginning of this answer. A mirror/copy of this document can be found here http://www.pdf-archive.com/2016/07/25/daiping-dsn14/daiping-dsn14.pdf.
Credits to: Daiping Liu and Haining Wang from College of William and Mary and Angelos Stavrou from George Mason University.