After some research we have found several interesting solutions, here are three papers that I found most relevant:
Anagram: A Content Anomaly Detector Resistant to Mimicry Attack by Wang et al. They construct a content anomaly detector called Anagram. Anagrams works by analyzing byte sequences using distinct n-grams to learn itself signatures of valid packets. According to the authors it has high accuracy and low false positive rates. Also, it can operate at high-speed making it useful for network intrusion detection. Since it is anomaly-based it has a greater chance of detecting zero-day exploits. And, because it is self-learning, it does not require a lot of attention from the supervisor.
Another interesting paper is Anomalous payload-based network intrusion detection by Wang et al. It is also fully automatic and can be runned unsupervised. As with other Anomaly-based IDS they have a training phase in which the system learn the frequency and standard deviation of the payload. This solution has also a high detection rate, near 100%, and a very low false positive rate.
The paper found in this answer https://security.stackexchange.com/a/24553/53067 has also been helpful.
None of the solutions above address serial communication. However, operating on byte level communication, we believe a similar solution can be applied on, for example: RS485 or CAN.