On a Linux system, is it at all possible to detect unknown keyloggers? Keyloggers which are new and haven't made it to any detection software's database?
Keylogger detection, as for viruses and other malwares, can basically be achieved through two methods (I simplify for answer's clarity) :
- Signature based detection
- Heuristic based detection
Obviously, an unknown keylogger will not get caught by signature-based detection products (it can be though, in case of code-reuse for ex.), and you will have to rely on heuristics or behavioral detection, which usually generate a more false-positive results.
Fortunately, the thing is that keylogger developers usually rely on well-known methods to develop their malicious code, and that allows researcher to quickly find and detect them. Such methods are for example:
- Using Loadable kernel Modules
- Using System Call Table hooking
- Using SSDT Hooking
- Direct Kernel Object Manipulation (DKOM)
But generally, if your system has been compromised at kernel level by an unknown malware, the only thing you can do is get rid of it and reinstall a clean one, as kernel level modifications can be almost impossible to detect if done properly.
The key item is whether you know whether there is a keylogger (or have reasonable suspicion) or you just want a way to automatically detect it.
In the first case an investigation is very likely to lead to something: data travelling back to the attacker (as others have pointed out), suspicious devices, evidence of tampering, and so on.
In the latter case the likelihood of automatically detecting a keylogger is very small: are you using off-the-shelf products? Can you measure timing of all systems' component, and compare with known, good values? Can you isolate the system in a Faraday cage to avoid passive keyloggers that transmit back data when irradiated (a-la NSA's TAO)? Do you have control of all the abstraction layers from boot to GUI, and can you cryptographically verify the software it runs (via some form of trusted boot)?
Yes but normally not an easy task
There is a framework exactly for this called Volatility: bunch of tools written in python
it will crawl through the memory to determine if any keyloggers are running check it out at https://code.google.com/p/volatility/
hope that helps
In theory, you could stop all the applications that you know are generating network traffic, then write a small piece of code to simulate lots of key typing events and monitor the network traffic to look for increased chatter over the wire. Even if the traffic is encrypted and sent in bulk packages, you could still have get a general idea about what your applications are doing and whether or not you were the target of a keylogger software. From that point on it would be up to you to investigate further, find the malicious app and take action to actively protect yourself and maybe backtrack the application to it's source.