9

I have some Linux servers, which from what I can tell are not vulnerable to the shellshock attack vector, but I am curious what the attack looks like in the logs. What does a successful attack look like in the appache2 logs? What would a successful attack look like in the system log?

What do failed attempts look like? I'd like to extend my black-list for shell-shock type attacks.

closed as too broad by Gilles 'SO- stop being evil', Xander, TildalWave, Steve, Adi Sep 28 '14 at 19:27

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 13
    A successful attack is not visible in the logs: good malware erases the log entries that would reveal it. – Gilles 'SO- stop being evil' Sep 26 '14 at 12:35
  • 3
    @r3wt Reading stdout? No, you either made a typo or are misunderstanding something (but I don't know what). Unless you log to write-once storage, good malware can erase log entries (not erasing log files, that can complicate forensics but makes the presence of the malware obvious). – Gilles 'SO- stop being evil' Sep 26 '14 at 12:41
  • 1
    @Gilles , when you edit a file on unix, you are creating an entirely new file in a new location on the disk, by unlinking the old file. if it hasn't been overwritten on the disk yet, it is still there, waiting to be discovered. In the scenario you are talking about, you can unmount the filesystem and do a binary search on keywords you might find in a access log file. – r3wt Sep 26 '14 at 12:51
  • 2
    @r3wt I know, but why do you assume that the malware would use this method, instead of overwriting the existing file? (Which may or may not leave traces on the disk depending on the filesystem.) You can't assume that malware is going to do what's convenient to you. – Gilles 'SO- stop being evil' Sep 26 '14 at 13:20
  • 2
    A successful attack looks like what that attack does. If I connec to your server and just ls /usr/bin to see what's installed it won't look the same as if I scp /etc/shadow. You can also get logs for what you're already paying attention to, are you logging all accesses to your Web page? What about other entry points, are you monitoring them? This question doesn't make much sense without more details on your environment and what you have to protect. – Steve Dodier-Lazaro Sep 26 '14 at 13:45
14

This is an entry from my access_log of what my coworker did to my test machine... :

10.11.12.13 - - [25/Sep/2014:16:00:00 -0400] "GET /cgi-bin/testing.cgi HTTP/1.0" 200 1 "-" "() { test;};echo \"Content-type: text/plain\"; echo; echo; /bin/rm -rf /var/www/"    

In my error log I saw a lot of this:

[Thu Sep 25 16:00:00 2014] [error] [client 10.11.12.13] /bin/rm: cannot remove `/var/www/icons/pie0.png': Permission denied

Bastard :-)

A well configured web-server won't be able to overwrite its logs, and in all but the tiniest environments, you should be using centralized logging to protect you from losing these kinds of log entries.

If you're successfully exploited, you may see lots of errors in the error_log which show failed attempts to access resources, execute programs or delete files. If they successfully escalate privileges, and you don't have off-site logging, then you may see no evidence.

  • 1
    Looks like you got pwned. ;-) – user42178 Sep 26 '14 at 16:54
5

If you have an Apache web server with a standard configuration, and a cgi scripts that makes use of bash, the log entry for a request may look identical in the following three cases:

  • A legitimate request is made.
  • A successful attack is performed.
  • An attempted attack fails because you have upgraded to a bash version that disables the vulnerable feature.

The only difference needed to turn a legitimate request into an attack is modification of one of the request headers, which is put into an environment variable.

If the User-agent header is used to perform the attack, it will be quite visible in the log access log, since that header is logged. But there are other headers which can be used to perform the attack, which are not logged by default.

If an attempted attack failed due to bash having been updated to the intermediate version that fixed the bug but still allowed functions to be specified through environment, you would see warnings in the error log. They may look like this:

[Thu Sep 25 20:46:51.483207 2014] [cgi:error] [pid 26424] [client 10.82.90.125:55631] AH01215: /bin/bash: warning: HTTP_ACCEPT_LANGUAGE: ignoring function definition attempt
[Thu Sep 25 20:46:51.483316 2014] [cgi:error] [pid 26424] [client 10.82.90.125:55631] AH01215: /bin/bash: error importing function definition for `HTTP_ACCEPT_LANGUAGE'
5

The most things you could notice at the moment are probes, scanning random servers for possible attack vectors, so its advised to patch.

A possible "attack" or at least a check could look a bit like this:

egrep "};|}\s*;" /var/www/logs/access*  xxx.xxx.xxx.xxx - -
[25/Sep/2014:12:12:12 +0100] "GET /cgi-sys/defaultwebpage.cgi
HTTP/1.0" 404 168 "-" "() { :;}; /bin/ping -c 1 xxx.xxx.xxx.xxx"

Where xxx.xxx.xxx.xxx is the IP address.

-2

I saw the CISO of Mandiant speak recently, he said when he is asked if there was always a pattern of behaviour that distinguishes an attacker (malware) from valid activity, he replies that if the attacker behaved exactly the same as valid worker we should thank them for doing work for free for us. Clearly the attacker will behave differently in some way from a valid worker. This implies that if we monitor behaviour/activity on a system, we should be able to detect if any malicous activity has taken place. Unfortunately system logs are likley to be tampered with by an advanced attacker so as well as shipping out to syslog and all the usual methods, you must monitor the whole system behaviour in some way to look for anomolies rather than just scrape one log. Sorry if this is an inconvenient answer but APTs are rather inconvenient... I appreciate this post does not actually answer the original posted question and is therfore not 100% helpful, but I do want people to remember not to focus too closely on a specific log entry and do not forget to monitor/analyse general behaviour as well

Not the answer you're looking for? Browse other questions tagged or ask your own question.