Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

My ruby on rails web application is hosted on Linode. Linode opened a ticket and blamed my Linode server for attacking other servers. The logs revealed that it's a SSH brute force attack which originated from my server. I have studied my source code, checked if there is any malicious script on my server but I found none.

  1. How can I prove that attack is not originating from my end?
  2. If however the attack is originating from my end, how can I detect which process is responsible for the attack?

My web application is running on Ubuntu 12.04.

share
1  
Proof to whose satisfaction? –  this.josh May 29 '13 at 6:35
    
Linode itself was compromised about two months ago. If they are trying to bully you instead of working together with you to find out the origin of the attack, it's time to find another hosting company. –  AnonymousLurker Jun 6 '13 at 21:19

3 Answers 3

Linode uses Xen, which means that you not only have an app; you have a whole operating system, complete with kernel and base libraries. The attack comes from your system, so it is completely in your jurisdiction. The attacker entered your system in some way, but once there, if he is at least half competent, he used a rootkit to transform his break-in into a permanent installation. The downside of it is that a rootkitted system will not show up from the inside of the system: the kernel would have been infected and will not show the attack process from the attackers.

(I am using here the assumption that once the attacker entered, he could escalate his rights into root level -- that's the prudent assumption, because operating systems are rarely rock solid against local attackers.)

This implies that your system should be wiped out and reinstalled from scratch; it has been compromised and can no longer be trusted. At that point, you would have two goals:

  1. Understand how the attacker came in, so that he won't do it again on the newly reinstalled system.
  2. Convince the Linode people that you are a victim, not an accomplice.

For the first point, this is a matter of analysis of all the logs and other traces that can be found in your system. So before scraping it, take a full copy of it (preferably, a byte-by-byte copy of its virtual hard disk) for ulterior analysis. But remember that if the attacker is reasonably good, he covered his tracks (that's what rootkits are for). Also, some auditing of your application code would be in order: the attacker entered in some way through one of the externally accessible services of your server, which includes your application. Another possibility is that the attacker guessed your SSH account password (after all, you know that this attacker does run brute-force attacks on SSH logins): on your new system, use a new, bigger, more random password (don't type it on your current system ! It is compromised, thus possibly grabbing password changes on-the-fly).

For the second point, this is a matter of trust and human relationships. You cannot prove that you are not the attacker; but the Linode people know that hijacked systems are an unfortunately common reality. Just exercise due diligence, such as reinstalling the system from scratch: going through that tedious task will go a long way towards establishing your good faith in that matter.

share
    
Just a side note, with Linode you don't need to copy the disk; just purchase additional disk space and do your new install on that new space, leaving your old disk partitions untouched to be mounted read-only by the new system for recovery and forensic purposes. The downside of virtual providers like Linode is you can't grab the physical disk and image it, the upside is that you don't have to. –  gowenfawr May 29 '13 at 14:58

You need to study the traffic coming from your server. It will reveal if your server is connecting to SSH services on other networks.

If you find no such evidence from the traffic generated from your server offer this proof to linode and have them either refute it or accept it.

If you find evidence you need to go through network traffic, logs, running processes and local files to determine what causes your server to behave this way.

When this is done you need to nuke the server and reinstall from a trusted source while also implementing the fix before it is exposed to the same attack vector that compromised it in the first place.

share

You can prove it, if you can find a compromised system of any way, shape or form on your server. You have a part of the proof. Having a single line of a log file is not proof. So, this is really hard to "proof". IT Security has a lot to do with the WOT (Web of Trust) You need to let them "trust" you on some way. The easiest way do to this is to hand over the source code. But this is not possible in most cases.

But still, if they have hard proof that your server has tried multiple logins on there servers with SSH you might have something going on. I'd make an image (harddrive, RAM) reboot, and hire a pentester.

About detecting the attack yourself. Put a Sniffer on the ethernet port of your server. The best way to do this is to use a SPAN-port. Do not sniff on the system itself.

So goodluck :)

share
1  
Rebooting is not recommended because a) if the system has been compromised, the attackers will know how to mitigate against a reboot and b) it might lose important volatile information that could be used for forencis; a pentester should perhaps only be hired after the OP has verified that his system is indeed compromised. –  Lex May 29 '13 at 8:07
    
About Incident Response a full image of the system should be made and than a reboot. Due to a lot of single shot exploits. And the memory image will hold the data anyhow ;) –  Stolas May 29 '13 at 10:00
    
Agreed. Quite an important step to note (making an image of the system). May I suggest you add this to the answer for completeness' sake? –  Lex May 29 '13 at 10:08
2  
If possible on your system, dump the memory or at least analyze it before rebooting. See superuser.com/questions/164960/… –  artifex May 29 '13 at 10:30

This site is currently not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .