Do these mysql and ssh issues mean my rails server was hacked?

Question

I'm running a rails application on an ubuntu server on slicehost. I got a pingdom alert that the server was unreachable, so I tried to load the website in my browser. It showed a Passenger error, saying that rails could not authenticate a connection to mysql using username:password. (Sorry this is a little vague; I did not save the error message before reloading the page later.)

At this point I tried to SSH in, but was confronted with an error about how my host key had changed, and this might be a man-in-the-middle attack. I did not attempt to log in by clearing that host's entry in the known hosts file.

I hesitated for a few minutes, but then got another pingdom alert saying my server was up. I was able to load it in the browser.

When I tried to SSH in again, I succeeded without any warning.

Total downtime was about 20 minutes. I checked the logs for rails, apache, mysql, and syslog. Aside from a 20 minute gap in the apache and rails logs, there was nothing interesting. (Several network access attempts blocked by iptables, but no more than usual.) No access to the root directory since 2009 (I use another user with sudo access).

I'm confused about how the first ssh attempt would fail with the wrong host key, but the second one would succeed. Does that make sense?

Does that sound like a successful/attempted attack? I don't know what either of those signs would point to.

I'd appreciate it if anybody could tell me what else I might want to check.

Answer 1

It doesn't sound like you were hacked to me, it sounds more likely that Slicehost had some sort of issue temporarily. The SSH message you got means that the key that was sent by the server was different from the one in your cache. Here are possible reasons that come to mind: - Another server may have been temporarily assigned the IP address, and then the problem was fixed. - The ssh service started up before all file systems were available and it responded improperly. Once the server was completely up sshd responded properly - Slicehost proxy incoming ssh connections somehow, and while the server was unavailable it responded with a default of some kind

I would not associate the symptoms with getting hacked, it sounds much more like a network/server infrastructure problem, so I would start with contacting Slicehost and asking them to correlate your problem with any events. A good company should own up to any issues they may have had at the time. If there were no events then you need to do a thorough check of your system as it is still possible you got hacked.

Answer 2

I don't know how Slicehost works, but before considering the worst, I would try to contact them in order to have more information.

As you are saying, you had a 20min down server, and when you tried to connect via SSH during this down time, your SSH key has changed.

It can be an attack, but it's also possible that when a server is down, slicehost still accept a SSH connection to an other server, a sort of fail over one, that doesn't have the same SSH key (of course!). That's why you should contact them.

And maybe they have more information about it, like a local hardware failure or a dead router.

If nothing like that happened, you should check if your webpages has been update (to include some virus your web pages) or if you don't have any worm installed by running an antivirus.

The last one will not be 100% sure and if you go down that road, I recommand you to regularly analyze your network connections for odd request (in/out) and also your html pages.

After all those procedures, if you still don't feel it, I would suggest you to do a clean reinstall of your whole server. That's what I'll do. And I'll try to locate what could I've been wrong in my configuration in order to avoid making the same mistake again.