7

The system runs a SSH daemon, and a nginx web server, both ports being open to the internet. We are part of a university so our ports are forwarded from the main router to the intranet.

At about 19:00, the SSH daemon stops responding to all requests for login. All passwords as well as SSH keys seem invalid. We responded by taking the machine offline two hours after the incident, just in case it was more than a "glitch".

I also checked the daemon logs, and there is no reason to suspect the process was killed, as it was still responding to password requests, just not allowing any user to login given correct credentials.

I've ruled out user error by looking through the shell history files. The only other option is that we were attacked. So my question is: is this a glitch with OpenSSH, or was my system breached?

I realize this is a very broad question with not much details, so I will provide any further information upon request.

EDIT Running On Fedora 20 Heisenbug with OpenSSH v.6.4p1, and Nginx v1.6.0 I'm not sure what auditing software is available in Fedora. Could anyone suggest

  • This is extremely broad question. It never happened to me to see sshd server rejecting all authentication attempts. It could help to provide some versions you are using (openssh, nginx). If your system provides some kind of auditing (RHEL, CentOS, Fedora), it can give you some traces of what was going on at that time from which address or account. – Jakuje Sep 23 '15 at 17:13
  • What was the overall uptime on the server and how often had the load on the CPU etc been reviewed? If you don't have a quick answer at hand suggest your focus a sysadmin on the box looking for resource depletion (memory, locks, inodes, etc) first. – zedman9991 Sep 23 '15 at 19:23
  • what do your logs say around that time? – schroeder Sep 23 '15 at 22:42
  • 2
    Fedora 20 has reached EOL (end of life). Please upgrade your server to the latest OS if you are concern about the security of your machine. – Question Overflow Sep 24 '15 at 9:07
  • 2
    Have you tried turning it off and on again? I'm serious: if a reboot fixes it, I'd be inclined to put it down to a glitch and move on. I don't think you'll get a good answer here; an investigation would require access to your servers. However, I have voted to leave your question open, in the hope that someone can provide more detailed assistance. – paj28 Sep 24 '15 at 10:01
1

This might have been the Kernel oom-killer in action.

How much RAM does your machine have? Does it have swap memory?

What could happen is when running out of memory, oom-killer will sacrifice a daemon and that daemon might have been the sshd one.

You could confirm this by looking in your messages log /var/log/messages or via journalctl.

| improve this answer | |
0

Hmmm ... your question isn't clear.

You wrote:

At about 19:00, the SSH daemon stops responding to all requests for login.

And then:

All passwords as well as SSH keys seem invalid.

Did openssh responded or not ? Cause, if openssh didn't respond, how can you tell that passwords and keys seem invalid. Moreover, what do you mean by "invalid" ? Did you got an "Permission denied" message ? or OpenSSH just gave up ?

That can be many things, from openssh bug (would be suprising) to network lag or even failure while trying to allocate memory or tty. Please be more precise.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.