Information Security Stack Exchange is a question and answer site for information security professionals. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When checking the auth log of a server with the command:

grep sshd.\*Failed /var/log/auth.log | less

I see thousands of lines like this:

    Jan 12 11:27:10 ubuntu-leno1 sshd[8423]: Failed password for invalid user admins from 172.25.1.1 port 44216 ssh2
    Jan 12 11:27:13 ubuntu-leno1 sshd[8425]: Failed password for invalid user phoenix from 172.25.1.1 port 20532 ssh2
    Jan 12 11:27:17 ubuntu-leno1 sshd[8428]: Failed password for invalid user piglet from 172.25.1.1 port 24492 ssh2
    Jan 12 11:27:22 ubuntu-leno1 sshd[8430]: Failed password for invalid user rainbow from 172.25.1.1 port 46591 ssh2
    Jan 12 11:27:25 ubuntu-leno1 sshd[8432]: Failed password for invalid user runner from 172.25.1.1 port 57129 ssh2
    Jan 12 11:27:34 ubuntu-leno1 sshd[8434]: Failed password for invalid user sam from 172.25.1.1 port 11960 ssh2
    Jan 12 11:27:37 ubuntu-leno1 sshd[8437]: Failed password for invalid user abc123 from 172.25.1.1 port 5921 ssh2
    Jan 12 11:27:40 ubuntu-leno1 sshd[8439]: Failed password for invalid user passwd from 172.25.1.1 port 21208 ssh2
    Jan 12 11:27:43 ubuntu-leno1 sshd[8441]: Failed password for invalid user newpass from 172.25.1.1 port 65416 ssh2
    Jan 12 11:27:46 ubuntu-leno1 sshd[8445]: Failed password for invalid user newpass from 172.25.1.1 port 26332 ssh2
    Jan 12 11:27:49 ubuntu-leno1 sshd[8447]: Failed password for invalid user notused from 172.25.1.1 port 51126 ssh2
    Jan 12 11:27:52 ubuntu-leno1 sshd[8449]: Failed password for invalid user Hockey from 172.25.1.1 port 14949 ssh2
    Jan 12 11:27:56 ubuntu-leno1 sshd[8451]: Failed password for invalid user internet from 172.25.1.1 port 35105 ssh2
    Jan 12 11:27:59 ubuntu-leno1 sshd[8453]: Failed password for invalid user asshole from 172.25.1.1 port 7916 ssh2
    Jan 12 11:28:02 ubuntu-leno1 sshd[8456]: Failed password for invalid user Maddock from 172.25.1.1 port 26431 ssh2
    Jan 12 11:28:05 ubuntu-leno1 sshd[8458]: Failed password for invalid user Maddock from 172.25.1.1 port 53406 ssh2
    Jan 12 11:28:09 ubuntu-leno1 sshd[8460]: Failed password for invalid user computer from 172.25.1.1 port 23350 ssh2
    Jan 12 11:28:15 ubuntu-leno1 sshd[8462]: Failed password for invalid user Mickey from 172.25.1.1 port 37232 ssh2
    Jan 12 11:28:19 ubuntu-leno1 sshd[8465]: Failed password for invalid user qwerty from 172.25.1.1 port 16474 ssh2
    Jan 12 11:28:22 ubuntu-leno1 sshd[8467]: Failed password for invalid user fiction from 172.25.1.1 port 29600 ssh2
    Jan 12 11:28:26 ubuntu-leno1 sshd[8469]: Failed password for invalid user orange from 172.25.1.1 port 44845 ssh2
    Jan 12 11:28:30 ubuntu-leno1 sshd[8471]: Failed password for invalid user tigger from 172.25.1.1 port 12038 ssh2
    Jan 12 11:28:33 ubuntu-leno1 sshd[8474]: Failed password for invalid user wheeling from 172.25.1.1 port 49099 ssh2
    Jan 12 11:28:36 ubuntu-leno1 sshd[8476]: Failed password for invalid user mustang from 172.25.1.1 port 29364 ssh2
    Jan 12 11:28:39 ubuntu-leno1 sshd[8478]: Failed password for invalid user admin from 172.25.1.1 port 23734 ssh2
    Jan 12 11:28:42 ubuntu-leno1 sshd[8480]: Failed password for invalid user jennifer from 172.25.1.1 port 15409 ssh2
    Jan 12 11:28:46 ubuntu-leno1 sshd[8483]: Failed password for invalid user admin from 172.25.1.1 port 40680 ssh2
    Jan 12 11:28:48 ubuntu-leno1 sshd[8485]: Failed password for invalid user money from 172.25.1.1 port 27060 ssh2
    Jan 12 11:28:52 ubuntu-leno1 sshd[8487]: Failed password for invalid user Justin from 172.25.1.1 port 17696 ssh2
    Jan 12 11:28:55 ubuntu-leno1 sshd[8489]: Failed password for invalid user admin from 172.25.1.1 port 50546 ssh2
    Jan 12 11:28:58 ubuntu-leno1 sshd[8491]: Failed password for root from 172.25.1.1 port 43559 ssh2
    Jan 12 11:29:01 ubuntu-leno1 sshd[8494]: Failed password for invalid user admin from 172.25.1.1 port 11206 ssh2
    Jan 12 11:29:04 ubuntu-leno1 sshd[8496]: Failed password for invalid user chris from 172.25.1.1 port 63459 ssh2
    Jan 12 11:29:08 ubuntu-leno1 sshd[8498]: Failed password for invalid user david from 172.25.1.1 port 52512 ssh2
    Jan 12 11:29:11 ubuntu-leno1 sshd[8500]: Failed password for invalid user foobar from 172.25.1.1 port 35772 ssh2
    Jan 12 11:29:14 ubuntu-leno1 sshd[8502]: Failed password for invalid user buster from 172.25.1.1 port 18745 ssh2
    Jan 12 11:29:17 ubuntu-leno1 sshd[8505]: Failed password for invalid user harley from 172.25.1.1 port 38893 ssh2
    Jan 12 11:29:20 ubuntu-leno1 sshd[8507]: Failed password for invalid user jordan from 172.25.1.1 port 64367 ssh2
    Jan 12 11:29:24 ubuntu-leno1 sshd[8509]: Failed password for invalid user stupid from 172.25.1.1 port 27740 ssh2
    Jan 12 11:29:27 ubuntu-leno1 sshd[8511]: Failed password for invalid user apple from 172.25.1.1 port 22873 ssh2
    Jan 12 11:29:30 ubuntu-leno1 sshd[8514]: Failed password for invalid user fred from 172.25.1.1 port 54420 ssh2
    Jan 12 11:29:33 ubuntu-leno1 sshd[8516]: Failed password for invalid user admin from 172.25.1.1 port 58507 ssh2
    Jan 12 11:29:42 ubuntu-leno1 sshd[8518]: Failed password for invalid user summer from 172.25.1.1 port 48271 ssh2
    Jan 12 11:29:45 ubuntu-leno1 sshd[8520]: Failed password for invalid user sunshine from 172.25.1.1 port 5645 ssh2
    Jan 12 11:29:53 ubuntu-leno1 sshd[8523]: Failed password for invalid user andrew from 172.25.1.1 port 44522 ssh2

It seems that I'm experiencing an ssh brute force attack. Is this a common occurrence, or am I being specifically targeted? What should I do now? Should I consider the attack successful and take measures?

-----Edit-------

The fact that the attack comes from an internal IP address is explained by this server having a ssh redirection from outside. It happened really quickly after opening the port, are every public IP scanned in the wild in search of an existing server behind ?

share|improve this question
30  
172.25.1.1 is I believe in the Private IP range (e.a. not rout able over the internet) so most likely this is someone near you, or a machine near you. – LvB Jan 15 at 10:17
    
Comments are not for extended discussion; this conversation has been moved to chat. – Rory Alsop Jan 18 at 23:09
up vote 23 down vote accepted

Quick note added about fail2ban, as a lot of people have been mentioning it: The frontend is a corporate firewall, the backend only sees the redirection/proxy that comes from the firewall. So no, 172.25.1.1 is not an internal machine compromised. Fail2ban in the backend would only block all possibility to use SSH for stretches at a time as it only sees failed attempts from 172.25.1.1. So please, read on for my answer.

There are no doubts, as other posts mention, it is painfully obvious you are under a brute force attack. However, that does not mean at all you are compromised in any way from the logs you have shown us.

Alas, these days ssh brute force attacks are all too common. Most of the time they are really automated, and you are not necessarily being targeted. As an anecdotal warning tale, some years ago the 1st day I setup new servers in an ISP provider, the ssh one open to the Internet got 200k+ ssh scanning probes on a single night.

As for the "internal IP address", either you are using SNAT or a 22/TCP proxy redirect at best, and the source Internet IPs do not show (which is not the best practice), or at worst your router/cable modem is compromised.

If you do really are having a SNAT/proxy SSH configuration, I do recommend you to think it over. You want logs of the real IP addresses, and not of your network.

As for measures, I do recommend some:

1) Do not allow passwords in SSH; only logins using RSA certificates;
2) Do not open ssh for the outside; restrict it to your internal network;
3) For accessing from the outside, access via a VPN; do not expose SSH to the Internet at large;
4) rate-limiting SYNs from the outside.

If you absolutely insist in still having the SSH Internet open to the outside, be aware that changing the default SSH port gives only a fake sense of security, and blocking temporarily IP addresses only slows the attack down as often we are talking about coordinated farms of zombie machines.

You may want to have a look at fail2ban if you do change your configuration to receive directly public IP addresses connecting to you; however take in account that if indeed any external IP arrives with the IP address of your gateway, you are effectively locking out all external SSH access using it.

There are also other caveats; please do note that nowadays zombies/malware take into account fail2ban, and will return after the default timeout period or take turns with different IP addresses (I have seen it happen); and even that if use mandatory RSA certificate authentication, the password attacks will still be logged, and will burn I/O, disk space and CPU cycles.

As for the VPN, you do not need dedicated hardware; it is trivial to configure a VPN in a Linux or FreeBSD server.

If you do not feel comfortable setting one up from scratch, I do recommend a VM with pfSense. https://www.pfsense.org for FreeBSD; strongSwan for setting up a VPN in a Linux server. https://wiki.strongswan.org/projects/strongswan/wiki/IOS_(Apple)

If your front-end server is linux, another alternative is port knocking, however due to it´s inherent working, I only recommend it for domestic settings. As correctly pointed out by @GroundZero, "port knocking is security by obscurity and an attacker can actively monitor your network traffic to discover the knocking sequence ".

"How To Use Port Knocking to Hide your SSH Daemon from Attackers on Ubuntu"

https://www.digitalocean.com/community/tutorials/how-to-use-port-knocking-to-hide-your-ssh-daemon-from-attackers-on-ubuntu

As an additional measure, you can also rate-limit with firewall/iptables rules SYNs in port 22. In this way, you wont rate limit your legitimate connections, and either iptables in linux, or most commercial firewalls allow that configuration. I have seen nasty tricks as bots attacking some daemons as fast as possible before security rules kick-in. However I do believe actually ssh has built-in defences against that.

To answer you about the rampant scanning, yes indeed. You have a lot of bad actors, zombie networks and malware constantly scanning the IP address space to find servers with compromised versions of sshd, vulnerable/old sshd versions, servers with default/bad passwords/known backdoors, and simply servers with openssh to gain a foothold in a non-privileged user either via brute force or dual pronged attacks via phishing.

After three successful attacks via phishing (in separate events) in my bastion host in my current work, I decided to do a dual ssh configuration in that all users are asked for a RSA certificate in sshd_config, and only the internal network allows password authentication, adding to the end of the configuration file as such:

# sshd configuration allowing only RSA certificates
Match Address 10.0.0.0/8,172.16.0.0/12,192.168.0.0/24
PasswordAuthentication yes

As another anecdotal scary tale, before I changed to mandatory RSA certificates, one of the phishing compromises was done in the exact week there were two kernel updates for vulnerabilities that allowed privilege escalation to root, and had I not updated on the spot, I would had been root compromised. (and if my memory does not fail me, it was around the 4th of July...hackers love to save that nasty attacks for holiday times)

As for graphs of actual live attacks real-time, have a look at the graphical interface of the Norse project. It is very educational.

http://map.norsecorp.com

for DDOS atacks too:

http://www.digitalattackmap.com/

For a free view of Checkpoint's central security monitoring network, where the agents are their customers:

https://www.checkpoint.com/ThreatPortal/livemap.html

To finish the answer: No, you probably are not compromised in any way. Yes, you have to take measures to increase your security level. I would advise a VPN tunnel/client from the outside to your corporate VPN server. (it is what I am doing actually)

I have also a last and important advice. To make sure a regular account was compromised, check your /var/log/auth.log authentication logs for a successful authentication. There are ways of logging into openssh with any account without it being registered in /var/log/wtmp and consequently not appearing in the last command.

Evidently that goes without saying, that if a regular account is compromised in an old machine without updates, all bets are off, and in the unfortunate case of privilege escalation to root, the logs can be compromised.

share|improve this answer
1  
Great answer, I do have an ssh redirection and your anecdote shows that this kind of things happen extremely quickly after opening to the outside. – syldor Jan 16 at 13:42
1  
By authentification logs, do you mean /var/log/auth.log ? That's where i found the logs, i didnt know about wtmp :) – syldor Jan 18 at 9:02
1  
And It's a Linux server behind a corporate firewall yes. – syldor Jan 18 at 9:03
1  
Note that port knocking is also a form of obfuscation and does not truly add security. However, it will stop most automated bots and will (in most cases) require an attacker to actively monitor your network traffic to discover the knocking sequence – GroundZero Jan 18 at 17:10
1  
I do agree with you @GroundZero. It can be useful in domestic settings, however I do use VPNs both at work and at home. – Rui F Ribeiro Jan 18 at 17:35

Yes it looks like you are experiencing a brute force attack. The attacker is in on a class B private address, so it is likely to be someone with access to your organization's network that is conducting the attack. From the usernames it looks like they are running though a dictionary of common usernames.

Have a look at 'How to stop/prevent SSH bruteforce' (Serverfault) and 'Preventing brute force SSH attacks' (Rimu Hosting) on how to take measures to mitigate some of the risk relating to SSH bruteforce attacks.

share|improve this answer
47  
Reminds me of this :) – Matt Lyons Jan 15 at 10:50
5  
meanwhile, to the OP, update your passwords. Disable PermitRootLogin if you don't need it. Better yet, disable password-based logins (ie, only RSA key logins). – Otheus Jan 15 at 14:38
1  
How likely is it that the network has been compromised (e.g., virus on a user's machine) rather than an actual person within the organization? – jpmc26 Jan 15 at 19:39
1  
@jpmc26 It is possible another machine has been compromised and the attacker may be attempting to pivot through the network. A virus could have been at play in any possible compromise. This is only conjecture though and remains one of many potential explanations. – TheJulyPlot Jan 16 at 7:40
2  
Many information to go through ! It will be a funny monday far for my usual programming. – syldor Jan 16 at 13:54

Yes, this looks exactly like a brute-force attack and after googling admins phoenix piglet rainbow it looks like this is the wordlist the attacker is using: https://github.com/hydrogen18/kojoney/blob/master/fake_users
Check out line 116 onwards. The wordlist is being used in the exact same order.
This appears to be a generic wordlist as it also present on other sites. e.g. http://src.gnu-darwin.org/ports/net/kojoney/work/kojoney/fake_users

@TheJulyPlot has provided some good information on what to do to mitigate this attack.

share|improve this answer
23  
Both of the links you provided are to the same piece of software, so of course they are the same word list. – Moshe Katz Jan 15 at 14:29
2  
Which is why i said "This appears to be a generic wordlist as it also present on other sites." – mk444 Jan 16 at 1:01

Yes, you're being bruteforced. But I don't think you should worry about any bruteforce you detect coming from the internet. You should, however, be worried about brute-force attacks coming from your own network.

Being bruteforced is very common, and as long you don't use passwords for SSH (or use good passwords), the attack won't be successful at all (especially with one guess every 3-4 seconds).

Though, the IP (172.25.1.1) is on the same network than you. This is the real problem, and you should check whether this machine is compromised as soon as possible.

share|improve this answer
    
"I don't think you should worry about any bruteforce you detect coming from the internet" I do. Install basic security software like fail2ban to prevent these attacks (a) stumbling on a correct password, (b) wasting your bandwidth, and (c) filling up your logs. – Lightness Races in Orbit Jan 17 at 16:50
1  
What Benoit meant is that the OP shouldn't be worried about this particular attack - as it is very common when carried out from the Internet - but instead about the fact that the attacker has escalated privileges and is now attacking from the internal network, having therefore access to the OP's organization's resources and servers. – dr01 Jan 17 at 18:47

You are surviving the attack, from appearances. SSH is doing what it's supposed to do. However, see below for some steps you should take ASAP to ensure continued survival.

Also, and unfortunately, a system inside the network has been compromised. Too common these days. You should ascertain the nature of this apparently interior attack, but do not assume that this system is compromised simply on the basis of this logfile. Anyone who runs a web-facing SSHD has seen this before, over and over, for years. Fix up the SSHD installation on this host, then investigate the system at 172.25.1.1.

Follow SSHD best practices such as these ...

Disable password authentication in favor of key-based authentication:

# grep PasswordAuth sshd_config | grep no
PasswordAuthentication no

As mentioned, disable root logins:

PermitRootLogin no

Add allowed users to sshd_config:

AllowUsers myusername the_other_sysop_guy

If you do the last one, the log message will change from 'invalid user' to "illegal user".

Depending on your business situation, you can also think about firewalling the SSHD port (allow SSH only from authorized IP addresses) or changing it to using a different port (which is really security by obscurity, but most attacks of this nature are actually scripted). Alas, the fact that the attacker appears to be inside your LAN might cause these to not offer as much help as they would otherwise.

Incidentally, that fact that the device is at 1.1 makes me wonder if they've gotten into your router? ;-)

share|improve this answer
1  
In addition to root, I also tend to forbid SSH logins to users with broad sudo privileges, since they are just as bad if compromized. On modern systems root itself often doesn't even have a password to begin with. – Dmitry Grigoryev Jan 16 at 19:56
    
While generally that seems a good idea, what do you do for the sysop who needs "broad" privileges? And what "modern systems" do you refer to ... no root password seems a tad counterintuitive ... unless there's no root account? :) – Kkinsey Jan 21 at 17:43

The standard look from the logs seem to be a common brute force attack as the names enlisted in the logs are from a standard dictionary being used. Even the common word-lists consider these as the primary target usernames. Also, the injection is from a Private Class B IP. Don't worry though, as long as the passwords that you have enforced are strong enough and you have a load balance to manage this wont be a problem for you. Look for measures to protect this in SSH Brute Force Protection measures in SSH.

share|improve this answer

Since all SSH logins to your server are redirected via the same local IP address, I would advise to use fail2ban with care, if you decide to use it. Installing fail2ban on the same server as sshd will result in the IP 172.25.1.1 being blocked on the spot. After that, nobody will be able to login via SSH to your server.

If you can install fail2ban on the server which redirects SSH traffic, do it. I use fail2ban on my public SSH server, and it does a great job limiting the attacker to 3 attempts in 10 minutes. Most "cracking" scripts the script kiddies are using will simply timeout after these 3 attempts, and won't bother me again for days.

share|improve this answer
1  
fail2ban has a look at logs for failed authentications do do it´s magic, and the frontend server wont have the logs. – Rui F Ribeiro Jan 17 at 10:37
    
(so placing it on a frontend server that redirects the service wont work as fail2ban wont have absolutely no ideia wether it is dealing with legitimate or failed attempts in the backend) – Rui F Ribeiro Jan 17 at 21:46

You can also slow down the password guessing attacker fairly easily.

In the file /etc/pam.d/sshd, you can add a line like this:

auth     optional   pam_faildelay.so   delay=7000000

On every failed sshd login attempt, the PAM module will wait 7 seconds. You may want to increase or decrease the delay, because if you fat-finger your own password, you wait 7 seconds to get another prompt. I would hazard a guess that most SSH password guessing programs are so unsophisticated that they don't have a timeout surrounding waiting for a new prompt, but it's possible that a really long delay would just cause better guessing programs to timeout at some point.

share|improve this answer
    
This one seems an interesting idea, and an obscure tidbit I was not aware of. – Rui F Ribeiro Jan 19 at 14:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.