Am I experiencing a brute force attack?

Question

When checking the auth log of a server with the command:

grep sshd.\*Failed /var/log/auth.log | less

I see thousands of lines like this:

    Jan 12 11:27:10 ubuntu-leno1 sshd[8423]: Failed password for invalid user admins from 172.25.1.1 port 44216 ssh2
    Jan 12 11:27:13 ubuntu-leno1 sshd[8425]: Failed password for invalid user phoenix from 172.25.1.1 port 20532 ssh2
    Jan 12 11:27:17 ubuntu-leno1 sshd[8428]: Failed password for invalid user piglet from 172.25.1.1 port 24492 ssh2
    Jan 12 11:27:22 ubuntu-leno1 sshd[8430]: Failed password for invalid user rainbow from 172.25.1.1 port 46591 ssh2
    Jan 12 11:27:25 ubuntu-leno1 sshd[8432]: Failed password for invalid user runner from 172.25.1.1 port 57129 ssh2
    Jan 12 11:27:34 ubuntu-leno1 sshd[8434]: Failed password for invalid user sam from 172.25.1.1 port 11960 ssh2
    Jan 12 11:27:37 ubuntu-leno1 sshd[8437]: Failed password for invalid user abc123 from 172.25.1.1 port 5921 ssh2
    Jan 12 11:27:40 ubuntu-leno1 sshd[8439]: Failed password for invalid user passwd from 172.25.1.1 port 21208 ssh2
    Jan 12 11:27:43 ubuntu-leno1 sshd[8441]: Failed password for invalid user newpass from 172.25.1.1 port 65416 ssh2
    Jan 12 11:27:46 ubuntu-leno1 sshd[8445]: Failed password for invalid user newpass from 172.25.1.1 port 26332 ssh2
    Jan 12 11:27:49 ubuntu-leno1 sshd[8447]: Failed password for invalid user notused from 172.25.1.1 port 51126 ssh2
    Jan 12 11:27:52 ubuntu-leno1 sshd[8449]: Failed password for invalid user Hockey from 172.25.1.1 port 14949 ssh2
    Jan 12 11:27:56 ubuntu-leno1 sshd[8451]: Failed password for invalid user internet from 172.25.1.1 port 35105 ssh2
    Jan 12 11:27:59 ubuntu-leno1 sshd[8453]: Failed password for invalid user asshole from 172.25.1.1 port 7916 ssh2
    Jan 12 11:28:02 ubuntu-leno1 sshd[8456]: Failed password for invalid user Maddock from 172.25.1.1 port 26431 ssh2
    Jan 12 11:28:05 ubuntu-leno1 sshd[8458]: Failed password for invalid user Maddock from 172.25.1.1 port 53406 ssh2
    Jan 12 11:28:09 ubuntu-leno1 sshd[8460]: Failed password for invalid user computer from 172.25.1.1 port 23350 ssh2
    Jan 12 11:28:15 ubuntu-leno1 sshd[8462]: Failed password for invalid user Mickey from 172.25.1.1 port 37232 ssh2
    Jan 12 11:28:19 ubuntu-leno1 sshd[8465]: Failed password for invalid user qwerty from 172.25.1.1 port 16474 ssh2
    Jan 12 11:28:22 ubuntu-leno1 sshd[8467]: Failed password for invalid user fiction from 172.25.1.1 port 29600 ssh2
    Jan 12 11:28:26 ubuntu-leno1 sshd[8469]: Failed password for invalid user orange from 172.25.1.1 port 44845 ssh2
    Jan 12 11:28:30 ubuntu-leno1 sshd[8471]: Failed password for invalid user tigger from 172.25.1.1 port 12038 ssh2
    Jan 12 11:28:33 ubuntu-leno1 sshd[8474]: Failed password for invalid user wheeling from 172.25.1.1 port 49099 ssh2
    Jan 12 11:28:36 ubuntu-leno1 sshd[8476]: Failed password for invalid user mustang from 172.25.1.1 port 29364 ssh2
    Jan 12 11:28:39 ubuntu-leno1 sshd[8478]: Failed password for invalid user admin from 172.25.1.1 port 23734 ssh2
    Jan 12 11:28:42 ubuntu-leno1 sshd[8480]: Failed password for invalid user jennifer from 172.25.1.1 port 15409 ssh2
    Jan 12 11:28:46 ubuntu-leno1 sshd[8483]: Failed password for invalid user admin from 172.25.1.1 port 40680 ssh2
    Jan 12 11:28:48 ubuntu-leno1 sshd[8485]: Failed password for invalid user money from 172.25.1.1 port 27060 ssh2
    Jan 12 11:28:52 ubuntu-leno1 sshd[8487]: Failed password for invalid user Justin from 172.25.1.1 port 17696 ssh2
    Jan 12 11:28:55 ubuntu-leno1 sshd[8489]: Failed password for invalid user admin from 172.25.1.1 port 50546 ssh2
    Jan 12 11:28:58 ubuntu-leno1 sshd[8491]: Failed password for root from 172.25.1.1 port 43559 ssh2
    Jan 12 11:29:01 ubuntu-leno1 sshd[8494]: Failed password for invalid user admin from 172.25.1.1 port 11206 ssh2
    Jan 12 11:29:04 ubuntu-leno1 sshd[8496]: Failed password for invalid user chris from 172.25.1.1 port 63459 ssh2
    Jan 12 11:29:08 ubuntu-leno1 sshd[8498]: Failed password for invalid user david from 172.25.1.1 port 52512 ssh2
    Jan 12 11:29:11 ubuntu-leno1 sshd[8500]: Failed password for invalid user foobar from 172.25.1.1 port 35772 ssh2
    Jan 12 11:29:14 ubuntu-leno1 sshd[8502]: Failed password for invalid user buster from 172.25.1.1 port 18745 ssh2
    Jan 12 11:29:17 ubuntu-leno1 sshd[8505]: Failed password for invalid user harley from 172.25.1.1 port 38893 ssh2
    Jan 12 11:29:20 ubuntu-leno1 sshd[8507]: Failed password for invalid user jordan from 172.25.1.1 port 64367 ssh2
    Jan 12 11:29:24 ubuntu-leno1 sshd[8509]: Failed password for invalid user stupid from 172.25.1.1 port 27740 ssh2
    Jan 12 11:29:27 ubuntu-leno1 sshd[8511]: Failed password for invalid user apple from 172.25.1.1 port 22873 ssh2
    Jan 12 11:29:30 ubuntu-leno1 sshd[8514]: Failed password for invalid user fred from 172.25.1.1 port 54420 ssh2
    Jan 12 11:29:33 ubuntu-leno1 sshd[8516]: Failed password for invalid user admin from 172.25.1.1 port 58507 ssh2
    Jan 12 11:29:42 ubuntu-leno1 sshd[8518]: Failed password for invalid user summer from 172.25.1.1 port 48271 ssh2
    Jan 12 11:29:45 ubuntu-leno1 sshd[8520]: Failed password for invalid user sunshine from 172.25.1.1 port 5645 ssh2
    Jan 12 11:29:53 ubuntu-leno1 sshd[8523]: Failed password for invalid user andrew from 172.25.1.1 port 44522 ssh2

It seems that I'm experiencing an ssh brute force attack. Is this a common occurrence, or am I being specifically targeted? What should I do now? Should I consider the attack successful and take measures?

-----Edit-------

The fact that the attack comes from an internal IP address is explained by this server having a ssh redirection from outside. It happened really quickly after opening the port, are every public IP scanned in the wild in search of an existing server behind ?

Answer 1

Quick note added about fail2ban, as a lot of people have been mentioning it: The front-end is a corporate firewall, the back-end only sees the redirection/proxy/internal address that comes from the firewall.

So no, 172.25.1.1 is not an internal machine compromised (and both the comment in the answer, and other answer here stating it is an internal machine are wrong when commenting that). It is one of the internal IP addresses of the firewall.

Fail2ban in the back-end would only block all possibility to use SSH for stretches at a time as it only sees failed attempts from 172.25.1.1. So please, read on for my answer.

There are no doubts, as other posts mention, it is painfully obvious you are under a brute force attack.

However, that does not mean at all you are compromised in any way from the logs you have shown us. Alas, these days ssh brute force attacks are all too common. Most of the time they are really automated, and you are not necessarily being targeted.

As an anecdotal warning tale, some years ago the first day I set up new servers in an ISP provider, the ssh one open to the Internet got 200k+ ssh scanning probes on a single night.

As for the "internal IP address," either you are using SNAT or a 22/TCP proxy redirect at best, and the source Internet IPs do not show (which is not the best practice), or at worst your router/cable modem is compromised.

If you do really are having a SNAT/proxy SSH configuration, I do recommend you to think it over. You want logs of the real IP addresses, and not of your network.

As for measures, I do recommend some:

1. Do not allow passwords in SSH; only logins using RSA certificates;
2. Do not open ssh for the outside; restrict it to your internal network;
3. For accessing from the outside, access via a VPN; do not expose SSH to the Internet at large;
4. rate-limiting SYNs from the outside.

If you absolutely insist in still having the SSH Internet open to the outside, be aware that changing the default SSH port gives only a fake sense of security, and blocking temporarily IP addresses only slows the attack down as often we are talking about coordinated farms of zombie machines.

You may want to have a look at fail2ban if you do change your configuration to receive directly public IP addresses connecting to you; however take into account that if indeed any external IP arrives with the IP address of your gateway, you are effectively locking out all external SSH access using it.

There are also other caveats; please do note that nowadays zombies/malware take fail2ban into account, and will return after the default timeout period or take turns with different IP addresses. (I have seen it happen.) Even if you use mandatory RSA certificate authentication, the password attacks will still be logged and will burn I/O, disk space, and CPU cycles.

As for the VPN, you do not need dedicated hardware; it is trivial to configure a VPN in a Linux or FreeBSD server.

If you do not feel comfortable setting one up from scratch, I do recommend a VM with pfSense. for FreeBSD; strongSwan for setting up a VPN in a Linux server.

If your front-end server is Linux, another alternative is port knocking. However due to its inherent working, I only recommend it for domestic settings. As correctly pointed out by @GroundZero, "port knocking is security by obscurity and an attacker can actively monitor your network traffic to discover the knocking sequence."

How To Use Port Knocking to Hide your SSH Daemon from Attackers on Ubuntu

As an additional measure, you can also rate-limit with firewall/iptables rules SYNs in port 22. In this way, you won't rate limit your legitimate connections, and either iptables in linux, or most commercial firewalls allow that configuration. I have seen nasty tricks as bots attacking some daemons as fast as possible before security rules kick-in. However I do believe actually ssh has built-in defenses against that.

To answer you about the rampant scanning, yes indeed. You have a lot of bad actors, zombie networks and malware constantly scanning the IP address space to find servers with compromised versions of sshd, vulnerable/old sshd versions, servers with default/bad passwords/known backdoors, and simply servers with openssh to gain a foothold in a non-privileged user either via brute force or dual pronged attacks via phishing.

After three successful attacks via phishing (in separate events) in my bastion host in my current work, I decided to do a dual ssh configuration in that all users are asked for a RSA certificate in sshd_config, and only the internal network allows password authentication, adding to the end of the configuration file as such:

# sshd configuration allowing only RSA certificates
Match Address 10.0.0.0/8,172.16.0.0/12,192.168.0.0/24
PasswordAuthentication yes

As another anecdotal scary tale, before I changed to mandatory RSA certificates, one of the phishing compromises was done in the exact week there were two kernel updates for vulnerabilities that allowed privilege escalation to root, and had I not updated on the spot, I would have been root compromised. (and if my memory does not fail me, it was around the 4th of July... hackers love to save those nasty attacks for holiday times)

As for graphs of actual live attacks real-time, have a look at:

• The graphical interface of the Norse project. It is very educational.
• A map of DDOS attacks.
• A free view of Checkpoint's central security monitoring network, where the agents are their customers.

For finishing the answer:

No, you probably are not compromised in any way.

Yes, you have to take measures to increase your security level. I would advise a VPN tunnel/client from the outside to your corporate VPN server. It is what I am doing actually.

I have also a last and important piece of advice: to make sure a regular account isn't compromised, check your /var/log/auth.log authentication logs for a successful authentication. There are ways of logging into openssh with any account without it being registered in /var/log/wtmp and consequently not appearing in the last command.

It goes without saying that if a regular account is compromised in an old machine without updates, all bets are off. And in the unfortunate case of privilege escalation to root, even the logs can be compromised.

Answer 2

Yes it looks like you are experiencing a brute force attack. The attacker is in on a class B private address, so it is likely to be someone with access to your organization's network that is conducting the attack. From the usernames it looks like they are running though a dictionary of common usernames.

Have a look at 'How to stop/prevent SSH bruteforce' (Serverfault) and 'Preventing brute force SSH attacks' (Rimu Hosting) on how to take measures to mitigate some of the risk relating to SSH bruteforce attacks.

Answer 3

Yes, this looks exactly like a brute-force attack and after googling admins phoenix piglet rainbow it looks like this is the wordlist the attacker is using: https://github.com/hydrogen18/kojoney/blob/master/fake_users
Check out line 116 onwards. The wordlist is being used in the exact same order.
This appears to be a generic wordlist as it also present on other sites. e.g. http://src.gnu-darwin.org/ports/net/kojoney/work/kojoney/fake_users

@TheJulyPlot has provided some good information on what to do to mitigate this attack.

Answer 4

Yes, you're being bruteforced. But I don't think you should worry about any bruteforce you detect coming from the internet. You should, however, be worried about brute-force attacks coming from your own network.

Being bruteforced is very common, and as long you don't use passwords for SSH (or use good passwords), the attack won't be successful at all (especially with one guess every 3-4 seconds).

Though, the IP (172.25.1.1) is on the same network than you. This is the real problem, and you should check whether this machine is compromised as soon as possible.

Answer 5

You are surviving the attack, from appearances. SSH is doing what it's supposed to do. However, see below for some steps you should take ASAP to ensure continued survival.

Also, and unfortunately, a system inside the network has been compromised. Too common these days. You should ascertain the nature of this apparently interior attack, but do not assume that this system is compromised simply on the basis of this logfile. Anyone who runs a web-facing SSHD has seen this before, over and over, for years. Fix up the SSHD installation on this host, then investigate the system at 172.25.1.1.

Follow SSHD best practices such as these ...

Disable password authentication in favor of key-based authentication:

# grep PasswordAuth sshd_config | grep no
PasswordAuthentication no

As mentioned, disable root logins:

PermitRootLogin no

Add allowed users to sshd_config:

AllowUsers myusername the_other_sysop_guy

If you do the last one, the log message will change from 'invalid user' to "illegal user".

Depending on your business situation, you can also think about firewalling the SSHD port (allow SSH only from authorized IP addresses) or changing it to using a different port (which is really security by obscurity, but most attacks of this nature are actually scripted). Alas, the fact that the attacker appears to be inside your LAN might cause these to not offer as much help as they would otherwise.

Incidentally, that fact that the device is at 1.1 makes me wonder if they've gotten into your router? ;-)

Answer 6

The standard look from the logs seem to be a common brute force attack as the names enlisted in the logs are from a standard dictionary being used. Even the common word-lists consider these as the primary target usernames. Also, the injection is from a Private Class B IP. Don't worry though, as long as the passwords that you have enforced are strong enough and you have a load balance to manage this wont be a problem for you. Look for measures to protect this in SSH Brute Force Protection measures in SSH.

Answer 7

Since all SSH logins to your server are redirected via the same local IP address, I would advise to use fail2ban with care, if you decide to use it. Installing fail2ban on the same server as sshd will result in the IP 172.25.1.1 being blocked on the spot. After that, nobody will be able to login via SSH to your server.

If you can install fail2ban on the server which redirects SSH traffic, do it. I use fail2ban on my public SSH server, and it does a great job limiting the attacker to 3 attempts in 10 minutes. Most "cracking" scripts the script kiddies are using will simply timeout after these 3 attempts, and won't bother me again for days.

Answer 8

You can also slow down the password guessing attacker fairly easily.

In the file /etc/pam.d/sshd, you can add a line like this:

auth     optional   pam_faildelay.so   delay=7000000

On every failed sshd login attempt, the PAM module will wait 7 seconds. You may want to increase or decrease the delay, because if you fat-finger your own password, you wait 7 seconds to get another prompt. I would hazard a guess that most SSH password guessing programs are so unsophisticated that they don't have a timeout surrounding waiting for a new prompt, but it's possible that a really long delay would just cause better guessing programs to timeout at some point.