20

For the second time my website seems to be the target of a large automated attack. It seems complex enough and very well executed. I have the following systems in place:

  • Captcha on 3rd failed login from IP
  • Account lock for 30 min after 5 failed login attempts (using same email)
  • Minimum password requirements (8 chrs, letter, number, capitals)
  • Failed login attempts return a non-specific error (i.e. your email or password is incorrect)
  • Rate limited requests (from same IP)

Over the last half an hour or so, my website has been 20,000 failed login requests. Each request is using a different email (from spot checking) and each one has failed with a 401 and no information. Each request is coming from a different public IP address (all seem to be coming out of Phoenix, Arizona from my manual spot check)

All of the requests are coming via an mobile app I built which loads the login webpage via a webview. Below is a sample of the full details from one request.

I can't think of a way to mitigate this attack. It seems like someone is fishing for email/ password matches. 99% of the emails are not in my system anyway, so it seems to just be a bot with a list of emails and passwords trying to gain access.

My questions are. Should I be worried about this? My biggest concern is the DDOS element with regards to system load. Why would someone even bother doing this? Are there any additional things I could be doing to mitigate the risk?

Sample payload:

{
    "path": "/auth/login/email",
    "method": "POST",
    "query": "POST /auth/login/email",
    "startts": 1598474644337,
    "endts": 1598474644342,
    "responsetime": 5,
    "node": {
        "name": "ip-XXX-XX-XX-XX",
        "version": "",
        "hostname": "ip-XXX-XX-XX-XX",
        "ip": "172.31.15.58"
    },
    "http": {
        "request": {
            "url": "/email",
            "headers": {
                "host": "api.domain.com",
                "x-forwarded-for": "XXX.XXX.XXX.XXX",
                "x-forwarded-proto": "https",
                "x-forwarded-port": "443",
                "x-amzn-trace-id": "Root=1-5f46c994-168fa61913c6b3a2153fe9dd",
                "accept-encoding": "gzip,deflate",
                "content-type": "application/x-www-form-urlencoded",
                "accept": "application/json, text/plain, */*",
                "appsecret": "12312312312313123123",
                "origin": "file://",
                "user-agent": "Mozilla/5.0 (Linux; Android 5.1.1; SM-G973N Build/LYZ28N; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/XX.X.XXXX.XXX Mobile Safari/537.36",
                "accept-language": "en-US,en;q=0.9",
                "x-requested-with": "myapp.bundle.app",
                "x-forwarded-host": "api.domain.com",
                "x-forwarded-server": "ip-XXX-XX-XX-XX.us-east-1.compute.internal",
                "connection": "Keep-Alive",
                "content-length": "45"
            },
            "clength": 45,
            "route_path": "/auth/login/email",
            "params": {},
            "query": {},
            "body": {
                "email": "{\"email\":\"user@domain.co.uk\",\"password\":\"realplaintextpassword\"}",
                "password": "{\"email\":\"user@domain.co.uk\",\"password\":\"realplaintextpassword\"}"
            }
        },
        "response": {
            "code": 401,
            "class": "client_error",
            "phrase": "Unauthorized",
            "headers": {
                "x-dns-prefetch-control": "off",
                "x-frame-options": "SAMEORIGIN",
                "strict-transport-security": "max-age=15552000; includeSubDomains",
                "x-download-options": "noopen",
                "x-content-type-options": "nosniff",
                "x-xss-protection": "1; mode=block",
                "vary": "X-HTTP-Method-Override, Origin",
                "access-control-allow-origin": "file://",
                "uuid": "1231y239hndn9u13u123",
                "server": "Apache",
                "x-ratelimit-limit": 10,
                "x-ratelimit-remaining": 9
            },
            "clength": 5
        }
    },
    "ip": "::ffff:127.0.0.1",
    "real_ip": "107.178.110.130",
    "port": 8081,
    "@timestamp": "2020-08-26T20:44:04.337Z",
    "api": {
        "path": "/auth/login/email",
        "query": "POST /auth/login/email"
    }
}
9
  • 12
    This is exactly the type of problem that web application firewalls, such as Cloudflare, aim to mitigate. Have you considered putting your site behind a WAF? – mti2935 Aug 26 '20 at 22:20
  • 9
    Why do you think this is coming from an app? I can almost guarantee that isn't the case – Conor Mancone Aug 27 '20 at 1:44
  • 3
    The solution to security is never to "just drop [insert x security appliance here] to cover up an underlying issue", its always best to have a secure application, then put the security appliance on top of that. – john doe Aug 27 '20 at 18:11
  • 9
    @contool there is literally nothing you can build into the app that can't be spoofed. There are a thousand questions here about exactly that. If you could make unspoofable headers then you could make uncrackable DRM and that would be worth, literally, a fortune – Conor Mancone Aug 28 '20 at 2:04
  • 8
    @contool even better, now you can immediately discard requests that don't also include captcha challenges. Make the captcha mandatory and impose an update to the app. Also, this isn't coming from an app. They Wiresharked a request and just duplicated its aspects. I pretend to be Firefox with curl all the time. – zero298 Aug 28 '20 at 2:45
16

Attack is automated. You can inspect packets coming from attack vs packets coming from your customers. It can be as simple as the HTTP user agent string or can be some TCP header difference (e.g. some strange flag). Then filter out on the firewall level.

4
  • 5
    Of a number of good recommendations, this is actually the only thing that did the trick. The hackers have used over 50k different IP addresses, usually not more than 10 times (max 49). I ended up creating a complex rule in my firewall which filters out a combination of 4 headers from the attacking requests and blocks them. – contool Aug 27 '20 at 20:37
  • 1
    You could also look here: community.sophos.com/products/unified-threat-management/f/… - this user saw a pattern in the unique IP's allowing them to ban large ranges effectively – TCooper Aug 28 '20 at 0:48
  • And if the attacker really wants to put you down, they will try to alter something (e.g. the user agent) to force you to write more complex rules – usr-local-ΕΨΗΕΛΩΝ Aug 28 '20 at 11:14
  • User agents are extremely easy to spoof. Filtering requests by suspicious user agents may stop very basic bots, but a bot can fairly easily be made to get around that by spoofing the user agent (e.g. to imitate a legitimate devise/browser) – stevec Aug 29 '20 at 6:48
7

My biggest concern is the DDOS element with regards to system load

Then your defences are not appopriate - unless you are blocking the packets before they get to your webserver they are consuming resources (although even if you drop the packets inside your network they will use your bandwidth, but that is likely to be less of a problem).

I've just implemented fail2ban on my sites (actually I got someone else to do the hard work) and its working a treat.

But do bear in mind that mobile devices are much more likely to be using shared IP addresses - ipv6 POPs or "accelerators".

8
  • 1
    I'll check out fail2ban - I haven't heard of it before. According to the site it detects malicious IPs and then blocks them, Unfortunately it looks like every request hitting my site is coming from a different IP. I've only manually checked them to be fair, and I've only gone through about 20 so perhaps they have a limited number that they recycle – contool Aug 26 '20 at 23:17
  • 1
    It would be more illuminating to say that it extracts IP address from log files and runs an action to respond. It comes with configuration/rules to identify failed ssh attempts and to add firewall rules but you can add your own rules and actions / there are lots of third-party configurations available. It is not restricted to just reading standard log formats - if you can enrich the indicators of unauthorized access in your application and write that to a log file with the IP address, fail2ban can read it and block the IP address – symcbean Aug 27 '20 at 11:31
  • 12
    How will fail2ban help if each request is coming from a different IP address? – marcelm Aug 27 '20 at 13:17
  • 1
    @symcbean OP states "Each request is coming from a different public IP address"; standard approaches with fail2ban don't help with that. I think you should either accept this statement from the OP at face value, or provide reasoning in your answer why OP's statement is wrong or doesn't apply to your answer. – marcelm Aug 28 '20 at 21:24
  • 1
    @Giacomo1968 It's nice that fail2ban works for you, but it doesn't answer the question that has been asked here. Also, I'm curious; does that mean your fail2ban blocks an IP after a single failed authentication attempt? Doesn't that greatly affect legitimate users? – marcelm Aug 28 '20 at 21:26
5

Attacks generally have an end time- attackers don't spend unlimited time on any one target. You could temporarily blackhole route Arizona logins from that ip range coming in via the app to a "we're sorry" page.

You could also leave them able to log in, but put captcha on first attempt vs letting them fail at all.

For unique ip analysis, it can help to throw the IP list in an excel spreadsheet and remove duplicates, see what vanished.

3

It seems to be a Credential Stuffing Attack. If the attacker is not using a large set of IP addresses and you can find out the source country and/or service provider, you can block the IP set with a traditional firewall even in your machines, or use a WAF provided by Cloud and CDN providers. They can even block DDoS attacks. Some of them have a free tier.

If the attack impacts your business and the security of your users, maybe you should consider solutions from Identity providers that have automatic and transparent countermeasures.

3
  • Unfortunately they are using a LOT of different IPs, I've counted over 50k in the last few hours. I did reach out to their ISP directly and they identified the "client" by IP so I issued a cease and desist. Out of interest, how would one go about determining the range or block of IPs assigned to an ISP? We do use some out of the box firewall countermeasures provided by AWS in WAF - specifically their own IP reuptation blacklists, but unfortunately none of these IPs have been flagged before. – contool Aug 27 '20 at 21:10
  • You'd make a WHOIS query for one of the IP addresses, either through the ICANN web form or with the command line tool, which is a lot more convenient if you want to do multiple queries. For Linux, it is available from your distribution, for Windows, you can get one from SysInternals. – Simon Richter Aug 28 '20 at 10:53
  • 2
    50K different IPs from a single customer of one ISP? Having the equivalent of a /16 network allocation is fairly expensive - in the order of $5 per IP, IIRC - so I would not expect an attacker to shell out that kind of money, not with their name attached on it. It seems more likely that the client got totally owned and is now hosting a botnet. – thkala Aug 29 '20 at 16:25
3

As you described, those attempts are pretty much indistinguishable from real logins of your clients. Even if you can narrow them down to IPs from specific location or some peculiarity in payload, blackholing them outright, as suggested in other answers is not a good option if you have any real clients from that location or whose software can realistically generate similar payload. Unless you think that dealing with alienating those client is simpler.

So before going for drastic measures do two other things to reduce impact: First: review your login handler. Profile and improve sub-optimal code and database access. If you primary login/password DB is still slow after that, try introducing more lightweight caching through simpler key/value stores, preferably in-memory.

Maybe you won't need to do anything else if implementing those measures already makes load from the bot negligible. Extra speed is good for your service in any case.

If bot is still taking too much resources after that, then do the second thing: reduce your resource consumption even further and slow bot to a crawl through of a lightweight reverse proxy in front of your service - those are often extremely optimized to handle very high load while taking much less resources than "real" service, thus lowering your chance to be DDoSed, but not fit to perform complex business logic.

You don't need anything complex though - set up a small rules that can be checked with something fast and simple like pattern matching on request without accessing any external storage and route everything that "path goes to login handler" + "have IP from specific range" + "...any other peculiarity you noted...". Network-based check like IP range can even be performed at firewall level, sending potentially problematic traffic to separate host assigned to this proxy, while letting the rest of traffic to avoid checks altogether.

Let anything that doesn't match you problematic traffic pattern pass directly to real service and stall potentially problematic requests for some extra time like 5 seconds before letting them go to real logic. Some proxies may even allow you to dynamically adjust delay depending on load. You don't have infinite resources, but neither does the attacker. As long as your mini-handlers on proxy are doing nothing but sleeping, it will be hard to completely flood it - attackers most often do limit amount of sessions open from same IP and wait for answer - they don't have infinite resources too, after all. With attack scripts generally written in dynamic languages that waste attacking device's resource faster than your optimized proxy compiled to binary, you can comfortably hold a big enough load. And if attacker uses a real browser, even headless, or automates your real application client, then you hold even bigger advantage. But even if proxy is completely overloaded, if you placed it on separate host and used firewall to route traffic, as mentioned above, it still won't disturb other clients.

With this setup your potential legitimate clients from networks with sources of bad traffic will only experience some delay on login, which is hardly noticeable by human unless he knows he's deliberately held up, as opposed from being completely denied from your service (i.e. self-DoS).

2
  • Thanks for the detailed response. I agree the ultimate solution is to improve the frontend, however, they seem to be using an emulated Galaxy s10 device using an old APK of ours, so they've essentially locked in an old interface to use that we can't update. In the end I've just gone for a number of chained firewall rules that identify a number of request headers. Small risk of me blocking off a subsection of actual users, but for the moment there's a big enough block that'll make it a pain for them to get back up and running. – contool Aug 27 '20 at 21:15
  • @contool So, as I've guessed, they're automating real app. In that case everything above will really help you if they do upgrade to latest. Keep using current solution as long as it works, but watch out for changes - obtaining latest .apk and running it instead is trivial after all. – Oleg V. Volkov Aug 29 '20 at 0:10
2

You are suffering a L7 attack based on your user email, so from my point of view you have the first option that is rate limiting how many times a URI can be called (by the client) per second, for example (if your backend supports that). The other option is to check on black lists, for example IPVoid, and check the reputation of the IPs that generate fails and build a small database, with this database later on you can block the IP addresses certain time for example.

Also you can look to JA3 signatures and try to build something in order to detect bogus clients before they hit your service.

Hope it helps

3
  • 6
    If you rate limit how many times theURI can be called you basically DoS yourself. Also detecting bogus clients is never fully possible and only rarely worth the effort – Conor Mancone Aug 27 '20 at 1:46
  • Update comment Connor :) – camp0 Aug 27 '20 at 8:45
  • Rate limiting is not going to be very effective when the source IP changes so rapidly, unless you can fingerprint the source some other way. – Simon East Aug 29 '20 at 22:43
2

Another possible denial of service vector is the automatic account lockout.

If they attempt five logins for a legitimate user, that user will be locked out from using your service for a few minutes, and might even have valid sessions terminated. Presumably, the attacker has a list of accounts they want to have service denied to, and they have stuffed that with a set of random email addresses to make it appear like a non-targeted attack.

Mitigation step one is to find whether the attacks all come from the same IP block, and if they have a sensible abuse handling process. If it's a "bulletproof hosting" operation, you're out of luck, but large providers do care about that kind of thing and don't want it on their network. This might be able to get the attack itself shut down.

Second, analysis: I'd filter the list of account names attempted so far down to the valid accounts, and further down to the list of accounts that were actually locked by the automatic lockout, to see if there are some common properties to some of these. It's still possible that the attack is random, but if it isn't, you definitely want to know.

7
  • yep - I lready have this in the system. Captcha after 3 attempts, half hour lock after 5, complete account lock after 10. They're using millions of different emails, most of them not even in my system. – contool Aug 27 '20 at 20:38
  • 1
    @contool, so if my goal is to get a particular account permanently locked rather than gain access to it, and I know just the email address, I have to solve seven captchas to do that? – Simon Richter Aug 27 '20 at 21:08
  • 1
    @contool, from the availablity point of view, there is absolutely an issue if I can lock other people's accounts permanently if I only know their email address. If I wanted to make it look like an accident, I'd probably spam a few million unrelated authentication requests at the same time -- that's why I suggested looking at whether any valid accounts got locked as a result of this. – Simon Richter Aug 28 '20 at 9:32
  • 1
    It's a good point. I just had a look and it seems that every user email is only attempted once and no-one was locked out thankfully. There's actually only a fairly small overlap between attempted logins and real user emails. It seems they've "acquired" a list of real email/password combos from somewhere and are trying their luck. I'd say I'm not the only target for this, just unlucky – contool Aug 28 '20 at 10:44
  • 1
    In that case I concur with @Logronoide that it's an attack that is trying out user passwords from leaked lists. If you have regular newsletters, you can add a note reminding users that they should use a different password on every service they subscribe to. – Simon Richter Aug 28 '20 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.