Getting numerous HEAD requests by Java user agents to resources that require authentication to view within a web application. Should I block them?

Question

I have recently started using Cloudflare's firewall in front of a web application. This app has a limited user base of selected applicants and they must log in to view anything. There is no public registration form and nothing within the portal can be accessed without an account.

Since moving the DNS to Cloudflare I can see we are receiving numerous daily HEAD requests to paths that are only accessible within the portal.

These requests come from one of two groups of IP addresses from the United States (we are not a US-based company; our own hosting is based in AWS Ireland region and we're pretty sure at least 99% of our users have never been US-based):

Java User Agents

• User agent is Java/1.8.0_171 or some other minor update version.
• The ASN is listed as Digital Ocean.
• The IP addresses all seem to have had similar behaviour reported previously, almost all against Wordpress sites. Note that we're not using Wordpress here.

Empty User Agent

• No user agent string.
• The ASN is listed as Amazon Web Services.
• The IP addresses have very little reported activity and do not seem at all connected to the Java requests.

Other Notes

• The resources being requested are dynamic URLs containing what are essentially order numbers. We generate new orders every day, and they are visible to everyone using the portal.
• I was unable to find any of the URLs indexed by Google. They don't seem to be publicly available anywhere. There is only one publicly accessible page of the site, which is indexed.
• We have potentially identified one user who seems to have viewed all the pages that are showing up in the firewall logs (we know this because he shows up in our custom analytics for the web app itself). We have a working relationship with our users and we're almost certain he's not based in the US.

---

I am aware that a HEAD request in itself is nothing malicious and that browsers sometimes make HEAD requests. Does the Java user agent, or lack of a user agent in some cases, make this activity suspicious? I already block empty user agents and Java user agents through the firewall, although I think Cloudflare by default blocks Java as part of its browser integrity checks.

Questions

1. Is there any reason why these might be legitimate requests that I shouldn't block? The fact it's a HEAD request from a Java user agent suggests no, right?

2. One idea we had is that one of the users is sharing links to these internal URLs via some outside channel, to outsource work or something. Is it possible some kind of scraper or something has picked up these links and is spamming them now? As I say, I was unable to find them publicly indexed.

3. Is it possible the user we think is connected has some sort of malware on their machine which is picking up their browser activity and then making those requests?

4. Could the user have some sort of software that is completely innocent which would make Java based HEAD requests like this, based on their web browsing activity?

Any advice as to how I should continue this investigation? Or other thoughts about what these requests are?

Answer 1

You should always expect that there are numerous attackers (bots, humans), that will try to find weaknesses in your application.

You should always consider the clients as not trusted. Some of your users may have some malware on their devices, that generates these requests (you said you don't know exactly if some users are in the US) or malware that steals information about user requests, sends to their backend and then generates requests from another IPs.

Some users can do that intentionally. They can have a good technical background and would try to find weaknesses in your system. It would be understandable, if they send such requests not from the same IP that they use for they normal work with your application.

In the public Internet anyone can send any requests on any of your ports. If you protected the important resources with authentication, why should you care who is sending what requests?

HEAD method is often used to check authentication or authorization. How good do you know your application? If you are working on a big application, may be you don't know that HEAD is really used in your application. Check that.

May be your management has ordered penetration testing and you observe one of its effects.

Anyone can set any headers. You should not trust user agent string.

If you prohibit HEAD requests, they can send GET requests. If you prohibit requests based on the agent string, they will use a copy of agent string from real user requests from Chrome of FireFox, and you will not be able to distinguish such requests from usual requests. If you prohibit one IP range or network, they will do that from another network, e.g. from AWS Ireland.

TLDR: I don't see any reason to block the traffic unless it generates high costs on your side (CPU, RAM) or unless it exhausts your resources and you see that it can lead to the denial of service.