What is the best way to mitigate malicious bot traffic on a Node Express server?

Question

My public website is getting a lot of get requests to endpoints like "wp-login.php", ".env", ".git/HEAD" and many others. This seems to be a non-trivial portion of my website's traffic and I'd like to discourage it or at least reduce its impact.

Option 1.

I've heard of tactics such as redirecting such requests to random 10 hour long youtube videos, like so: https://twitter.com/nick_craver/status/720062942960623616 Is this effective against a significant portion of these actors?

Option 2.

I could block ips that make requests to such endpoints by using something like this:

var http = require('http');
http.createServer(function (req, res)
{
    var ip = req.ip || req.connection.remoteAddress || req.socket.remoteAddress || req.connection.socket.remoteAddress;
    if (ip == '127.0.0.1') // exit if it's a particular ip
        res.end();
}

source: https://stackoverflow.com/questions/12349251/restrict-access-to-node-js-based-http-server-by-ip-address

or something like this: https://www.npmjs.com/package/express-ipfilter (though this seems to require a redeployment to add new ips to)

Option 3.

My current plan is not especially elegant, but the idea is to just never respond to a request to any of the known get requests I'm receiving, not sending them a redirect to youtube:

const bannedGetRequests = ["latest/dynamic/instance-identity/document","wp-login.php" ...]; <-about 1,500 that I've got in my logs now.

bannedGetRequests.forEach(request => {
    app.get(request, function (req, res) {
        return; //I'm not calling res.end() here on purpose, I'm hoping this will leave their connection open awaiting a response I'm never going to send, if this slows them down even a few milliseconds, it seems worthwhile to me.
    });
});

Option 4.

Pay cloudflare. Their free tier does not seem to support bot suppression. Maybe it's not as big a deal as I think it is. I don't really want to go through what looks like a huge amount of work to setup some kind of support from Cloudflare only to find out it doesn't even address my concern. Furthermore, this is a web application, I don't really think that using a CDN is going to be especially helpful to me. I don't have very many users (aside from these bots I guess).

Is there anything I'm missing? Are any of these options objectively superior/inferior to others?

Additional information: This is running on an AWS elastic beanstalk which puts an Nginx server (from AWS) in front of it that I'm not sure I have any control over.

Answer 1

There are likely many tools and custom scripts out there that accomplish what you are trying to do. However, I think fail2ban is a good example. It ships with profiles "apache-badbots", "apache-noscript", and others that that will follow your logs and temporarily ban anyone who is in violation of the rules, e.g. trying to access .php pages on a NodeJS server. These profiles can be modified to suit your configuration.

That said, this type of activity is very normal on the internet, and there is likely very little risk in leaving it be.

Answer 2

Cloudflare's Attack Mode does offer bot suppression for free, however the screening it offers is very aggressive (5 second wait before going to the site, captcha, etc).

You should configure your site to run "silently" - configure it to not respond to pings, close any ports you aren't using, and timeout any request that doesn't lead directly to a valid page.

But honestly, I don't see why you're concerned. Unless this traffic is doing damage, it seems like you're putting a lot of effort into something that can't hurt your site. These vulnerability scrapers will always exist, and there isn't a cheap & easy way to screen them out. If you're concerned about Denial of Service, and your site can't hold up to the "ambient noise" of bots, you might consider upgrading your infrastructure.