Return to Answer

wget --spider -t 1 --read-timeout=5 [target url]

this is the initial url database builder bot component. It's only requesting the HEAD, that is, checking to see if the file exists. This will generate a server response code. As some people here indicated, correctly, one excellent strategy is to simply use this step to send them a 404, once their bot has been detected. The bot is software, not a person, so it will merely shrug its virtual shoulders and go, fine, page doesn't exist, next url. This is a very good strategy, and the one least likely to raise red flag alerts.

wget -t 1 -Nc --read-timeout=5 [target url]

Note again, we're timing it out again, which terminates the big file idea, and makes that a total waste of your server resources.

wget --spider -t 1 --read-timeout=5 -U "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0" [target url]

this is the initial url database builder bot component. It's only requesting the HEAD, that is, checking to see if the file exists. This will generate a server response code. As some people here indicated, correctly, one excellent strategy is to simply use this step to send them a 404, once their bot has been detected. The bot is software, not a person, so it will merely shrug its virtual shoulders and go, fine, page doesn't exist, next url. This is a very good strategy, and the one least likely to raise red flag alerts. It's also telling the server that it's firefox on windows 10, or it could be anything else they chose to enter in there, but fake lists of useragents that are switched or rotate are common features of decently designed bad bots.

wget -t 1 -Nc --read-timeout=5 -U "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0" [target url]

Note again, we're timing it out again, which terminates the big file idea, and makes that a total waste of your server resources, and sending a fake useragent.

Then there are non bot type things like DDOS attackers, which use zombie net pc's usually, or hijacked servers (see #6, this is one reason they will spide -r to find sites running software with known security issues, or zero days. You can actually discover usually when a security issue appears in the blackhat world by simply looking for requests for certain common software urls). Hijacked servers are a premium product in the underworld because they tend to have access to much more bandwidth and cpu/ram than a regular personal pc device.

I had a client who had the terrible idea of running their own IIS webserver, which was at the time so radically insecure that, as in the OP case, the second I put it online, I saw instant bot probes for common IIS access points, even though there had never been a link to the IP. I told my client 6the next morning that he had to shut down the IIS instance and give up, since he was never going to be able to maintain secure local systems, luckily for me, he did.

You're not opening 10s of thousands of accesses, you're searching for the patterns, then seeing if you can detect patterns that can be solved with programming responses. There's also firewall tools that can be programmed to block requests that fit certain rules, but those are harder to configure correctly.

Then there are non bot type things like DDOS attackers, which use zombie net pc's usually, or hijacked servers (see #6, this is one reason they will spider - to find sites running software with known security issues, or zero days. You can actually discover usually when a security issue appears in the blackhat world by simply looking for requests for certain common software urls). Hijacked servers are a premium product in the underworld because they tend to have access to much more bandwidth and cpu/ram than a regular personal pc device.

I had a client who had the terrible idea of running their own IIS webserver, which was at the time so radically insecure that, as in the OP case, the second I put it online, I saw instant bot probes for common IIS access points, even though there had never been a link to the IP. I told my client the next morning that he had to shut down the IIS instance and give up, since he was never going to be able to maintain secure local systems, luckily for me, he did.

You're not examining 10s of thousands of accesses in your logfiles, by the way, you're searching for the patterns, using tools that are made for that job (like awk/sed/grep), then seeing if you can detect patterns that can be solved with programming responses. There's also firewall tools that can be programmed to block requests that fit certain rules, but those are harder to configure correctly.

Also, it's important to understand that there's no such thing as bots per se, what there are are usually single purpose bots:

Also, it's important to understand that there's no such thing as bots per se, what there are are usually single purpose bots. A 'bot' per se is a piece of software that in almost all cases automatically requests urls in its database, it may also as part of its function create that database of urls, or it may use commercially available databases. Some bots scan automatically the entire IP ranges of the internet, they are just looking for things, others are targeted, and, as with search bots, follow links. Note that if you have not blocked all admin type pages/directories in robots.txt, you can't tell which bot is ok and which isn't, so that's the first step you take. You don't need the whole file path, just the start of it that makes it unique as a path, like: /admin