I have been intermittently hit by what appeared it may be a botnet. There was no logic to what it was doing in many cases, often hitting the same URLs 10,000s of times per day.
It came and went. But recently I was hit by a giant number of IPs, with up to 1,000 requests per second, i.e. enough to kill my server. It was not apparent if this was web scraping, DDOS or something else.
It followed the same footprint as the previous hits; that is the bulk of the requests came from Vietnam, Thailand, Korea, Japan, Indonesia, Brazil, Serbia, Pakistan and The US.
However this time there were evidently IPs from more or less every country, all residential.
From reading I have heard reports of a similar problem for others. The solution is to block certain AS number ranges of IPs. This can be done in CloudFlare, like:
(ip.geoip.asnum in {17451 17974 17586 41786 3786 17506 4721 17511 2497 63859 2514 45899 8452 7552 4766 3462 7713 29614 39832 26599 4760 12389 9829 9299 18403 45015 198961 36947 24560 9318 36903 23969 55836 47331 45758 45595 6697 45609 25019 8048 8151 27775 9304 28573 17552 132199 5384 4788 263535 9121 133481 4713 37457 6830 17501 4804 17488 38266 28885 16735 17676 6147 3215 24086 53006 24309 17924 34984 3320 4609 9198 8708 24863 38264 29975 2516 45271 131269 16190 10620 29465 23700 131090 22047 3352 8402 24835 35819 262186 18881 16509 50710 30722 17917 9541 17639 58224 3269 6799 2905 36873 17858 268030 18809 266532 30986 39891 2119 45629 12479 13999 27699 24186 23860 10139 204578 6400 53180 133982 7738 263137 14618 13489 42298 45916 22085 16010 22927 11796 10481 15895 23889 5617 29256 39651 22884 3243 8400 11427 36925 31549 5410 2527 33771 2860 27747 1267 12880 15399 200064 132165 31042 9329 6849 52760 37168 14117 36992 13188 25229 131596 23693 37705 9506 7418 37105 16135 3301 24940 262617 9269 9498 14754 2609 17465 8376 43766 4775 139526 18207 8369 17072 6805 37075 21003 198668 5769 3209 45769 27947 15802 8386 23674 12353 24757 12430 28548 1136 12322 19429 12083 17809 9009 7303 8167 264158 35805 42961 17882 15704 42863 15557 47524 17665 23944 3816 37611 8473 33915 9824 264527 31334 1299 45543 35699 10318 36884 44244 45143 18209 8544 4780 25144 37693 38623 37492 29518 45011 18001 8881 16086 12252 4771 8346 25472 5416 3329 29119 5391 4761 6821 12978 55430 36351 18126 37347 131207 45903 24323 9050 45184 12578 26615 27831 9873 12735 6713 21299}) or (ip.geoip.asnum in {10297 7922 35862 701 577 5089 1221 20115 20001 209 22773 6128 7545 7018 6327 5650 2856 20057 13285 11796 10796 35228})
That is a list of most AS number that are hitting me.
If you are not using CloudFlare take those numbers and make rules for your server from here:
https://www.enjen.net/asn-blocklist/
I pass them over to JS browser testing in CloudFlare, which is blocking about 99% of the attack.
I suspect this is a service, like Proxy Rack, which is paying ISPs for use of their IPs. So by blocking/bot testing all traffic from those ISPs, you stop the attack and the vast majority of your traffic is unaffected.