0

I have a website and I notice a lot of bots traffic, I guess trying to extract data from the site. I managed to detect some traffics with scripts, or IP rate limits, but I see I can't detect access that uses Selenium/webdriver.

Does anyone know how can these be detected?

I did see this post: https://stackoverflow.com/questions/33225947/can-a-website-detect-when-you-are-using-selenium-with-chromedriver and starting to check this, but from my tests not all of them are possible.

P.S. Specific parts (clicks, links etc) of our site make our backend work hard and accessing them for scrapping disrupt our regular clients work, therefore I need to block access to those pages but solutions like re-captcha seems too harsh (at least for our sales). So please don't ask me to let it be.

EDIT

My only question is - does anyone know and willing to share how to detect selenium/webdriver controlled browsers in my website?

I really don't care about the other run-rounds, they are not my responsibility.

closed as off-topic by schroeder Oct 11 '17 at 10:05

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be about Information security within the scope defined in the help center." – schroeder
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    Detecting them would probably be more resource-intensive than its worth, since they are driving an actual browser and are therefore difficult to impossible to fingerprint. What might be more useful is combining throttling with the NoCaptcha version of ReCaptcha, so that a session with an unrealistic number or frequency of requests triggers triggers it. This is minimally invasive for most users. Beyond that, look to harden up your backend with aggressive but intelligent caching and by scaling so that impact is minimal. – Stephanie Oct 10 '17 at 6:31
  • As this looks to be a duplicate of the SO question, I'm going to have to close this one. You say that you have tested, but you do not talk about what worked and what didn't. The linked question has a lot of options, but unless we know what's different, we will just be flailing in the dark. – schroeder Oct 11 '17 at 10:05
  • I also see that you duplicated this question there. – schroeder Oct 11 '17 at 10:07
  • I have no problem with that, didn't get any relevant answers as it is (I actually tried deleting before posting there, but the site didn't let me) – Eyal H Oct 11 '17 at 10:08
  • That's because people have votes on responses – schroeder Oct 11 '17 at 11:06
2

You may not want to hear it, but you're asking the wrong question.

Specific parts (clicks, links etc) of our site make our backend work hard and accessing them for scrapping disrupt our regular clients work

You're being inadvertently DoSed. If something as trivial as public website scraping is enough to impact production/revenue, your architecture is flawed. Chasing after bots and trying to identify every last one of them is not the answer-- there will always be new bots and new methods of scraping.

I'm constantly amazed at the number of questions involving web scraping that appear to be school assignments that pop up on SO. You have no end of up-and-coming adversaries.

Furthermore, an unscrupulous competitor could just suppress your traffic altogether by running a bot swarm of their own against you. I've had to work for places that asked me to do similar, so they definitely exist.

You need to take steps to achieve the results of these requests without letting any number of arbitrary visitors kick off these resource-intensive tasks. Caching is going to be your answer, or restricting these particular operations to authenticated users.

  • Thanks for the effort to answer, I am aware of all these issues, and we do have people working on those. My task is to block/confuse the bots at the moment and understand the alternatives. – Eyal H Oct 10 '17 at 18:27
0

Most bots/scrapers/legitimate attackers won't be using Selenium. They'll just make the web requests (aka non-interactive). As a result, user-agent and really any other header/data based filtering won't work. You'd have to do behavior based analysis to stop the traffic. You can start a massive research study on behavior of natural users in an attempt to block bad traffic but that's really an expensive solution to an easily solved problem (fix the backend!)

Until you fix this (per Ivan) you can use one click reCAPTCHA. It's a temporary stop-gap measure to stop your site from breaking until you fix your performance issue. Remember that this is essentially punishing your users (forcing them to fill a captcha in some cases) and is an awful UX.

-1

I would also monitor inbound IP traffic on our web server.

You might also apply boot blocking rules to your .htaccess file, once done you will see plenty of rejected requests in your logs.

  • That's what our 'IP rate limits' does, more or less, I'm looking something more like left-overs from the selenium. – Eyal H Oct 10 '17 at 13:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.