I'm trying to implement security against scraping on my website to prevent basic scraping techniques.
Google seems to have a very good protection against scraping, but it's so good that I'm unable to understand its mechanism.
I was trying to make an http GET request as "normal user" using normal browser headers and query parameters.
It was working all fine before certain number of requests, then it displayed 503 error page notifying me that unusual traffic was detected, it also contained my external ip address.
What's weird, is that from my normal chrome browser there were absoloutely no errors when making request to that certain url, but with my custom http requests it kept displaying status 503.
I was almost certain that proxy server could bypass such protection, but I was wrong - even though website displayed different ip address, I kept receiving status 503 error.
Main ---- Method: GET URL: https://www.google.com/search Data (Query parameers) ---------------------- q: "this+is+example" ie: utf-8 oe: utf-8 start: 0 Headers ------- 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36'
The information that was sent from my browser was generated by Chrome - I was logged in, therefore session cookies were sent within the headers as well.
If not http rate IP rate limiting and cookie rate limiting, how could Google identify such scraping bot? Is there any other method that can offer such protection?