I'm making a bot to scrape data from a website for study purposes. But I don't want them to catch it and block my ip. Basically what I do is scraping few images and few lines of texts.(using dom and html parsing with regex :- I know It's horrible , but this is for a simple set of webpages) What I already tried is running the bot in 2-3 minutes time intervals and scrape few data and turn off, will that be enough to not getting caught? Else , what should I do ? Most importantly what is your idea of making a bot which is not going to get caught.


This is just for research and study purposes , not violating the site's security policies .

Im just worried about if they see my unusual bot traffic and block my ip .

*trust me my bot is innocent *

closed as off-topic by Steffen Ullrich, Arminius, S.L. Barth, techraf, Matthew Dec 11 '16 at 9:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to break the security of a specific system for you are off-topic unless they demonstrate an understanding of the concepts involved and clearly identify a specific problem." – Steffen Ullrich, Arminius, S.L. Barth, techraf, Matthew
If this question can be reworded to fit the rules in the help center, please edit the question.

  • If they "catch" your bot it's apparent that they don't want scrapers on their site which you should respect. A better question could be "Which techniques do websites use to block bots/crawlers?" – Arminius Dec 11 '16 at 7:26
  • 1
    Since you fear getting caught you probably want to do something which is against some usage or security policy of the target site. Helping you with such activity just for your personal gain is off-topic here. – Steffen Ullrich Dec 11 '16 at 7:28
  • @Arminius just for research purposes. I respect all the site's policies – lasan Dec 11 '16 at 7:30
  • 5
    @lasan: if there is no site policy against your activity then there should be no fear of getting caught. Also, "just doing research" does not mean that your activity is conform to their policy. – Steffen Ullrich Dec 11 '16 at 7:36
  • 1
    @lasan: again: why they should block you if it is not against their (explicit or implicit) policy? Usually it is enough to behave nicely like all the other bots, i.e. don't stress the site too much by trying high speed crawling but be slow when scraping. And follow their robots.txt – Steffen Ullrich Dec 11 '16 at 7:48

You could route your traffic through Tor. If you get blocked, you can simply change to a new virtual circuit with a different exit node. You could achieve this by integrating a Tor client library into your scripts, e.g. libtor. You could also set your system's HTTP proxy to a local SOCKS proxy capable of connecting to Tor. Some good options have been discussed on SuperUser.SE. This could fail if the servers can detect and block connections from the Tor network.

Other than that, you could try routing your traffic through a plain old HTTP proxy or VPN. Even this could fail if your script has a particular user agent fingerprint that can be identified and blocked.

You could also access pages through Google's internet cache or the Internet Archive. You may be able to cleverly use Google queries to do your scraping for you, sidestepping your concerns entirely.

Any attempt to evade detection will slow your crawling, possibly a lot. Tor will be very slow. Furthermore, this is arguably a malicious use of Tor. However, this method could be ethical in the case of a legal penetration test in which you attempt to simulate attackers.

Ultimately, you aren't engaging in a penetration test, so evading detection is not really necessary. If you're worried about being blocked, you could talk to the site owners and seek their permission to crawl the website. Perhaps they don't care, as is usually the case with publicly available websites.

  • That's what to do if you know that you won't respect site policies, not to innocently download some pages... – Serge Ballesta Dec 11 '16 at 8:05
  • @serge but there's lot to learn from his answer, if I'm doing a website I can learn the ways the bots work and prevent it – lasan Dec 11 '16 at 8:08
  • I had to downvote this answer. Going through Tor seems like a really bad choice for multiple reasons: 1. Tor will slow your crawling efforts. 2. You're abusing the general attitude of Tor. That is not what Tor is for. If 3. You don't answer his question. This question looks to avoid getting caught not dealing with consequences. 4. Websites (especially smaller ones) most of the times won't notice a bot (except for some headerchecking) (I've had a website where I programmatically requested specific pages up to 300 times a day.) Could you put my concerns into your answer? That would get my +1 – FMaz Dec 11 '16 at 8:46
  • @Krazor Sure. I don't understand your third point, but I'll have a go. – user68527 Dec 11 '16 at 11:52
  • I like. :) I was just saying that using Tor won't stop you from getting caught. It just makes it easier to 'unban' and continue (e.g. switching nodes) – FMaz Dec 11 '16 at 13:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.