Information Security Stack Exchange is a question and answer site for information security professionals. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am going to develop a simple CMS site on top of Google App Engine using webapp2. I am wondering how avoid malicious data scrapers to deliberately get the pages in order to make the app consume traffic beyond the free quota (I think 1GB per day) and hence cause Google to charge me for the junk traffic. Thanks

share|improve this question

There are two types of automation that you should be concerned with in regards to automation.

On one side of the coin there are search engine spiders that will pull down whatever they can, you can direct these spiders with a robots.txt and sitemap.xml. Which is a simple problem to solve.

On the other side of the coin there are the (very uncommon) content scrapers. These are virtually indistinguishable from search engine spiders. They often use the same user-agent, and use about the same amount of bandwidth as a search engine spider becuase they both want to be up-to-date. To find a spider that is breaking the rules you could specify a blacklist request handler in your robots.txt, and if anyone hits that page, you could ban them. Although, the number or malicious spiders is very, very small, and this could be leveraged as a CSRF-DoS attack, whereby an attacker forces a victim into loading the blacklist request handler.

... that being said 1GB is TON of text, and its very unlikely that you would go over this limit unless you where the victim of a DoS attack.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.