I am going to develop a simple CMS site on top of Google App Engine using webapp2. I am wondering how avoid malicious data scrapers to deliberately get the pages in order to make the app consume traffic beyond the free quota (I think 1GB per day) and hence cause Google to charge me for the junk traffic. Thanks

1 Answer 1


There are two types of automation that you should be concerned with in regards to automation.

On one side of the coin there are search engine spiders that will pull down whatever they can, you can direct these spiders with a robots.txt and sitemap.xml. Which is a simple problem to solve.

On the other side of the coin there are the (very uncommon) content scrapers. These are virtually indistinguishable from search engine spiders. They often use the same user-agent, and use about the same amount of bandwidth as a search engine spider becuase they both want to be up-to-date. To find a spider that is breaking the rules you could specify a blacklist request handler in your robots.txt, and if anyone hits that page, you could ban them. Although, the number or malicious spiders is very, very small, and this could be leveraged as a CSRF-DoS attack, whereby an attacker forces a victim into loading the blacklist request handler.

... that being said 1GB is TON of text, and its very unlikely that you would go over this limit unless you where the victim of a DoS attack.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .