How to Secure REST API from being crawled when serving public content ?

Question

I have a ReactJs web app, an Android app that both consuming a REST API built with django, the whole project is for media/news, the biggest part of the REST API is public content, there is a little part where user has secured API endpoints with Token based authentication.

The problem we are facing is that there are more people who crawl our REST API and duplicate our content in many places and we can't keep track of those and stop them, so we are thinking of some way that we can make the REST API HARDER to be crawled.

My current suggested solution: Encryption but my current understanding of encryption, is that you encrypt the REST API responses from the server, and in the client in javascript you decrypt it, but you still need a decryption key or token, so you will face other problems of saving that token in the client side.

Is there any better method of securing the api than encryption ?

What are the best practices in saving the token in the client side?

Thanks.

Answer 1

I used to write web scrappers as a job, and here are my two cents on the toughest issues I faced when I was on the other side of your problem.

1. Load some or all of the content via several AJAX calls. If the scrapper isn't tailored to your site, this will stop the scrappers. In practice, you could answer with a JSON which would have only the beginning of the content and a nonce token which would be requested to fetch the next part of the content.

2. Additionally, if you know that your REST api should only be used from your React app, you can add a long duration token from Django to the main React template. Then the REST API would require the content ID as well as that unique token. In your database, you would assign that token to a given IP address and make it expire in a reasonably small amount of time. This would require bots to fetch the HTML side of the app first to get that token and then fetch the content itself.

3. Throttling was usually an annoyance. The implementations I encountered were based on the IP address and users were allowed 10 requests per second. That's much more than what a typical human would do. To pass that, my code would have to keep track of the time when ten requests were sent, and then on timeout request ten more pieces of content.

4. Add a honeypot to the disallow of your robots.txt. Nefarious bots don't follow the rules. Also part of that same job, we would redistribute the content and had others trying to scrape it. This was the most effective way of stopping most bots: they were allowed three hits to the honeypot after which they blacklisted for 36 hours unless they succeeded in solving a captcha (we were using an external tool for that). For the Django integration, we used coded that up as a middleware so it would apply to all pages. In order for this to have a minimal performance issue, we used Django's cache to store which IPs were blacklisted or whitelisted (one would be added to the whitelist if they succeeded in solving the captcha).

Answer 2

Stopping others from crawling your public website/API is a difficult problem.

You can try rate limiting and blocking client IPs that make too many quick requests to the REST API. This can stop those who are using simple methods for crawling your REST API, but it won't help you out with crawlers that use more advanced techniques- I am not sure what you particular situation is.

Also, services like https://www.cloudflare.com/ can provide protections against data scraping/web crawling that may be easy for you to implement as a quick attempt to address the problem.