How to prevent users from downloading information from our search engine?

Question

We have developed a search engine (simple search, no login required) where users can search details in the database (Apache Lucene) by entering their name (et. al.), then clicking the search button. The search invokes a asynchronous HTTP GET request (an AJAX call on the same domain) to the server which in turn calls the search engine. The response is a JSON object.

I have disabled the "Allow cross origin access" policy on the server. It seems that an attacker is continuously hitting the search server programmatically to download the data. We know this because the number of hits on the search server is much larger than the one shown by Google Analytics for the index page.

Other developers have suggested the following:

1. Create a session for the search.
2. Put captcha on the index page and verify it on the server.
3. They suggest that someone can still programmatically make GET requests with parameters and hence perform a search, explained by the huge difference between page visits and visits to the index page.

This confuses me:

1. Whether captcha is really required for an application which only fetches information, apart from making the usability worse?
2. If I have disabled "cross origin access", how can someone programmatically invoke calls to the server?

Are there better ways to approach this (especially captcha)?

Answer 1

If the user invokes over x amount of queries per x amount of time which seems impossible via a human then you'll want to generate a captcha to prevent bots.

If the user invokes a silly amount of requests just ban his/her IP for x amount of time as it's clear they are using some form of botting.

You could generate a key everytime the user searches (This will prevent basics bots but is by-passable by complexed bots as they will read the key and throw into parameters). If they are missing the key then display an error.

So, let's pretend you have Default.aspx and Search.aspx would set the key set in form data as hidden field. So, default will direct you to Search.aspx with a key. If you go directly to Search.aspx you won't have key so you can't search. Once you hit "Search" button it will pass it to server and you'll just validate that key. So have hidden field such as:

<input id="hifKey" type="hidden" runat="server" value="{getfromserver}" />

This key would be only validate for one search, after validated and used generate a new key.

This would prevent basics bots but do bare in mind it's still bypassable as all you need to do is create a .NET application and use HttpWebRequest and read out the HttpWebResponse data to Default.aspx then to get a key then do search on Search.aspx and keep reading next key from <input id="hifKey" type="hidden" runat="server" value="{getfromserver}" />

Answer 2

As Paul mentioned, using a key will prevent basic bots to crawl your website, but more advanced scripts will bypass that easily. Also note that modern bots are capable of running Javascript as a real user would do, and that includes Google Analytics JS code.

A reasonable balance in terms of security versus user-experience, would be to configure quotas and limits to the searches, as Google does, and present a CAPTCHA only if this limit is exceeded.

Note that some CAPTCHA systems can be automatically solved by scripts using Optical Character Recognition, so you'd want to consider this when choosing an implementation.

Answer 3

Disabling Allow cross origin access only prevents cross-site HTTP requests made by JavaScript from another website on which an arbitrary user is surfing.
'Real' bots, which are not restricted within a JavaScript environment for example, are not affected by this option.

You now have to differentiate between requests initiated by users and those requests which are only faked by bots. There are several options to do so:

• user login mechanisms
  Require your users to have dedicated accounts on your system. Problems occur if user accounts get being stolen or hijacked.

• keys/token as Paul has mentioned _source
  This is very easy to circumvent since the bots just need to replay what normal users would be doing.

• limited usage quotas
  you limit the access to x requests within a set time interval for a specific IP address. Unfortunately, distributed bots with different IP addresses may also overcome the limit. You can combine this idea with the user system approach in order resolve the issue with distributed bots.

Answer 4

We know this because the number of hits on the search server is much larger than the one shown by Google Analytics for the index page.

Your premise is flawed (it may be correct, but it's still flawed). GA only shows you who has co-operated with GA. If a user's javascript is disabled (doesn't seem to apply in your case), GA is blocked etc. etc. the request will not show in your analytics.

You need to examine your logs for bot-like activity: alphanumeric sequences, keywords humans are unlikely to type etc. and then just block that IP range. Note, however, that any group using a proxy or NAT process can easily look like a bot if all you examine is frequency. Large companies and smaller ISPs (like VPN providers) will fall into this category.

You can also be mean - instead of blocking the bot, either create a tarpit ( very slow responses) or a result bomb (small compressed results that expand to 50Pb).