Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

We have developed a search engine (simple search, no login required) where users can search details in the database (Apache Lucene) by entering their name (et. al.), then clicking the search button. The search invokes a asynchronous HTTP GET request (an AJAX call on the same domain) to the server which in turn calls the search engine. The response is a JSON object.

I have disabled the "Allow cross origin access" policy on the server. It seems that an attacker is continuously hitting the search server programmatically to download the data. We know this because the number of hits on the search server is much larger than the one shown by Google Analytics for the index page.

Other developers have suggested the following:

  1. Create a session for the search.
  2. Put captcha on the index page and verify it on the server.
  3. They suggest that someone can still programmatically make GET requests with parameters and hence perform a search, explained by the huge difference between page visits and visits to the index page.

This confuses me:

  1. Whether captcha is really required for an application which only fetches information, apart from making the usability worse?
  2. If I have disabled "cross origin access", how can someone programmatically invoke calls to the server?

Are there better ways to approach this (especially captcha)?

share|improve this question
3  
Google uses rate limiting to do this. If there are too many requests from the same IP over a period of time, it forces you to enter a captcha. –  Mike Apr 15 at 7:39
    
Are you sure it's an attacker and not, say, a mis-behaving proxy? Can you sniff some of the data to see if it changes or it's just the same content over and over? Can you pinpoint the IP address by region or it's distributed? If it's distributed it's likely to be a botnet. Are you sure your service can't be used for spamming or web scraping of other domains? –  lorenzog Apr 15 at 13:49

4 Answers 4

If the user invokes over x amount of queries per x amount of time which seems impossible via a human then you'll want to generate a captcha to prevent bots.

If the user invokes a silly amount of requests just ban his/her IP for x amount of time as it's clear they are using some form of botting.

You could generate a key everytime the user searches (This will prevent basics bots but is by-passable by complexed bots as they will read the key and throw into parameters). If they are missing the key then display an error.

So, let's pretend you have Default.aspx and Search.aspx would set the key set in form data as hidden field. So, default will direct you to Search.aspx with a key. If you go directly to Search.aspx you won't have key so you can't search. Once you hit "Search" button it will pass it to server and you'll just validate that key. So have hidden field such as:

<input id="hifKey" type="hidden" runat="server" value="{getfromserver}" />

This key would be only validate for one search, after validated and used generate a new key.

This would prevent basics bots but do bare in mind it's still bypassable as all you need to do is create a .NET application and use HttpWebRequest and read out the HttpWebResponse data to Default.aspx then to get a key then do search on Search.aspx and keep reading next key from <input id="hifKey" type="hidden" runat="server" value="{getfromserver}" />

share|improve this answer
    
doesn't disallowing cross origin access, prevent execution of my server side scripts from the host which are not same as mine ? –  krammer Apr 15 at 10:10
    
Yes, it does but the attacker will be accessing your server side scripts indirectly via your search page instead of directly. So, disallowing cross origin just helps prevent basics bots as its much hard to perform the job. –  Paul Apr 15 at 10:31

As Paul mentioned, using a key will prevent basic bots to crawl your website, but more advanced scripts will bypass that easily. Also note that modern bots are capable of running Javascript as a real user would do, and that includes Google Analytics JS code.

A reasonable balance in terms of security versus user-experience, would be to configure quotas and limits to the searches, as Google does, and present a CAPTCHA only if this limit is exceeded.

Note that some CAPTCHA systems can be automatically solved by scripts using Optical Character Recognition, so you'd want to consider this when choosing an implementation.

share|improve this answer

Disabling Allow cross origin access only prevents cross-site HTTP requests made by JavaScript from another website on which an arbitrary user is surfing.
'Real' bots, which are not restricted within a JavaScript environment for example, are not affected by this option.

You now have to differentiate between requests initiated by users and those requests which are only faked by bots. There are several options to do so:

  • user login mechanisms
    Require your users to have dedicated accounts on your system. Problems occur if user accounts get being stolen or hijacked.

  • keys/token as Paul has mentioned source
    This is very easy to circumvent since the bots just need to replay what normal users would be doing.

  • limited usage quotas
    you limit the access to x requests within a set time interval for a specific IP address. Unfortunately, distributed bots with different IP addresses may also overcome the limit. You can combine this idea with the user system approach in order resolve the issue with distributed bots.

share|improve this answer

We know this because the number of hits on the search server is much larger than the one shown by Google Analytics for the index page.

Your premise is flawed (it may be correct, but it's still flawed). GA only shows you who has co-operated with GA. If a user's javascript is disabled (doesn't seem to apply in your case), GA is blocked etc. etc. the request will not show in your analytics.

You need to examine your logs for bot-like activity: alphanumeric sequences, keywords humans are unlikely to type etc. and then just block that IP range. Note, however, that any group using a proxy or NAT process can easily look like a bot if all you examine is frequency. Large companies and smaller ISPs (like VPN providers) will fall into this category.

You can also be mean - instead of blocking the bot, either create a tarpit ( very slow responses) or a result bomb (small compressed results that expand to 50Pb).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.