Block requests containing URL of other websites

Question

GET / HTTP/1.1" 200 166113 "-" "Mozilla/5.0 (compatible; dsada/2.0; +http://www.asd.com/search/asddd.html)

I found this request repeated continuously in my logs. How can I prevent this (requests containing URLs of other websites)?

Answer 1

If you're sure that you don't want to get your site indexed on this perticular Search Engine, then you can block the Crawler by modifying .htaccess file.

In .htaccess file you need to match the bot with the specific User Agent and respond with a custom response.

This can be done with Apache RewriteEngine(similar functionalities would be available for other HTTP Servers too). Just add following lines in your .htaccess file :

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (www.asd.com/search/asddd.html) [NC]
RewriteRule .* - [R=403,L]

What the above Rewrite Rule will do is, match the User Agent of each request with the given, and will respond 403 Access Forbidden for the one which matches. You can generate other response codes too.

Source : .htaccess RewriteRule

Also see how to block bots/crawlers in Master htaccess file by nikosdion.

Answer 2

If I didn't misunderstood, you mean the URL of asd.com.

The visit is made by a crawler/robot of a search engine. The string starting with Mozilla and including the URL is the User Agent provided by its HTTP request. It tells the server of your website what kind of client it is, and the URL tells you which search engine does it belong.

They simply fetch the contents of your website and don't do any harm on it, so there is no need to prevent them from appearing.