New answers tagged

2

If you're sure that you don't want to get your site indexed on this perticular Search Engine, then you can block the Crawler by modifying .htaccess file. In .htaccess file you need to match the bot with the specific User Agent and respond with a custom response. This can be done with Apache RewriteEngine(similar functionalities would be available for other ...


1

If I didn't misunderstood, you mean the URL of asd.com. The visit is made by a crawler/robot of a search engine. The string starting with Mozilla and including the URL is the User Agent provided by its HTTP request. It tells the server of your website what kind of client it is, and the URL tells you which search engine does it belong. They simply fetch the ...


1

Those credentials have to be placed somewhere. If you decide to place them at a place like /usr/lib/cgi-bin, you'd have to grant your web user access to that directory structure which would in turn make your system more vulnerable. It is more common to place them one level above the web root as a compromise between the two. This way you don't open much of ...



Top 50 recent answers are included