Tell me more ×
IT Security Stack Exchange is a question and answer site for IT security professionals. It's 100% free, no registration required.
up vote -3 down vote favorite

I searched the internet like this inurl:ftp -inurl:(http|https). I found many FTP hosts. I can add or remove files from some hosts.
How does Google get information about FTP servers? How can one avoid indexing of one's FTP server(s) in Google?

share|improve this question
Does this belong on Webmaster's Stack Exchange? – George Bailey Mar 5 at 15:14
should i delete this? – messi fan Mar 5 at 15:27
They were probably referenced as links from other pages. – Jeff Ferland♦ Mar 5 at 15:35
This is not a security question, rather this question concerns how Google's web spidering process works. This question would be better suited for webmaster.se or plain old SO. – grauwulf Mar 5 at 15:59
@messi, If you pop over to the Chat, or the Meta area (links at the top of this website), then you can get some advice from there on making good questions, and what to do about bad ones. – George Bailey Mar 5 at 16:20

3 Answers

up vote 2 down vote

Google apparently scans new domain names, and infers from a name like www.example.com or ftp.example.com that there may be an HTTP or FTP server responding there, and thus worth indexing. They will also follow links discovered in other Web pages; this domain-based indexation is used by Google to explore and reference sites which have not been linked from other sites (yet).

To prevent indexation of your FTP, you can:

  • Put a robots.txt file in your server. See this page for details. Most Web crawlers will honour such a file in an HTTP server; Google also looks for it in FTP servers (but Google claims such support to be "Google-specific").
  • Disable anonymous login. Instead, enforce use of a specific login+password pair; you can then publish the login and password on an explanatory Web page. Google's robot will not be able to "understand" that Web page and won't go beyond anonymous login.
share|improve this answer
I can confirm independently and from my server logs that this is indeed correct. Also worth pointing out (in case it wasn't clear from Tom's answer) is that Google does not resort to URL/port probing as it was recently discovered Bing does and excuses that as 'beta'. A robots.txt with Disallow: ftp://* following the User-agent: * in HTTP root and Disallow: / in FTP root will stop Google from crawling your FTP server. Should stop other crawlers respecting robots.txt, too. – TildalWave Mar 6 at 8:08
up vote 0 down vote

Google indexes FTP servers in exactly the same way that they index web servers. For details try something like http://lmgtfy.com/?q=How+do+search+engines+work%3F&l=1#

Basically, they start with a bunch of popular web pages, and follow all the links in them (which will include FTP servers) and then follow all the links in those and so on and so on.

There is a standard way to request that search engines and such do not index your site, using a file called robots.txt. A good source of information about this mechanism is http://www.robotstxt.org.

share|improve this answer
up vote 0 down vote

AFAIK, Google reaches servers whether they are HTTP or FTP by using a crawler. So if a website refers to an FTP server, the content will be indexed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.