I searched the internet like this
inurl:ftp -inurl:(http|https). I found many FTP hosts. I can add or remove files from some hosts.
How does Google get information about FTP servers? How can one avoid indexing of one's FTP server(s) in Google?
Google apparently scans new domain names, and infers from a name like
ftp.example.com that there may be an HTTP or FTP server responding there, and thus worth indexing. They will also follow links discovered in other Web pages; this domain-based indexation is used by Google to explore and reference sites which have not been linked from other sites (yet).
To prevent indexation of your FTP, you can:
- Put a
robots.txtfile in your server. See this page for details. Most Web crawlers will honour such a file in an HTTP server; Google also looks for it in FTP servers (but Google claims such support to be "Google-specific").
- Disable anonymous login. Instead, enforce use of a specific login+password pair; you can then publish the login and password on an explanatory Web page. Google's robot will not be able to "understand" that Web page and won't go beyond anonymous login.
Google indexes FTP servers in exactly the same way that they index web servers. For details try something like http://lmgtfy.com/?q=How+do+search+engines+work%3F&l=1#
Basically, they start with a bunch of popular web pages, and follow all the links in them (which will include FTP servers) and then follow all the links in those and so on and so on.
There is a standard way to request that search engines and such do not index your site, using a file called robots.txt. A good source of information about this mechanism is http://www.robotstxt.org.