How can an attacker use robots.txt?

Question

I want to understand how the robots.txt file can be use by an attacker. I know it can contain a list of paths and directories. Is that all or can we find more information in it?

Answer 1

That's all. What you see in robots.txt is all there is.

What makes it useful for attackers is that site administrators sometimes use robots.txt to hide sensitive information. If "https://www.example.com/sensitive_info" should remain hidden, it shouldn't be crawled by bots, so it should be in robots.txt. However, putting it there exposes it to attackers as well.

Answer 2

Web developer or web admin thinks that robots.txt is only to tell web crawlers what to look and what to avoid. That's actually a good part.

But here is the catch. Pentesters always include the check for robots.txt for gathering any sensitive information or gaining information of paths which are even tough to guess. So making Pentesters job easier.

A piece of information like this would give an attacker an idea which technology you are using and what path to look for.

User-Agent: *
Disallow: /extend/themes/search.php
Disallow: /themes/search.php
Disallow: /support/rss
Disallow: /archive/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/ 

Here /wp-admin/ is of attacker's interest. And sometimes you will get paths so easily which would be otherwise tough for crawlers too!.

Even nmap has script to check any vulnerabilities related to it.

nmap -sV --script http-wordpress-enum <target>

Even portswiggers has an article on it. Please go through this to understand why and what you should actually write in robots.txt from the security perspective. portswigger robots.txt kb

Answer 3

One class of attack perpetrated through /robots.txt is attacks on availability of archives of information previously publicly available under a domain name.

A speculator can extort a ransom from a domain name's former owner

When a domain name changes hands, its new owner can rewrite /robots.txt to advise search engines and archiving services not to index paths on web servers in origins within that domain name. Many speculators will buy domain names on a drop list immediately after they expire, switching / to a parking notice on a lightweight web server and /robots.txt to the following to prevent search engines' crawlers from overloading the server with traffic:

User-agent: *
Disallow: /

But once Internet Archive's Wayback Machine service has archived an HTML document, it used to use the current contents of /robots.txt, not the contents at the time the site was archived, to determine whether or not to make the available to the public. This means the new owner of a domain name could rewrite /robots.txt to cause Wayback Machine to deny access to the archive until the previous owner buys back the domain name from the speculator at an extortionate rate.

A site owner can cover up past policy statements

After a company or government agency is under new management, it can make the previous management's statements of policy difficult or impossible to retrieve. The Bush administration had been accused several times of using /robots.txt to cover up documents related to United States military involvement in Iraq in 2003 and 2007 (#1, #2, #3). And when Barack Obama was about to leave office as President of the United States in 2016, the public speculated that the incoming administration would try to erase information provided by the Obama administration related to environmental protection and other causes unattractive to Trump's political party, as reported by Valerie Volcovici of Reuters. For this reason, Internet Archive crawls .gov in more depth at the end of each term.

Wayback Machine changed its policy

When Internet Archive learned of these ransom and memory hole attacks, it changed how Wayback Machine interprets an origin's current /robots.txt, first on U.S. government and military sites and later on the web at large. Instead, since sometime in 2017, Internet Archive uses an email address for site operators to request exclusion from Wayback Machine.

Answer 4

If you check it manually, it provides you more info to check for. It tells your web crawler which links it should avoid scanning.