30

I want to understand how the robots.txt file can be use by an attacker. I know it can contain a list of paths and directories. Is that all or can we find more information in it?

  • 3
    Very related: Robots.txt security strategy – Michael Sep 20 at 13:15
  • Crawl-delay: xxxx directive can be used to know on which crawl rate the server could start to ban clients. And then tune the bots on that rate. – AccountantM Sep 21 at 13:04
  • 3
    @AccountantM are you saying the Crawl-delay directive can have the disastrous consequence of bots abiding by the specified rate? – Roman Odaisky Sep 21 at 19:59
  • @RomanOdaisky It's not "disastrous" but a bot tuned for the rate limit is worse than a bot that sends requests on a slower rate. – AccountantM Sep 22 at 9:44
48

That's all. What you see in robots.txt is all there is.

What makes it useful for attackers is that site administrators sometimes use robots.txt to hide sensitive information. If "https://www.example.com/sensitive_info" should remain hidden, it shouldn't be crawled by bots, so it should be in robots.txt. However, putting it there exposes it to attackers as well.

  • 38
    Even if it's not hidden info, path in robots.txt are often more resource intensive and can point hackers to the best page to target for a (D)DoS attack – Sefa Sep 19 at 12:52
  • 3
    Why would one put infos he wants to remain hidden, specifically there? – Zaibis Sep 20 at 11:15
  • 1
    See first line of @Jassi's answer :"Web developer or web admin thinks that robots.txt is only to tell web crawlers what to look and what to avoid. " – Mawg Sep 20 at 11:50
  • 1
    @Zaibis to prevent search engine bots from crawling it :) – beppe9000 Sep 20 at 11:50
15

Web developer or web admin thinks that robots.txt is only to tell web crawlers what to look and what to avoid. That's actually a good part.

But here is the catch. Pentesters always include the check for robots.txt for gathering any sensitive information or gaining information of paths which are even tough to guess. So making Pentesters job easier.

A piece of information like this would give an attacker an idea which technology you are using and what path to look for.

User-Agent: *
Disallow: /extend/themes/search.php
Disallow: /themes/search.php
Disallow: /support/rss
Disallow: /archive/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/ 

Here /wp-admin/ is of attacker's interest. And sometimes you will get paths so easily which would be otherwise tough for crawlers too!.

Even nmap has script to check any vulnerabilities related to it.

nmap -sV --script http-wordpress-enum <target>

Even portswiggers has an article on it. Please go through this to understand why and what you should actually write in robots.txt from the security perspective. portswigger robots.txt kb

  • 6
    The link doesn't not contain the information claimed. It's even more basic than the content already in this answer. – Ben Voigt Sep 19 at 20:58
  • 7
    The problem is when a robots file is configured like in this example. It is backwards. Much better to disallow all but allow what you do want visited/used/seen (although the file would be backwards - start with list of allowed then end with a Disallow: /*) By specifically disallowing paths, that is when the info on what you are using, etc is exposed. – ivanivan Sep 20 at 2:43
  • 1
    Thanks you guys , i understand the use of this file now ! – human_garbage Sep 20 at 8:08
  • 4
    Keep in mind that 99.99% of scans check for '/wp-admin' and other common urls of interest anyway – beppe9000 Sep 20 at 11:53
  • 1
    "Allow" is not in the robots.txt specification. The most used search engines support it, but there are many other services which do not. – Oskar Skog Sep 22 at 5:35
4

One class of attack perpetrated through /robots.txt is attacks on availability of archives of information previously publicly available under a domain name.

A speculator can extort a ransom from a domain name's former owner

When a domain name changes hands, its new owner can rewrite /robots.txt to advise search engines and archiving services not to index paths on web servers in origins within that domain name. Many speculators will buy domain names on a drop list immediately after they expire, switching / to a parking notice on a lightweight web server and /robots.txt to the following to prevent search engines' crawlers from overloading the server with traffic:

User-agent: *
Disallow: /

But once Internet Archive's Wayback Machine service has archived an HTML document, it used to use the current contents of /robots.txt, not the contents at the time the site was archived, to determine whether or not to make the available to the public. This means the new owner of a domain name could rewrite /robots.txt to cause Wayback Machine to deny access to the archive until the previous owner buys back the domain name from the speculator at an extortionate rate.

A site owner can cover up past policy statements

After a company or government agency is under new management, it can make the previous management's statements of policy difficult or impossible to retrieve. The Bush administration had been accused several times of using /robots.txt to cover up documents related to United States military involvement in Iraq in 2003 and 2007 (#1, #2, #3). And when Barack Obama was about to leave office as President of the United States in 2016, the public speculated that the incoming administration would try to erase information provided by the Obama administration related to environmental protection and other causes unattractive to Trump's political party, as reported by Valerie Volcovici of Reuters. For this reason, Internet Archive crawls .gov in more depth at the end of each term.

Wayback Machine changed its policy

When Internet Archive learned of these ransom and memory hole attacks, it changed how Wayback Machine interprets an origin's current /robots.txt, first on U.S. government and military sites and later on the web at large. Instead, since sometime in 2017, Internet Archive uses an email address for site operators to request exclusion from Wayback Machine.

2

If you check it manually, it provides you more info to check for. It tells your web crawler which links it should avoid scanning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.