I want to understand how the robots.txt file can be use by an attacker. I know it can contain a list of paths and directories. Is that all or can we find more information in it?
That's all. What you see in robots.txt is all there is.
What makes it useful for attackers is that site administrators sometimes use robots.txt to hide sensitive information. If "https://www.example.com/sensitive_info" should remain hidden, it shouldn't be crawled by bots, so it should be in robots.txt. However, putting it there exposes it to attackers as well.
-
38Even if it's not hidden info, path in robots.txt are often more resource intensive and can point hackers to the best page to target for a (D)DoS attack – Sefa Sep 19 '19 at 12:52
-
3
-
1See first line of @Jassi's answer :"
Web developer or web admin thinks that robots.txt is only to tell web crawlers what to look and what to avoid." – Mawg says reinstate Monica Sep 20 '19 at 11:50 -
1
Web developer or web admin thinks that robots.txt is only to tell web crawlers what to look and what to avoid. That's actually a good part.
But here is the catch. Pentesters always include the check for robots.txt for gathering any sensitive information or gaining information of paths which are even tough to guess. So making Pentesters job easier.
A piece of information like this would give an attacker an idea which technology you are using and what path to look for.
User-Agent: *
Disallow: /extend/themes/search.php
Disallow: /themes/search.php
Disallow: /support/rss
Disallow: /archive/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Here /wp-admin/ is of attacker's interest. And sometimes you will get paths so easily which would be otherwise tough for crawlers too!.
Even nmap has script to check any vulnerabilities related to it.
nmap -sV --script http-wordpress-enum <target>
Even portswiggers has an article on it. Please go through this to understand why and what you should actually write in robots.txt from the security perspective. portswigger robots.txt kb
-
6The link doesn't not contain the information claimed. It's even more basic than the content already in this answer. – Ben Voigt Sep 19 '19 at 20:58
-
7The problem is when a robots file is configured like in this example. It is backwards. Much better to disallow all but allow what you do want visited/used/seen (although the file would be backwards - start with list of allowed then end with a
Disallow: /*) By specifically disallowing paths, that is when the info on what you are using, etc is exposed. – ivanivan Sep 20 '19 at 2:43 -
1
-
4Keep in mind that 99.99% of scans check for '/wp-admin' and other common urls of interest anyway – beppe9000 Sep 20 '19 at 11:53
-
1"Allow" is not in the robots.txt specification. The most used search engines support it, but there are many other services which do not. – Oskar Skog Sep 22 '19 at 5:35
One class of attack perpetrated through /robots.txt is attacks on availability of archives of information previously publicly available under a domain name.
A speculator can extort a ransom from a domain name's former owner
When a domain name changes hands, its new owner can rewrite /robots.txt to advise search engines and archiving services not to index paths on web servers in origins within that domain name. Many speculators will buy domain names on a drop list immediately after they expire, switching / to a parking notice on a lightweight web server and /robots.txt to the following to prevent search engines' crawlers from overloading the server with traffic:
User-agent: *
Disallow: /
But once Internet Archive's Wayback Machine service has archived an HTML document, it used to use the current contents of /robots.txt, not the contents at the time the site was archived, to determine whether or not to make the available to the public. This means the new owner of a domain name could rewrite /robots.txt to cause Wayback Machine to deny access to the archive until the previous owner buys back the domain name from the speculator at an extortionate rate.
A site owner can cover up past policy statements
After a company or government agency is under new management, it can make the previous management's statements of policy difficult or impossible to retrieve. The Bush administration had been accused several times of using /robots.txt to cover up documents related to United States military involvement in Iraq in 2003 and 2007 (#1, #2, #3). And when Barack Obama was about to leave office as President of the United States in 2016, the public speculated that the incoming administration would try to erase information provided by the Obama administration related to environmental protection and other causes unattractive to Trump's political party, as reported by Valerie Volcovici of Reuters. For this reason, Internet Archive crawls .gov in more depth at the end of each term.
Wayback Machine changed its policy
When Internet Archive learned of these ransom and memory hole attacks, it changed how Wayback Machine interprets an origin's current /robots.txt, first on U.S. government and military sites and later on the web at large. Instead, since sometime in 2017, Internet Archive uses an email address for site operators to request exclusion from Wayback Machine.
If you check it manually, it provides you more info to check for. It tells your web crawler which links it should avoid scanning.
Crawl-delay: xxxxdirective can be used to know on which crawl rate the server could start to ban clients. And then tune the bots on that rate. – AccountantM Sep 21 '19 at 13:04