25

The Google crawl bot uses the user agent "Googlebot", and I am wondering if this knowledge can be used maliciously.

Let's say in a random website, the Googlebot user agent is allowed to access an admin panel, could this be considered a vulnerability, or is this just simply impossible?

  • 3
    A user agent string isn't and shouldn't be used practically as an authentication method without some other means of authentication. – Sheerforce Feb 23 '16 at 2:16
  • 10
    One thing that might be interesting is where paywalled content-providers want to be indexed. You might assume that they rely on more than the user agent but I wouldn't bet on it. – Chris H Feb 23 '16 at 10:14
  • 44
    I find that some news sites, and Quora especially, are a lot less pushy about forcing me to sign up when I am Googlebot. I use user agent switcher to toggle my UA between some common values, like Googlebot, the FB crawler, etc. It's kind of funny, and a little sad, that I have to impersonate a bot to get decent service. – Dan Ross Feb 23 '16 at 12:21
  • @DanRoss And to think I've been entering glassdoor.com/Reviews/Lockheed-Martin-Reviews-E404.htm and glassdoor.com/Reviews/Lockheed-Martin-Reviews-E404_P2.htm , etc..., into Google and clicking the links like a caveman. – MonkeyZeus Feb 23 '16 at 18:15
  • 3
    @DanRoss Hehe, now that I've posted my comment I see that glassdoor.com removes the paywall for links from this site as well :-) – MonkeyZeus Feb 23 '16 at 18:17
34

Yes. You should research what exactly goes in an HTTP request.

The user agent is easily changed as it is set by the client. There are many tools that allow you to change your user agent for your browser. If you are using curl, you can also just craft your own HTTP request and set the user agent to whatever you like.

If a web application depends on a user agent for security purposes, this is 100% vulnerable.

  • You answered my question of is it possible to modify it, but you didn't answer the question of can it be used maliciously. – Xavier Feb 23 '16 at 2:18
  • 1
    Okay, I understand what you mean now. So you are capable of using it as a authentication method, but it's not a recommended method. Thanks! – Xavier Feb 23 '16 at 2:21
  • 6
    @Xavier you should never use any data provided by the client for authentication. – Neil Smithline Feb 23 '16 at 2:29
  • 18
    @NeilSmithline that's not a very helpful comment. You always need some sort of client input for authentication. – d1str0 Feb 23 '16 at 2:30
  • 11
    By "data", Neil obviously meant information such as user agent, OS, etc, not fields sent by the client such as username/password (even those need to be validated but that's a different matter). He is absolutely right that you cannot trust this information (again, user agent, OS) to authenticate a user, since all of these can be spoofed. – Simon Feb 23 '16 at 2:54
20

Yes, a modified user agent can be used for malicious purposes. However it is unlikely that your scenario of the googlebot UA being used for privileged access.

How is UA more likely to be used? If your application parses the user agent to take some action, and you do not properly sanitize that input, Cross Site Scripting will be a very likely outcome.

In a similar vein if the user agent is parsed by some sort of server side scripting engine remote code execution could also be possible. Shellshock was a great example of that.

  • 13
    Several paywalled sites allow the Googlebot to index their pages, then ask human visitors for payment. – Simon Richter Feb 23 '16 at 10:52
  • 1
    Back in my very (very) early days I wrote code to display web server logs as HTML. Did I properly sanitise the user agent before displaying it? I did not. Fortunately it was me who first developed and deployed a javascript injection attack against my own code, and not some enterprising bad guy. – Steve Jessop Feb 23 '16 at 12:39
  • @SimonRichter That's interesting. Wow. – Mark Buffalo Feb 23 '16 at 14:52
  • I don't think the UA string can be used for XSS, as it can't be set by malicious code (that does not already control the victims browser). – Bergi Feb 23 '16 at 16:27
  • 1
    @Bergi I guess I should clarify. In this case it's stored cross site scripting. The UA is logged by the app and displayed in a separate panel back to the user. That is where the XSS is, and often may occur. – KDEx Feb 23 '16 at 16:42
13

A recent example can be found in a Joomla exploit. Joomla stored the user agent in the session, and this allowed for a use-after-free vulnerability in PHP to be exploited.

  • Another example: an XSS in an CMS that shows the user agent in the administrator interface. – Sjoerd May 2 '16 at 12:18
1

Yes, it's possible that it can be utilized for malicious purposes depending on how the application is written.

User-agents should never be used for any form of authentication and poses a huge risk from a security perspective.

1

Another example which User-Agent strings are also used, is in SIP. While your Blacklists/whitelists are often defined based on User-Agent strings.

For example the security scanning tool sipvicious uses the User-Agent (friendly-scanner). This User-Agent can be blacklisted so requests containing "friendly-scanner" are rejected, or better yet receive no response.

It is also a method to whitelist, an example would be in a IP-PBX scenario you can define the User-Agent of the SIP Phones/endpoints you are using and only allow their registration requests to be processed.

In this cases someone spoofing the User-Agent could use scanning/attack tools to circumvent the first layer of security to hopefully retrieve more information about the system(s) that reside in the SIP network or execute an exploit.

0

In addition to the answers above, an attacker can also launch SQLi attacks against a target website by supplying malicious input via the value for the User agent. For example:

User-Agent: NoAgent"; DROP TABLE users

More on this at: http://sechow.com/bricks/docs/content-page-4.html

https://security.stackexchange.com/a/1334

Also, example of how the user agent can be modified using Curl:

curl -A "Googlebot/2.1" http://example.com

0

While other answers detail if it is modifiable (you can), I'd like to talk about the malicious part of it.

You should never rely on user-agents alone.

Some D-link routers reportedly had a backdoor for their admin panel, which allowed everyone who set their user-agents to xmlset_roodkcableoj28840ybtide to login without password. You can read a blog post about it here.

Not the answer you're looking for? Browse other questions tagged or ask your own question.