25

The Google crawl bot uses the user agent "Googlebot", and I am wondering if this knowledge can be used maliciously.

Let's say in a random website, the Googlebot user agent is allowed to access an admin panel, could this be considered a vulnerability, or is this just simply impossible?

Improve this question
5
  • 3
    A user agent string isn't and shouldn't be used practically as an authentication method without some other means of authentication.
    – Sheerforce
    Feb 23, 2016 at 2:16
  • 10
    One thing that might be interesting is where paywalled content-providers want to be indexed. You might assume that they rely on more than the user agent but I wouldn't bet on it.
    – Chris H
    Feb 23, 2016 at 10:14
  • 44
    I find that some news sites, and Quora especially, are a lot less pushy about forcing me to sign up when I am Googlebot. I use user agent switcher to toggle my UA between some common values, like Googlebot, the FB crawler, etc. It's kind of funny, and a little sad, that I have to impersonate a bot to get decent service.
    – Dan Ross
    Feb 23, 2016 at 12:21
  • @DanRoss And to think I've been entering glassdoor.com/Reviews/Lockheed-Martin-Reviews-E404.htm and glassdoor.com/Reviews/Lockheed-Martin-Reviews-E404_P2.htm , etc..., into Google and clicking the links like a caveman.
    – MonkeyZeus
    Feb 23, 2016 at 18:15
  • 3
    @DanRoss Hehe, now that I've posted my comment I see that glassdoor.com removes the paywall for links from this site as well :-)
    – MonkeyZeus
    Feb 23, 2016 at 18:17

7 Answers 7

Reset to default
34

Yes. You should research what exactly goes in an HTTP request.

The user agent is easily changed as it is set by the client. There are many tools that allow you to change your user agent for your browser. If you are using curl, you can also just craft your own HTTP request and set the user agent to whatever you like.

If a web application depends on a user agent for security purposes, this is 100% vulnerable.

Improve this answer
8
  • You answered my question of is it possible to modify it, but you didn't answer the question of can it be used maliciously.
    – Xavier
    Feb 23, 2016 at 2:18
  • 1
    Okay, I understand what you mean now. So you are capable of using it as a authentication method, but it's not a recommended method. Thanks!
    – Xavier
    Feb 23, 2016 at 2:21
  • 6
    @Xavier you should never use any data provided by the client for authentication.
    – Neil Smithline
    Feb 23, 2016 at 2:29
  • 18
    @NeilSmithline that's not a very helpful comment. You always need some sort of client input for authentication.
    – d1str0
    Feb 23, 2016 at 2:30
  • 11
    By "data", Neil obviously meant information such as user agent, OS, etc, not fields sent by the client such as username/password (even those need to be validated but that's a different matter). He is absolutely right that you cannot trust this information (again, user agent, OS) to authenticate a user, since all of these can be spoofed.
    – Simon
    Feb 23, 2016 at 2:54
20

Yes, a modified user agent can be used for malicious purposes. However it is unlikely that your scenario of the googlebot UA being used for privileged access.

How is UA more likely to be used? If your application parses the user agent to take some action, and you do not properly sanitize that input, Cross Site Scripting will be a very likely outcome.

In a similar vein if the user agent is parsed by some sort of server side scripting engine remote code execution could also be possible. Shellshock was a great example of that.

Improve this answer
7
  • 13
    Several paywalled sites allow the Googlebot to index their pages, then ask human visitors for payment.
    – Simon Richter
    Feb 23, 2016 at 10:52
  • 1
    Back in my very (very) early days I wrote code to display web server logs as HTML. Did I properly sanitise the user agent before displaying it? I did not. Fortunately it was me who first developed and deployed a javascript injection attack against my own code, and not some enterprising bad guy.
    – Steve Jessop
    Feb 23, 2016 at 12:39
  • @SimonRichter That's interesting. Wow.
    – Mark Buffalo
    Feb 23, 2016 at 14:52
  • I don't think the UA string can be used for XSS, as it can't be set by malicious code (that does not already control the victims browser).
    – Bergi
    Feb 23, 2016 at 16:27
  • 1
    @Bergi I guess I should clarify. In this case it's stored cross site scripting. The UA is logged by the app and displayed in a separate panel back to the user. That is where the XSS is, and often may occur.
    – KDEx
    Feb 23, 2016 at 16:42
13

A recent example can be found in a Joomla exploit. Joomla stored the user agent in the session, and this allowed for a use-after-free vulnerability in PHP to be exploited.

Improve this answer
1
  • Another example: an XSS in an CMS that shows the user agent in the administrator interface.
    – Sjoerd
    May 2, 2016 at 12:18
1

Yes, it's possible that it can be utilized for malicious purposes depending on how the application is written.

User-agents should never be used for any form of authentication and poses a huge risk from a security perspective.

Improve this answer
1

Another example which User-Agent strings are also used, is in SIP. While your Blacklists/whitelists are often defined based on User-Agent strings.

For example the security scanning tool sipvicious uses the User-Agent (friendly-scanner). This User-Agent can be blacklisted so requests containing "friendly-scanner" are rejected, or better yet receive no response.

It is also a method to whitelist, an example would be in a IP-PBX scenario you can define the User-Agent of the SIP Phones/endpoints you are using and only allow their registration requests to be processed.

In this cases someone spoofing the User-Agent could use scanning/attack tools to circumvent the first layer of security to hopefully retrieve more information about the system(s) that reside in the SIP network or execute an exploit.

Improve this answer
0

In addition to the answers above, an attacker can also launch SQLi attacks against a target website by supplying malicious input via the value for the User agent. For example:

User-Agent: NoAgent"; DROP TABLE users

More on this at: http://sechow.com/bricks/docs/content-page-4.html

https://security.stackexchange.com/a/1334

Also, example of how the user agent can be modified using Curl:

curl -A "Googlebot/2.1" http://example.com

Improve this answer
0
0

While other answers detail if it is modifiable (you can), I'd like to talk about the malicious part of it.

You should never rely on user-agents alone.

Some D-link routers reportedly had a backdoor for their admin panel, which allowed everyone who set their user-agents to xmlset_roodkcableoj28840ybtide to login without password. You can read a blog post about it here.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .