HTTP authentication: identify the client

Question

We have a web-site, where each user has to log-in to access a private area. Inside the private area, there are some information about the user, a profile, including some sensitive information (home address, bank details, etc.)

Naturally, we follow best practices here (strong hashing, SSL with the correct algorithms, check for weak passwords, etc...).

We also offer an API base on OAuth for third parties. In this way the user knows what a registered application can access, the application does not have to store the user credentials, we can limit what the application gets... all the usual benefits. For example, the API can read the profile info, but the sensitive information is blanked out or omitted.

The problem is: we have a third party application that decided that using the API was no good; instead, it asks the credentials, store them (in plaintext...), and then authenticates using a POST. And reads user info doing screen-scraping. It actually then uses only the "public" information, but it could access also those sensitive information we would like not to disclose.

I am not particularly worried that the app will leak the user information (I think they did this way because they are sloppy/lazy, not malicious), but still this is less then desirable behavior. And it made me wonder: is there a way to handle this situation at the root?

I want to tackle the problem in the most effective way, and I am starting to wonder: is there a way to let the user login ONLY through your page? To avoid that someone else authenticates using just a POST? To the best of my knowledge, you can make it difficult to do (using one-time ids in the page/headers, using client-side (Javascript) hashing, etc.) but there is no way to be 100% sure the post comes from "your page". But I am no security expert, so I may be missing something. And in any case, which is the best way to mitigate the problem, making it impractical enough (but with little or no burden upon the user - no captcha - which will not solve the issue, anyway)?

EDIT: to be clear: I am not talking about solutions for "preventing" screen-scraping. We already have mitigations to make it hard (and others to make it even harder down the line) that will make this particular app desist. I am talking about the login process: what can you do to be sure the username and password came from your own form? One of the "selling-point" of OAuth is that you give your credentials to someone (google, twitter) you trust; indeed, you are redirected to a form on the "trusted" domain. Besides looking at the url, the provenience of the credentials is enforced somehow? If so, how?

Answer 1

If they're screen-scraping, break their screen-scraper by changing your webpage. Name the "password" field "username" and the "username" field "password". Introduce a randomly-named hidden form field with random contents that needs to be posted back during login. Introduce changes to the page structure that don't change the appearance. Be creative.

Screen-scraping relies on "landmarks" on the page to find the important content. If those landmarks are constantly shifting, the person writing the screen-scraper will need to constantly update it to keep it working, and hopefully will get the message that they shouldn't be doing things this way.

Answer 2

Using a CAPTCHA is usually a good way of discouraging automation. This may discourage legitimate logins though, and the third party can always pass the captcha on to the user.

You could potentially blacklist the IP addresses that the third party are using. They can get around this by changing IP addresses, but the hassle of doing that might be enough of a deterrent.

It occurs to me that what the third parties are doing may in fact be illegal in some jurisdictions. If your terms and conditions deny access to third parties not using the OAuth API, then I think this is pretty clear-cut. (please get proper legal advice though)

Even if it is not illegal, you could warn your legitimate users about the third party applications that store their data improperly.

Answer 3

This is what I got so far (it is a mix of the other (2) answers I got till now, and personal investigation: For "general" discouragement of screen-scraping:

• You can randomize the id/class of the content which is being "scraped". (but the scraper could parse the HTML more throughly)
• For extra evilness, you can even add some other content, with similar id/class, which contains similar but wrong content and it is hidden by css (but the scraper could parse the CSS too)
• Ban user-agent (but the scraper can fake it)
• You could ban the IP(s) (but you might want to allow legitimate apps - and web access - from the same client)

Now, you could avoid all the hassle (at least in my case, where everything is behind a login) by "securing" the login, or better, identify the client as an authorized one. The problem is that in the case of a web-application, the "client" exposes everything in plain sight: html, css, JS.

Therefore, you do not have a place for "secrets", on the client: everything the browser sees, the scraper sees as well.

Anyway, the focus of this question was on making login through the web page as difficult to replicate as possible. So far, that's what I have thought off:

1. You can ask for CAPTCHA at login (but this is bad for usability, and it can just be passed onto the user)
2. You can do two-factor authentication (same drawback)
3. You can use something like a CSRF token (but since you are not in a session - yet - the client can just "scrape" the token too and send it along the POST login)
4. You can use client-side hashing, to hash the 'CSRF-like' token, and ask for it in the answer (but this can be "defeated", either reading the JavaScript and implementing the same hashing, or by executing the JS directly)

You can make it harder and harder, but as the scraper can mimick a legitimate web client in every aspect (and present to the user what he/she wants too). If you implement at least 3 or 4, the "automatic login" (just posting the username and password) does not work anymore, and it might be a strong incentive in using OAuth instead.

So no, to the best of my knowledge is impossible to guarantee the login will come from "your form", because in the end it is just a POST, and there is no way to assert the identity of "your form". (Related: OpenID spoofing/phishing)