2

We have an API openly accessible through a data visualization on a website and an app. We'd like to make the life of potential data scrapers a bit more difficult.

First thing we did was to add some throttling, but in our case the line between legit and malicious request rate can be blurry. It also doesn't keep scrapers from taking their time.

It's currently easy to see how our requests are built and create new ones, so we thought of making this more difficult by generating some kind of hash using chosen values (request parameters, current time, value in cookie, etc.) on the client side and checking it server side. The code generating the token would naturally need to be obfuscated on the client side.

So the question is, how useful can such a protection be? It appears to come down to how much we can obfuscate the token generation. How efficient is this, particularly in javascript?

  • how would the success of this endevor be measured? how many scrapes/day you have now? If you want to see some decent practice, look at gmail's view-source/DOM. all the identifiers are randomized and humanly meaningless, certainly doesn't make it easier to scrape email... – dandavis Oct 10 '17 at 2:50
  • We don't have scrapes for now (none meaningful at least), this would be more of a preemptive move. We're really just looking for relatively easy ways to deter most scrapers. – Jukurrpa Oct 10 '17 at 3:10
  • 1
    ok, simply reverse what's easy; i like working on data sources that are named semantically, delivered in json/csv or other plain text, have predictable parameters that are statically named, that arrive over http, that don't need some complicated auth or cookie value, and that do the same thing each time I request data the same way. The less of that stuff i see, the less "bother-worthy" it is... – dandavis Oct 10 '17 at 3:14
  • Yep, what you cited is exactly how our requests are right now. Open up developer tools and you instantly know how to build new requests and parse them. That's why the hash thing seemed like an easy way to add some difficulty to that. – Jukurrpa Oct 10 '17 at 3:17
1

Not terribly useful I think. The attacker doesn't even have to understand your obfuscation - if they have access to the routines (I assume they do since its on the client) the attacker can just steal your code and run the same parameters through to create their hash.

If the server serves up previous pages that this is requested from, then you can add a one time token/nonce (a bit like a CSRF token) to the API. This would make the subsequent request pretty much impossible to replicate without requesting the previous page first and getting their own token.

But even with this strategy, a scraper could always go back to an entry point page/URL and play through the sequence of requests as if they were a normal user. This is likely to be a valid attack no matter what obfuscation you use since it's going to be indistinguishable from a valid user.

  • So I guess having access to the routines helps but they'd still need to figure out which parameters to pass (not sure how well this can be hidden). Also it would basically force an attacker to decompile/unobfuscate code which can be much more deterrent that just having to check his browser's network requests. – Jukurrpa Oct 10 '17 at 3:14
  • @Jukurrpa: not really, i always simply roboticize the user controls by scripting mouse clicks and sucking the data from the visible DOM (if it's not ready to go in the ajax returns). works everywhere, no matter the backend technology, auth methods, server-side filters, etc. i used to mess with curl, but nothing has nothing anymore, except the in-browser view, which always has everything... – dandavis Oct 10 '17 at 3:18
  • That's a good point but the browser view isn't as easy to parse as it's a map, barely has any text fields or anything. It would be much more of a pain (and way slower) than parsing the json – Jukurrpa Oct 10 '17 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.