1

We run scraping jobs on behalf of users. We use webkit-gtk as browser to do the scraping. Each job is executed as a separate process

So as not to mix user data we have isolated following data per user.

  1. Cookies
  2. Flash cookies
  3. Cache files
  4. HTML 5 storage

Do I need to worry about any other types of files that normal browser sesson will save? The "browser" does not have any other plugins apart from flash. Browser does not save any password and auto form fill is disabled.

Improve this question
0

1 Answer 1

Reset to default
1

You seem to have the basic user data covered, but I'm not sure why try to isolate it like that. I would run each user's browser instances with an in-memory home directory/storage area. Then when the process terminates, all of the data is destroyed, no risk of leaving anything behind. Headless Webkit like PhantomJS provide flags specifically to move these data to their own directories, which can be placed on a tmpfs ramdisk. (Webkit-gtk seems to be a library, so I assume you've wrapped this in your own program?)

Improve this answer
1
  • All the scrape jobs need authentication. Sites "remember" computer in various ways. So we have to recreate exact same data rather than destroying it to minimize security questions. And yes, we use java wrapper for webkit-gtk.
    – user871199
    Jul 22, 2014 at 14:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .