0

I'm working on web scraping and data mining for Alexa top-ranked websites in the world.

I was doing that with node.js, Axios (for HTTP request), on a Windows machine (especially on WSL bash). After I was running the top 500 or 1000 websites, I started getting some malicious attacks, security problem notifications from my security software on Windows (Norton security)

enter image description here

I didn't think it would be a security concern to web scrape using a decent scraping/web request library, and it's the first time to do web scraping to such a big list of URLs, I'm surprised I got attacks through the program.

  1. Is it safe to do web scraping to such a random big list of URLs, or the attack notifications are ignorable?
  2. If not very safe, what can I do to my code or used library, language, or Windows setting?
  3. If I use Linux or Ubuntu, will it be no problem?

1 Answer 1

2

I'm surprised I got attacks through the program.

The word "attack" is likely an exaggeration here. Probably you just hit a site which has a very low reputation and has served malware in the past, i.e. "Malicious Domain" in the message. Nothing attacked you've actively here.

Is it safe to do web scraping to such a random big list of URLs, or the attack notifications are ignorable

It depends a lot on what you are doing in detail. If you just make HTTP requests and responses and follow links with some simple program than it likely can be ignored. If you post-process data like converting images etc it might be more of a problem since this might trigger security issues in complex libraries. Of course this assumes no newly introduced security issues by programming errors in your code or used libraries, i.e. data from the internet should always be treated as untrusted and no assumptions about their content and format should be made without explicit verification.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .