I saw some suspicious errors being generated on my site based on pages that were requested. My error is logging the path that the user is trying to access. Because of these errors (and the paths that they were trying to access) I created my own blacklist process where I can blacklist someone from my site based on ip address and/or username.

After implementing this, I didn't see ANY errors of that kind....until today.

Now, before I go ahead and blacklist this person, I'd like to make sure that it isn't a legitimate search engine just trying to build it's database with all links available from my site.

So, my question, is there a way to see what company an ip address is assigned to?

Or, do those crawlers from search engines only go to pages that exist?

migrated from programmers.stackexchange.com Sep 13 '14 at 14:54

This question came from our site for professionals, academics, and students working within the systems development life cycle.

  • Why the downvotes? – ganders Sep 12 '14 at 17:59
  • 1
    @ganders: about the downvotes, you may focus on your first two questions and remove the last one. You may also rephrase the title: as-is, it is too broad and can only be answered by a book, maybe several books. – Arseni Mourzenko Sep 12 '14 at 18:11
  • @MichaelT thanks. Had no idea the complexity of this...I honestly thought that it was as easy as "if user tried accessing this path, and it's never existed, then...", or "if ip address = ..., then...", or etc. Now I know, thanks everyone. Might be looking for some books on Amazon now... – ganders Sep 12 '14 at 18:39

is there a way to see what company an ip address is assigned to?

You can find additional information about an IP address by using WHOIS services. ARIN, is a good start, and will redirect you to other registries (such as LACNIC or RIPE) when needed.

For most activity from legit search engine bots, the WHOIS will also contain the name of the company. For example, the WHOIS for (one of the IP addresses used by Google crawler) will indicate that the IP address is registered by Google Inc.

You may also be interested by the lists of IP ranges by crawler, such as this one.

Or, do those crawlers from search engines only go to pages that exist?

Not necessarily. A crawler doesn't always know that the page doesn't exist:

  • A page could have existed and indexed in the past and was removed later,
  • An internal link can lead by mistake to a resource which doesn't exist,
  • An external link can lead by mistake to a resource which doesn't exist.

Here are some of the errors that I'm getting, should I blacklist this person?

It's your responsibility to decide to blacklist the IP or not.

  • Thanks, based on your 3 bullet points, and that those pages have NEVER existed on my site, it's pretty easy to draw the conclusion that it's probably a hacker. Thank you! – ganders Sep 12 '14 at 18:11
  • 1
    @ganders: not necessarily. I've seen lots of activity which looks strange at the first glance, but which then appears to be perfectly legit. The actual queries you quoted might as well come from something similar to an RSS client. – Arseni Mourzenko Sep 12 '14 at 18:14
  • Ok. I'm wondering if I should even mess with blacklisting people since it sounds like it's hard to determine whether they are nefarious or not... – ganders Sep 12 '14 at 18:16
  • 2
    Are they harmful to your business? If yes, ask yourself exactly how much loss do they cause? A few actual hacking requests per minute equals a few kilobits of bandwidth and a small amount of computing power wasted—not good, but you can live with it. On the other hand, if you spend three hours per month dealing with blacklisting, it means that your company spends $400-$600 for your salary doing just that: it would be much less expensive to ignore those requests. – Arseni Mourzenko Sep 12 '14 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.