I saw some suspicious errors being generated on my site based on pages that were requested. My error is logging the path that the user is trying to access. Because of these errors (and the paths that they were trying to access) I created my own blacklist process where I can blacklist someone from my site based on ip address and/or username.
After implementing this, I didn't see ANY errors of that kind....until today.
Now, before I go ahead and blacklist this person, I'd like to make sure that it isn't a legitimate search engine just trying to build it's database with all links available from my site.
So, my question, is there a way to see what company an ip address is assigned to?
Or, do those crawlers from search engines only go to pages that exist?