How can I detect that my page is requested by robot, but not user's browser? I'm aware of basics tricks:
- Watch for incorrect headers or urls. For example, urls with hash or header with full url - GET www.yoursite.com/test
- Detect that several unrelated pages were requested directly by some IP (not good enough)
- Watch for missing headers
- Watch for obsolete User Agents
Is there any other ways to detect robots?
UPDATE: some good robots identify themselves in User-Agent header:
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)