32

I have a web application that the Yandex spider is trying access into back-end a few times. After these spider searching, there are few Russian IP addresses that try to access back-end too and they failed to access.

Should I block Yandex or take another action?

Update:

The Yandex spider visits a back-end URL about once per 2-3 day. We did not release any back-end URL at the front-end.

The "back-end" meanings: the web application's interface just allowing our administrative to manage the application

Improve this question
8
  • 2
    You should look up the IP addresses to see if they are real Yandex IP addresses, or not. For instance, looking at my own access logs, the most common IP address by far identifying itself as Yandex is 100.43.81.141, which turns out to be legitimate. By contrast, 104.238.95.146 is not
    – Michael Hampton
    May 9, 2016 at 9:55
  • 8
    What's the point? Scans won't stop when you block Russian search engines. It's only a matter of time before Chinese, Nigerian and Moroccan hackers pick you up.
    – Dmitry Grigoryev
    May 9, 2016 at 11:06
  • 5
    What is the "back-end" that you are talking about? Normally your "back-end" (middleware, databases and such) should not be even reachable from the internet.
    – mustaccio
    May 9, 2016 at 18:23
  • 1
    Do you have a proper robots.txt to inform the various spiders on which parts of your backend should or should not be accesed? In the absence of that, expect any linked URLs to your backend to be crawled by various well-meaning bots.
    – Peteris
    May 9, 2016 at 22:54
  • 3
    No, Yandex feeds DuckDuckGo.
    – Chloe
    May 10, 2016 at 1:29

4 Answers 4

Reset to default
78

Should i block Yandex

Why?
First, if the bot is a legitimate search engine bot (and nothing else), they won't hack you. If not, blocking a User agent won't help, they'll just use another one.
If your password is good, fail2ban is configured, the software is up to date etc., just let them try. If not, you need to fix that, independent of any Yandex bots.

To make sure the problem is actually Yandex, try disallowing it in robots.txt and see if it stops.
No => not Yandex.

(Did set up a new webserver some weeks ago. One hour after going online, had not even a domain yet, a "Googlebot" started trying SQL injections for a non-existent Wordpress. It was fun to watch, as there were no other HTTP requests. But I did not block Google because of that.)

Improve this answer
4
  • 16
    Reminds me of when one of those Windows worms went around. Our router at the time was a Linux box and my brother setup a script to make the PC speaker beep whenever our IP was scanned (and also to log the information of the infected machine). My dad said for a while it sounded like someone was playing a video game, due to the high frequency of attacks pinging. There's also a variety of maps like map.norsecorp.com/#
    – Wayne Werner
    May 9, 2016 at 18:23
  • 2
    Nevertheless it is disturbing that the crawler or its affiliates effectively harvested the backend middleware URL and was crawling it directly. I'm seeing the same pattern myself and it is unsettling to see a crawler target unpublished URLs.
    – Brad Hein
    Sep 18, 2016 at 3:01
  • @BradHein For commonly used internet software, these URLs are known, it's no reason to believe someone hacked you
    – deviantfan
    Sep 18, 2016 at 7:26
  • @deviantfan That is true of commonly used internet software. Did OP state that his software is common? I see Yandex commonly target custom-designed software by its unpublished URLs however, so it is using unscrupulous means to gather these otherwise hidden URLs.
    – Brad Hein
    Sep 20, 2016 at 14:00
9

Along with agreeing with @deviantfan 's answer and specifically with this point

First, if the bot is a legitimate search engine bot (and nothing else), they won't hack you. If not, blocking a User agent won't help, they'll just use another one.

I would like to point out that as Yandex as well as another search engine bots in general might not intentionally want to access your backend. Remember bots are crawling the sites by following the links, so imagine if the bad guys would put some of your backend's urls in some other website's pages, and the search engine simply indexed those pages and now is trying to follow the links from there. So, it will look like the search engine is trying to access your backend - but it just crawling the net: it does not know that it is your backend.

Similar thing might happen by accident. Lets say a non-tech savvy user posted a url in some forum, that is only accessible when you are logged in - by crawling the search engine will try to follow those links and you will end up seeing logs as I assume you did.

UPDATE: I think you might want to set in your robots.txt rule to disallow yandex to access specific urls. Btw, you better define specific rule with its name, I am not sure, but it might happen, that yandexbot can ignore User-agent: *, so you can do smth like this(according to your backend urls)

User-agent: Yandex
Disallow: /admin/*

So, in this way you will disallow it to try to access backend urls - matching that pattern, but at the same time it(yandexbot) will be free to crawl another pages of your website.

Improve this answer
0
3

You should not block the legitimate Yandex bot, but you could verify that it is in fact the legitimiate bot, and not someone just using the Yandex User-Agent.

From: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml

  • Determine the IP address of the user-agent in question using your server logs. All Yandex robots are represented by a set User agent.
  • Use a reverse DNS lookup of the received IP address to determine the host domain name.
  • After determining the host name, you can check whether or not it belongs to Yandex. All Yandex robots have names ending in 'yandex.ru','yandex.net' or 'yandex.com'. If the host name has a different ending, the robot does not belong to Yandex.
  • Finally, make sure that the name is correct. Use a forward DNS lookup to get the IP address corresponding to the host name. It should match the IP address used in the reverse DNS lookup. If the IP addresses do not match it means that the host name is fake.

In fact, almost all big search-engines provide similar ways of verifying the User-Agent. The way this works is because someone can spoof the reverse DNS lookup, but not the forward DNS of that spoofed address.

Improve this answer
1

While I agree with the answer from @deviantfan I'd like to add that - since your initial question sounds like the route to hell I saw someone go down a few years ago - even if you try blocking a spider completely as opposed to just telling it to go away via robots.txt (which you probably should do for the admin parts anyway), you'll end up in an unending game of whackamole that you can't win.

A few years ago someone on another site I read was ranting about how Google/Microsoft/Yandex/etc were all "DDOSing" his website. Apparently the site was "private, and only a few of his friends should have any access, and it shouldn't show up in any search engines". However apparently at least one of its users was posting links to internal locations that the various web spiders were finding. (As opposed to them just knowing about www.whatever.tld from DNS records.)

The person running it decided that having a robots.txt file so the spiders who found links to it would just read the file and go away was unacceptable. They needed to ignore his site forever and never touch it ever again under any circumstances. (He never gave a reasonable explanation of why this was the only acceptable option. As apparently were doing something about whoever was posting links to internal locations on his site that the spiders were finding, or just requiring authentication so the bots would just get dumped to a content free login page.)

Instead he just started blacklisting all the web spider IPs so that any request attempts they made timed out without a response. Once he did that, he discovered that, instead of following a single link getting the robots.txt file and going away for some length of time, the spiders had much shorter retry intervals when they errored out and began hitting his site from numerous different IPs (presumably the spiders trying to route around what they assumed was a network error).

This was the point that his IP blacklist exploded to thousands of addresses and he started ranting about being DDOSed by the search engines. Sadly the person had gone full on crazy at that point and refused any and all attempts to actually address his real problem instead of what he was convinced was the correct solution.

Improve this answer
6
  • 1
    I have seen that behavior when a web server was improperly configured, and 'normal' spider traffic would actually cause loss of service. Add to that an inexperienced admin, and you've got someone with a full time job of looking up IPs and adding them to IPTables block chains.
    – David Wilkins
    May 10, 2016 at 16:20
  • "a full time job of looking up IPs and adding them to IPTables block chains" .. oh man .. this guy -> img1.etsystatic.com/010/0/5175241/…
    – a20
    May 11, 2016 at 5:09
  • To be fair, though, the person does have a right for his private server to not be accessed in the first place. Even getting a robots.txt error and going away /is/ wrong. If it's private then they shouldn't even scan his server. Who gave them the right to access it?
    – user64742
    May 11, 2016 at 5:36
  • 1
    @TheGreatDuck If someone doesn't want anyone to see smething, why put it on a public website? ...
    – deviantfan
    May 19, 2016 at 10:15
  • He didn't. It's his private website. Someone from the Internet accessed it and shared it. /He/ never gave permission for it to be shared. It was private. Technically he should have asked his friend or the site to remove the link. However, I can see why he had an issue. He never shared the link. He never gave permission for it to be accessed. So it shouldn't be accessed. Yes, he can't blame Google but he could demand that the privacy be restored so it stops finding and accessing. That's not at all an unreasonable request if it's meant to be a private server.
    – user64742
    May 20, 2016 at 1:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .