The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

74

Should i block Yandex Why? First, if the bot is a legitimate search engine bot (and nothing else), they won't hack you. If not, blocking a User agent won't help, they'll just use another one. If your password is good, fail2ban is configured, the software is up to date etc., just let them try. If not, you need to fix that, independent of any Yandex bots. ...


59

This isn't really a great question for stackexchange as Google is keeping its algorithms secret so all we can really do is make guesses about how it works, but my understanding is that the new system will analyze your activity across all of Google's services (and possibly other sites that Google has some control over, such as websites that have Google ads). ...


23

CloudFlare serves as a guard between your webserver and the client. Every content the client receives got provided by your webserver and filtered by CloudFlare. This way, CloudFlare obfuscates email addresses by filtering them using a regex before delivering it to the client. If your website contains the email <a href="mailto:s@scha.bz">s@scha.bz</...


23

Since you have the logs I suggest that you look for usage of the login form. Did the try to login at all? Most often this is just a scan that looks for interesting sites and stores them for later use. This behaviour is extremely common and is common place in almost every http log with a internet facing web service. First of all you should look at the ...


16

I also use to be amazed by this thing. So, what I did, in Chrome open incognito mode, then browse a site that has the new Google CAPTCHA and tick the box. Well, it didn't get me through, instead it shows a series of images and asked me to select images related to one image. This shows that Google is constantly tracking our behavior to determine if we are ...


16

Simple bot behaviour and "normal user" behaviour are noticeable different, and most bots tend to be relatively simple, since it works for the majority of sites. For example, consider arriving on Security.SE: A human loads the page, there is a delay of a few seconds upwards whilst they read the first few questions, then you get a request for a page, followed ...


12

In addition to James' and Matthew's answer (which are both valid points by the way): Obviously services like CloudFlare have a bunch of detection methods to decide whether or not a client is allowed through their various layers of protection. They have a lot of information on their website about these features but you probably won't find specific rules ...


9

Along with agreeing with @deviantfan 's answer and specifically with this point First, if the bot is a legitimate search engine bot (and nothing else), they won't hack you. If not, blocking a User agent won't help, they'll just use another one. I would like to point out that as Yandex as well as another search engine bots in general might not ...


9

The Majestic project is a distributed web crawler, which explains why you get such a lot of different source IP addresses. It is not malicious, that is it does not attack your site and it does not even uses lots of resources (800 requests a day is not much). Like most proper bots Majestic even includes a URL in the user-agent string and if you visit this ...


9

This looks like an open HTTP proxy scan to me. The HEAD or GET request is not normally followed by http://, but only by the local path. If your server acts as an open HTTP proxy, the attacker is trying to hide behind it and you should close it. This will get your server blacklisted pretty soon. Make sure your server is not acting as an open proxy, then ...


8

When you click on I'm not a robot it sends over an HTTP request to google with the whole bunch of useful information things like Your IP Address Your country Timestamp Information from your browser such as the way you move your cursor just before entering the checkbox. How you are scrolling the page before the click. The time interval between different ...


7

Those UDP packets all seem to go to a public DNS server. Considering the destination port is 53, it is highly likely you are doing DNS queries and not something malicious.


6

As mentioned by echelon, Zeus source code is available in GitHub. Availability of its source code (leaked in 2011) is one of the reasons many modern botnets are evolved from Zeus. Be careful when infecting with your botnet several VM/computers you control, you don't want the to infect real user machines with your toy botnet! For additional security you ...


5

This anti-bot library belongs to Akamai. Clients execute the javascript and post to /_bm/data on a site behind the Akamai CDN. The CDN's bot manager (bm) then makes decisions based on the submitted data such as slowing down responses to suspected bot clients, or downright returning 403s. The abck cookie is returned upon submission to the /_bm/data endpoint. ...


4

Most secure is very subjective. Even Google's Captcha system has been known to be broken on ocassion, one example out of many. Treat Captcha as a speed bump to deter people from putting in a lot of effort to break it. Checkboxes if your speed bump is good enough is... maintained by a trusted company, regularly updated, and is useable. I know you said ignore ...


4

I question your premise. A properly implemented site should be made secure without a Captcha. Once you have accomplished the appropriate level of security you need, adding a captcha should be thought of only as a convenience to prevent bots from submitting forms or taking certain actions. In other words, whether a bot performs the action, or a person does, ...


4

They are performing port scans, not trying to establish a connection. This specific scanning technique is called a SYN scan. The idea is that you don't have to complete the entire TCP handshake to find out if a port is open. A SYN scan is often preferred to establishing a full connection because it's faster and less noticeable. From the nmap guide: SYN ...


4

What I see there are two different users, one (139.x.x.x network) presumably being a script kiddie running an exploit scanner, and the other (46.x.x.x network) being a presumably legitimate user. That, and you forgot to anonymize your server's address (45.x.x.x network) in the last line. Bottom line: Given no other signs of intrusion, no reason to panic.


4

If I had the means to automatically detect a 0-day exploit in a piece of software, the vendor of that software has the same potential, thus these exploits would be caught before the software was ever released. Such things are done, of course, and they are called Source Code Analysis. They do exactly what you are trying to do: Automatically look for ...


3

Theoretically it is possible, but it takes a lot of skill and effort - much more than it is worth. The captchas are created to be hard for machines, and easy for humans. This is the perpetual arms race between spammers and webmasters. It's easier to have a human verify the captcha's for this hypothetical bot network. Gaining rep by fake posts is difficult, ...


3

You should not block the legitimate Yandex bot, but you could verify that it is in fact the legitimiate bot, and not someone just using the Yandex User-Agent. From: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml Determine the IP address of the user-agent in question using your server logs. All Yandex robots are represented by a ...


3

If I understand your question correctly you wish to redirect certain IP-adresses to a different domain? If that is the case you can use this in your .htaccess file: RewriteEngine On RewriteBase / RewriteCond %{REMOTE_HOST} 46\.4\.123\.1 RewriteCond %{REQUEST_URI} /index\.html$ RewriteRule .* /blocked.html [R=301,L] change the RewriteCond and RewriteRule, ...


3

While all of them lead to a 404 Not Found, CPU cycles are wasted in processing these requests ... Somewhere CPU cycles "must be wasted" to filter out these requests. But it depends on the kind of requests and of your server and application setup how much cycles this will be and where exactly they will be needed. If there is a clear source of these requests ...


3

To limit its chances of success, make sure your users' passwords are strong and all security patches are applied on time. From your description it looks like a simple automated brute force activity trying to break into your system in some simplest scripted ways. This on the Internet is roughly equivalent to white noise in radio transmission. It's always ...


3

Almost always, but not always. It could be a user behind a misconfigured proxy, or a browser extension, or (more likely) someone using telnet/netcat: netcat server 80 GET / HTTP/1.0 <enter> <enter> And done. You can ban users without User-Agent set, but it's dead easy to set one. curl, wget, aria2, python and almost every single command/...


2

A lot of botnet are actually "commercial" software, in the sense that you have to pay like a licence to get them (even underground commerce remains a commerce... actually you may even find that botnet developers actually offer paid support for their tools!). However, you may find older botnets or versions (ie. software widely known by current anti-viruses) ...


2

The router is like a automated door that would: Let anyone open it from the inside Require a passcode to open from the outside There are multiple ways to bypass the normal behaviour, some are exploiting flaw in the implementation of the router, other flaw in the protocols and some in design. Let's say you invited a friend to stay at your home, then you go ...


2

I have found that these automated, undirected scans are usually scanning IPs randomly and do not include a Host: header or include a bogus Host: header. Filtering out requests with bad Host headers can reduce the nuisance logging a bit and possibly reduce the impact on your server, depending on the server's architecture. This doesn't make your server much ...


2

This is either just a simple spidering, penetration testing, browser randomization, or a mix of some of those. Web Spiders Many web-spiders allow you to randomize your user-agent while siphoning the contents of a website. This is rather trivial to implement, and some of my web spiders do the same thing. However, it is bad design to randomize user-agents ...


Only top voted, non community-wiki answers of a minimum length are eligible