I need to manage my website's requests. Because of both I need to detect suspect requests and block that IP and calculate the number of users which visit my website.

So I've created a table like this:

// requests
| id |  UserId_or_IP  |  unix_time  |
| 1  | 12353          | 1339412843  |
| 2  | 23353          | 1339412864  |
| 3  | | 1339412894  |
| 4  | 3422           | 1339412899  |
| 5  | 342232         | 1339412906  |
| 6  | 2353           | 1339412906  |
| 7  | 7785           | 1339412951  |
| 8  |  | 1339413640  |
| 9  | | 1339413621  |
| 10 | 5462           | 1339414490  |
| 11 | 342232         | 1339414923  |
| 12 | | 1339419901  |
| 13 | 8007           | 1339424860  |
| 14 | 8007           | 1339424822  |
| 15 | 2353           | 1339424902  |

And I insert a new row into that table for every request (opening every page of my website means one request).

And here is my query in the index.php: (index.php is a page which all pages pass from it)

INSERT INTO requests (id, UserId_or_IP, unix_time)
            VALUES   (NULL, :id_or_ip, UNIX_TIME());

All I need to know: What I'm doing is correct or there is a better approach to control requests? As you know That query makes my webpage-loading slow (a bit), because there is an INSERT query before loading every page. Well has it worth? In fact I'm trying to understand what's the standard way for doing that.

closed as off-topic by Anders, LvB, Matthew, Xander, Steffen Ullrich Jun 24 '16 at 21:04

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be about Information security within the scope defined in the help center." – Anders, LvB, Matthew, Xander, Steffen Ullrich
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    how is recording the ID\IP a method of controlling requests? – schroeder Jun 21 '16 at 9:20
  • 2
    ok - so when you say "suspect requests" you really mean "high volumes of requests from a single source". The one problem with your plan is shared IPs. If a large company is accessing your site, you will end up with what looks like high volume from a single source. Unless you have other method determining "suspicious requests" you might have to plan for high volumes. – schroeder Jun 21 '16 at 9:26
  • 1
    Or a web crawler, getting your content into search engines, or a user who really likes your content and wants to read it all... You've put the content on your website - surely you want it to be read? – Matthew Jun 21 '16 at 9:37
  • 1
    @stack any website is public, and anyone can copy it. If that's what you want to prevent, then you need to approach the problem very differently. – schroeder Jun 21 '16 at 10:56
  • 1
    A suspect request is one that indicates non-usual behaviour that might be linked to unwanted behaviour. What that means for you will be unique. Whether or not you need to rate-limit requests is a question for your app and what your servers can handle. – schroeder Jun 21 '16 at 11:17

In fact I'm trying to understand what's the standard way for doing that.

On a properly configured web server, the web server will log all received requests in a log file (the exact content of this log file is usually more or less configurable, depending on the server software used). There are also standard means to centralize the logs if you want to like syslog which will allow you to store and analyze the logs on a different machine.

The content of this log can then be analyzed by external software in order to either:

  • Generate statistics, there are plenty of monitoring software depending on your exact software and needs,
  • Detect suspicious behavior, here you will search for IDS / IPS software, depending if you need Detection only (IDS), or Protection too (IPS). Fail2Ban for instance is a classical example of such software for Linux/Unix platforms: it will monitor your server log files, detect suspicious behavior according to some set of rules and optionally trigger an action (send an alert email or update firewall rules to ban the offender for instance).

You did not asked for it in your question, but speaking of standard practice, generally the logs are also analyzed in order to analyze the health of the server software and detect possible issues and outage. As for IDS/IPS, you have the option to have a simple detection & alert system or automatic actions.

With this, usage stats + health stats + security issues, you get a pretty complete view of your server activity.

  • Since OP wants to protect their website, IDS/IPS discussion should include Web Application Firewalls (WAF). In this case ModSecurity is the heavy favorite. – user2320464 Jun 21 '16 at 18:13
  • @user2320464: The OP did not asked "How can I protect my website?", this would be an overwhelmingly broad question where WAF would be just a part of the equation, he asked for a standard way to store all received queries and to analyze them in order block offenders and generate statistics. WAF are positioned in front of the server before the requests are received, so they are not part of this. Moreover, you talk about ModSecurity which is an Apache-only module, the OP did not restrict his question to a single web server and actually uses Nginx according to its comments. – WhiteWinterWolf Jun 22 '16 at 8:49
  • I agree this is a broad question and needs more specifics. Only wanted to point out that when protecting websites/web servers specifically, one should also include WAFs with IDS/IPS. ModSecurity supports lots of platforms now, even IIS! – user2320464 Jun 23 '16 at 6:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.