Odd “duplicate” HTTP requests

Question

An unexpected duplicate HTTP GET operation triggered an "unusual event" report on my web site. The duplicate came from the web site of "Trendmicro.com" who seem to be in the internet security business. It's just my guess, but I suspect they collect sample queries on behalf of their clients, then submit those queries themselves and perform some kind of analysis on the results.

This is usually innocuous, but I wonder what the implications would be for any of traffic which is intended to have side effects, if the server is not aware that another, similar request may appear.

For example, my case in point the request was intended to update a user ranking database, and the duplicate triggered a hacking alert. Without the check, the update would have occurred twice.

edit: It's also relevant that the "original" request originated from an IP in Australia, and the "Duplicate" arrived a few seconds later, from an IP locatated in San Francisco. There were a hundred or so HTTP requests in the session, but only 2 or 3 were duplicated.

Answer 1

Trend Micro are a fairly big player in security (well at least I have heard of them).

One of their products is a browser toolbar to warn users of malicious websites. I agree with your guess, that they are fetching pages that their toolbar has seen their users fetch, so they can decide whether it is malicious.

There is a little info on the Trend Micro spider here: http://www.webmasterworld.com/search_engine_spiders/3657953.htm

Ideally they would only replay GET requests, and GET requests should be idempotent (have no side-effects). If your user data is getting changed as the result of a GET request, then I would recommend changing that to a POST. Not only the Trend Micro bot, but any robot or search engine spider might come along and do a GET on any link and cause an unwanted result for you.

Answer 2

The issue is totally dependant on what the server is intended to do. A "replay attack" is an old and quite valid approach to attacking almost any system or protocol. Bumping up a user's rank is a pretty good use, since it would certainly change the state of the system and serve to benefit someone. Buying two orders on someone's credit card would be even worse!

It's something that should be factored into the design of a system. Things that I've seen used as preventatives:

• time out before repeat - the system makes a given account wait a certain amount of time before repeating an action (Stack Exchange has done this in places)
• requiring a nonce, when digital signatures are used. - if you ask to make an update, my server gives you a unique piece of meaningless data, you submit your request and the data, and sign it all. I won't let you submit the same peice of meaningless data twice - you have to ask, sign, send over and over which requires that the user's credentials be available.
• point to point privacy - keeps the man in the middle out

Answer 3

The clue here is where you say you received a duplicate GET request. The rest of your question almost put me off course, thinking some client (or another server on its behalf for whatever reasons) duplicated POST requests, or even duplicated complete request headers (cookies, e.t.c.). But what you describe could as well be read as "same URL was requested".

Anyway, the most common reason for duplicate GET requests come from what are known as caching proxies. Some such products can be detected by seemingly lacking any support for compression (gzip, deflate,...) showing in your logs as duplicate requests but the response size will be larger, such as Blue Coat System's ProxySG (apparently one of the largest players in the proxy appliance business). Others might change User Agent string accordingly to identify themselves to web admins reading access logs, Web Application Firewalls, e.t.c, and some repeat requests completely synonymously to original requests, and only the IP address might differ, if even (if the client uses the proxy also as gateway, then the IP address will remain exactly the same, and repeated requests will follow the cache expiration headers of requested contents).

These caching proxies can be a bit obnoxious at times (see for example this ranting on Blue Coat System's solutions), but they aren't considered as any particular security threat. Well, at least not any differently than with all other proxies - at the end of the day, they are Men in The Middle and untrusted hosts can be considered extremely risky to use to the end users. But a lot of networks do use them for various reasons, such as local caching to reduce bandwidth needs to remote hosts, speed up response times, even cache multiple old copies of accessed documents for accessibility. And in a sense, Google is the biggest proxy we know of. What I'm saying is there is no indication the request you describe was duplicated with any malicious intentions, and such request duplications shouldn't cause any problems on your server end (besides an occasional unnecessary bandwidth hogging), if your web applications are written properly, to re-authorize users whenever necessary, and use other means of identifying individual users than GET request's URI parameters (such as cookies, that non-malicious caching proxies shouldn't attach to their duplicate requests for caching purposes).

I'm not saying what happened wasn't a malicious attempt, what I'm saying is what you describe doesn't prove it either way, and it could as well have been a benign visitor of the caching proxy variety.