We're looking for help/advice from website security experts. Our company develops web-based process management software and hosts it for our customers. The software sends emails to users when they have new tasks in the system, or reminder emails when those tasks become overdue.
We've recently started seeing probes in the web server logs that indicate something is taking one of the URLs from those emails, partially obfuscating the parameters using a ROT-13 approach, and sending it to our server. Then it takes the response page (usually a sign-in or error page) extracts the page contents, constructs a postback URL with those contents included as parameters, and sends it to the server. Then the sequence ends. This usually happens within seconds of the email being sent, and independent of whether or not the user's email client is active, indicating that it's being intercepted and processed somewhere in between.
There's no apparent danger from this behavior, and at first we thought it might be some kind of anti-malware agent deployed inside one of our customer's environments. The "attacks" would come from one of two datacenters (one in New York and one in the UK), even though the user wasn't in either of those locations, and would rotate through a variety of IPs on specific subnets. However, we just took a new customer live, and we're seeing it happen for a lot of their users, and with the same behavior: probing "attacks" on our servers from one of two datacenters within seconds of sending an email to those users.
Here are the datacenter IP addresses we're seeing the attacks from:
- 126.96.36.199-167 - above.net
- 188.8.131.52-207 - as29017.net
We've looked but haven't found any reports of similar behavior, and we're hoping that the StackExchange community might be able to give us some suggestions on where to look, or how to proceed with our investigation.
- Is this explanation of the problem clear?
- Has anyone else seen something like this?
- Is it some kind of "innocent" anti-malware behavior, or indicative of an actual attack?
Thanks ahead of time for any suggestions, and let us know if there's any additional information we can provide that would help.
UPDATE: Thanks munkeyoto. We'll look into contacting AboveNet.
With regards to your suggestion, the e-mail is actually successfully delivered to the customer unchanged. We believe something is just searching the emails for URLs to attempt an attack. Once it finds a target URL, in this case our web application, it modifies the URL before using it to contact our application.
Our server sends out large numbers of email each day to many different customers, and this behavior only occurs with two specific sites. Because of that, we believe that the emails are being scanned after they've left our network, somewhere along the SMTP chain. We also don't believe it's in the user's specific email client because it happens even when their machine isn't turned on. And since the probing attacks are coming consistently from two specific subnets inside data centers, even though the emails were sent to completely different customer companies, we think that points to something inside those data centers.
Anyway, we'll follow up with AboveNet and see if they can offer advice.
UPDATE: In response to the question about PHP/WordPress: Neither. It's an ASP.NET application suite written entirely in C#.
UPDATE: In response to the question about how the URLs are modified and probed.
Here's what a normal sequence might look like for our application when someone would normally click one of these URLs from an email (simplified for demonstration purposes):
2013-11-13 22:14:53 XXX.XXX.XXX.XXX GET /ExampleApplication/TaskList.aspx
2013-11-13 22:14:53 XXX.XXX.XXX.XXX GET /ExampleApplication/SignIn.aspx ReturnUrl=%2fExampleApplication%2fTaskList.aspx
And instead here's what we're seeing when the probes occur:
2013-11-13 22:10:23 XXX.XXX.XXX.XXX GET /ExampleApplication/TaskList.aspx
2013-11-13 22:10:23 XXX.XXX.XXX.XXX GET /ExampleApplication/SignIn.aspx ReturnUrl=%2sRknzcyrNccyvpngvba%2sGnfxYvfg.nfck
So in both cases a request to the TaskList.aspx page is made. The server responds with a redirect to a SignIn.aspx page (because the user hasn't authenticated yet). In the normal case, the browser agent simply does a GET on that redirect page. In the probe case, the agent modifies the ReturnUrl argument by ROT-13'ing the text before executing the GET. Then it does a second GET with a mostly correct ReturnUrl, but with additional concatenated parameters containing the names of HTML fields from the SignIn page. It almost looks as if it's attempting to use potentially hidden fields on that page to hack into the system via the parameters (there aren't any, and it wouldn't work).
Let me know if this is the kind of information you're looking for. Thanks.
It took a few calls with AboveNet (now part of Zayo), but we were finally able to determine that one of their customers is an anti-malware firm based in the UK, providing services to two of our common customers. They were scanning all incoming emails and probing any hyperlinks to identify potential hazards and/or vulnerabilities in the destinations. They've added our application servers to a whitelist and the probes are no longer occurring. Thanks to @munkeyoto for the suggestion to contact them.