Email server compromise?

Question

My company sends around 1M emails per month through various providers (Sendgrid etc.). One of emails we send is a "sign up" email, which contains a link for a user to finish registering their account.

This morning around 5:21am we sent a batch of around 200 to a company in CA. Within a minute our system logged 200 errors. The link in the email goes to a registration form which contains a hidden numeric ID and some other info so we can match them up to the right account. The system errored in that the ID submitted wasn't numeric.

When we looked into it, all of the errors were employee emails at the same company. So we setup some more verbose logging to catch the form submits, resent one email to one person and within 3-5 seconds we had the error and form data. Something replaced every form field - including the numeric ID and other hidden fields) with what looks like gibberish email addresses:

lrls******@mailru.com
txiat*****@walla.co.il
etc. 

The only fields it interpreted were the password field and password verification fields (it didn't get verification right but did enter two non-email strings that would be good passwords)

The IPs that generated the errors all came from two companies - ColoUp in DC and ColoCrossing in Buffalo NY. While ColoCrossing appears legit, ColoUp is another story. When you dig into their services you see that a colo rack is $11,500,000 / month. Most pages are a mix of English and Arabic writing.

My question is, is this a compromised email server, or is their mail being routed through a malicious proxy, or is this some sort of hyper aggressive email filter that they have installed?

Answer 1

Publicly accessible forms that send an email to an arbitrary address can be leveraged as part of the cover-up for an attack. By flooding the inbox of a victim, an important message containing evidence of a hack might be ignored in a storm of messages.

Dima Bekerman wrote a firsthand account of what that looks like. I've included the most relevant portions below, but the whole post is interesting and definitely worth a read.

I only noticed that something was odd when I opened Gmail one night and found hundreds of registration confirmations to numerous services I’d never heard of. What’s more, I was receiving a similar email every few seconds.

When most of the noise had been cleared, I found an Amazon email hidden among the junk. It informed me that my purchase—one I hadn’t made—would be delivered within 24 hours.

TLDR: Don't let bots send email to arbitrary targets from your infrastructure.

Answer 2

I think any answer you get here is going to be speculative; we're unlikely to have enough information to fully troubleshoot this, but I suspect this is intentional rather than a compromise.

 

One possibility is that it's a (poorly conceived) phishing detector.

Given a link meeting certain requirements (for example: a link with unknown reputation plus from an unknown sender or perhaps the message was flagged as potential phish), a phishing detector may be employed to further research the link.

It would make sense that that link would be anonymized so it can't be easily tracked back to the recipient. This provides some privacy and also prevents accidental actions like "subscribe". It may additionally seed a spam honeypot.

I find it irresponsible that it's using third party mail providers for these fake addresses, as it could consume those third parties' resources by sending spam or transactional emails.

It's also inappropriate to bombard your server so much. A responsible implementation of a security vetting crawler would only hit your landing page once, or maybe once per geography they're protecting (to ensure attack sites don't customize based on geo-location).

 

Perhaps it's retaliation against spammers and aggressive marketers.

This would be somewhat similar to how Blue Frog operated 15+ years ago. Blue Frog's methodology was to report spam (like SpamCop does) and when that fails, to send instructions to its end user clients, which would result in hitting spammers' web servers with automated complaints (one per reported spam). These complaints mentioned the process and how to stop it.

One of the world's biggest spammers at the time even wrote "Blue [Frog] found the right solution to stop spam, and I can't let this continue."

This could be a similar (albeit cruder and less transparent) mechanism, giving a small bit of load to your server in response to your email campaign. Spam works because it's cheap; send out a million emails, get a few hundred visits to your website, sell a few dozen items, run a profit. This wouldn't pan out if your response rate was 100% because your servers would fall over, harming your profits unless you beef up your infrastructure, which also harms your profits.

It's missing all of Blue Frog's procedure though: no complaint (right?), no attempts to contact your network provider (right?), no explanation in the form submissions on your site with instructions on how to stop the attack. Perhaps the lack of contact is designed to avoid Blue Frog's fate? (They were DDoSed out of existence.)