Online Shop got hacked, fixed but is still under attack - what to do?

Question

an xt-commerce-based online shop from a friend of mine got hacked recently. The attacker mailed that the shop is vulnerable, he showed the extracted data (basically he dumped the databases and cracked the md5 hashes, which were not salted) and asked for some bitcoins.

My friend engaged a security company, which performed a (rather bad) webapp scan using accunetix web scanner. The output was almost unreadable as it contained almost 98% false positives (spread over some 40-50 pdf pages). So he asked me to perform a second scan and I found 4-5 SQL injections and some XSS, an outdated version of Apache as well as PHP and MySQL. The SQL injections were patched and I verified the patches, as I couldnt perform any SQLi anymore.

Now, the users of the shop were informed about the breach and were asked to change their password. However, some hours ago, the hacker managed to place a fake order in xt-commerce.

I wonder what the owners of the shop could do now?

1. The did not enable MySQL logging as this would fill memory quickly.
2. I asked them to inspect the Apache logs to see what happened during the point of time were the fake order was placed.
3. I asked them to look for PHP/Web-Shells, which might have been placed on the webserver.
4. I wonder if it is possible to somehow scan the contents of the database for abnormalities?
5. Of course there is the possibility of still having security holes in the web app.
6. If the attacker chose to exploit the webserver, php-daemon or the mysql server, we would have a hard time to trace this back, right? Should we thus turn on filesystem logging?

Could you suggest me what further steps to take or maybe just tell me your gut feeling about this situation?

Answer 1

Generally speaking you should take close to the following steps:

Secure your site and regain access

• Lock down the site and prevent all external access. Try to avoid "contamination" in the process by modifying potentially compromised files

• If you need to retain an on-line presence then set up something like a separate landing page hosted on an isolated server/account. However, you can't leave your site up and running when you know it's vulnerable, especially if it's an eCommerce site. You'd be opening yourself up to substantial legal liability for potential losses if you did so.

• Change all your passwords and work with service providers if you've lost access to any of your accounts.

Identifying back doors

• Download a full copy of your site.

• Use a Diff tool such as WinMerge to compare the downloaded files to your most recent backup and a fresh copy of your cart software (in the version you're running).

• Inspect every variance you find, record any variances you find before reverting them. Keeping a record allows you to search for reoccurring instances in the future.

• Try and look for any malicious data in your database. There's no easy way to do this in particular look for new users and search for things like "exec" (the PHP function for executing code from a string) in things like templates which might be stored in the database. Use what you find as clues for what to search for. Comparing against backups can also be handy, especially for tables which don't change much.

• If you're able to just throw out your code base and start with a fresh file set (ie. from a freshly downloaded package from your cart software) then strongly consider this.

Resolve vulnerabilities

• Make sure you're running the absolute latest version of whatever cart software you're using. If your cart isn't maintained any more then consider switching.
• Vet any other sources of code such as 3rd party plugins. Remove any which aren't absolutely necessary, aren't actively maintained or don't have a strong reputation.
• If you have any clues from things such as logs then investigate them.

Restoring site and prevention

• Once you're confident that the back doors have been removed and vulnerabilities resolved then you can restore access.

The did not enable MySQL logging as this would fill memory quickly. I asked them to inspect the Apache logs to see what happened during the point of time were the fake order was placed.

Strongly consider external logging. Options such as S3 and Glacier from AWS are ridiculously cheap and can ensure your logs aren't compromised in a future attack.

• Consider implementing a Web Application Firewall, such as CloudFlare. Do not rely on this to resolve your current vulnerabilities, it's only a first line of defence.

• Improve isolation where possible, for example if your site consists of a cart and a blog ensure they're both hosted as separate users/accounts or ideally separate servers.

• Ensure your software is kept up to date, you have strong passwords, etc.

• Consider implementing routine file integrity checks. Some software has this built in where basically you can automatically compare the checksums of files against the clean file checksums so you can be notified if a file is modified unexpectedly.

• Review your file permissions, ensure the web process can't write to any of your code files (and can write to an absolute minimum of other files in general).

Answer 2

I would do a few more things to clean up the site:

1. Change every password

   Yes, before you do anything, change all admin passwords, ftp, ssh, MySQL and so.

2. Move all website's files to an inaccessible folder

   Take the site offline and put a Under Maintenance sign to give you time to fix everything. This will disable any changed file from the attacker. Shells will be unusable too.

3. Reinstall xt-commerce from scratch

   Patching the current xt-commerce is not enough. There are plenty of places to put a backdoor, and it's easy to bypass one even with very careful examination. If there's custom themes or plugins installed, reinstall from scratch. Try don't copy any of the resources from the old site, treat them as infected. Don't try to fix any infection, create from scratch.

4. Patch the new xt-commerce to use a better password storage

   If the default password storage from xt-commerce does not implement salting, something is very wrong. Salting is required for every password management since the launch of Intel Pentium MMX. Or even earlier.

5. Take a full database dump and inspect it for signs of abnormalities

   Any sign of eval, replace, decode, exec or system will be suspicious. Take a close look on those entries.

6. See if there's any strange account created

   With the database dump in hand, see if any account have more privileges than it should have. Maybe the attacker created a disguised admin account.

7. Inspect the log files

   This will show you how the attacker gained access to the site. As some PHP shells can use POST, custom headers and cookies to send and receive data, the default Apache log configuration can not show everything. Add the %{Set-Cookie}o directive to the configuration to log cookies, and log POST data too.

Answer 3

My gut is that you don't know how long ago you've been hacked.

By patching the site and combing for known vulnerabilities, you're basically hoping the hacker doesn't know what he's doing.

The first thing the hacker probably did was to delete the logs and install multiple backdoors. Even if you properly patch the system, even if you scrub every vulnerability, there could still be outbound backdoors, extra administrative accounts, etc. These can be kernel-level and not detectable without shutting down and scanning the media by mounting the volume externally on a known-good machine.

If you can find out when the breach occurred, you can restore from backups older than that, patch the systems and possibly restore more recent data.

If you can't figure out when the breach occurred, take the company down. Rebuild a second website, bring in the old data, and comb the user lists carefully. If you get hacked again, you can always repeat this process, activate more logging, dig deeper into your data etc, it's fast to rebuild a second or third time.

It's very important to remember: Bring no code or binaries over from the old site. You're rebuilding from scratch.

If that's not financially possible, and you don't have known secure copies of your code, then you were coasting on good luck for a while. You might have to pay the extortion or just keep hammering at the site and hope that you outsmart the hacker. Be sure to get a complete backup. And hope they don't have code set to cause damage on shutdown/startup.

Answer 4

"However, some hours ago, the hacker managed to place a fake order in xt-commerce."

Placing a fake order does not amount to hacking. Seems to me thats a normal usage of the site. If they are logging in as an admin, what account are they using? Change the password. If this is because of access to the admin account, then do not discount the fact that attack may have happened because of a keylogger software. Thus apart from the site infrastructure you may want to look for any malicious software that the user may have installed. Thus you may need to change those passwords immediately and stopping using that particular computer until it has been formatted or the issue has been resolved or the malicious software has been removed.

Answer 5

You can scan the site to find if the hacker had put some javascript using the admin account. If he did it he can keep stealing cookies from site users!

You should also change all passwords including MySQL password and FTP password.