57

Backstory
My sites and VPS were stolen from me. The hosting company and I were locked out and unable to access it. They weren't able to create a temp password for access because the attacker blocked it. The last time I was logged into WHM, root control was taken and all HDDs were no longer bootable. I believe the same person used a worm to monitor my desktop remotely. 5 pcs and 5 mobile devices, bricking my ddwrt r7000 router with PIA VPN killswitched. A dozen or so mint/ubuntu vms were taken over. Many usb drives were made to be write-only.It was relentless. I stopped trying to figure out what was going on and reformatted all devices.

I am now waiting on the server image and a memory snapshot, as well as an rsync copy. Upon transaction ill get a fresh server image...certainly of a different IP, unless they aren't to be convinced.

Here is the email I received today:

Blockquote This ticket was just assigned to me. I have made a backup of the account >out of the way of the backup processes here.

[2018-03-25] pkgacct completed

I was reading over some of the conversations and would like to just make sure we are on the same page.

We can not dd ( bit for bit copy ) the current state because the account does not have a storage medium that can handle it. If you wanted to add keys that we can rsync over ssh to a remote destination just give us the destination of the output file and we will be happy to help with that.

We normally do not keep hacked operating systems around unless there is some specific interest. What I find interesting is the openVPN software:

uperior.hosting.com [home]# ifconfig as0t0 Link encap:Ethernet HWaddr inet6 addr: Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:436367538 errors:0 dropped:504 overruns:0 carrier:0 collisions:0 txqueuelen:200 RX bytes:0 (0.0 b) TX bytes:26310498062 (24.5 GiB)

asbr0 Link encap:Ethernet HWaddr inet addr: Bcast: Mask: inet6 addr: Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:431222324 errors:0 dropped:0 overruns:0 frame:0 TX packets:1492069 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:20595591150 (19.1 GiB) TX bytes:634373123 (604.9 MiB)

If you are using this server as a production WebServer I would recommend using CentOS 7 and installing something like 'Cockpit' instead of using a VPN through a cPanel server on CentOS 6.

What is in the scope of our support is to insure you have a path to and from the server while you salvage the barebones data. I will conclude with some options and questions. (The following are my answers)

  1. An inquiry about the openvpn activitu

  2. Letting them know, again, I want a copy of the server image, a memory snapshot, and all logs available

  3. Fresh installation of cPanel/WHM

  4. Fresh IPs for sites and VPS

  5. SFTP info

Blockquote

Is having a VPN, IDS, firewall, honeypot enough? Have I left anything out?

VPS is a from Bluehost, running CentOS 7 with any alternative to WordPress...

closed as too broad by Steffen Ullrich, Anders, Xiong Chiamiov, Xander, Rory Alsop Mar 7 '18 at 22:17

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 9
    Long question but essentially you ask to if it would be possible to create logs which were not written in the first place since they were disabled. Unless you have a time machine to enable the logs before the attack occurred this is not possible because it would mean to create reliable information out of nothing. And then the only other question is to ask for suggestions so that it will not happen again: figure out how the attack happened in the first place and make sure that this way (and all the others) is closed. There is not much more to say since your exact setup is essentially unknown. – Steffen Ullrich Mar 4 '18 at 18:42
  • 18
    First, you should obtain a copy of your compromised server, e.g. as a VM or disk image. Then identify the attack vector. Once you have it, fix the vulnerability, then re-image your server a.k.a « nuke it from orbit » as advised by your provider, and redeploy your website from source. Audit the data backup before restoring it. While inconvenient, an instance of a hack should not be a fatality if you plug the holes appropriately. Get the necessary expert help if possible. – korrigan Mar 4 '18 at 18:48
  • 3
    As far as bringing the culprit to justice though, it’s a job for digital forensics experts, and even then there’s no guarantee of a result. I’d be more concerned to be back in business as a first order or action. – korrigan Mar 4 '18 at 18:51
  • 10
    WordPress on Bluehost... Well that's why you got hacked. Get a better hosting provider. And if you aren't proactively managing updating WordPress core, plugins, and themes, get a service who will do that. – Kallmanation Mar 5 '18 at 14:37
  • 5
    @ChrisNevill The English language, being a Germanic language, necessarily has ambiguities when using pronouns, and the meaning of pronouns is contextual. "He" is the de rigueur in modern writing. It is often used instead of "they" which can cause more semantic confusion due to it being (ab)used as singular. The most commonly used pronoun is thus the sex-indefinite semantical singular "he". – forest Mar 7 '18 at 14:19
3

Most hosting providers make available a small OS image (the purpose of which is to facilitate initial OS install when a client wants some custom OS, or to fix errors preventing the OS from booting; typically called “rescue system” or something) which you can boot and gain access to the compromised VPS without ever launching any malware the attackers could have put there. You just mount your volume(s), copy all the necessary data and examine it at your leisure.

If you can’t do this, your only concern is not to grant the attackers any further attack vectors. Don’t SSH to other machines from the compromised one, make sure you aren’t using SSH agent forwarding, X11 forwarding and the like. Just tar the interesting files and then scp the tar file from your local machine. Or do something like ssh -C root@compromised-host cat /dev/the-root-device >compromised-root-dev.img if you know what to do with the image then.

Or maybe just ask the hosting provider to make a filesystem image and send it to you in some way (which would amount to the same procedure I started with).

  • 1
    Well, if you convince the hosting provider to get a filesystem image of your VPS for you, to get it they would either boot their rescue OS image (which maybe you can do yourself) or log in to the machine containing the VPS and just read the data (which you surely can’t), depending on virtualization technology. – Roman Odaisky Mar 6 '18 at 15:56
  • Bluehost severely dropped the ball in providing this image for me. 5 phone calls all to whatever technical specialist level 3s and no image ever sent. Never again BH. I do wish I could have tried your method as it seems to be of greatest potential value and leaving room for proper hardening, etc. – Preston Bennett Mar 2 at 3:27
90

I'll start with what to do with your current system:

  1. Get in and make a backup of everything.
  2. Unless you can demonstrate major losses ($10k+), I wouldn't even begin to think about involving law enforcement. They have their hands full, and given the current patterns on the internet, it's highly likely that your culprit is in a different country than you are. Nobody is going to do an extradition process for hacked Wordpress sites. (Sorry, I know it's hard to hear, but it's the reality.)
  3. Burn the current server to the ground.
  4. Consider every password on your old server compromised.

Now, how do you build a new server to avoid this happening again? I'm going to make a few assumptions based on what you wrote in your post:

  • Multiple Wordpress installs.
  • mod_php on Apache
  • CPanel/WHM

Here's a few recommendations:

  1. Get your new server.
  2. Do a fresh installation of your applications, and setup strong passwords/credentials that have nothing to do with your old ones.
  3. Configure SELinux to limit the exposure of each site as much as possible.
  4. Be careful what wordpress plugins you install. They have a much worse security track record than wordpress core.
  5. Ensure that directories that are writable by the webserver are never interpreting files as PHP.
  6. Use 2-factor authentication for everything you can.

Without being able to do digital forensics on your system, it's hard to know for sure, but I'm going to go out on a limb and guess what happened:

  1. Attacker runs scanner looking for vulnerable Wordpress installs.
  2. Attacker finds vulnerable Wordpress on your server. Gets RCE as webserver.
  3. Dumps password hashes for wordpress databases.
  4. Cracks password hashes, one of them matches a WHM/CPanel password.
  5. Gets into CPanel. Maybe as an admin, or maybe the version of CPanel had a bug that allows privilege escalation to admin.

You talk about fear of retaliation and the attacker coming after you again and again. Unless this is personal (a vendetta of some sort), I wouldn't worry about that. Attackers like this will just move on to their next compromised host. Just don't give them another chance.

  • 25
    I'd recommend making a FULL backup of the hacked system (eg via rsync -va host:/ /copy_of_hacked_system --exclude /sys --exclude /proc --numeric-ids), and doing a detailed diff against the new system, if it is set up on the same OS version etc. - this could already give you a clue about what had been surreptitiously modified. – rackandboneman Mar 5 '18 at 1:19
  • 21
    @rackandboneman If the system has not been shut down, I'd also recommend doing a full memory snapshot of the system for later analysis. It can turn up a lot more than what you'll find on the disk. – forest Mar 5 '18 at 2:33
  • 6
    Concrete advice on passwords: When we say strong, we mean long and random. Save them in a (potentially shared) password database, but these should mostly be service authenticators and only need to live in a file. – Riking Mar 5 '18 at 20:00
  • 9
    Additionally to the above, please be sure to be transparent and caution your users that their account could have been compromised. It's not just you that was hacked, all your users now probably have their email address along with whatever personal information your site stored in a database the attacker will use in the future. Everyone must change their passwords, and not just on your site... Don't be that corporate guy who tries to hide the fact that you were compromised for months or years. – Drunken Code Monkey Mar 6 '18 at 1:19
  • Depends if the server image you get is in a format you can mount and/or boot with a rescue system medium attached. – rackandboneman Mar 6 '18 at 8:53
21

David's answer gave some excellent recommendations (which I highly recommend you follow). I will focus my answer instead on your specific fears in hope to alleviate them.

What I wish to accomplish is to acquire as much information about the hacker as possible. I want to download my legacy backups. I want to get in and out as soon as possible.

You say you want to get as much information about this person as possible. This may not be practical if they used any form of anonymity. Post-break in analysis is still important, of course. The first step would be to do a complete backup of the disk. If you have not already shut the system down since it was compromised, you may also be able to take a snapshot of the memory on the system, making it possible to do later forensic analysis on it. Memory will contain far more information about the attacker as they cannot reliably control what stays in memory, whereas it is very easy to prevent valuable information from being saved to persistent storage. Because of this, it's likely that you will not be able to obtain the disabled logs unless they already existed and were subsequently deleted.

My precautions are to stay behind a killswitched vpn and spend as little time there as possible.

There are a number of methods hackers can use to get the original IP behind a VPN, due to the architecture of VPNs. Using Tor or a similar anonymity network would be preferable. Spending as little time as possible may not be advisable as you may miss something important. Staying on a little bit longer does not increase the chance that you are detected. The hacker may even think you are another hacker who has broke in using the same vulnerability. If they notice you, chances are regardless of who they think you are, they'll simply disconnect you from the server.

The server-side SSH logs among others likely contain your home address anyway.

Is there any way to somehow recover these disabled logs? What else should I be looking at if the goal is to compile enough data to turn this guy into the authorities?

If the logs were disabled, you likely cannot retrieve them. If they were still written but merely deleted, you may be able to recover them from unallocated filesystem space. Unless the system has not shut down since the compromise and you are able to take a full snapshot of the system's memory, it is unlikely that you will be able to retrieve the information that was given in the logs.

As David already mentioned, you will not likely get much from involving law enforcement. The best you can get from obtaining this information is knowing more about how the hacker broke in, making it possible for you to close the hole they exploited and avoid it in the future.

My fear is that, even with a re-imaged server and all the hardening I'm capable of, this guy knows my site domains and will likely attack again. I suppose this should be a separate question, but should I prepare myself for having to altogether give up my domains due to his skill level? Id hate to do it, but I fear he will do all he can to wreck the server and sites again, then proceed as before to destroying my home network and all devices.

Honestly, you don't have to worry about retaliation. You are just random collateral to their activities, not a carefully selected victim. They likely won't even remember your domain in a week. Of course these are generalizations, and if you have an actual enemy with a personal vendetta against you, things are a little different. There are a few possible classifications that may fit your attacker:

  • Curious - A hacker who breaks into a site because they are curious or want to improve their hacking skills will do so often with little regard for the results of their actions, but they are not out to get you. If you lock them out, they may want to find a way to get back in, but they will likely not be angry. They could even see it as a challenge.

  • Criminal - A purely criminal hacker has something to gain by attacking servers, such as money or digital resources. They won't waste time with revenge. These attackers are like water. They seek the path of least resistance.

  • Automated - It is very possible that no person attacked your site. A criminal enterprise wants to maximize profits, so to be more efficient, they often automate attacks. A script may scan the internet for vulnerable services and attack them. After breaking in, it will do their dirty work and attempt to establish persistence. You cannot make a script angry.

  • Script kiddie - A script kiddie is a junior hacker who thinks they know everything. They are more likely to seek revenge because they may take the act of you recovering your server as personal. They do not have the necessary skills to do any real harm to your home network. The worst they can do is attempt to DDoS you. Don't worry about them.

  • Personal enemy - If you have an actual enemy with a personal vendetta against you, you have more to worry about. Since you gave no indication that this is the case, I'll assume it isn't.

The chances are his skill levels are not as high as you think. Breaking into a Wordpress site and adding a backdoor does not take a professional. As such, I would not worry about getting rid of your domains. It is absolutely possible to protect from this hacker using the advice given in another answer. As soon as someone else's site is more vulnerable than yours, a hacker moves on.

  • 6
    I wouldn't call script kiddies "hackers" because that's like calling somebody who can install a light bulb an electrician. Most of the time they're just running scripts and manually feeding the outputs to other scripts, perhaps with some manual FTP use thrown in. If you keep up with your security patches and don't do something too stupid, script kiddies are rarely a problem. – wizzwizz4 Mar 5 '18 at 18:31
  • 7
    It's wrong to associate a "script kiddie" attitude with a "script kiddie" skill level. There's no guarantee that someone with more skill would have a more mature attitude. – jpaugh Mar 5 '18 at 20:35
  • 2
    @wizzwizz4 Note that sometimes, local laws and/or property regulations stipulate that only licensed electricians (sometimes even only those belonging to the trade union) are allowed to install light bulbs. This was the case in parts of Australia until 1998, and might still be the case in certain environments, like trade shows, hospitals and certain shared living spaces. – Nzall Mar 6 '18 at 10:14
  • 2
    @LateralTerminal In general, a VPN puts you in a pool with a number of other people in the same NAT (since that's how VPNs work). These other people can then often see you through a local address allowing them to send netbios probes or sometimes even connect to services on your computer. PIA in particular is vulnerable to this due not isolating individual users from each other. You can also use lsrr to get someone's real address due to the hardware used by many commercial VPNs. Additionally there is always the issue of TCP/IP stack fingerprinting that works through VPNs. – forest Mar 7 '18 at 1:57
  • 3
    @PrestonBennett Someone attacking so many devices implies a very dedicated attacker who has a vendetta against you. BTW, Kali itself is very insecure. Do not use it if someone is trying to attack you. – forest Mar 7 '18 at 2:00
9

In this case you don't know how the attacker was able to enter, so you are right to be concerned that the exact same thing will happen again even if you do a complete reinstall. To deal with this problem, you should configure your system for remote logging. A dedicated separate computer saves system logs which cannot be tampered with. You should configure your web server access log to send a real time copy over to the logging server as well as anything security related, in addition to the regular system log. The logs should be separate so that if one gets too big and needs to be trimmed down that it won't affect the others.

You don't know how they got in. Your own computer could be compromised and the password could have been intercepted. Someone at the VPS provider could have accessed it, or someone renting another VPS along side yours could have exploited something. Assuming that those things didn't happen, the next step may be to ensure that everything vulnerable is updated on the new install, and then configure security software such as SELinux. Be sure SELinux policy violations are sent to the remote logging server.

  • Thanks for your input. Remote logging will most definitely be implemented. – Preston Bennett Mar 6 '18 at 9:16
7

My precautions are to stay behind a killswitched vpn and spend as little time there as possible.

That is infosec voodoo. Assuming you're behind a router with PAT, the attacker is going to have no way to connect back to you. In fact, the VPN creates a tunnel through the PAT layer, so you're actually increasing your exposure by using it! (Althought the VPN policy may prevent inbound connections.)

The much greater risk is that if the attacker still has access to the compromised system, then typing in your password is dangerous, i.e. SSH interactive login or sudo password could be stolen by a root user on the compromised system. If you have public key authentication configured, then that would be safe to use as long as you don't use sudo, however see my note below about backups...

The best practice is to remove the hard drive from the compromised machine and forensically analyze it on a separate machine (a professional would use a forensic write blocker) so that the attacker has no ability to interfere.

As a VPS customer, you have no ability to do this yourself. Unless your provider is willing to do it for you (unlikely since it would disrupt other tenants on the same hardware), you are better off deleting the VPS and starting from scratch.

Hopefully I can get what I need from backups and [snip]

Backups should always be stored on a separate system. If your only backups are stored on the compromised service, then you have put yourself in a very bad position. If you have to type in a password to get to the backups, then the attacker can steal your password.

Even if you don't need to type in a password (e.g. public key authentication), you would be restoring data that has already been breached. If the attacker installed some backdoor previously, then that backdoor would be saved in the backups, and you would end up restoring that backdoor onto your new server! Or if the attacker used some other means to compromise the system, then the attacker could have inserted a backdoor into your backups in order to regain access after you re-image.

Is there any way to somehow recover these disabled logs?

Yes, it is possible to recover deleted log data using forensic analysis, but for the reasons stated above, this is unlikely to be an available option for you.

As an alternative, your provider may have logs about what IP addresses are connecting to their systems, and they may be willing to share that data with you if you ask nicely.

My fear is that, even with a re-imaged server and all the hardening I'm capable of, this guy knows my site domains and will likely attack again.

I'm not really sure what you mean by "domains". You can keep your existing hosting account and domain names, assuming that your password has been reset to something secure.

The most likely scenario is that you were compromised due to an unpatched vulnerability in WordPress or one of your plugins (or custom code if you've written any of it yourself). To prevent a future compromise, you need to start from scratch (don't use a backup of data) and make sure that all of your applications and plugins are the latest version. Disable any plugins that you don't absolutely need.

It sounds like you might be a bit over your head. You are managing complex software by yourself, and you don't know how the previous compromise occurred. Maybe you should move your site to a provider that offers better patching and security, such as hosting through WordPress.com, so that the responsibility isn't 100% on you.

  • Lots of helpful insight and suggestions. Your efforts are appreciated. As far as the domain goes, whomever did try to sell it and i caught it at the last moment...Any other alternatives to wordpress.com? I am now more capable of keeping a site safe now, though i will concede it took about a month for my EC3 instance to be taken down. This of course is likely due to a tampered backup, though I combed through all the obvious things. But no Im not that good. At least I got the content back so the site simply meant to help others will be back soon. Any other hosting suggestions? – Preston Bennett Mar 2 at 3:25
2

Is there any way to somehow recover these disabled logs?

Yes.

If the entire virtual machine is yours (and you aren't sharing it with someone else), get a backup of your entire virtual machine. Do that immediately. That will include whatever logs that your machine has.

Also ask Bluehost for whatever relevant logs they can provide. Do that immediately so that they don't lose the logs. (Many places will cycle out logs over time, so make sure that someone knows to preserve logs that are believed to be potentially interesting.)

Then focus on getting a safe machine operational. (As BlueHost mentions, this likely involves an OS reload.) Ensure that your data is safe. (e.g., if your data includes information about accounts, and includes an account that the attacker uses to gain access, that is not safe.) Have that safe machine use the safe data that you value. Do all that as quickly as possible.

What else should I be looking at if the goal is to compile enough data to turn this guy into the authorities?

Where is this attacker from? Who is this attacker?

As forest notes, "authorities" are unlikely to care much. Chances are that you may have been attacked by some other compromised machine. Therefore, the person who owns that machine is also a victim, and trying to legally attack that person will probably not benefit anybody very much. What you need to do is to get after the person who really initiated the attack. Unfortunately, with several methods of redirection being possible, many attackers are not feasibly traceable.

My fear is that, even with a re-imaged server and all the hardening I'm capable of, this guy knows my site domains and will likely attack again.

Yup. That's a concern. Until the person is stopped, there is such potential. Hopefully you are more successful at preventing the next attacks. Ongoing education and efforts should get you to that point, hopefully sooner rather than later. In the mean time, ensure you have good, accessible backups, and sleep well at night knowing that you can restore from backup.

I suppose this should be a separate question, but should I prepare myself for having to altogether give up my domains due to his skill level?

Nope. Don't give in.

Id hate to do it, but I fear he will do all he can to wreck the server and sites again,

That could happen if you don't secure the server more successfully. Try to spend time doing that. Also, make sure you have good backups. Don't just trust Bluehost. I'm not trying to say anything bad about Bluehost. I'm saying, don't just trust one backup solution, especially if trouble is expected.

then proceed as before to destroying my home network and all devices.

Oh, now that's just terrible. Why would damage to a remote virtual machine, run on a professional hosting site, affect your home network? There's some major security problems if that occurs.

Is this even worth it? He's capable of far worse than what he did and I am in fear of retaliation.

Get backups of your web site.
Get backups of important data on your home network.

If you do a good job of that, then even if thievery ends up being possible (a copy of confidential information may be stolen), you can still recover.

  • 2
    I think OP is worried that the attacker will try to destroy his home network in revenge, not his hosting company. – forest Mar 5 '18 at 7:10
  • 1
    Your answer gives the impression you are saying the « disabled logs » can be retrieved, which is incorrect. – korrigan Mar 5 '18 at 11:07
  • 1
    OP can get the logs that may contain information before they were disabled. – TOOGAM Mar 6 '18 at 2:29
1

Just get an image of the web server drive. Mounting it on a fresh virtual machine is unlikely to cause significant hazard. Code from the mounted drive should not run, you can peruse the data at your leisure.

It is most certainly not safe to connect to a compromised machine with a remote access protocol. While servers are more constantly available to be compromised, client software connecting to a malicious server can be exploited (if an exploit exists).

Attempting to restore your web content from the content on the mounted drive is probably ill advised. Modifications to code files can be quite hidden and difficult to find. Using PHP, they can easily provide a backdoor to your next VPS machine...

Stuff in the SQL database might be modified to contain code to attack visitors to your website.

If you really want to harden your site get rid of WordPress. Wonderful things are being done with static site generators (Gatsby, Jekyll, etc...)

  • Thanks so much. Will install the image on VM for later analysis. Can you clarify on your remote desktop caution? My plan is to retrieve the BH provided root pass, then proceed to WHM (or will that be a separate password?) to retrieve backups and save copies of his dbs. I will take your wordpress related suggestion under high advisement. Do you or anyone have a recommendation of the best generators (i know you gave 2) that can be use on a hosted or self-hosted site? Self-hosting seems like a different beast, leaving my network further at risk... – Preston Bennett Mar 6 '18 at 7:00
  • 1
    @PrestonBennett My understanding of WHM is that it will do nothing unless you boot the exploited machine (and have it connected to the internet, no less). This is a bad idea, don't do it. Virtual machines use disk image files that are conceptually equivalent to a physical HDD. Get this disk image and access it in an offline way, without ever running files or booting the operating system contained therein. – trognanders Mar 6 '18 at 9:35
  • 1
    @PrestonBennett Again, the value of your restored backups is highly suspect... you should probably try to use backups that were not stored in the virtual machine's disk during the exploited period. Bluehost's logs might sort of help you find some clues of the culprit... but there will not be anything in the vm disk image that will help you very much. Keep the disk image around unmodified... that can be evidence later if you convince anyone to investigate. – trognanders Mar 6 '18 at 9:38
  • 1
    @PrestonBennett Aren't you kind of tired of the self-hosted thing right now? Static site generators build plain HTML that you can host literally anywhere. Amazon S3 would probably be my pick. DO NOT self host static HTML files... it is all risk, no reward. Gatsby would probably be my pick for a new project. – trognanders Mar 6 '18 at 9:42
  • 1
    @PrestonBennett Have you looked at the static site generators? Can you accomplish your goals using one? Using IAAS style resources (VPS, EC2, etc...) to host WordPress means you are responsible for basically everything. The web control panel and customer service line make it seem safe and easy, but both feelings are misguided. Amazon S3 (among other things) can publicly host static files via HTTP, such as those generated by your static site generator. This is more of a PAAS style service, where Amazon is responsible from the CIA/AAA of the service. They are pretty good at it. – trognanders Apr 11 '18 at 10:12
0

As some information alluded to here - if you used these compromised credentials anywhere else you must change your passwords. Preferably, you should use a good tool like Keepass or Lastpass that allows you to protect any number of strong, generated, random passwords, and you only have to remember the one main passphrase.

  • keepass was exploited in my home system before i knew it was comprimised. Thanks for the suggestions. – Preston Bennett Mar 6 '18 at 6:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.