4

Earlier today when attempting to ssh into my remote server, I got the standard man in the middle warning relating to inconsistent public keys. In fact the key-type wasn't even correct. My server uses ssh-rsa, and whatever was on the other end was using ecdsa-sha2-nistp256. I'm the only one with access to the server, and I know for certain that the keys were not changed during this time.

I resisted the urge the reset my client's known_hosts file, and waited a few hours. SSH started working again with the correct public keys and everything seems fine. However it's a little unsettling, so I'd like to understand what's going on here.

I've ruled out an eavesdropper on the client subnet. I tried SSH clients from EC2 instances and over VPN, and got the same malicious public keys. I had ssh sessions that were previously open, that remained uninterrupted through the events. Self-SSH'ing to localhost and the subnet IP worked fined, but SSH'ing from the server to the public IP showed the malicious key.

The sever sits behind a common household router, with port 22 forwarding enabled. The server ISP is Comcast Business, so I don't know if this is could be related to the web-page injections that Comcast has been doing more frequently lately. Any good ideas on what the likely cause of this is?

  • I don't have experience dealing with MITM attacks, but if you are the only with access to that server I would strongly recommend you to change the port (22) to another one (be creative and don't set it to 222 or 2222). You won't see bots trying to login to your server anymore (if you see, then its a direct attack). – lepe Nov 17 '15 at 0:52
  • 1
    Were you addressing your server by address or by name? If by name, you may be suffering a DNS based attack. – JaimeCastells Nov 17 '15 at 1:56
  • @JaimeCastells But I was using IP addresses directly, so I can rule out DNS. God suggestion though. Thanks. – user79126 Nov 17 '15 at 8:39
  • @lepe Great suggestion. Seems like an easy change to make, that should definitely reduce the few thousand random bot login attempts I get a day. – user79126 Nov 17 '15 at 8:39
2

I have no idea about sources of attacks, so strictly speaking, this won't answer your question. However, I think there are some things you can do to figure out who or what may be engaged in this. It will require extreme care on your part. Also, it's been a few days, so first you need to see if it's still happening.

You should set up your server to determine who's doing this (by IP). Turn on logging on the server and your firewall was my first thought. See if someone is knocking on your front door and testing SSH credentials. Note the times in the logs. If it isn't you, examine the logs for IP addresses of the attacker(s).

You can do a lot with an IP address. Reverse lookup won't tell you everything, but it will tell you a lot. Is the address a cable modem, a US government address or an endpoint in China or Bulgaria? Perhaps, it terminates inside Comcast? Then, you can call up Comcast and ask them, WTF?

No matter what, DO NOT SCAN OR ATTACK THAT IP. You have no idea who or what you're dealing with. You're almost certainly ill equipped to handle whomever is on the other side of this thing if it's any kind of sophisticated attacker. You don't want the Eye of Sauron upon you.

However, you don't know if the attacker has reached your server, yet, or if the logs are showing some random scanning of your ssh port, although you should be able to figure this out. So, if you feel like continuing, jump to your client computer.

Look over your entire client and server set up, the credentials used to authenticate the server, the usr/pwd and other secrets used to authenticate the client. Keep a copy of all of this information, or just note it all and be prepared to change it later. Make a copy of your client's known_hosts file.

Test running an IP snooping program on your client, something like wireshark. Become familiar with how it works and make sure you can capture the ssh/tcp/ip session with the attacker/server. As it will be over SSH, you won't see the content. However, (just like NSA meta data collection) you will know the remote IP address. You might be able to know that without wire shark because the IP should appear in your client's ARP database (arp -a). However, wire shark will give a little bit more information (and it's cool to watch the packet exchange). Also, compare IPs from your server logs with IPs from your client. Are there matches? Are they related?

You've gone pretty far and collected a lot of data. You've logged at your server (and may have seen the MITM IP), you've passively set up an SSH session from the client, you can see the offending host (MITM), you have knowledge of the IP, perhaps that's far enough.

If you feel like digging deeper, you can. However, this one becomes a bit more dangerous for your home network. You need to be very confident about your ability to protect that network from the inside. Honestly, I'm not sure you should take this step, but here's how far you can go.

You could set up what amounts to a honeypot. Replace your machine on the backend with another old machine. I'm not sure if your backend needs to be live 24x7, so this may not be possible. If you can, just disconnect your backend network and leave only the honeypot in place. If you have a real router, isolate the honeypot so all incoming connections are to it. Harden it as best you can. Set up a dummy account with very different username/password and whatever other secrets you need. Set up some dummy information, maybe even load a web server on there and a mysql database. Something that looks vaguely real. Also, the best honeypot is actually running a VM, where you've virtualized the honeypot running on top of the real hardware. It can be harder to break out of a VM and do damage.

Turn on logging on the honeypot on everything. Log incoming connections, log shell session histories, log everything you can. Run wire shark on the real computer underneath the VM honeypot. Then, perform the whole shebang. Connect from the client via ssh (run wireshark), use your honeypot credentials and complete the SSH login. Are you on your honeypot or on some foreign machine? Then, go home and check your honeypot logs. Did someone connect in? Can you leave the pot around for a few days? It might take a little while.

This last bit doesn't get you a whole lot more information, but you will know two things if you see someone log in. (1) You are definitely dealing with an active attacker and (2) you have the IP of the machine used to engage in the attack.

At that point, I'd hand things off to the authorities. It's a tough call who to go to. If you get this far and can't find anyone who will listen to you, PM me and I might be able to help get you to someone (casual acquaintance type of thing).

Lastly, throw away the honeypot. Smash it, destroy it, never use it again. That thing cannot be trusted, they might have dug into its BIOS. Never again plug it into a network you care about. Change every single password you have on everything. Modem, router, machines, accounts. Consider re-imaging and/or re-installing firmware on any device used in the honeypot scenario. Get that firmware/image before you start the honeypot project and keep it well secured. In fact, use an old modem for the honeypot project and just disconnect everything else and smash that modem, too, when you're done. If I were you, I would be completely paranoid when running the honeypot scenario.

I hope this helps.

  • 1
    Woah, Andrew. I'm floored by how thorough, detailed and well-written this answer is. I wish I could upvote you x100. I'm very tempted to try the honey-pot scenario if/when this issue pops again. (It's been a few days so far, and no attempts, so I might not get the chance). You should know that someone is very thankful, and in a much better position security position, because you decided to share your expertise. – user79126 Nov 20 '15 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.