I'm trying to track down a cause on an Internal Server Failure for an Apache server. The IP address changed recently, so I thought the crash might be related. A quick search of the old IP:
$ sudo grep -IR --exclude-dir="/var/run" '45.78.157.165' /var | egrep -v '(audit|access)'
/var/log/secure:Jun 21 14:08:34 cryptopp sshd[19729]: error: Received disconnect from 199.91.135.157: 3: com.jcraft.jsch.JSchException: reject HostKey: 45.78.157.165 [preauth]
/var/log/secure-20160626:Jun 21 14:08:34 cryptopp sshd[19729]: error: Received disconnect from 199.91.135.157: 3: com.jcraft.jsch.JSchException: reject HostKey: 45.78.157.165 [preauth]
/var/log/secure-20160626:Jun 21 14:08:34 cryptopp sshd[19729]: error: Received disconnect from 199.91.135.157: 3: com.jcraft.jsch.JSchException: reject HostKey: 45.78.157.165 [preauth]
com.jcraft.jsch.JSchException: reject HostKey is unusual, so I wanted to gather more information. A quick search revealed com.jcraft is some sort of Java Secure Channel. Then:
$ whois 199.91.135.157
...
NetRange: 199.91.132.0 - 199.91.135.255
CIDR: 199.91.132.0/22
NetName: BCS-HQ-USA
NetHandle: NET-199-91-132-0-1
Parent: NET199 (NET-199-0-0-0-0)
NetType: Direct Assignment
OriginAS:
Organization: Bluecoat Systems, Inc. (BLUEC-8)
RegDate: 2011-02-04
Updated: 2012-03-02
Ref: https://whois.arin.net/rest/net/NET-199-91-132-0-1
OrgName: Bluecoat Systems, Inc.
OrgId: BLUEC-8
Address: 420 N Mary Ave.
City: Sunnyvale
StateProv: CA
PostalCode: 94085
Country: US
RegDate: 2007-01-23
Updated: 2013-01-28
Ref: https://whois.arin.net/rest/org/BLUEC-8
OrgAbuseHandle: LAMAS1-ARIN
OrgAbuseName: Lama, Soni
OrgAbusePhone: +1-408-220-2200
OrgAbuseEmail: [email protected]
OrgAbuseRef: https://whois.arin.net/rest/poc/LAMAS1-ARIN
OrgTechHandle: LAMAS1-ARIN
OrgTechName: Lama, Soni
OrgTechPhone: +1-408-220-2200
OrgTechEmail: [email protected]
OrgTechRef: https://whois.arin.net/rest/poc/LAMAS1-ARIN
I know what Bluecoat is, and I know what they do. I think the behavior is highly unusual, but I want to refrain from making leaps.
To provide a little more context, this is abasically a Linux LAMP server, with no additional software. The VM provided by VirtWire, and the system is virtualized CentOS 7.2. It runs an SSH server, and Apache, MediaWiki and MySQL. Only Admins have SSH access for administration. I patch it regularly, and it usually lags by 3 to 7 days, but not much more.
My question is, why is the server trying to callback to Bluecoat (or why is it receiving connections from them)? Are there any legitimate use cases I may have missed?
Zeroing in on the exception and IP block:
$ sudo egrep -IR --exclude-dir="run" --exclude-dir="spool" '(com.jcraft|199.91.135)' /var
/var/log/secure:Jun 21 14:08:34 cryptopp sshd[19729]: error: Received disconnect from 199.91.135.157: 3: com.jcraft.jsch.JSchException: reject HostKey: 45.78.157.165 [preauth]
/var/log/secure:Jul 1 20:58:28 cryptopp sshd[1235]: error: Received disconnect from 1.55.196.144: 3: com.jcraft.jsch.JSchException: Auth fail [preauth]
/var/log/secure-20160620:Jun 13 13:18:01 cryptopp sshd[23246]: error: Received disconnect from 103.207.36.185: 3: com.jcraft.jsch.JSchException: Auth fail [preauth]
/var/log/secure-20160620:Jun 13 21:03:09 cryptopp sshd[28964]: error: Received disconnect from 116.99.253.189: 3: com.jcraft.jsch.JSchException: Auth fail [preauth]
/var/log/secure-20160620:Jun 13 21:04:20 cryptopp sshd[28971]: error: Received disconnect from 116.99.253.189: 3: com.jcraft.jsch.JSchException: Auth fail [preauth]
/var/log/secure-20160612:Jun 8 21:09:59 cryptopp sshd[3712]: error: Received disconnect from 46.35.253.161: 3: com.jcraft.jsch.JSchException: Auth fail [preauth]
/var/log/secure-20160626:Jun 21 14:08:34 cryptopp sshd[19729]: error: Received disconnect from 199.91.135.157: 3: com.jcraft.jsch.JSchException: reject HostKey: 45.78.157.165 [preauth]
/var/log/secure-20160626:Jun 21 14:08:34 cryptopp sshd[19729]: error: Received disconnect from 199.91.135.157: 3: com.jcraft.jsch.JSchException: reject HostKey: 45.78.157.165 [preauth]
I believe this is a related question: What's the purpose? Strange login attemps “sshd[*] Received disconnect from **...: 11: Bye Bye [preauth]”.
jschis a Java implemention of SSH protocol(s) fromjcraft.comwho are a company that writes Java software. SSH disconnect reason 3 is keyexchange which is consistent with the text. It appears some machine at Bluecoat is setup so it connects to your address, perhaps due to bad DNS somewhere, but once it connects and sees your hostkey it realizes it doesn't want to connect to you and disconnects. If it only happened once, or rarely, I'd ignore it; if it's frequent enough to be an issue I'd try to complain to them.