This question had already been brought up here, but was poorly answered with generic answers, like
reinstall everything and no actual solution and cause.
So here's my story: There is a new virus out there which creates a process called "WATCHBOG" which eats all CPU from the server. After further investigation, I found out that this virus is mining cryptocurrency.
Server specs: OS: Ubuntu server 16.04.5 LTS Software: Apache2 with virtualhosts, MYSQL, PHP5.6, PHP7
When I found out that this web-server is infected, web pages still worked but awfully slow. The first thing I see is the watchbog process which eats all CPU, so I tried to kill it, but it reappears instantly, and everything is so slow that it is almost impossible to operate the server. Next what I thought- "ok I need to terminate this process" So I thought I will put "* * * * * killall watchbog" in crontab , so I opened crontab, and found out that its compromised as well. I removed this new entry from crontab, i deleted the watchbog file, and after a minute or two everything magically reappeared. Crontab got compromised again to run some kind of remote script and the watchbog process was up and eating cpu again.
I tried to find anything useful on the web, and found these articles:
Here's is the most helpful one: https://sudhakarbellamkonda.blogspot...50061219193777
After following this article the watchbog virus still reappeared so I came up with this solution:
Open a screen session as root and then run this loop: ( while true ; do killall watchbog ; done )
and leave it running in background by detaching screen session with CTRL+A+D.
I posted as well this solution in that blog.
and here is one other post, but nothing really is helpful there
So I tried fighting this virus many ways, changed passwords, reinstalled SSH, etc, with no luck. Meanwhile I created a new Ubuntu server with the latest 18.04.1 LTS. We installed all latest webserver stuff, enabled UFW, opened web and ftp ports, then migrated WWW data and SQL, changed IP back to original servers IP, and..... there it is AGAIN!
The virus came back on the new machine
I think probably this virus infects the system by using some vulnerability in web-server software, So we found out that WGET and CURL is responsible for distributing the virus around the system, now we are trying to understand how it got there.
If you have any tips, please help me to find the vulnerability.
P.S. This is my first question, judge me softly please :)