Automatic robot crawler to check web pages authenticity?

Question

Our servers seem to be commonly hacked, mostly injecting malicious codes/frames to our web pages or changing downloadable files with backdoors and so on.

Is there any tool that can crawl our website pages, codes, files etc and make a hash database and if anything changed inform me by email instantly so I can fix it as fast as possible?

Checking everything manually is difficult.

Answer 1

If you had static sites, it might make sense to check the files on the frontend, but if you have a dynamic site like a CMS, it will be very hard to detect changes via crawling, because by definition the pages should be changing their content frequently.

On the backend, you can check that your code has not been hacked or modified by using file integrity monitoring software, like tripwire. This would let you know if someone changed the source files, added new files, etc. However, this will not help you detect reflected XSS that is in the database, etc. You could write some scripts to check database integrity.

However, if you have vulnerabilities that allow getting root privileges they may be able to then turn off the file integrity monitoring. If you run your webserver in a chroot/jail, this could reduce that likelihood.

Where possible, you might also want to look into migrating your information-only content to a static site generator like jekyll or pelican. This would help you reduce your footprint.

Answer 2

The GNU wget utility has options for recursive crawling and download; it is quite convenient to automatically retrieve a full site. You could use that to automate the comparison of your current site status with what you expect. However, note the following:

• This will give you only the result. For instance, any PHP code is evaluated server side, so you don't see it when downloading, only what the code currently chose to send you. A cunning attacker could try to detect your crawler and serve it the "normal" site, while sending the defaced pages to all others.

• This assumes that you have a "normal site image" to compare with. This is easy for sites with only static contents, but very difficult for sites with dynamic contents.

• This is like paying policemen to follow criminals and dutifully write down each crime in big ledgers. Masterful accounting, high crime rate. Crime detection is good, crime prevention is better.