I’m using Linux based VPS hosting, and a firewall and mod_sec help to keep most hackers out. However, over the last couple of weeks I noticed several entries in mod_sec showing that an unknown domain was attacked. Does this mean that the VPS is compromised (I do not see any unknown DNS entries) and other domains are getting a free ride, or is it just pointing to a sloppy setup of the VPS server?
If mod_sec is showing that an unknown domain was attacked then its possible that you still have a default vhost configuration in place on the server.
The default vhost on most distros has no ServerName associated to it, so it will respond to all requests that do not match a ServerName directive in another virtual host. Its probably a good idea to disable the default vhost if you don't need it.
As packs mentioned in his comment, posting some log files would help with diagnosing it further.
Make sure that mod_security is blocking attacks instead of just logging them.
Mod Security is going to produce a lot of log entries. Most of these issues are nothing to worry about. Often times its a bot trying a handful of attacks against a large number of hosts. Chances are you aren't vulnerable.