First of all, I assume that you have already located the IP address of the attacker, using an IDS for example. In this case:
- Are there ways to determine that the attacker is using a VM at all?
One way I can think of, provided you have the IP, is to find the MAC of the attacker. Usually it will point to the manufacturer, which will be vmware or some other similar.
nmap can be used for this sort of fingerprinting.
- Are there ways to determine Which machine on the network is the host system?
This is a really interesting question - and I don't have an answer, and I am not sure one exists.
I suppose you can locate the network segment if you have a proper switching infrastructure, not sure if you can do anything more than that remotely.
Of course, if you are really determined, you can try to find the IP the opposite way:
Assuming you have a managed switch, try locking MAC addresses to ports, one by one. Eventually you will stop having communication with the guest, when you lock the mac address of its host to the switch port.
Also you should be able to see if more than one mac addresses are connected to one port, if using a decent switch (unfortunately I don't have a link available, but I've read about functionality like that).