There has been some work done that I've heard of like anti-sniff, which looks to detect machines in promiscuous mode using timing information.
The idea being that machines in promiscuous mode will have to process all packets that they see so if there are large amounts of traffic that need processed the system will be busy and slower to respond to directed traffic.
This sort of approach, if it's still practical, wouldn't work in every scenario. For example if a host doesn't have an IP address it can still potentially sniff traffic and it wouldn't be possible to detect it using this approach.
However it's one possible approach that could be explored.