Recently I downloaded Open Shell application and checked it via VirusTotal. Most famous antiviruses like Bitdefender, Nrothon, Kaspersky found nothing. But some of antiviruses thing there are some bad stuff in it.
A lot of security and administration tools get picked up as something like "Potentially unwanted program" it may be that Open Shell has been detected as being used in conjunction with attacks (which is perfectly plausible)
The 'W32.AIDetectVM.malware2' flag could plausibly mean 'Virtual Machine' i.e. 'we have detected that this application allows for arbitrary code to be executed, which could be used to circumvent malware protection' and flagged as something you need to pay attention to. If you hadn't installed it intentionally finding a shell on your system you didn't put there is not a good sign!
Whilst it may be a false positive, it is possible that you may have a version which has been back-doored and it genuinely is dangerous in a way that only these vendors have detected; whether you trust them is a matter of risk assessment.