VST Plugins are DLL files available to be put into a folder in the Music Tool which then can use this as a plugin.
Does VST have a mechanism not to execute malicious code inside the dll file?
Can I safely install them without anti-virus?
A VST is just a DLL which exposes a specific interface. As such, it can contain arbitrary code, and can do anything that an EXE could do. Treat it as such. If you don't trust it, don't run it.
Keep in mind that virus scanning will only protect you against known signatures, so something unknown or custom might still catch you out. Running the plugin through VirusTotal or a similar service may help, but ultimately you can never be sure and there is always a risk.
My advice would be to research the plugin and the person/company who wrote it, and the site you downloaded it from if it's coming from somewhere generic. Verify their trustworthiness by reading reviews, looking at their other work, and seeing which communities they're active in and trusted by.