I've been trying to figure out the same thing. So far I've found
Universal Extractor Universal Extractor 2.0 (updated fork) which can extract the files from many types of installers. It's basically a frontend for many different freeware/open-source utilities (such as innounp and 7-zip) that can extract files from installers and other archive file formats. This is good for very simple installers that just extract some files into a folder. For some installer types, you can even get the script files. For instance, with Inno installers, innounp is used:
It recovers portions of the installation script (.iss file), including the registry changes and the compiled Innerfuse/RemObjects Pascal Script, if available.
For installers such as NSIS (which a lot of ransomware has been using lately), there's some limited decompilation tools to get the original script:
There's also some limited static-analysis programs I've found, such as pestudio (closed source, don't have enough rep to post more than 2 links), which will give you some 'indicators' of various levels of suspicion about what things the executable does. For instance, "The file modifies the registry", "The file references the Clipboard", "The file ignores Address Space Layout Randomization", etc.
As mentioned in other answers, there's some sandboxing/virtualization solutions such as sandboxie (closed source). Though I have a hard time trusting it, since it's closed source and needs to install system drivers and whatnot. And the interface and access-permission settings aren't great (it's blacklisting only, rather than block by default and whitelist, so you can't feasibly limit the programs to accessing only the directories they need to).
I'll update this if I find anything else.