I have an encrypted file and the program.exe through which it was encrypted. When I want decrypt file, I need to know the password with which the file was encrypted. Can you help me some tool for brute-force attack to this password, which will be usable for .exe file? When I paste wrong password, I get a popup window displaying message: wrong password.
The oldy but goldy one is John the Ripper which is open source and can be extended to your "program.exe". Needless to say, you need to know how the executable was encrypted. You cannot use John, or even your own extended version, to decrypt container formats for which the format and choice of algorithms isn't known.
Kerckhoff's principle states that the security of a crypto-system should not depend on static information such as the algorithm. Unfortunately that does not mean that it is necessarily easy to decrypt if the protocol or algorithm is not known or kept secret.
You may be able to identify the format of the file by using the Unix
file command, other tools or even a text or hex editor. Looking at the header of the file does indeed make most sense initially, as safesploit muses.
Without more information about how the file was encrypted, I cannot give specifics. Forgot password to 7-Zip archive, may give insight.
Beyond this, I would suggest extracting the file header information. This should contain the hash of the password. Then you could identify the encryption algorithm and rounds used by
program.exe and use the same KDF to brute force the hash to identify the password.