I've just understood what you're actually asking. I'll leave the old answer below, just in case someone needs that.
The signtool application allows you to specify an option named 'verify'. You don't need to get access to the encrypted hash value, it will verify it for you. It probably isn't possible to do it manually if you prefer. Maybe when you use the verbose mode at the signing time it will show you the hash value, in case you really want to see it.
Signing only the binary sounds like a good idea if there wouldn't be any metadata files/ configuration/ anything that doesn't compile. I didn't use Windows for a while, but maybe you can use makecat instead of signtool, I think it should allow bigger app sizes. Hope this helps.
The old answer, explaining technical things:
It looks for me like you are trying to understand how digital signatures work and how you can implement the mechanism. If I'm wrong, please rephrase your question and try to explain step by step what you are doing, as for an algorithm pseudo-code. I will try to explain what I think you want to know.
When you sign a document: you can read the document (so it's not encrypted), you just want to make sure it's valid so you look for its signature and stamp. The signature and the stamp tell you if the document is valid/ verified. In the same way, when you use digital signatures, you always send 2 pieces of data: the executable in your case (not hashed, not encrypted, you can "read" it) plus the signature. So you have the executable, you don't have the signature.
To create the signature:
- hash the executable. The executable is usually large so it would take a lot of time to encrypt it. Luckily, if you use a good hashing algorithm, there is an infinitesimal probability that 2 files will give the same hash string. So you can be sure enough that the hash represents your executable (even if it's just a meaningless string) and use it instead of using the whole executable further in this process.
- generate a pair of private-public key. Your certificate will contain the public key and the private key must be kept secret. The nice thing here is that you can always encrypt with any of the public key or private key and decrypt with the other one. Never in a different way or with a different key. So if you know for sure that the public key belongs to someone (the certificate is trusted), then if they encrypt something with the private key, then you can decrypt it using the public one. If anyone else encrypted the data, because the private key is secret and they don't have access, whatever key they generated they use, you won't be able to decrypt it.
- encrypt the hash with the private key generated at step 2. The value obtained here is your signature.
Now that you have both the file and the signature, send them to the other end or use them however you want. Once the information is received on the other end, how do they verify that it's valid? They have the document, they only want to be sure it wasn't changed by someone else. They also have the signature. So they follow the following steps:
- decrypt the signature value using the public key (the certificate). If they cannot decrypt the value, then someone forged the signature (they changed that value with another one). If they can decrypt it, then they know the signature is the valid one. After decrypting, they obtain the hash value of the executable (step 1 above, when creating the signature). They know this hash value identifies the valid executable. This is the source of truth.
- hash the received document (the executable) with the same algorithm as used when creating the signatures. When using a good hashing algorithm, no matter how many times you will hash the same document, the obtained hash value will always be the same. So if the document was not changed, the hash value will be identical with the one obtained at step 1. If they are not equal, it means that now when you hashed the received document, the hash value was different so the document was modified/ replaced by someone else. So you cannot trust it.
- verify if the 2 hashes are equal (if they're not, as explained at step 2, the document is not the true one, it was modified by someone else). If they are equal, you can trust both the document and the signature.
This steps should also answer you how you will have the hash values being equal and how will receive the hash value from the signature value.
One last thing, please use secure algorithms, such as SHA-256 instead of SHA1 at the very least and eventually (not really necessarily) EC instead of RSA.