so that if anyone complains about what they received I can check if the file is really the one I sent or if anyone modified it.
The traditional way of checking if a file is modified is to use
the "md5sum" utility, or the (probably superior) "sha256sum" utility.
Many organizations use this technique to verify that files, downloaded from "mirror sites" that are conveniently nearby (and therefore faster than downloading from the "original site" on some other continent), haven't been maliciously or accidentally modified somewhere on the path from the original, to the mirror site, to the final destination.
a b c d e
Using those utilities, someone with the original version of the file generates a short md5 hash or a short sha256 hash (or both) of the original file,
and stores that short hash in some text file somewhere.
(Typically on the "original site" and on every mirror site).
Later, someone with a questionable file runs the same utility, producing a fresh hash value.
If the fresh hash value is identical to the hash produced from the original file (that hash is typically obtained from the "original site", possibly using a web browser and https and TLS), then you can be confident that the questionable file is, in fact, identical to the original file.
Some paranoid people download the hash sum from several different "mirror sites" and other sources.
(This is quick, because the hash sum is very short; much faster than re-downloading the entire file from all those sources).
The likelihood that all those sources have been subverted becomes more and more unlikely as more sources are consulted.
(Since you are generating the original executable, be aware that recompiling exactly the same source files using exactly the same compiler usually produces a slightly different executable file, resulting in a completely different hash.
See "How do you verify that 2 copies of a VB 6 executable came from the same code base?"
for details).
There are many utilities that build on top of md4sum or sha256sum.
md5deep and hashdeep can make a "summary" of an entire folder at one point in time -- perhaps your home directory, or the /bin directory, or the root folder of an entire hard drive. Then later it can tell you exactly which files have been added, deleted, or changed since that summary was made. When the hashes match, you can be sure executable files haven't been infected by a virus after the original snapshot -- this even detects new viruses that are not yet recognized by any virus scanner. (Virus scanners are still useful to detect files that were infected by a virus before the original snapshot).
bittorrent uses sha1 to confirm that pieces of a file downloaded from completely untrusted machines are identical to the original pieces of that file.
rsync uses md5sum to quickly check if the local file is identical to the master copy of a file on some remote file server. If they are different, it assumes the local copy is some obsolete version, and automatically downloads the differences between the two files, modifying the local file until it becomes identical with the remote master file.
I'm pretty sure some protocol could be developed using a self-signed certificate that would work just as well as using sha256sum; but I don't see how this new protocol would give any advantages over the commonly-used sha256sum check.