Unfortunately the answer is yes, keeping in the spirit of your methodology (which from a "pure" security standpoint is terrible, but a good starting point that's better than a sticky note under the keyboard; it's good that you're asking the question!). This method is fine to keep random people out of your password list, but a determined attacker will go through this easily. (See brute-force attacks for an idea how)
Since you're saving the file with the password (possibly the correct password) and then re-writing the file, there is a good chance that the file is not actually being overwritten on disk (and a simple spelling error could leave a copy around too). This means that a search of the file system by an attacker could find the bytes that were stored by your file.
If you didn't write the file with the password and just calculated your checksum with the password in memory, it makes this process slightly more secure (nothing on disk ever has any password attempt), but as mentioned by other answers, it's still based on the question being answered being a closely guarded secret.
As mentioned by others, some sort of USB keyfob for each person would potentially be a better option. This would also allow you to just remove someone's lost keyfob from the allowed set of fobs if they misplaced it.