So I'm building an application that is designed for sharing files across a local network, and eventually across the Internet. I want to have a system for verifying the identities of computers, something like Bluetooth pairing that requires mutual agreement between two computers before any files can be shared between them, but is relatively simple for the user. I also want to protect against MITM attacks so that files sent from one computer to another are not modified in transit or read. How can I achieve all of this?
While your question is very broad and thus it is not clear if the answer really fits your problem a way to achieve what you describe is to use TLS with client certificates. This is an established technology (i.e. what you use with
https:// links) and widely implemented in a variety of libraries and tools on server and client side. Client certificates don't actually allow you to identify the system by itself but only the user of the certificate, i.e. in theory they could be shared between multiple systems. But by generating a new client certificate for each new installation you can essentially identify the system where the software is used.
Maybe you could implement some kind of handshake system
-> Want to connect //gives one identity <- Your identity is recognized, please check my identity //gives one identity -> Your identity is recognized // Establish connection
I don't know much more about those kind of systems but check gpg and wpa it might give you a hint.