Depending on your goal, you may want to use different architectures:
1. Use a centralized server for authentication
This would be similar to how some videogames act. A centralized server would handle authentication and authorization, but actual data transfer would happen directly between clients for performance reasons.
This approach works well, since authentication can be treated like any other client/server scenario. Passwords can be stored securely in one centralized database, while the server doesn't need to worry about actual traffic between clients.
The downside of this is if your centralized server is unavailable (maintenance, Denial-of-Service attack, obsolescence of service, etc.), then the entire application becomes unusable.
2. Use public-key cryptography
You stated in your question that you do not wish to use PKI. Unfortunately, PKI excels at precisely this kind of scenario. A client can generate a new key pair, which is akin to a new "identity". The client can then publish their public key, allowing others to verify their identity. Whether or not the corresponding private key corresponds to a real-life identity is not an easy problem to solve, but one you don't necessarily need to solve. If a virtual identity (akin to a username/password combination) is enough, then PKI solves this problem well.
Advantages of this system would be that it is very resilient, requiring basically no maintenance, since there is no centralized server involved. Two users, who have each other's public keys, can authenticate each other until the end of days.
Disadvantages is that if you have multiple devices, migration becomes less simple than just "entering your credentials" on the other device. You either need to copy your private key to the other device, or you generate a new private key there and have the keys sign each others public key. In a sense, Key A says "I guarantee that Key B is also me. Yours truly, Key A" and vice versa Key B says "I guarantee that Key A is also me. Yours truly, Key B".
This system works well, but requires your users to be a bit more involved. Depending on who your target audience is, it could be a reasonable assumption to make that your users will understand how PKI works. Otherwise, telling them to copy a file will be something most users can do.
3. Zero-Knowledge Proofs
Basically, you want to prove to someone that you know something, with them knowing anything what that thing is. This seems like an impossible task, but it is doable. Wikipedia provides a set of abstract examples, but for the sake of completeness, I will include one here:
Imagine you have two differently-colored balls, and a color-blind friend. To your friend, the balls seem identical. You want to prove to him that they are differently-colored, without revealing which ball is which color.
Your friend shows you the balls, then places them behind his back and randomly decides to switch them or not. He then asks you whether or not he switched the balls. If the balls are indeed differently-colored, then you can see the difference and say whether or not they were switched.
The first time this is done, your friend believes that there is a 50% chance that you could have just guessed it correctly, so he repeats the task. You succeed again, and now there is a 25% chance it was just another lucky guess. With each repetition, his confidence doubles. After 80 repetitions, your friend is more than 99.9999999999999999999999 % confident that you weren't just guessing, which "proves" to him that you indeed are able to differentiate the balls, even if you did not disclose how, or which ball is which color.
To be less abstract, the Secure Remote Password Protocol implements a Zero-Knowledge Proof, which allows a server to verify whether or not a client knows a password without storing the password. The following image demonstrates the workflow:
Image credit to thinbus-srp
I'll be honest with you, I've never worked with SRP before, but it seems to be something worth looking into.
Why can't I just store hashes?
A password is a so-called "shared secret". Authentication with a "shared secret" works by sending the server my secret, and the server checks whether or not this shared secret matches what the server previously stored. However, in your case, you don't want the involved parties to know the secret - after all, then they could just impersonate you. You're tasked with solving the problem of whether or not someone knows something, without knowing what it is. Let me show you why this doesn't work:
- If you just send plaintext passwords, then every client needs to store every plaintext password of every user they encounter - and are thus able to impersonate every user they encounter. Not good!
- Just hash them, right? Imagine you "register" with me, telling me the hash of your password is
04C11DB7.... How can you authenticate to me then? You could send me your password in plain text, which means I can steal it. You could also send me that hash again, but then I could just in turn send your hash to other people. By just hashing everything, you made the hash the new password.
- Just add more hashes, right? No, it doesn't work. No matter how many layers you apply, I can always just take what you send me and send that to other people.