Is it possible to use RSA public and private keys to authenticate two computers (or to make sure they're both the right computers)? If so, how?



Yes, it's possible. Computers Alice and Bob both have one private key and one public key each.

Alice chooses a random token and encrypts it with Bob's public key and its own private key. Bob decrypts the token with its private key and Alice's public key, then reencrypts it with Alice's public key.

Upon receiving the original token from Bob, Alice knows that Bob must know Bob's private key, therefore it must be Bob. And since the token was decrypted with Alice's public key, Bob knows that Alice is in possession of Alice's private key, which means it is Alice.

And since nobody else except Alice and Bob are now privy to the random token that has been exchanged, both Alice and Bob may use it as the key to a symmetric cypher algorithm suitable for exchanging further communication (they might also use again RSA, but it's more expensive computationally):

  ALICE = Alice's public key; bob = Bob's private key

  Alice chooses K

  Alice encrypts with BOB,
  then with alice

                   ----> alice(BOB(K)) ----> 

                                                Bob knows ALICE and bob,
                                                so can retrieve K
                   <---- bob(ALICE(K)) <----
                      (or just ALICE(K))
              (or even K, if K need not be secret)

  Alice gets confirmation

                   <---- AES256(K, MESSAGE[0]) ---->
                   <---- AES256(K, MESSAGE[1]) ---->
                   <---- AES256(K, MESSAGE[N]) ---->

During such a conversation, K may also get refreshed from time to time.

Real life example (well, sort of)

In this example Alice and Bob are two friends and they both have the OpenSSL utility. They communicate through a public channel (e.g. their Facebook pages) where they can post binary files. OpenSSL actually can produce ASCII files that would be much better for this purpose, but let's keep as near the above as possible (in real real life we'd use some OpenSSL scripts that would automate the whole thing and work much better, or some tool that would wrap OpenSSL to do the same things with an easier interface).

There are some caveats that I'll mark with "VULN". See at bottom.

To begin with, Alice creates a RSA public/private key pair and extracts the public key. Bob does the same (his files will be named bob-*) (VULN-1).

openssl genrsa -out alice-both.pem 1024

openssl rsa -in alice-both.pem -out alice-public.pem -outform PEM -pubout

Now both Alice and Bob publish their public keys on their facebook page. Anyone can read them.

Now Alice wants to exchange a private conversation with Bob, maybe some files. Alice wants to be sure that Bob is cognizant of their source, and she wants Bob the be the only one to be able to read the files.

This can be done using asymmetric encryption only, but it would be too slow and inefficient due to size constraints and computational overhead. So they need to use symmetric encryption, and for this, a shared secret (above, this was called K), that only they know.

They do so by choosing a K, and communicating it over a secure channel. The secure channel is provided by asymmetric encryption over a public channel (facebook).

Bob being a gentleman, it's Alice who chooses a passphrase.

She does so by creating a text file with the passphrase...

echo 'the magic words are Squeamish Ossifrage' > phrase.txt

and encrypting it with Bob's public key downloaded from Bob's facebook page (VULN-2)

    openssl rsautl -encrypt -inkey bob-public.pem -pubin -in phrase.txt -out file1

Now, only Bob can decrypt the file. The problem remains how to make Bob sure that this is from Alice. Alice could cut "file1" in pieces small enough to be encrypted and send them all, but it's not necessary; she can sign the file with her private key.

    openssl dgst -sha256 -sign alice-both.pem -out file1.sign file1

Now Bob receives the two files. He first verifies the signature against Alice's public key (VULN-2 again).

    openssl dgst -sha256  -verify  alice-public.pem -signature file1.sign file1

and he gets "Verified OK" (if anyone else's certificate had been used, for example his one, he would have gotten "Verification Failure").

So, verified against the key published on Alice's facebook page, the "file1" is at least as trustworthy as Alice's facebook account.

Bob proceeds in decrypting with his own private key.

openssl rsautl -decrypt -inkey bob-both.pem -in file1 -out secret

Now Bob knows the secret phrase (it's in "secret") and can encrypt a document using that phrase.

openssl enc -in secret-document.txt -out secret.bin -e -aes256 -k "$( cat secret )"

or, equivalently,

openssl enc -in secret-document.txt -out secret.bin -e -aes256 -k "the magic words are Squeamish Ossifrage"

secret.bin is again posted publicly on Bob's page.

Alice on the other hand just runs the corresponding decoding command

openssl enc -in secret.bin -out plain.txt -d -aes256 -k "$( cat phrase.txt )"

and recovers "plain.txt".

  • VULN-1: Alice and Bob must be sure that nobody else has access to their private keys. They can lock them with a passphrase that needs to be entered for OpenSSL to access the keys, for example.

  • VULN-2: if Eve gains access to Alice's facebook page, or can induce Bob's computer to believe that some other computer is actually the Facebook web site, she can trick Bob into downloading Eve's public key believing it's Alice's. If Eve is capable of doing the same to Alice, she can pose with her as Bob. So Alice "talks" to Eve, using a format that Eve can decrypt, and Eve reencrypts it after tampering and sends it along to Bob, both ends being none the wiser. This would be a man-in-the-middle attack. That's why the authenticity of the public keys must be unquestionable.

  • Since neither Alice nor Bob have reason to use the keys they have published (they own the originals), once gained access to both Facebook pages Eve could replace both keys with others of her own making, and continuously monitor both pages in order to download any newly posted file and replace it with the appropriately decoded-tampered-and-reencrypted version. Since Alice and Bob aren't likely to verify their own files, and they haven't published the files' signatures (and even if they did, they might not notice they've been changed), they won't notice a thing. Not really likely to happen with something as public as a Facebook page, but in other scenarios the key distribution might be at risk. Of course, as soon as someone out of Eve's purview attempts to communicate with Alice or Bob and fails, the game's in the open.

  • does Alice encrypt with ALICE(BOB(K)) or alice(BOB(K)) May 27 '14 at 21:35
  • alice(BOB(K)): it uses its private key (lower case), and Bob's public key (upper case). If it used its public key ALICE, then Bob couldn't possibly decrypt the token, as it would need Alice private key -- which it hasn't.
    – LSerni
    May 27 '14 at 22:25
  • ok, with OpenSSL, how would you encrypt with a private key? I googled it and apparently anyone would be able to decrypt it. May 27 '14 at 22:43
  • No, they wouldn't. Just a moment, I'll post a "real life" example -- except that I'll have to make some slight modification to keep it from being unwieldy for two human beings.
    – LSerni
    May 27 '14 at 23:24
  • awesome, thanks! So you sign the file with the public key! I Thoth you meant encrypt. May 28 '14 at 0:20

It is possible. RSA algorithm works as mentioned in link.

And you can implement it using various languages. here shows it can be implemented with java. And many languages provides inbuilt packages if RSA algorithm.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.