Ignore authorized_keys and known_hosts for now.
I don't think the "pair of big primes" is in the public digest, it's more akin to a hashed scramble, irreversible and asymmetric. The split you speak of is done with the private key (somehow).
RSA is not used for symmetric decryption of your (ssh) state channel, new, ephemeral AES-128/AES-256 type keys are used (actually Chacha now?), modern stream cyphers rolling one byte per time. Every new ssh connection probably has new ephemeral stream cyphers generated, and long running sessions maybe shift keys each hour!
Apart from the SSL wrapping the entire connection, the ephemeral keys are sent somehow using a blend of a portion of the (presumed) prime number that only a certain person could hold? Dunno how this part works, say your challenge is the number 2 and the public key 4162.574257426 only the owner could deduce the ephemeral authenticate magic phrase maybe is 840840 because (420420 / 101) * 2 = 840840.
It is immune to a replay attack because a new challenge number or nonce is generated at the server side: lets say the new nonce is 3, your secret primes (oops, 420420 is not a prime my bad) will produce the answer 12487.722772277 that being 420420 / 101 but multiplied by 3 in this case, changing maybe called a nonce (nonsense).
The old RSA private key files are specially formatted. The two large primes are extracted from this file an agent on your machine similar to:
openssl rsa -noout -text -inform PEM -in 4096-id_rsa
RSA Private-Key: (4096 bit, 2 primes)
87:2b:87 (greatly shortened enormous string)
publicExponent: 65537 (0x10001)
83:01 (shortened quite long enormous string)
ae:c1 (I can see two on my terminal at once)
b4:47 (I can see two on my terminal at once)
41 (not sure why - these are also bigger than my primes)
8b:e5 (not sure why - these are also bigger than my primes)
ff (not sure why - these are also bigger than my primes)