You probably mixing up a bit what is happening between client and server when they communicate over encrypted channel (ssh or tls it doesn't matter that much - the pattern anyway involves two kind of ciphers).
In essence, there are two kinds of cryptographic constructs that used: symmetric and asymmetric ciphers (I would omit describing other nitty-gritty details involving integrity of a data etc).
What is generally happening, is the following (example ssh with key verification and password log in disabled):
Asymmetric crypto
Client has a private key (usually a file called id_rsa) and server has a corresponding public key (located in a known_hosts file).
When client want to ssh into a particular host it establishes a TCP connection to it sending it's own username. Server matches a corresponding public key from known_hosts to this particular username and sends some kind of challenge back to the client.
This challenge is solvable only with a corresponding private key that client should have on it's machine. Client solves the challenge and sends the results back to the server which verifies the solution and lets user log in (or rejects the client if the solution is not correct - which means the private key is wrong).
After that point, if the user was granted an access, symmetric crypto starts to play it's role.
Symmetric crypto
User was authenticated by a server and session key is generated on server side which then send to a client encrypted with it's public key (again from a known_hosts).
This session key is a secret key of a some much faster then the asymmetric crypto symmetric cipher (i.e. AES). This key then is used to encrypt all the traffic between both sides as it was securely passed to a client and both sides now has that secret known only to them.
After the end of a session (user logs out) the session key plays no role and is not used for any other session (it's generated randomly and a probability that the same key would be generated twice is negligible).
It's a very simplified description but it would at least make things a little bit more clear.
What happens is that asymmetric crypto only authenticates a user and helps to exchange secret key in a secure way. All the communication then is encrypted with the same session secret key of some symmetric cipher.
As of your question about omitting storing a secret on a client and using some secure one-way transfer:
ssh is a protocol of communication and the described above pattern is a part of it's 'mode of operation'. It can't be changed (correct me if I'm wrong) - if you want just to push some data to a remote host without even authenticating the sender (which is strongly discouraged) you can figure out something like a shell script that encrypts data with some public key (client side,
gpg is a way to go) and sends it over a telnet to a server which decrypts it with it's private key (again gpg).
Though the above described pattern could be made secure with enough amount of effort (hmac, digests, signing etc) it really makes little sense to bother with all of this, as you still have to store secret somewhere (this time it's on a server side, or even on both sides if something like hmac is used).
