When we authenticate with a server we authenticate with a public key (asymmetric encryption) to transfer a symmetric key. Why don't we keep using the asymmetric key pair instead of the symmetric key?
There are several reasons why this can't work in real world (even though it sounds quite ok with the basic knowledge you are presenting).
SSH servers support also password authentication. What asymmetric key would you use in that case?
Transferring huge amount of data and serving many clients would make the host keys extremely prone to the various attacks. In cryptographic algorithm, you need to rekey (exchange the keys) after some time or some amount of transfered data. That would be impossible unless you would need to redistribute the server public key very frequently.
Last but not least as mentioned in the comments, the CPU cycles to achieve the same security strength with RSA is gradually higher than AES and it is similar with other algorithms.