I'm having trouble understanding exactly what you are asking but I'll take a whack...
As you point out, the private/public key pair that is associated with the server can be used for encrypting data sent to the server because only the server has the private key to decrypt. But there is no secure way for the server to send data to the client. All the client has is the server's public key which, as its name suggests, is public. So encrypting data with the private key and sending to a client is readable by everyone. So, without a symmetric key, there's no way to encrypt data being sent to the client.
Note that a different encryption model than SSL could have the client generating a public/private key pair for every SSL handshake. The server would then use the client's public key to encrypt data sent to it. The reason this isn't done is, as you stated, asymmetric encryption is slow (and less robust in other ways) so this is a bad choice.
Also, because a unique key is used for the transfer of data, you can establish forward secrecy. Forward secrecy assures that even if the private key gets discovered at a future date, it is still impossible to decrypt a recording of the communication.