I would say that if you will just connect there, nothing should happen. Except you give him your IP so he can theoretically focus you. If you have correctly configured your client.
It really depends on the security of your client setup and another servers you use. If you authorize / sudo yourself using the key the probability he will be able to exploit another server is quiet low.
But many times I also saw tunelling over SSH setup in the way the servers were able to access each other over the SSH channel opened by the user. And keys were stored on "compromised server".
As I said in the beginning. It really depends on many other things, I would say, its hardly possible but not probable if other security recommendations will be followed.