First, obviously, if there's a bug in the SSH client, it might allow the server to execute code there. A really bad bug could even allow a man-in-the-middle to exploit the client. The rest of the attack surface consists of where SSH sends data that it receives from the server.
The text received through the SSH connection is, in your case, sent to a terminal. Terminals parse escape sequences — that's how they display colors, move the cursor, etc. There can be bugs in the escape sequence parser. Additionally, some escape sequences may request things that go beyond affecting the display, such as changing the terminal's title, resizing the window, or injecting input data (e.g. a command to query the cursor position or the window title). Modern terminals disable the truly dangerous commands (the ones that cause injected input that can be arbitrary text set through another escape sequence). For more on this topic, see Can "cat-ing" a file be a potential security risk?, How can I protect myself from this kind of clipboard abuse?, and How to avoid escape sequence attacks in terminals?. The solution is to verify that your terminal doesn't allow dangerous sequences. For added safety, instead of opening a terminal where you run a shell where you run
ssh directly in the terminal, and disable any keyboard handling it may have (
EscapeChar none in the OpenSSH client configuration); this way, even if a malicious server manages to inject input, there's no risk that it would be executed as a shell command on your machine.
SSH can also forward network connections, allowing the server to reach the client besides the terminal. This can bypass a firewall restriction. Two common forwarded network connections carry risks of their own. X11 forwarding gives the server access to your local X server; this is pretty much equivalent to giving shell access to the server as it allows things like injecting input in other applications (a useful feature for things like macro recorders and window manager-based input remapping, but lacking proper firewalling). SSH agent forwarding lets the server use your private keys to potentially log in elsewhere. So when you connect to untrusted server, make sure that you don't activate any forwarding. Turn on forwarding selectively for known servers.
Additionally there is a privacy exposure if you send a default username or a public key offer.