What is the expected behavior when a server hello contains an
unexpected reply for a TLS extension (e.g. ALPN)?
I assume you're when you say "unexpected reply" you're suggesting the server responds with an ALPN "selected protocol" value that isn't among those proposed by the client.
Should I (as a client) terminate the connection or just ignore the
extension? Is it defined in the RFC?
While RFC 7301 isn't explicit, I believe that the implicit expectation is that the client should terminate the TLS handshake - just as the server would if it didn't support any protocols that the client advertised:
In the event that the server supports no protocols that
the client advertises, then the server SHALL respond
with a fatal "no_application_protocol" alert.
Should the behavior be different for different extensions?
Different extensions may handle this case differently - the two most common handlers being "abort the handshake" and "ignore the extension attempt." By their very nature, extensions are often tolerant of differing levels of support and implementation maturity.