If the question is "Why do we need to encode characters other than < and >", this is because corner cases possibly exist. A naive example would be when the input from a text box is used to generate, say, an href, or an image source on the fly.
Possiblities:
<img src="&{alert('XSS Vulnerable')};">
The special character “&” is sometimes interpreted as a new JavaScript code segment (entity).
" [event]='code'
In many cases it may be possible for an attacker to insert an exploit string, with the above syntax, into a HTML tag that should have been like:
<A HREF="exploit string">Go</A>
resulting in:
<A HREF="" [event]='code'">Go</A>
<b onMouseOver="self.location.href='http://bad.com/'">text</b>
As the client cursor moves over the bolded text, an intrinsic event occurs and the JavaScript code is executed.
There are many more corner/obscure (in today's world) cases where encoding just < or > may not be enough. Look up http://www.technicalinfo.net/papers/CSS.html for a decent (if not slightly outdated) run through of similar cases.