My question is, if any characters are allowed in the alt attribute of an img tag (except for double quotes), is XSS possible? Valid image or not.
Assuming you are surrounding with double quotes, this would be fine. However, you might want to also escape <>, even though mainstream browsers won't have a problem with it, just in case someone has a broken user agent. It's valid within an attribute, not not required in a double quoted attribute: http://www.w3.org/html/wg/drafts/html/master/syntax.html#attributes-0
You haven't mentioned how the attribute is quoted, so I've covered all possibilities of attribute quoting. This answer reflects the precondition that double quotes are already stripped as you mentioned in your question.
Yes, if the attribute itself is single quoted or unquoted:
are both vulnerable. The first is vulnerable to the following sequence
so the tag becomes
and the second is vulnerable to this sequence
which will be rendered as
Regarding the attribute being double quoted, as well as restricting or encoding double quote characters (
As you cannot guarantee that user controlled data will not contain an "ambiguous ampersand", you should either remove or properly encode this characters.
Now whether an XSS attack is possible with access to only the
If the value
However, it is best to do things properly when it comes to security, you never know if this escaping faux pas could be turned into something else evil by something such as mXSS.