HSTS, the HTTP Strict Transport Security mechanism, is defined by RFC 6797.
The relevant section is section 12.1, No User Recourse, which states (in part) that (my emphasis):
Failing secure connection establishment on any warnings or errors
(per Section 8.4 ("Errors in Secure Transport Establishment")) should
be done with "no user recourse". This means that the user should not
be presented with a dialog giving her the option to proceed. Rather,
it should be treated similarly to a server error where there is
nothing further the user can do with respect to interacting with the
target web application, other than wait and retry.
Essentially, "any warnings or errors" means anything that would cause
the UA implementation to announce to the user that something is not
entirely correct with the connection establishment.
The above is non-normative implementation advice. However, the referenced section 8.4, Errors in Secure Transport Establishment is normative, and states (in full) that (my emphasis):
When connecting to a Known HSTS Host, the UA MUST terminate the
connection (see also Section 12 ("User Agent Implementation Advice"))
if there are any errors, whether "warning" or "fatal" or any other
error level, with the underlying secure transport. For example, this
includes any errors found in certificate validity checking that UAs
employ, such as via Certificate Revocation Lists (CRLs) [RFC5280], or
via the Online Certificate Status Protocol (OCSP) [RFC2560], as well
as via TLS server identity checking [RFC6125].
Here, "UA" means "user agent", which is the technical term for among other things a web browser. In RFCs, "must" really means "must":
- MUST This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification.
So if a client connects to a website over HTTPS, which announces HSTS in its HTTP response, then if the web browser allows the user to proceed with the connection in light of an untrusted certificate, it is acting in violation of certainly the intention and very likely the letter of the HSTS standard.
Note the term known HSTS host; because a HSTS header is delivered only after the first request has been sent, and a TLS connection must be established before the first request can be sent, the browser cannot know before that has been done whether or not the remote host uses HSTS. This vulnerability can be avoided by the owner of the web site using HSTS Preloading. Once a UA has established a host as a HSTS host, whether through preloading or through a HSTS header in a response, that information is supposed to be cached, ideally persistently, by the UA for the indicated amount of time.
TL;DR:
For the reasons outlined above, in the presence of any SSL/TLS problems with a host announcing HSTS, absent HSTS preloading, the first request might succeed (because the non-HSTS behavior of allowing clicking through a SSL/TLS warning is the default, and the host has not yet had the chance to announce anything else). However, any connections initiated after the first request completes (assuming this happens within the HSTS max-age period) or in the presence of HSTS preloading are required to fail because at that point the host is a known HSTS host.
It seems reasonable to state that failure of the UA to comply fully with the MUST requirements of a relevant, implemented specification is a bug. For that reason, the behavior you describe of Firefox 4 through 36 would appear to have been a bug.