First of all, thanks for the interesting question. I did not know about the details of CSRF before and had to look up the answer to your question myself, but I think I know the correct explanation for Django's behavior now.
The Django developers are treating HTTP and HTTPS refers differently because users expect different things from insecure and secure web services. More specifically, if a web page is using transport layer security, users expect to be protected against man-in-the-middle attacks, meaning they trust in the principle that even if someone sat directly between them and the remote server and intercepted every single message, they couldn't make any use of that information. Note that this is not expected of plain HTTP connections.
Now consider the following scenario, quoted from a Django dev's post here :
- user browses to http://example.com/
- a MITM modifies the page that is returned, so that is has a POST
form which targets https://example.com/detonate-bomb/ . The MITM has
to include a CSRF token, but that's not a problem because
he can invent one and send a CSRF cookie to match.
and so includes the CSRF cookie, a matching CSRF token and the
user's session cookie, and so will be accepted.
I did not instantly understand this attack myself, so I'm gonna try to explain the details. Note first that we are looking at a page that displays forms over plain connections but submits data via SSL/TLS. Part of the problem, as I understand it, is that the cookie and hidden form value (aka "the CSRF token") are only compared against each other, not against any value that is stored server-side. This makes it easy for the attacker to supply their victim with a cookie-token-combination that will be accepted by the server - remember, the page displaying the form is not secured, so
Set-Cookie headers and the contents of the form itself can be spoofed. Once the manipulated form is submitted (via injected JS, for example), the server sees a perfectly valid request.
Referer checking is the answer to this exact problem. Checking these headers, only requests originating from
https://example.com will be accepted at another endpoint of
https://example.com. Insecure pages from the same domain will be treated as completely untrusted, and rightly so.
Now to come back to the question why plain HTTP requests are treated differently, we just have to imagine a site that doesn't use encryption at all. In that case, a man in the middle could also spoof the
Referer headers sent with the actual form data, so checking those does not provide any additional security. In other words: There is no protection against CSRF attacks by a man in the middle - but, as I mentioned earlier, users do not expect this kind of security from plain HTTP sites.
Regarding your question about how other web frameworks handle this attack vector, I honestly have to say I don't know.