I'm working on an article for work and seeking a list of the top ~500 websites (ranked by Alexa) that still don't have support for HSTS. Does such a list exist? I'm not getting any interesting hits from my Google queries.

Other than check all the websites one-by-one, would anyone be able to offer a method for accomplishing this?

  • 3
    It would be pretty easy to script this, I'd imagine. – multithr3at3d May 18 at 2:46
  • What would one look for in such a script? I'm a decent BASH coder, I could try to create something automated GET requests, but how would know if a particular queried site was using HSTS or not? – user178357 May 18 at 3:19
  • 6
    You could use curl to make a HTTP request, and check the HTTP response headers for the Strict-Transport-Security header. – multithr3at3d May 18 at 3:21
  • 1
    Regular or static HSTS? Static HSTS is compiled into a list, but far fewer websites use static HSTS. – forest May 18 at 8:48
  • 1
    @user178357 You would want to use HEAD requests, not GET. – forest May 19 at 0:06

Assuming you already have a newline-delimited list of domain names you wish to check, use a script such as this one which takes the list in a file specified by the first argument or stdin:

#!/bin/bash

while IFS= read -r s; do
    curl -sI "https://$s" | grep -qiE "^Strict-Transport-Security:" || echo "$s"
done < "${1:-/dev/stdin}"

exit 0

This will output a list of domains to stdout that are up, but which do not send the standard HSTS headers over HTTPS. This does not mean that certain subdomains do not use HSTS, and it does not mean that all the sites which do send HSTS headers are doing so correctly. Checking whether or not the HSTS is syntactically valid would require much more complex regex.

Dynamic HSTS

@forest's answer covers this well, so not to retread ground, you will want to connect to each site and check the returned headers, this could be done using HEAD if you wish to reduce the amount of data sent back.

Static HSTS

Firefox

You will want to use one of the following links:

Chrome

Chrome is slightly harder, first you need to use this:

https://omahaproxy.appspot.com/all.json

to find the hash for a given release, then load

https://chromium.googlesource.com/chromium/src/+/hash/net/http/transport_security_state_static.json?format=TEXT

where hash is the hash you got from the previous step.

You'll need to Base64 decode the result.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.