When I use online tools to test www.bestbuy.com URL's, I get a timeout connection error.
By online tools I mean HTTP Status Code Checkers or Redirection Checkers.
I expect to receive a 200 HTTP Status Code, same as if I were visiting the website with a normal web client.
My goal is to understand exactly what is causing this behaviour (technology and configuration). I'm really curious about this implementation since it can be useful for future projects.
WHAT I TRIED
I tried different methods to replicate the connection timeout error, but couldn't achieve my goal.
First thing I did was to check different online tools. Tested at least 20 of them, and everytime I receive a connection timeout response. Example:
My first thought was they were blocking specific IP ranges, but some of the tools I used were not that famous. Only tool that didn't hang up is W3 Techs Sites Info, but it may be old data.
In this case I tested with different configurations of User Agents and HTTP Headers, using exactly the same as if requesting the URL with my browser.
Some User Agents get a 403 Forbidden HTTP Status code, but most of the time I still get a 200.
I very much doubted that the online tools that I used ALL left some kind of fingerprint. I mean, in the end is still a simple HTTP request, right?
But to leave no stone unturned I still quickly set up two local HTTP Status code checkers, one in PHP using cURL and the other in Python using Flask and Requests.
In this case I also tried different setups, HTTP Headers and configurations. The only time I receive a timeout is if I don't set up a User Agent at all, but in all the other scenarios I still always get a 200.
I also researched online about Akamai and Nginx configurations that could be causing this behaviour, but found nothing definitive.
Tried visiting the website through the TOR network and I get a 403 Forbidden.
So my final conclusion is that of course there are different network security measures in place, but I really don't know what is causing the timeout connection error using online tools.
There is the theory of the blocked IP ranges, but as I said before I used multiple tools, even not so famous ones, so my doubt persist. And if that is the case I still don't know exactly what kind of configuration could block all those online tools or if a public list exist because is the first time I see this happen when analyzing a website for competitive research.
I even reach out one employee at BESTBUY in charge of network and IT management, but unfortunately couldn't help with this issue.
I am probably missing something because I'm not an expert in this matter and maybe I am tackling the issue from the wrong perspective, so thought the only place I could get a definitive answer was here at StackExchange.
Help a poor fellow, cannot sleep with unsolved problems and I'm running out of my coffee supply :D