2

PROBLEM

When I use online tools to test www.bestbuy.com URL's, I get a timeout connection error.

By online tools I mean HTTP Status Code Checkers or Redirection Checkers.

EXPECTED BEHAVIOUR

I expect to receive a 200 HTTP Status Code, same as if I were visiting the website with a normal web client.

MY GOAL

My goal is to understand exactly what is causing this behaviour (technology and configuration). I'm really curious about this implementation since it can be useful for future projects.

WHAT I TRIED

I tried different methods to replicate the connection timeout error, but couldn't achieve my goal.

TEST 1

First thing I did was to check different online tools. Tested at least 20 of them, and everytime I receive a connection timeout response. Example:

  • web-sniffer.net
  • builtwith.com
  • urlitor.com
  • httpstatus.io
  • tools.searchbrain.it/chain-of-fools
  • redirect-checker.org
  • internetmarketingninjas.com/header-checker

My first thought was they were blocking specific IP ranges, but some of the tools I used were not that famous. Only tool that didn't hang up is W3 Techs Sites Info, but it may be old data.

In this case I tested with different configurations of User Agents and HTTP Headers, using exactly the same as if requesting the URL with my browser.

TEST 2

Tried to replicate the behaviour with a browser. Tested different user agents and HTTP headers. Also navigated the website without javascript, cookies or cache enabled, but nothing.

Some User Agents get a 403 Forbidden HTTP Status code, but most of the time I still get a 200.

TEST 3

I very much doubted that the online tools that I used ALL left some kind of fingerprint. I mean, in the end is still a simple HTTP request, right?

But to leave no stone unturned I still quickly set up two local HTTP Status code checkers, one in PHP using cURL and the other in Python using Flask and Requests.

In this case I also tried different setups, HTTP Headers and configurations. The only time I receive a timeout is if I don't set up a User Agent at all, but in all the other scenarios I still always get a 200.

TEST 4

I also researched online about Akamai and Nginx configurations that could be causing this behaviour, but found nothing definitive.

Tried visiting the website through the TOR network and I get a 403 Forbidden.

CONCLUSION

So my final conclusion is that of course there are different network security measures in place, but I really don't know what is causing the timeout connection error using online tools.

There is the theory of the blocked IP ranges, but as I said before I used multiple tools, even not so famous ones, so my doubt persist. And if that is the case I still don't know exactly what kind of configuration could block all those online tools or if a public list exist because is the first time I see this happen when analyzing a website for competitive research.

I even reach out one employee at BESTBUY in charge of network and IT management, but unfortunately couldn't help with this issue.

I am probably missing something because I'm not an expert in this matter and maybe I am tackling the issue from the wrong perspective, so thought the only place I could get a definitive answer was here at StackExchange.

Help a poor fellow, cannot sleep with unsolved problems and I'm running out of my coffee supply :D

closed as off-topic by Teun Vink, Steffen Ullrich, Serge Ballesta, Arminius, Rory Alsop May 27 '17 at 14:30

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be about Information security within the scope defined in the help center." – Teun Vink, Steffen Ullrich, Serge Ballesta, Arminius, Rory Alsop
If this question can be reworded to fit the rules in the help center, please edit the question.

0

Without seeing the internals of Best Buy's network and their firewall/network device configuration it would be hard/impossible to come to a definitive answer for this.

With that said, we could speculate about possibilities.

you've already eliminated possible problems with request headers by trying a variety of options.

The most likely alternative, to me, would be that the application is filtering based on the source IP address of the request.

I notice when I visit that page from the UK that it asks which country I'm from, recognizing that I'm not in the USA/Canada/Mexico. So it's likely that they're using some form of GeoIP lookup database to establish that fact.

It could be that their system is doing additional checking on the source IP addresses used to contact it and is recognizing that they're not from ranges belonging to residential ISPs in the US (I'd expect most of these services to be hosted in a hosting company/cloud company network somewhere).

As it could determine that the source IP address is unlikely to be a real customer at that point, it could choose to block it.

Now this could very well not be the case, however if they're not filtering based on something in the request format, the other piece of information that they could be making a choice based on would be source IP address.

Not the answer you're looking for? Browse other questions tagged or ask your own question.