Unknown client-side redirect

Question

I'm having an issue analyzing the traffic with a client's website. I'm using ZAP proxy to see all request/responses. Now, here's the problem. Say I make a request to: www.example.com/index.html I never ever see that exact request in the proxy. That request triggers a whole lot of other requests for js scripts, and also requests to www.example.com/api/documents, www.example.com/api/users, www.example.com/api/resetpasswords, etc... but never the main request. When i look for the index.html in all the packets, the only match I can see is in the referer field of all the example.com/api requests, each and every one of them have index.html in the referer field.

What kind of redirection is in place here? I tried to wget the main index.html file and its mainly a lot of scripts being called, as well as some comments about angular js. There are no 30X redirects involved, and if there were redirects inside of some javascript, i should see the main index.html request that subsequently loads the javascripts. How is this redirection happening?

Answer 1

If the client is a browser, there is not guarantee to what exactly happens. The index could be cached. Also some browsers show redirects in the network console while never send by the server. Start by disabling caches, cookies and other client side storage (or use a private navigation mode). Btw, AngularJS uses extensive caching, so I almost certain its the cache.

Update

There are no 30X redirects involved, and if there were redirects inside of some javascript,

No, there are indeed no redirects involved, but there are a lot of 304 responses. They explain the behavior you ask about, and what happens is the following:

To limit the number of requests, webservers can use a technique called an entity tags. If the webserver supports this and is configured to use them (Microsoft IIS is by default), upon request the server will calculate a hash for this document (based on last time changed) and includes this hash in the response (ETag header). In the next requests the client can send this hash to the server, which matches the hash with the current ETag. If they match, the server can respond with an empty 304 response (not modifier).

This empty response if what you see. Some browsers will not list these responses, because in essense 'nothing has changed'. If you disable the cache, the client will omit the If- headers in the request, causing the server to resend the document.

With caching and ETags:

Without caching an ETags: