I am trying to create a small network auditing script that tracks HTTP and HTTPS domain names visited by certain hosts using the libpcap wrapper for Nodejs. At its best the script would be able to give a printout of the names of the websites that a host visited as well as a timestamp of when the visit occurred.

My initial thought is to do a standard capture on the LAN with a packet filter to monitor only for "udp port 53" traffic, inspect only DNS questions, and then save them to a hashmap by ip or MAC address. However this method will give somewhat cloudy results as any asset loaded on a page will make another DNS request. Going to say, https://amazon.com, actually triggers ~20 hostname lookups. Is there any sensical method of separating hostnames visited by a browser from ones that are requested simply for assets? One method I can think of would be to employ some sort of "debounce" timer where my script listens only to the first DNS question request made every 1000 milliseconds (give or take), that way I could at least ignore the first wave of assets loaded by any web page. Any thoughts on if this is a solid method for tracking webpages visited (both http and https) or suggestions for ways to improve it?

  • Visiting amazon.com should generate at most one DNS request, which should be cached for subsequent HTTPS requests. – Bob Brown Mar 5 '15 at 21:21
  • You shouldn't be trying to filter the useful from the non-useful data at the time of collection, this is a task for post-processing, maybe the additional data will be useful later on - it definitely won't be useful if you've not collected it. – StampyCode Mar 5 '15 at 23:46

This cannot work as you envision.

For one thing, web browsers are not the only thing that cause DNS lookups: anything (such as a network time client) that needs to convert domain names into IP addresses or vice-versa will do so.

For another, you cannot count on the browser doing a DNS lookup every time it loads a page. The DNS resolver should cache the results so that, for example, Amazon won't be looked up more than once per minute, while visiting my home website shouldn't produce more than one lookup per hour. Additionally, it's possible to browse websites by IP address, which doesn't generate any DNS traffic at all.

A third problem, and one that will trouble any tracking system, is that a page load doesn't equal a page view. Some browsers perform "predictive loading" to pre-load the page they think the user is most likely to visit next, and some anti-virus programs do the same thing to pre-scan links for damaging content (one antivirus famously DDoS'd a number of sites with a poorly thought-out implementation of this).

Your best bet is to set up a transparent proxy that logs all traffic going through it. This will let you track page-load activity for HTTP, and the IP addresses for HTTPS traffic.


Perhaps, as a start, use the Adblock domain list and block some of these domains which will prune some of the traffic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.