48

I have seen increased 'HEAD' requests in my webserver access.log. What are these requests for? Should I disable this method in my webserver configs?

63

No.

Relevant quote from the link:

HEAD

Asks for the response identical to the one that would correspond to a GET request, but without the response body. This is useful for retrieving meta-information written in response headers, without having to transport the entire content.

If you disabled it, you'd just increase your throughput cost. A person can get the same information with a GET, so if they were trying to do something malicious, they could just use a GET. Except, this way, they're being nice and not forcing you to send the request body.

EDIT: I don't know what the requests would be from, although I can certainly think of uses. Anyone else who knows or wants to chip in, please do so. I'm kinda curious, myself. Hence, community wiki.

  • reqests are for random directory names and come from "DirBuster-0.12 (owasp.org/index.php/Category:OWASP_DirBuster_Project)" So I guess some reconnaissance is going on – hnn Jul 9 '14 at 19:28
  • Hmm... You could blacklist their IP. Or just heavily log and watch traffic from that IP without letting them know you're doing it; maybe with a transparent proxy. Watch for any change (because if there is any change... Or you could just contact their ISP. If they have a domain (which you can use dig -x to discover, or just normal dig with .in-addr.arpa and a reversed address), I'd ask them directly first, then contact their ISP. – Parthian Shot Jul 9 '14 at 19:48
  • 9
    Good answer! HEAD requests are commonly used by proxies or CDN's to efficiently determine whether a page has changed without downloading the entire body. (esp large media files) – Barett Jul 10 '14 at 1:00
  • 1
    Someone is probably running wget -m http://yoursite as a cron job. If you disable HEAD, you'll have no fun. – Damon Jul 10 '14 at 10:40
  • 2
    HEAD is also useful to tell if a webpage exists before fetching it - e.g. if you've got some custom web-scraping code and need to see if the page exists then branch your logic from there. – Adonalsium May 16 '18 at 20:15
18

Everything Parthian said was spot on. HEAD requests are a like a 'short' GET request that avoids the network extra traffic and potentially the rendering overhead of a GET request.

There are a variety of reasons you, your browser, or your search engine may want to do a HEAD request. Some websites may just be pulling meta information off you, and your smaller response is to your benefit. More likely your browser or search engines are probably using HEAD requests to see if their cached versions of your pages are still up to date.

The Response header's "date" and "expires" field should be used by clients when your page is cached to determine when the next time they should visit your site for an update. Also the response headers may sometimes include a modified date that could also be used to indicate when your page needs to be updated.

-5

You block the head request and you watch for increases in GET or HEAD request from the scummy scrapers.. THEN YOU BLOCK THEIR IPS. Funny thing is that their BOTS are so STUPID that they don't take the hint and keep coming back for more attempts. At which point you send them off on a 301 redirection to some where else ( ie someone elses web site and bandwidth. Take for example that 98% of all INTERNET search traffic comes from just Google (86% market share ) and BING and YAHOO taking up the remaining 12% ) - there is ABSOLUTELY NO REASON TO ALLOW ALL THE SCUMMY head tester to have any access. You don't need them ! They are expendable. And there is no reason to encourage their traffic.

My site handles close to 1 million visitors per day... of the 1 million -500,000 are from pointless content scrapers and barely used search engines. By blocking those clowns I free up 50% of my bandwidth and server cycles to handle the LEGIT traffic. Its too bad that the clever campers behind Apache and Linus never gave us a NULL that we could send the useless traffic off to.

  • 1
    I take it you've never heard of DaaS. – forest May 18 '18 at 1:27
  • 3
    You still do not address if HEAD is ever useful. You are outlining a process for blocking some kinds of requests to your site but never if HEAD is useful. Are you aware that some legitimate client-side caching programs use HEAD? – schroeder May 18 '18 at 7:27
  • This reads like Holden Caulfield got a job in web development.. – Amalgovinus Apr 10 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.