I have seen increased 'HEAD' requests in my webserver access.log. What are these requests for? Should I disable this method in my webserver configs?
Relevant quote from the link:
If you disabled it, you'd just increase your throughput cost. A person can get the same information with a GET, so if they were trying to do something malicious, they could just use a GET. Except, this way, they're being nice and not forcing you to send the request body.
EDIT: I don't know what the requests would be from, although I can certainly think of uses. Anyone else who knows or wants to chip in, please do so. I'm kinda curious, myself. Hence, community wiki.
Everything Parthian said was spot on. HEAD requests are a like a 'short' GET request that avoids the network extra traffic and potentially the rendering overhead of a GET request.
There are a variety of reasons you, your browser, or your search engine may want to do a HEAD request. Some websites may just be pulling meta information off you, and your smaller response is to your benefit. More likely your browser or search engines are probably using HEAD requests to see if their cached versions of your pages are still up to date.
The Response header's "date" and "expires" field should be used by clients when your page is cached to determine when the next time they should visit your site for an update. Also the response headers may sometimes include a modified date that could also be used to indicate when your page needs to be updated.