In the Drupal logs on my dev site there are "page not found" reports that are obviously from a bot trying out well known urls (e.g. /wp-login). But I have set up apache basic auth and I am the only person who knows the password! If I go to those URLs in a browser, I get a 401, not a 404. What could possibly be happening?

I tried asking this on the Drupal stack exchange and they weren't having any of it, but I can't help thinking it's some Drupal weirdness.


The auth config is in /etc/apache2/ and is:

<Directory /var/www/html>
  AllowOverride All
  AuthType Basic
  AuthName "Authentication Required"
  AuthUserFile "/etc/htpasswd/.htpasswd"
  Require valid-user
  Order allow,deny
  Allow from all

When I go to a nonexistent url in a browser, I get the Basic Auth popup and I see this in the apache logs:

x.x.x.x - - [20/Feb/2019:23:44:15 +0100] "GET /horde3/imp/test.php HTTP/1.1" 401 920 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"

But when the bot did it, it got a 404! - - [20/Feb/2019:21:43:57 +0100] "GET /horde3/imp/test.php HTTP/1.1" 404 21401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"

I'm less convinced it's a Drupal issue now, after seeing this in the Apache logs.

  • could you please add the link to the question at the Drupal stack exchange? – Refineo Feb 20 at 22:13
  • It was the exact same question. I got 3 downvotes and then it was put on hold, so I deleted it. – naomi Feb 20 at 22:18
  • Could there be any user agent based logic on the server side? Different pages presented to browsers compared to crawlers? – Natanael Feb 20 at 22:18
  • Could you please also add more details, for example, in each situation (401, 404), what are the urls, i.e. do they all start with http: or https: ? – Refineo Feb 20 at 22:19
  • 1
    Someone suggested it might be that the basic auth is on https but not http. There is a redirect from http -> https in the vhost config, so I'd have thought it woudn't matter. But I've added the basic auth lines to the http vhost as well and will see if that helps. – naomi Feb 20 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.