This is a situation that I have encountered on a number of occasions. I will conduct a web search in Google, a link to a file (.pdf for example) will take you to that file on the server. It opens up in your browser, you can save the file.
But when you backtrack through the URL to find out more about the website that holds it, and if there is other information that can aid your search, you are denied access. You are either given an error message or taken to the log in page for a web portal. Which can be accounted for by configuring the server to prevent directory traversal.
While I understand the lack of a robot.txt file could contribute to this, some of these sites would not rely on these for protection since a rogue web crawler would just ignore the file anyway. Not to mention the robot.txt would enumerate a lot of your files and directory structure to a hacker.
So it would be too much to presume that some of these sites were not protected by physical firewalls, and given the server authentication which denied moving to other parts of the site, that crawling the site could be done without having log in credentials.
So what loop holes is the Google crawler exploiting that allows it to bypass possible firewall, server authentication, and portal protection to find out the directory structure of that server and create direct links that allow someone without credentials to see that file?