On servers that need only limited access I simply use
iptables to restrict access to the IP addresses that are allowed to see the site. Other IP addresses will behave exactly as you'd like, timing out as if there's nothing there.
For each (range of) IP address(es) allowed define and run:
# /sbin/iptables -A INPUT -p tcp -s $ip --dport 80 -j ACCEPT
$ip with the desired IP (range))
For IPv6 you can do the same thing except run
Then drop everything else:
# /sbin/iptables -P INPUT DROP
# /sbin/ip6tables -P INPUT DROP
Make sure to allow SSH from your IP as well of course, or you'll lock yourself out if the machine is remote.
Also remember that by default, these rules don't survive a reboot. Where to save the rules differs even between Linux distributions but on Debian I use:
# /sbin/iptables-save > /etc/iptables/rules.v4
# /sbin/ip6tables-save > /etc/iptables/rules.v6