Great question. As far as I can tell, location is not special. There are other permissions that are more dangerous, or that generate a higher level of user concern.
So why do platforms ask specially about location? I am not sure, but I have some conjectures. First, location has some privacy implications, which many users might care about. Asking about location helps provide notice and consent, which are two core themes of protecting user privacy. Second, many apps use location information (many more than some of the other dangerous or concerning permissions we might consider).
That said, I still think your question is entirely fair. Mobile platforms are currently in a relatively young state. It is not clear whether the current approach of asking only about location is optimal, or whether this practice will survive. I would not be surprised if things look a bit different a decade from now, and if other resources start to be handled like location currently is.
For instance, consider the recent flap over Path, which was grabbing users' contact information and uploading it to Path's servers. It turned out that Path was not special; many other widely used iPhone apps were doing the same thing. Some users were upset to find that apps were grabbing all of their contact information, without asking them first. So, Apple responded by promising that future versions of iOS will prompt the user for permission before allowing the app to gain access to contact data -- just like happens with location. This suggests it is possible we might see some evolution of the platform where location is not so unique any longer, and where the platform prompts for permission for several other resources as well.
P.S. iOS currently prompts the user for two resources: access to location, and ability to create push notifications (which interrupt the user and can be annoying to some users). So there is already one other situation that generates a permission prompt beyond location access.