As forest@ suggested, you can use a robots exclusion file - which like he also said only compliant web crawlers will follow. But, this is like opening your house for everybody to visit and putting the sign "please don't look inside" on the open door to the bedroom . Guess what, there are visitors who are curious why they should not look and will exactly do this.
Thus, it is only a way to deter polite visitors like the major search engines. If this is enough for you then you might go for it.
... and only have people access that service, if they know that it exists and how to access it
This can be realized by either having a non-public link or by requiring more than just the link - i.e. make the door to the bedroom invisible or just lock the door with a key.
A non-public link can be simply realized by creating some file or directory which is not linked from somewhere else on your page (and also should not be visible in some automatically generated directory index - i.e. switch these off). Crawlers which don't know up-front about this link will not be able to crawl it since it is not linked from any place they've visited so far and thus they don't know it exist. Of course, the link must stay non-public for this to work, thus make sure that nobody publishes it.
Much better instead is to require a secret additionally to the link itself. This can be done by require authentication for a directory and setup some users, then give everybody not only the link but also the (maybe personalized) login credentials. Even if crawlers know the link they don't have the credentials and can thus not access it. There are numerous resources on how to setup authentication, like Password protect a directory using basic authentication.