This will not help you. Anyone finding and scanning you from the outside is almost certainly blindly scanning IP space, not looking up names and trying them, and removing yourself from DNS won't impact them a bit.
This will help, but as you've pointed out, making it work with all your widgets may end up being a pain.
- robots.txt entry to avoid search engine exposure
This will not help very much. It might cut down on honest search engines indexing you, but only stops well behaved crawlers. And the chances that you opponent is stumbling across you via a web search are very, very low - see above, they're probably blindly scanning IP space and hitting you.
Update to respond fully to @Co Lega's comment
Oh, you want solutions, too? Crap, that's the hard stuff! 8)
One solution that might work for you is to temporarily grant IP-based access based on authentication. This is similar to port knocking or POP-before-SMTP. It sounds like you want to test apps on the dev server, and therefore can't add application layer authentication to the dev server without conflicting with applications which may already have their own authentication. You don't want to maintain firewall rules because your users may not be predictable in their source IP, or because you don't want to maintain a shifting set of rules.
The idea, then, is that people perform an authentication to the dev server for the sole purpose of opening up access from their IP for a limited amount of time - like a dynamic and temporary firewall rule. Because you're altering IP access, their browser and/or application don't need to be maintaining credentials, the application server doesn't need to maintain state for that level of access, etc. etc. They browse to the server auth page, authenticate, and their IP is cleared for access to the rest of the site for 30 minutes, 2 hours, 2 days, whatever you want. They test the apps without any need to allow for wrapping that access in authentication, because their IP has been granted access based on that out-of-band (e.g., not the application) authentication.
One downside is that IP source is not always as secure; if they're behind a proxy at the library anyone else behind that proxy is granted access too. At that point you want to look at a VPN solution.
Looking at your question, I'm not sure how well that fits your original goal. It's aimed at more of a dev crowd than end users, and it sounds like you may be thinking end users. In that case, perhaps the answer is to harden the applications to survive Internet exposure - which is a good idea anyway.