Local buffer overflows are relatively easy to understand: throw some input at an interface and see if the process fails with a core dump or similar. However, in my mind, this kind of exploit works well only when the attacker has had unlimited access to test the interfaces and see how their input affects the process.
However, for things like locked-down appliances (firewalls, proxies, etc.), high-end commercial-grade servers, or specialized software where an attacker cannot easily acquire it to play with, how does one successfully gather the feedback necessary to mount an successful attack?
(I realize if the asset is of sufficient value, a motivated attacker will purchase such a specialized device, but I'd like to know -- in theory -- how it might be done without local access.)
Edit: I'm also assuming the attacker does not have access to a generic remote shell.