I have a specific sample I'd like feedback on, but my view is an answer about general techniques is more valuable for this site. I'll leave the details in, in case anyone googling this bumped into the same service.
What are general techniques and resources for identifying an unknown service, or something on a non-standard port that isn't being talkative?
Does the behaviour below ring a bell with anyone? (Are there steps I have missed to further identify the service?)
While on an engagement, we have encountered an open port, 10001. As you might've guessed, as far as I can tell it does not respond to protocols usually used on that port. My search has not been completely exhaustive but I have fuzzed the first three bytes and found a response for the first byte.
Observations on my specific unknown service:
When sent a capital i,
I\n, it responds
I211568 \x00 \x00
Messages seem to be null- and newline-terminated; anything but those after the capital i do not affect the behaviour, but a null before it will impair the response.
nmap shows it as
scp-config, once as
Turns out it was the Remote Port of a Swann DVR