As I cannot find a direct answer to this on this site, I'll provide a quick answer.
There are two primary methods for locating shells:
- shell signatures
- anomalous network connections
Meterpreter, for example, has a very distinct signature and AV tools can find these. PHP shells and others can likewise contain unique elements that might be detected by automated tools. Custom shells are difficult to find this way.
But a shell, in your context, is something that has a network connection. You want to look for
- known processes that have no business connecting to the network ("why is
calc.exe connecting to a Linode server?")
- unknown processes that have a network connection ("what is
sefhwebfh and why is it connecting to a Linode server?")
- known processes that have a legitimate need to connect to the network, but the connections are unusual ("why is
httpd making outbound connections to a Linode server?"
(Note, "Linode" is a generic example.)
To look for these, you need to:
- know your OS's tools to list processes and any network connections they make. It's then important to baseline what is "normal" (either through documentation or from personal experience) to compare against.
- run packet captures (or similar, like IDS, packet inspection on the firewall, etc.) to listen to the wire to look for anomalies from the network baseline.
What do you do once you find a shell?
Once you discover the shell, you can no longer be confident about anything on the server. You have no idea if permissions have been escalated currently or in the past. While a full forensic investigation might be interesting and satisfying, you can never be sure that you have all the information to declare the machine safe or cleaned.
Your response should be to capture an image of the machine for an investigation into the impacts of the event, then nuke the machine from orbit.