I currently use mod_security in combination with maldec and block almost all attempts to upload shells, leading them to error 406. However, I found one "problem" if I upload Shell through an ftp client then I can use shell in hosting. my question is how to block the opening of the shell even if uploaded via ftp? any mod_security rule to use and if so what? I also use cPanel
ModSecurity is a Web Application Firewall. FTP is not Web traffic, HTTP and HTTPS are. This means ModSecurity cannot be used for filtering FTP traffic not can it be used to filter mail traffic.
Also, ModSecurity does not protect against misuse of hacked accounts or severe server misconfiguration in general. If somebody is able to use FTP to upload malicious code which then can get executed as part of the website (like a webshell) then this can only be done if you either allow anonymous upload to your web hosting (misconfiguration - just don't!) or that the username and password to your account is known to the attacker.