With a closed user group where everyone trusts the self-signed CA it can be achieved, that an attacker with access to the mail server or the communication between mail servers can not see the original content of the mail, since this content is encrypted. The attacker can see meta information of the mail though, i.e. sender and recipient of the mail, maybe also subject and of course traffic pattern about size, time etc.
This is assuming that everyone trusts the self-signed CA, the process of issuing, using and revoking certificates is well protected and everyone always and properly uses S/MIME for end to end security of mail. These assumptions are not trivially to achieve.
In a more open user group with outside users this gets even harder. It is unlikely that these external users will simply trust your self-signed CA. It is more unlikely that they will be able to use S/MIME for every recipient, since they most first somehow obtain the certificate for this recipient in a secure way.
Does PGP mail still play a role in this case? Thank you.
PGP and S/MIME are alternativ technologies with the same goal of achieving end to end mail security. They are not used together to achieve end to end security, but only one or the other.
But essentially all what I said about S/MIME is kind of true for PGP too. The main difference between these technologies is how trust in the peer is achieved. S/MIME uses a PKI with (possible multiple) central CA issuing certificates and trust in the peers certificate derived from trust in the CA. PGP uses a web of trust, where trust in a key is derived from the trust one has to others and the trust others have in the peers key.