PKCS#7 is an old standard from RSA Labs, published later on as an "informational RFC". Then, a new versions was produced, as an "Internet standard", i.e. with the seal of approval from the powers-that-be at IETF; a new name was invented for that: CMS. Newer versions were subsequently defined: RFC 3369, RFC 3852, RFC 5652. You can consider CMS and PKCS#7 to both designate the same standard, which has several versions.
CMS (PKCS#7) is a format for applying encryption, signatures and/or integrity checks on arbitrary binary messages (which can be large). It can be nested. It is the basis for some protocols such as time stamping. It is also frequently (ab)used as a container for X.509 certificates: the
SignedData type includes a field for a set of signatures (and that set can be empty), and also a field to store arbitrary X.509 certificates which are considered as "potentially useful" for whoever processes the object.
CMS is nominally backward compatible: there are version fields at various places, and a library which understands a version of CMS should be able to process messages from older versions; moreover, such a library should also produce messages which are compatible with other versions, inasmuch as it is feasible given the message contents. See section 5.1 for an example: the "version" fields are set to the minimal values which still make sense given what else is in the structure.
S/MIME is a protocol for doing "secure emails". S/MIME uses CMS for the cryptographic parts; what S/MIME adds over CMS is the following:
Rules for encoding CMS objects (which are binary) into something which can travel unscathed in emails (which are text-based). These rules build on MIME, so basically Base64 with some headers.
Canonicalization rules for signed but not encrypted emails, with "detached signatures". When an email is just signed, and the recipient might not be S/MIME-aware, it is convenient to send the signature as a CMS
SignedData object which does not contain the email data itself; the signature is joined to the email as an attachment. It is then important to define exactly how the email contents are to be encoded into a sequence of bytes for the purpose of signature verification (a single mis-encoded bit would imply verification failure).
Restrictions and usage practices on the actual CMS objects: how much nesting is allowed, what types may appear...
Some extra attributes so that a sender may publish, in his signed messages, what kinds of cryptographic algorithms it supports and similar metadata.
In a way, we can think of S/MIME as the glue between CMS and emails.
OpenSSL is a library which implements some protocols, including some versions of PKCS#7 and CMS and S/MIME. The library also comes with command-line tools which expose, as a command-line interface, some functionalities of the library. The tools won't support anything that the library does not implement (the contrary would be surprising, to say the least), but the converse is not true: the library may implement features to which the command-line tools give no easy access, or no access at all.
Generally speaking, the command-line tools are not very "serious". They are convenient for some manual operations or a few scripts, but for robust applications you should use the library directly (it is a C library, but bindings exist for many programming languages).
Among the command line tools are the subcommands
smime. Confusingly, the
cms command is meant to be used for S/MIME, and supports S/MIME, and turns out to be usable as a "generic" CMS support tool only as a byproduct. The OpenSSL developers created the new command
cms in order to support recent versions of S/MIME (and CMS) because they felt that they could not update the
smime command without breaking interface compatibility: third-party scripts using
openssl smime would have become invalid, if the parameters for that command has been adjusted to account for newer S/MIME versions. So they just created a new command, called
Theoretically, you should use
openssl cms for everything related to S/MIME; but since that command gives access to the OpenSSL support code for the "new" versions of S/MIME and CMS, it necessarily allows you to produce S/MIME messages which are perfectly valid but use features that older implementations of S/MIME might not support. This is summarily described in the man page. Extensive tests with other S/MIME implementations (e.g. Thunderbird, Outlook...) are needed to know exactly what you can do with S/MIME without making your emails unreadable by other people.