You've evolved to mostly right, but to add several points
and expand on @CoverosGene' answer more than I felt comfortable doing in an edit:
X.509 defines a certificate (and some other things not relevant here) in ASN.1,
a (very!) general data structuring method which has several defined encodings, of which
DER Distinguished Encoding Representation is quite common and is used here.
PEM format -- for several types of data of which a certificate is only one --
is much as you say just binary (DER) data encoded in base64 (edit) broken into lines normally every 64 chars (but there are variations), plus header and trailer lines
consisting of dashes + BEGIN or END + the type of data, in this case CERTIFICATE + dashes.
Those lines look redundant to a human but they are expected and mostly required by software.
PEM (Privacy Enhanced Mail) was actually a complete standard for secure email that has now
been mostly forgotten (see below) except for its encoding format.
(edit) As of 2015 there is RFC 7468 describing in detail most use of 'PEM' formats for modern crypto data.
PKCS#7 was defined by RSA (the company, not the algorithm) as a multi-purpose format
for encrypted and/or signed data. It was turned over to IETF and evolved into
CMS Cryptographic Message Syntax in RFC 2630, then RFC 3369, then RFC 3852, then RFC 5652,
hence the wording of the Windows (inetopt) prompt. "PKCS#7" is often used to mean both
the original RSA PKCS#7 and the IETF successor CMS, in the same way "SSL" is often used for
both the original Netscape protocol and the IETF successor TLS Transport Level Security.
The .p7b or .p7c format is a special case of PKCS#7/CMS: a SignedData structure containing
no "content" and zero SignerInfos, but one or more certificates (usually) and/or CRLs (rarely).
Way back when this provided a standard way to handle (edit) the set of certificates needed to make up a chain (not necessarily in order).
PKCS#7/CMS is (are?) also ASN.1 and depending on circumstances can be either DER or BER,
a closely-related encoding with some very minor differences that most DER decoders handle.
While PKCS#7/CMS like any DER or BER object can be PEM-formatted, I've not seen any implementation other than
openssl (edit) it is rare for certs. (Java CertificateFactory can read PKCS7/CMS-certs-only from DER or PEM, but CertPath.getEncoded writes it only to DER.) In contrast both DER and PEM formats for a single cert are common.
PKCS#7/CMS is also used as the basis for S/MIME secure email (multiple rfcs starting from 5751).
Basically PEM encoded PKCS#7 into ASCII text which email systems of the 1980s could easily handle, while
S/MIME represents CMS as MIME entities which are encoded in several ways modern email systems can handle.
OpenSSL confused matters by implementing, in order: a pkcs7 command which handles
the certs-CRLs-only case not full PKCS#7; a crl2pkcs7 command which actually handles
CRLs and certs, but again not the rest of PKCS#7; a smime command which actually handles both S/MIME
and PKCS#7/CMS for most cases of encrypted and/or signed messages; and a cms command which actually handles
both S/MIME and PKCS#7/CMS for a more complete set of cases.
So I would describe the options as: a cert in PEM or DER format; a (single) cert
in a PKCS#7 container or for short just p7, and mention PEM only in the rare case it applies;
or a cert chain in PKCS#7 or p7. The semantic difference between a single cert and a cert chain
is at least as important as the format difference between a cert by itself or in a container.
And this doesn't even reach the widespread confusion between a certificate by itself (for some other entity, most often a CA root or anchor) and the combination of privatekey PLUS certificate -- or usually chain -- that you use to prove your own identity, for example as an SSL/TLS server or when signing S/MIME email. That uses the originally-Microsoft PFX Personal Information Exchange format, or its standardized form PKCS#12 or "p12".
